Event replay in Microservices - Architecture Diagram

System Overview - Event replay

This system allows replaying past events in a microservices architecture to rebuild state or recover from errors. It stores events in an event store and replays them to services that consume these events, ensuring data consistency and fault tolerance.

Architecture Diagram

User
  |
  v
Load Balancer
  |
  v
API Gateway
  |
  v
Event Producer Service ---> Event Store (Event Log) <--- Event Replay Service
                                   |
                                   v
                            Consumer Services
                                   |
                                   v
                               Database
                                   |
                                   v
                                 Cache

Components

User

client

Initiates actions that generate events

Load Balancer

load_balancer

Distributes incoming requests evenly to API Gateway instances

API Gateway

api_gateway

Routes requests to appropriate microservices

Event Producer Service

service

Generates and publishes events to the event store

Event Store (Event Log)

event_store

Stores all events in order for replay and auditing

Event Replay Service

service

Reads events from the event store and replays them to consumer services

Consumer Services

service

Processes events to update state and business logic

Database

database

Stores the current state updated by consumer services

Cache

cache

Speeds up read operations by storing frequently accessed data

Request Flow - 9 Hops

User → Load Balancer

Load Balancer → API Gateway

API Gateway → Event Producer Service

Event Producer Service → Event Store (Event Log)

Event Store (Event Log) → Event Replay Service

Event Replay Service → Consumer Services

Consumer Services → Database

Consumer Services → Cache

Cache → User

Failure Scenario

Component Fails:Event Store (Event Log)

Impact:New events cannot be stored, so event replay and state updates stop. Consumer services do not receive new events, causing stale data.

Mitigation:Use event store replication and backups to restore quickly. Consumer services can continue reading from cache but will have stale data until event store recovers.

Architecture Quiz - 3 Questions

Which component is responsible for storing all events for replay?

A Load Balancer

B Cache

C Event Store (Event Log)

D API Gateway

Design Principle

This architecture uses an event store as a single source of truth for all events, enabling reliable event replay to rebuild system state. It separates event production from consumption, improving fault tolerance and scalability.

Practice

1. What is the main purpose of event replay in a microservices architecture?

easy

A. To balance load between microservices

B. To rebuild system state by reprocessing stored events in order

C. To send real-time notifications to users

D. To encrypt data during transmission

Solution

1. Step 1: Understand event replay concept

   Event replay means using stored events to reconstruct the current state of a system by processing them again in the order they occurred.

2. Step 2: Identify the main purpose

   This process helps recover system state after failures or to debug by looking at past events, not for notifications, load balancing, or encryption.

3. Final Answer:

   To rebuild system state by reprocessing stored events in order -> Option B

4. Quick Check:

   Event replay = rebuild state [OK]

Hint: Event replay means replaying past events to restore state [OK]

Common Mistakes:

• Confusing event replay with real-time messaging
• Thinking event replay balances load
• Assuming event replay encrypts data

2. Which of the following is the correct way to ensure events are replayed in the right order?

easy

A. Ignore event order since it doesn't affect state

B. Replay events randomly to speed up processing

C. Replay only the latest event to save resources

D. Store events with timestamps and replay by sorting them chronologically

Solution

1. Step 1: Understand importance of event order

   Events must be replayed in the exact order they occurred to correctly rebuild system state.

2. Step 2: Identify correct ordering method

   Using timestamps to sort events chronologically ensures the correct sequence during replay.

3. Final Answer:

   Store events with timestamps and replay by sorting them chronologically -> Option D

4. Quick Check:

   Correct event order = chronological replay [OK]

Hint: Replay events by timestamp order to keep state consistent [OK]

Common Mistakes:

• Replaying events randomly
• Skipping older events
• Ignoring event order

3. Given the following event log stored as tuples (timestamp, event):
[(1, 'create'), (3, 'update'), (2, 'update'), (4, 'delete')]
What is the correct order of events during replay?

medium

A. [('update'), ('create'), ('delete'), ('update')]

B. [('delete'), ('update'), ('create'), ('update')]

C. [('create'), ('update'), ('update'), ('delete')]

D. [('update'), ('delete'), ('create'), ('update')]

Solution

1. Step 1: Sort events by timestamp

   Sort the list by the first element (timestamp): 1, 2, 3, 4.

2. Step 2: Extract event names in sorted order

   Events in order: 'create' (1), 'update' (2), 'update' (3), 'delete' (4).

3. Final Answer:

   [('create'), ('update'), ('update'), ('delete')] -> Option C

4. Quick Check:

   Sorted timestamps = 1,2,3,4 [OK]

Hint: Sort by timestamp, then list events in that order [OK]

Common Mistakes:

• Ignoring timestamp order
• Mixing event sequence
• Assuming original list order is correct

4. A microservice tries to replay events but the system state is incorrect after replay. Which issue is most likely causing this?

medium

A. Events were replayed out of order

B. Events were encrypted during replay

C. Events were replayed multiple times in parallel

D. Events were filtered by type before replay

Solution

1. Step 1: Analyze replay error cause

   Incorrect system state after replay usually means the event sequence was not preserved.

2. Step 2: Identify the most common cause

   Replaying events out of order breaks the state reconstruction logic, causing errors.

3. Final Answer:

   Events were replayed out of order -> Option A

4. Quick Check:

   Out-of-order replay = wrong state [OK]

Hint: Check event order first when state is wrong after replay [OK]

Common Mistakes:

• Blaming encryption which doesn't affect replay order
• Assuming parallel replay is always safe
• Filtering events without understanding impact

5. You want to add a new feature that analyzes historical user actions using event replay. Which design choice best supports this without affecting live system performance?

hard

A. Replay events asynchronously from a separate event store copy

B. Replay events synchronously on the main database during user requests

C. Replay only the latest event repeatedly for analysis

D. Skip event replay and query live data directly

Solution

1. Step 1: Understand impact of replay on live system

   Replaying events synchronously during user requests can slow down or disrupt the live system.

2. Step 2: Choose design for performance and safety

   Using a separate copy of the event store and replaying asynchronously isolates analysis from live traffic, preserving performance.

3. Final Answer:

   Replay events asynchronously from a separate event store copy -> Option A

4. Quick Check:

   Async replay on copy = no live impact [OK]

Hint: Use async replay on separate store to avoid live system load [OK]

Common Mistakes:

• Replaying synchronously blocking live requests
• Analyzing only latest event missing history
• Ignoring benefits of event replay for analysis