Event replay in Microservices - Scalability & System Analysis

Scalability Analysis - Event replay

Growth Table: Event Replay System Scaling

Scale	Users / Events	System Changes
100 users	~10K events/day	Single event store instance; simple replay; low latency
10K users	~1M events/day	Partition event store; add read replicas; batch replay; introduce caching
1M users	~100M events/day	Sharded event store; distributed replay workers; event compaction; asynchronous replay
100M users	~10B events/day	Multi-region event stores; advanced partitioning; replay throttling; event archival; CDN for event snapshots

First Bottleneck

The event store database is the first bottleneck. As event volume grows, the database struggles to handle high write and read throughput for storing and replaying events. This causes increased latency and potential data loss during replay.

Scaling Solutions

• Horizontal scaling: Add more event store nodes and partition events by user or event type to distribute load.
• Read replicas: Use replicas to offload replay reads from the primary event store.
• Caching: Cache frequently replayed event sequences to reduce database hits.
• Batch processing: Replay events in batches asynchronously to smooth load.
• Event compaction: Summarize or snapshot event streams to reduce replay size.
• Multi-region deployment: Deploy event stores closer to users to reduce latency.
• Throttling: Limit replay request rates to prevent overload.
• Archival: Move old events to cheaper storage to keep active event store performant.

Back-of-Envelope Cost Analysis

• At 1M users generating 100M events/day (~1157 events/sec), event store must handle ~1200 writes/sec plus replay reads.
• Storage needed: Assuming 1KB per event, 100M events/day = ~100GB/day; requires scalable storage and retention policies.
• Network bandwidth: For replay, streaming event data can consume significant bandwidth; e.g., 1K replays/sec * 1MB replay size = ~1GB/s peak.
• Compute: Replay workers must be scaled horizontally to process event streams without delay.

Interview Tip

Start by explaining the event replay flow and identify the main components. Discuss how event volume affects storage and replay latency. Highlight the event store as the bottleneck and propose scaling strategies like partitioning and caching. Use concrete numbers to justify your choices and mention trade-offs like consistency vs. availability.

Self Check

Your event store database handles 1000 QPS. Traffic grows 10x to 10,000 QPS. What do you do first?

Answer: Add read replicas and partition the event store to distribute load horizontally. This reduces pressure on a single database instance and maintains replay performance.

Key Result

The event store database is the first bottleneck as event volume grows; horizontal scaling with partitioning and caching is key to maintain replay performance.

Practice

1. What is the main purpose of event replay in a microservices architecture?

easy

A. To balance load between microservices

B. To rebuild system state by reprocessing stored events in order

C. To send real-time notifications to users

D. To encrypt data during transmission

Solution

1. Step 1: Understand event replay concept

   Event replay means using stored events to reconstruct the current state of a system by processing them again in the order they occurred.

2. Step 2: Identify the main purpose

   This process helps recover system state after failures or to debug by looking at past events, not for notifications, load balancing, or encryption.

3. Final Answer:

   To rebuild system state by reprocessing stored events in order -> Option B

4. Quick Check:

   Event replay = rebuild state [OK]

Hint: Event replay means replaying past events to restore state [OK]

Common Mistakes:

• Confusing event replay with real-time messaging
• Thinking event replay balances load
• Assuming event replay encrypts data

2. Which of the following is the correct way to ensure events are replayed in the right order?

easy

A. Ignore event order since it doesn't affect state

B. Replay events randomly to speed up processing

C. Replay only the latest event to save resources

D. Store events with timestamps and replay by sorting them chronologically

Solution

1. Step 1: Understand importance of event order

   Events must be replayed in the exact order they occurred to correctly rebuild system state.

2. Step 2: Identify correct ordering method

   Using timestamps to sort events chronologically ensures the correct sequence during replay.

3. Final Answer:

   Store events with timestamps and replay by sorting them chronologically -> Option D

4. Quick Check:

   Correct event order = chronological replay [OK]

Hint: Replay events by timestamp order to keep state consistent [OK]

Common Mistakes:

• Replaying events randomly
• Skipping older events
• Ignoring event order

3. Given the following event log stored as tuples (timestamp, event):
[(1, 'create'), (3, 'update'), (2, 'update'), (4, 'delete')]
What is the correct order of events during replay?

medium

A. [('update'), ('create'), ('delete'), ('update')]

B. [('delete'), ('update'), ('create'), ('update')]

C. [('create'), ('update'), ('update'), ('delete')]

D. [('update'), ('delete'), ('create'), ('update')]

Solution

1. Step 1: Sort events by timestamp

   Sort the list by the first element (timestamp): 1, 2, 3, 4.

2. Step 2: Extract event names in sorted order

   Events in order: 'create' (1), 'update' (2), 'update' (3), 'delete' (4).

3. Final Answer:

   [('create'), ('update'), ('update'), ('delete')] -> Option C

4. Quick Check:

   Sorted timestamps = 1,2,3,4 [OK]

Hint: Sort by timestamp, then list events in that order [OK]

Common Mistakes:

• Ignoring timestamp order
• Mixing event sequence
• Assuming original list order is correct

4. A microservice tries to replay events but the system state is incorrect after replay. Which issue is most likely causing this?

medium

A. Events were replayed out of order

B. Events were encrypted during replay

C. Events were replayed multiple times in parallel

D. Events were filtered by type before replay

Solution

1. Step 1: Analyze replay error cause

   Incorrect system state after replay usually means the event sequence was not preserved.

2. Step 2: Identify the most common cause

   Replaying events out of order breaks the state reconstruction logic, causing errors.

3. Final Answer:

   Events were replayed out of order -> Option A

4. Quick Check:

   Out-of-order replay = wrong state [OK]

Hint: Check event order first when state is wrong after replay [OK]

Common Mistakes:

• Blaming encryption which doesn't affect replay order
• Assuming parallel replay is always safe
• Filtering events without understanding impact

5. You want to add a new feature that analyzes historical user actions using event replay. Which design choice best supports this without affecting live system performance?

hard

A. Replay events asynchronously from a separate event store copy

B. Replay events synchronously on the main database during user requests

C. Replay only the latest event repeatedly for analysis

D. Skip event replay and query live data directly

Solution

1. Step 1: Understand impact of replay on live system

   Replaying events synchronously during user requests can slow down or disrupt the live system.

2. Step 2: Choose design for performance and safety

   Using a separate copy of the event store and replaying asynchronously isolates analysis from live traffic, preserving performance.

3. Final Answer:

   Replay events asynchronously from a separate event store copy -> Option A

4. Quick Check:

   Async replay on copy = no live impact [OK]

Hint: Use async replay on separate store to avoid live system load [OK]

Common Mistakes:

• Replaying synchronously blocking live requests
• Analyzing only latest event missing history
• Ignoring benefits of event replay for analysis