Event replay in Microservices - Scalability & System Analysis

Scalability Analysis - Event replay

Growth Table: Event Replay System Scaling

Scale	Users / Events	System Changes
100 users	~10K events/day	Single event store instance; simple replay; low latency
10K users	~1M events/day	Partition event store; add read replicas; batch replay; introduce caching
1M users	~100M events/day	Sharded event store; distributed replay workers; event compaction; asynchronous replay
100M users	~10B events/day	Multi-region event stores; advanced partitioning; replay throttling; event archival; CDN for event snapshots

First Bottleneck

The event store database is the first bottleneck. As event volume grows, the database struggles to handle high write and read throughput for storing and replaying events. This causes increased latency and potential data loss during replay.

Scaling Solutions

• Horizontal scaling: Add more event store nodes and partition events by user or event type to distribute load.
• Read replicas: Use replicas to offload replay reads from the primary event store.
• Caching: Cache frequently replayed event sequences to reduce database hits.
• Batch processing: Replay events in batches asynchronously to smooth load.
• Event compaction: Summarize or snapshot event streams to reduce replay size.
• Multi-region deployment: Deploy event stores closer to users to reduce latency.
• Throttling: Limit replay request rates to prevent overload.
• Archival: Move old events to cheaper storage to keep active event store performant.

Back-of-Envelope Cost Analysis

• At 1M users generating 100M events/day (~1157 events/sec), event store must handle ~1200 writes/sec plus replay reads.
• Storage needed: Assuming 1KB per event, 100M events/day = ~100GB/day; requires scalable storage and retention policies.
• Network bandwidth: For replay, streaming event data can consume significant bandwidth; e.g., 1K replays/sec * 1MB replay size = ~1GB/s peak.
• Compute: Replay workers must be scaled horizontally to process event streams without delay.

Interview Tip

Start by explaining the event replay flow and identify the main components. Discuss how event volume affects storage and replay latency. Highlight the event store as the bottleneck and propose scaling strategies like partitioning and caching. Use concrete numbers to justify your choices and mention trade-offs like consistency vs. availability.

Self Check

Your event store database handles 1000 QPS. Traffic grows 10x to 10,000 QPS. What do you do first?

Answer: Add read replicas and partition the event store to distribute load horizontally. This reduces pressure on a single database instance and maintains replay performance.

Key Result

The event store database is the first bottleneck as event volume grows; horizontal scaling with partitioning and caching is key to maintain replay performance.