Bird
Raised Fist0

Event Sourcing and CQRS with Message Queues

HLDmediumInterview freq: medium
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Event Sourcing and CQRS with Message Queues
Designing a scalable event sourcing system with CQRS pattern using Kafka for message streaming
mediumMessage Queues and Event StreamingFreq: medium
⚙️
Functional Requirements
  • Capture all state changes as immutable events in an append-only log (event sourcing).
  • Separate write model (command handling) from read model (query handling) following CQRS.
  • Publish events asynchronously to downstream consumers via a message queue.
  • Support rebuilding read models from event streams for eventual consistency.
  • Allow multiple read models optimized for different query patterns.
  • Provide APIs for commands (writes) and queries (reads) with consistency guarantees.
📊
Non-Functional Requirements
  • Support 2 billion daily active users generating 40 events per user per day.
  • Ensure 99.99% availability for command and query APIs.
  • Achieve 100ms P99 latency for read queries on read models.
  • Handle peak write throughput of up to 1 million events per second.
  • Retain event data for at least 1 year with replication for durability.
❓
Clarifying Questions to Ask
  • What is the expected event size and variability in event types?
  • Are read models eventually consistent or do some require strong consistency?
  • What is the expected query load compared to write load?
  • How many different read models or projections are expected?
  • Is there a requirement for event versioning and schema evolution?
  • What is the acceptable recovery time objective (RTO) for read model rebuilds?
Out of Scope
✗ Designing the domain model or business logic for commands and events.
✗ Implementing user authentication and authorization mechanisms.
✗ Real-time analytics or complex event processing on event streams.
✗ Cross-data center replication and disaster recovery strategies.
📌
Assumptions
  • 2 billion DAU generating 40 events per user per day
  • Average event size is 100 bytes
  • Peak write load is 10% of average write load
  • Read queries are 5 times the write QPS
⚡
QPS Calculation
write qps2,000,000,000 × 40 / 86,400 ≈ 925,926 events/sec
peak write qps925,926 × 1.1 ≈ 1,018,519 events/sec
read qps925,926 × 5 ≈ 4,629,630 queries/sec
💾
Storage Estimate
daily storage925,926 events/sec × 100 bytes × 86,400 sec ≈ 8 TB/day
retention 365 days8 TB/day × 365 ≈ 2.92 PB
with replication 3x2.92 PB × 3 ≈ 8.76 PB total storage
🖥️
Server Estimate
kafka brokersAssuming 1 TB disk per broker, need ~9 brokers for storage + 3x replication = 27 brokers
read model workersTo handle 4.6M QPS with 10k QPS per worker, need ~463 workers
🏗️
Architecture Diagram
Architecture Diagram
          +-------------+
          |  Command    |
          |  API Layer  |
          +------+------+
                 |
                 v
          +-------------+
          |  Write      |
          |  Model      |
          +------+------+
                 |
                 v
          +-------------+
          |  Kafka      |
          |  (Event Log)|
          +------+------+
                 |
        +--------+--------+
        |                 |
        v                 v
 +-------------+   +-------------+
 | Read Model  |   | Read Model  |
 | Worker 1    |   | Worker 2    |
 +------+------|   +------+------|
        |                 |
        v                 v
 +-------------+   +-------------+
 | Query API   |   | Query API   |
 +-------------+   +-------------+
🧩
Components
Command API Layer
Receives client commands, validates and applies business logic to generate events
Why thisStateless REST/gRPC servers scale horizontally to handle 1M+ QPS writes with load balancing
Why notMonolithic app server would bottleneck and complicate scaling at 1M QPS
Write Model
Applies commands to current state, generates immutable events for state changes
Why thisIn-memory or NoSQL store (e.g., Cassandra) for fast state access and event generation
Why notRelational DBs like PostgreSQL have poor write scalability and complex sharding at 1M QPS
Kafka (Event Log)
Durable, ordered, replicated event log for publishing events asynchronously to consumers
Why thisKafka supports 10M+ writes/sec with partitioning, replication, and retention policies
Why notRabbitMQ lacks Kafka’s throughput and partitioned log semantics needed for event sourcing
Read Model Workers
Consume events from Kafka, update read-optimized projections for queries
Why thisKafka consumer groups enable scalable parallel processing and fault tolerance
Why notPolling databases for changes causes high latency and inconsistent read models
Query API Layer
Serve read requests from read models with low latency and high availability
Why thisStateless API servers with caching (e.g., Redis) provide sub-100ms P99 query latency
Why notDirect queries to write model cause high load and stale data due to eventual consistency
🔄
Data Flow
1
Client → Command API Layer
Send command request
Commands validated and processed synchronously
2
Command API Layer → Write Model
Apply command to current state
Generates immutable event if command succeeds
3
Write Model → Kafka
Publish event to Kafka topic
Event appended to durable, replicated log
4
Kafka → Read Model Workers
Consume events asynchronously
Workers update read-optimized projections
5
Client → Query API Layer
Send query request
Queries served from read models with low latency

Practice

(1/5)
1. A web application needs to handle millions of concurrent users sending requests simultaneously. Which component is primarily responsible for managing client connections and distributing requests to backend services?
easy
A. Database server, because it stores all user data and handles queries
B. Load balancer, because it manages incoming client connections and routes requests
C. Cache server, because it speeds up data retrieval for clients
D. Message queue, because it buffers requests for asynchronous processing

Solution

  1. Step 1: Identify the role of the load balancer

    The load balancer manages client connections and distributes incoming requests evenly across backend servers to prevent overload.

  2. Step 2: Eliminate other components

    The database server handles data storage but does not manage client connections; cache servers speed up data retrieval but don't handle connections; message queues buffer asynchronous tasks but don't manage live client connections.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Load balancer routes client requests to backend servers [check]
Hint: Load balancer directs traffic, not stores or caches data [check]
Common Mistakes:
2. A payment system must ensure that duplicate payment requests from clients do not result in multiple charges. Which component is primarily responsible for handling idempotency in this scenario?
easy
A. The load balancer, by filtering duplicate requests before they reach backend services
B. The database, by enforcing unique constraints on payment amounts and timestamps
C. The client SDK, by retrying requests only once and caching responses locally
D. The transaction processing service, by storing and checking unique request IDs before processing

Solution

  1. Step 1: Identify idempotency enforcement point

    The transaction processing service must track unique request IDs to prevent duplicate charges, as it controls payment execution.

  2. Step 2: Eliminate other options

    The load balancer cannot reliably detect duplicates due to statelessness; the database unique constraints cannot cover all idempotency cases; client SDK retries are unreliable and cannot guarantee global idempotency.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Transaction service stores request IDs -> prevents duplicate processing [check]
Hint: Idempotency enforced where payment logic executes [check]
Common Mistakes:
3. In a 45-minute system design answer, at step 4 you have identified key components and now start sketching data flow. What happens at this step in the component trace?
easy
A. You map how data moves between components to validate interactions
B. You choose cloud providers before understanding data flow
C. You finalize the database schema without considering data flow
D. You write API endpoint code to test data flow assumptions

Solution

  1. Step 1: Trace the role of step 4 -- sketching data flow

    This step validates how components interact and ensures data moves correctly, critical for system correctness.

  2. Step 2: Exclude premature or unrelated actions

    Finalizing schema or coding before data flow understanding risks misalignment; choosing cloud providers is premature.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Data flow mapping -> validates component interaction [check]
Hint: Map data flow before design details [check]
Common Mistakes:
4. A public API receives unpredictable bursts of traffic that can overwhelm backend services. Which component is best suited to protect the system from overload?
easy
A. A distributed cache like Redis to store frequent requests for faster responses
B. A rate limiter that enforces request quotas per client IP or API key
C. A load balancer that evenly distributes requests across servers
D. A message queue to buffer incoming requests before processing

Solution

  1. Step 1: Identify the protection mechanism against traffic bursts

    Distributed cache (A) improves response time but doesn't limit request rate; load balancer (C) distributes load but can't prevent overload from bursts; message queue (D) buffers but can be overwhelmed if bursts are too large.

  2. Step 2: Recognize rate limiter role

    Rate limiter (B) enforces quotas per client, directly preventing backend overload by rejecting or delaying excess requests.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Rate limiter enforces quotas to prevent overload [check]
Hint: Rate limiter controls request bursts to protect backend [check]
Common Mistakes:
5. If the health check module fails to run, what is the exact failure observed in the load balancer's routing behavior?
medium
A. Load balancer reroutes traffic randomly ignoring server health
B. Load balancer blocks all traffic until health checks resume
C. Load balancer routes only to a cached subset of servers marked healthy previously
D. Load balancer continues routing to all servers, including unhealthy ones, causing client errors

Solution

  1. Step 1: Understand health check module role

    Health checks detect unhealthy servers; without them, no updates occur to remove bad servers.

  2. Step 2: Trace routing behavior without health checks

    Load balancer routes to all servers indiscriminately, including unhealthy ones, causing client errors.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    No health checks -> no removal of unhealthy servers -> client errors [check]
Hint: No health checks means no unhealthy server removal [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard