Bird
Raised Fist0

Event Sourcing and CQRS with Message Queues

HLDmediumInterview freq: medium
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Event Sourcing and CQRS with Message Queues
45 min · senior
senior

You are designing an event sourcing system with CQRS pattern for a startup with 50 million daily active users. The system must handle high write throughput of user-generated events, ensure strong consistency for command processing, and provide low-latency query responses. Events must be encrypted at rest and in transit. Offline delivery of events to clients is required. The system must maintain P99 latency under 200ms for query reads.

Constraints
  • End-to-end encryption of events at rest and in transit
  • Support offline event delivery with reliable replay
  • P99 query latency under 200ms at 50M DAU scale
  • Event ordering guarantees per aggregate for consistency
  • High write throughput: ~1M events/sec sustained
  • Separation of command and query workloads (CQRS)
📋
Guided Steps
1/5
Step 1
Choosing the Event Store
Which storage technology will you use to store the immutable event log, and how will you partition it to scale to 1M events/sec?
💡 Hint
Consider a write-optimized, distributed log with partitioning by aggregate ID for ordering guarantees.
Key Decision
Use Apache Kafka with topic partitioning keyed by aggregate_id to ensure ordered, scalable writes and replayability.
Interviewer Expectation
Mid: Names Kafka as event store. Senior: Explains aggregate_id as partition key to guarantee ordering and linear scalability of writes.
⚠️Common Wrong Answer
Using a relational database for event storage leads to write bottlenecks and poor scalability at 1M events/sec.
Step 2
Implementing CQRS Separation
How will you separate command processing from query reads to meet the P99 < 200ms latency requirement?
💡 Hint
Think about using separate data stores optimized for writes and reads, and how to keep them in sync asynchronously.
Key Decision
Use Kafka Streams to process commands and update a read-optimized store like Cassandra asynchronously for low-latency queries.
Interviewer Expectation
Mid: Describes separate command and query stores. Senior: Details Kafka Streams for event processing and Cassandra for fast range queries with aggregate_id partition key.
⚠️Common Wrong Answer
Using a single monolithic database for both commands and queries causes contention and high query latency.
Step 3
Ensuring Event Encryption
How will you ensure events are encrypted both at rest and in transit without sacrificing throughput?
💡 Hint
Consider native encryption features in messaging and storage layers and client-side encryption options.
Key Decision
Enable TLS for Kafka communication and use Kafka's support for encryption at rest with AWS KMS-managed keys; optionally, apply client-side envelope encryption for sensitive payloads.
Interviewer Expectation
Mid: Mentions TLS and encryption at rest. Senior: Explains Kafka's integration with KMS for seamless encryption and client-side encryption trade-offs.
⚠️Common Wrong Answer
Relying only on network encryption (TLS) without encrypting data at rest risks data exposure if storage is compromised.
Step 4
Supporting Offline Event Delivery
How will you design the system to support offline clients that need to receive missed events reliably upon reconnection?
💡 Hint
Think about durable message retention and client-side cursors or offsets for replay.
Key Decision
Leverage Kafka's durable retention and consumer offsets to allow clients to resume event consumption from last acknowledged offset after offline periods.
Interviewer Expectation
Mid: Describes durable retention and offsets. Senior: Explains how consumer groups and offset commits enable exactly-once replay semantics for offline delivery.
⚠️Common Wrong Answer
Using ephemeral message queues without retention causes lost events for offline clients.
Step 5
Designing the Query Store for Low Latency
What technology will you use for the read model to achieve P99 query latency under 200ms at 50M DAU scale?
💡 Hint
Consider a horizontally scalable NoSQL store with fast range queries and tunable consistency.
Key Decision
Use Apache Cassandra with partition key as aggregate_id to enable O(1) range scans for event-sourced state queries with tunable consistency.
Interviewer Expectation
Mid: Names Cassandra for read store. Senior: Explains partition key choice and how Cassandra's LSM-tree design supports fast queries and high write throughput.
⚠️Common Wrong Answer
Using relational databases for read model causes slow queries and poor horizontal scalability.
🔌
API Design Challenge
Sketch 3 core API endpoints for the system: one for submitting commands, one for querying event state, and one for acknowledging event processing.
Hints
  • How does the client submit a command that results in an event?
  • How does the client query the current state derived from events?
  • How does the client acknowledge receipt or processing of events for offline delivery?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Use Kafka with aggregate_id partition key for event ordering and scalability
  • Separate command and query stores with Kafka Streams and Cassandra
  • Implement encryption at rest and in transit using Kafka TLS and KMS integration
  • Support offline delivery with Kafka consumer offsets and durable retention
Good to Have - 30%
  • Client-side encryption for sensitive data
  • Explain trade-offs of consistency models in Cassandra
  • Describe exactly-once processing semantics with Kafka consumer groups
Bonus - 10%
  • Discuss monitoring and alerting for event lag and processing failures
  • Propose schema evolution strategies for event versioning
Anti-patterns to Avoid
Using relational databases as event store causes write bottlenecks and poor scalability at 1M events/sec
Single monolithic database for commands and queries leads to contention and high latency under load
Relying solely on TLS without encryption at rest risks data exposure if storage is compromised
Using ephemeral queues without retention causes lost events for offline clients who reconnect
HTTP polling at 50M DAU causes 50 million reconnects per minute just to check for messages, collapsing the system under load
Ignoring partition key design leads to out-of-order events and inconsistent state across aggregates
✅
Model Answer
▼ Reveal

The system uses Apache Kafka as the event store with aggregate_id partitioning to ensure ordered, scalable writes. Kafka Streams asynchronously processes commands and updates a Cassandra read store optimized for low-latency queries. Encryption is enforced via TLS and Kafka's integration with AWS KMS, with optional client-side encryption for sensitive data. Offline delivery is supported by Kafka's durable retention and consumer offset management.

Key Decisions
  • Use Apache Kafka with aggregate_id as partition key to guarantee event ordering and scale to 1M events/sec
  • Implement CQRS by separating command processing (Kafka Streams) and query reads (Cassandra) asynchronously
  • Enable TLS for all Kafka communication and use Kafka's integration with AWS KMS for encryption at rest
  • Support offline clients by leveraging Kafka's durable retention and consumer offset management for reliable replay
  • Apply client-side envelope encryption for sensitive data to enhance security beyond transport and storage encryption
  • Apache Kafka as the event store with aggregate_id as partition key for ordered, scalable writes
  • Kafka Streams for processing commands and updating read models asynchronously
  • Apache Cassandra as the read-optimized store with aggregate_id partition key for low-latency queries
  • TLS encryption for Kafka communication and Kafka's encryption at rest integrated with AWS KMS
  • Client-side envelope encryption for sensitive event payloads to enhance security
  • Kafka consumer groups and offset management to support offline event delivery and replay
  • Separation of command and query workloads (CQRS) to optimize for throughput and latency
  • Use of consumer offset commits to enable exactly-once processing semantics for event delivery

Practice

(1/5)
1. A system must enforce a maximum number of API calls per user per minute to prevent abuse. Which component in an API Gateway architecture is best suited to handle this requirement?
easy
A. The API Gateway applies rate limiting before routing requests to backend services
B. The backend microservices implement rate limiting individually to distribute load
C. A centralized database tracks request counts and enforces limits synchronously
D. Clients are responsible for self-throttling their request rates

Solution

  1. Step 1: Identify rate limiting responsibility

    The API Gateway is the entry point and can efficiently enforce rate limits before requests reach backend services, preventing overload.

  2. Step 2: Eliminate other options

    The backend microservices implement rate limiting individually to distribute load spreads rate limiting, risking inconsistent enforcement and backend overload. A centralized database tracks request counts and enforces limits synchronously adds latency and complexity by synchronous DB checks. Clients are responsible for self-throttling their request rates trusts clients, which is insecure.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    API Gateway enforces rate limits upfront to protect backend [check]
Hint: Rate limiting belongs at the gateway entry point [check]
Common Mistakes:
2. In a system using Zookeeper for leader election, at step 4 in the request trace, the follower node receives a notification that the leader's ephemeral node has disappeared. What state change does the follower node perform at this step?
easy
A. The follower node attempts to create a new ephemeral node to become leader
B. The follower node restarts its local cache without leader election
C. The follower node ignores the event and continues as follower
D. The follower node immediately assumes leadership without coordination

Solution

  1. Step 1: Understand Zookeeper ephemeral node disappearance

    Leader's ephemeral node disappearance signals leader failure.

  2. Step 2: Follower reaction

    Follower tries to create ephemeral node to become new leader; immediate assumption without coordination risks split-brain.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Follower creates ephemeral node -> attempts leadership [check]
Hint: Ephemeral node gone -> followers try to create leader node [check]
Common Mistakes:
3. For a large-scale code repository system supporting millions of users, which branching strategy best balances performance and user flexibility?
medium
A. Use a centralized locking mechanism to serialize all branch creation and merges
B. Use lightweight branch pointers referencing immutable commits with eventual consistency
C. Use full branch copies stored separately for each user to avoid conflicts
D. Use synchronous replication of branch metadata across all servers before allowing merges

Solution

  1. Step 1: Analyze branching strategies for scale

    Lightweight branch pointers referencing immutable commits allow fast branch creation and merges without locking, supporting high concurrency.

  2. Step 2: Evaluate other options

    Centralized locking serializes operations, hurting performance; full branch copies waste storage and complicate merges; synchronous replication adds latency and reduces availability.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Immutable commits + lightweight pointers -> scalable branching [check]
Hint: Immutable commits enable lightweight, scalable branches [check]
Common Mistakes:
4. For a system requiring minimal key remapping and uniform load distribution when nodes are frequently added or removed, which approach is best?
medium
A. Use rendezvous hashing without virtual nodes
B. Use modulo hashing with a fixed number of partitions per node
C. Use a centralized directory that reassigns keys on every node change
D. Use consistent hashing with virtual nodes and assign multiple tokens per physical node

Solution

  1. Step 1: Evaluate modulo hashing with fixed partitions

    Modulo hashing causes large remapping on node changes, poor for frequent changes.

  2. Step 2: Compare consistent hashing with virtual nodes

    Virtual nodes spread load evenly and reduce remapping, ideal for frequent node changes.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Virtual nodes in consistent hashing balance load and minimize remapping [check]
Hint: Virtual nodes smooth load and reduce remapping on node changes [check]
Common Mistakes:
5. For a time-series IoT data ingestion system requiring high write throughput and efficient range queries by device and time, which data modeling approach is best?
medium
A. Partition by timestamp and cluster by device ID
B. Partition by device ID and cluster by timestamp within each partition
C. Use a single partition key for all devices and cluster by timestamp
D. Use a relational database with indexes on device and timestamp

Solution

  1. Step 1: Analyze query and write patterns

    Queries filter by device and time ranges; writes are high volume per device.

  2. Step 2: Evaluate partitioning strategies

    Partitioning by device ID localizes writes and queries; clustering by timestamp supports efficient range scans. Single partition key causes hotspots; partitioning by timestamp scatters device data, hurting queries; relational DBs can't handle high write scale.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Partition by device ID + cluster by timestamp -> balanced writes and efficient queries [check]
Hint: Partition by device, cluster by time for time-series data [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard