Bird
Raised Fist0

Message Queue Design - Producers, Consumers, Brokers, Partitions

HLDmediumInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Message Queue Design - Producers, Consumers, Brokers, Partitions
45 min · senior
senior

You are designing a message queue system for a startup with 50 million daily active users (DAU). The system must support encrypted messages end-to-end, guarantee offline delivery with message persistence, and maintain a P99 end-to-end latency under 200ms. Messages are exchanged in conversations between users, and ordering must be preserved per conversation. Producers and consumers connect globally, and the system must scale to handle peak throughput of approximately 1 million messages per second. The system should also support efficient consumer reconnection and cursor-based message sync.

Constraints
  • End-to-end encryption of messages at rest and in transit
  • Offline delivery with durable message persistence
  • P99 end-to-end latency under 200ms
  • Ordering guarantees per conversation (strict ordering within a conversation)
  • Support for 50M DAU with peak throughput ~1M messages/sec
📋
Guided Steps
1/5
Step 1
Partitioning Strategy for Ordering and Scalability
How would you partition messages to ensure strict ordering per conversation and scale to 1M messages/sec?
💡 Hint
Consider using conversation_id as partition key to keep all messages of a conversation on the same partition.
Key Decision
Use Apache Kafka with conversation_id as the partition key - this ensures all messages in a conversation are routed to the same partition, preserving strict ordering and enabling linear scalability by adding partitions.
Interviewer Expectation
Mid-level: Names Kafka and partitioning by conversation_id. Senior: Explains why conversation_id is better than sender_id (which scatters messages across partitions causing out-of-order delivery) and how Kafka partitions enable parallelism and ordering guarantees.
⚠️Common Wrong Answer
Using sender_id as partition key - causes messages from the same conversation to be split across partitions, breaking ordering guarantees and complicating consumer logic.
Step 2
Message Persistence and Offline Delivery
How will you persist messages to support offline delivery and ensure durability under high throughput?
💡 Hint
Consider a distributed log storage system optimized for high write throughput and sequential reads.
Key Decision
Leverage Kafka's durable log storage with replication factor ≥3 to persist messages reliably and enable offline consumers to replay messages from the log for offline delivery.
Interviewer Expectation
Mid-level: Mentions Kafka durability and replication. Senior: Details how Kafka's log compaction and retention policies support offline delivery and how replication ensures durability despite broker failures.
⚠️Common Wrong Answer
Using a relational database for message storage - relational DBs do not scale linearly for high write throughput and cause high latency for sequential reads.
Step 3
Encryption and Security
How will you implement end-to-end encryption ensuring messages are encrypted at rest and in transit without breaking ordering or delivery guarantees?
💡 Hint
Separate transport layer encryption from application-level encryption; consider encrypting message payloads client-side.
Key Decision
Use TLS for transport encryption between clients and brokers, and implement client-side encryption of message payloads before sending to Kafka to ensure end-to-end encryption without broker access to plaintext.
Interviewer Expectation
Mid-level: Mentions TLS for transport security. Senior: Explains client-side encryption to protect data at rest and in transit, and how this design preserves ordering and delivery guarantees without broker decryption.
⚠️Common Wrong Answer
Relying only on TLS without client-side encryption - brokers can access plaintext messages, violating end-to-end encryption requirements.
Step 4
Consumer Sync and Acknowledgement Mechanism
How will consumers efficiently sync missed messages after reconnecting and acknowledge processed messages to avoid duplicates?
💡 Hint
Use Kafka consumer offsets and implement cursor-based sync with offset commits.
Key Decision
Use Kafka consumer groups with offset commits stored in Kafka’s __consumer_offsets topic, enabling consumers to resume from last committed offset for efficient cursor-based sync and exactly-once processing semantics.
Interviewer Expectation
Mid-level: Describes Kafka consumer offsets and commits. Senior: Explains how offset management enables efficient reconnection, how to handle offset commit failures, and how to design idempotent consumers to avoid duplicates.
⚠️Common Wrong Answer
Using HTTP polling for sync - causes massive load spikes and high latency at 50M DAU reconnects.
Step 5
Broker Scaling and Fault Tolerance
How will you scale brokers and handle broker failures without impacting message availability or ordering?
💡 Hint
Use Kafka’s partition leader election and replication to handle broker failures transparently.
Key Decision
Deploy Kafka brokers in a cluster with partitions replicated across multiple brokers (replication factor ≥3), enabling leader election on failure and ensuring availability and ordering guarantees without downtime.
Interviewer Expectation
Mid-level: Mentions Kafka replication and leader election. Senior: Details failure scenarios, how ISR (in-sync replicas) work, and how to tune replication and min.insync.replicas for durability vs latency trade-offs.
⚠️Common Wrong Answer
Single broker deployment - causes single point of failure and loss of availability during broker downtime.
🔌
API Design Challenge
Sketch 3 core API endpoints for the message queue system: how producers send messages, how consumers receive/sync messages, and how consumers acknowledge message processing.
Hints
  • How does the client send a message to a conversation?
  • How does the client sync missed messages after reconnecting?
  • How does the client acknowledge successful message receipt?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Correct partitioning strategy using conversation_id as Kafka partition key to preserve ordering
  • Use of Kafka’s durable replicated log for message persistence and offline delivery
  • Client-side encryption combined with TLS for end-to-end encryption
  • Efficient consumer sync using Kafka consumer offsets and cursor-based resume
  • Broker fault tolerance using Kafka replication and leader election
Good to Have - 30%
  • Idempotent producer configuration to avoid duplicates
  • Use of Kafka log compaction and retention policies
Bonus - 10%
    Anti-patterns to Avoid
    ✅
    Model Answer
    ▼ Reveal

    Design uses Apache Kafka as the core messaging infrastructure, partitioning messages by conversation_id to preserve strict ordering and scale to 1M messages/sec. Messages are durably stored with replication factor ≥3 for offline delivery and fault tolerance. TLS secures transport, while client-side encryption ensures end-to-end encryption without exposing plaintext to brokers. Consumers use Kafka consumer groups with offset commits for efficient cursor-based sync and exactly-once processing. Kafka cluster handles broker failures transparently via leader election and ISR management.

    Key Decisions
    • Partition messages by conversation_id using Kafka partition key to maintain ordering
    • Persist messages in Kafka’s replicated log with replication factor ≥3 for durability and offline delivery
    • Encrypt messages client-side before sending to Kafka, combined with TLS for transport encryption
    • Use Kafka consumer groups with offset commits stored in __consumer_offsets topic for efficient consumer sync and exactly-once semantics
    • Deploy Kafka brokers in a cluster with leader election and ISR to handle failures without downtime
    • Apache Kafka with conversation_id as partition key for strict ordering and linear scalability
    • Kafka’s durable replicated log storage for offline delivery and message persistence
    • TLS for transport encryption combined with client-side payload encryption for end-to-end security
    • Kafka consumer groups with offset commits enabling efficient cursor-based consumer sync and exactly-once processing
    • Kafka cluster with replication factor ≥3 and leader election for fault tolerance and high availability
    • Idempotent producer configuration in Kafka to avoid duplicate messages during retries
    • Use of Kafka’s log compaction and retention policies to manage storage and support offline consumers
    • Monitoring and alerting on Kafka consumer lag to detect and handle slow consumers proactively

    Practice

    (1/5)
    1. In the request flow for generating a course completion certificate, at step 5 the Certificate Service verifies user progress. What happens if the progress data is stale due to delayed updates from the Progress Worker?
    easy
    A. Certificate Service rejects certificate generation and returns an error
    B. Certificate Service retries fetching progress data until fresh data is available
    C. Certificate Service issues a provisional certificate flagged as pending verification
    D. Certificate Service immediately issues the certificate ignoring progress freshness

    Solution

    1. Step 1: Trace Certificate Service behavior at Step 5

      Certificate Service requires up-to-date progress to ensure course completion before issuing certificates.

    2. Step 2: Evaluate handling of stale progress data

      Retrying (Certificate Service retries fetching progress data until fresh data is available) would cause indefinite blocking; provisional certificates (Certificate Service issues a provisional certificate flagged as pending verification) complicate trust; ignoring freshness (Certificate Service immediately issues the certificate ignoring progress freshness) risks invalid certificates.

    3. Final Answer:

      Option A -> Option A

    4. Quick Check:

      Certificate Service rejects generation if progress data is stale [check]
    Hint: Reject certificates if progress data is stale [OK]
    Common Mistakes:
    2. In a Saga-based order processing system, at Step 4 the Saga Coordinator receives a compensation event after a payment failure. What is the Coordinator's exact state transition at this step?
    easy
    A. Marks the saga as completed successfully and publishes a confirmation event
    B. Ignores the compensation event and waits for manual intervention
    C. Transitions the saga to a compensating state and triggers rollback actions for prior steps
    D. Retries the payment step immediately without changing saga state

    Solution

    1. Step 1: Trace Saga Coordinator on compensation event

      Upon payment failure, the coordinator must trigger compensating transactions to undo prior successful steps.

    2. Step 2: Confirm state transition

      The saga moves to a compensating state, ensuring rollback actions are executed to maintain consistency.

    3. Final Answer:

      Option C -> Option C

    4. Quick Check:

      Compensation event triggers rollback state transition [check]
    Hint: Compensation event -> saga enters rollback state [check]
    Common Mistakes:
    3. If the distributed lock service is unavailable, what exactly breaks in the hotel booking system's reservation flow?
    medium
    A. Reservations proceed normally but double bookings increase due to lack of coordination
    B. Reservation requests block indefinitely waiting for locks, causing user timeouts and failed bookings
    C. Inventory service marks rooms as available incorrectly, causing UI to show wrong availability
    D. Payment processing fails because it depends on the lock service for transaction ordering

    Solution

    1. Step 1: Identify lock service role in reservation flow

      The lock service ensures exclusive access to room inventory during reservation to prevent double booking.

    2. Step 2: Trace failure impact

      Without the lock service, Inventory cannot reliably mark rooms as unavailable, leading to stale or incorrect availability states shown to users (Inventory service marks rooms as available incorrectly, causing UI to show wrong availability). Reservations may still proceed but with inconsistent inventory state. Blocking indefinitely (Reservation requests block indefinitely waiting for locks, causing user timeouts and failed bookings) is unlikely as fallback mechanisms exist. Payment processing (Payment processing fails because it depends on the lock service for transaction ordering) is independent of locking. Double bookings (Reservations proceed normally but double bookings increase due to lack of coordination) may increase but the immediate failure is incorrect inventory state.

    3. Final Answer:

      Option C -> Option C

    4. Quick Check:

      Lock service failure causes stale inventory availability state [check]
    Hint: Lock service failure -> stale inventory state, not immediate booking block [check]
    Common Mistakes:
    4. If the token assignment metadata is lost or corrupted in a consistent hashing system with virtual nodes, what exactly breaks?
    medium
    A. Only new nodes fail to receive keys, existing nodes continue serving correctly
    B. Requests cannot be routed to correct nodes, causing key lookups to fail or go to wrong nodes
    C. The system continues routing keys correctly but load balancing is uneven
    D. The system falls back to modulo hashing automatically without data loss

    Solution

    1. Step 1: Understand role of token assignment metadata

      Token metadata maps virtual nodes to physical nodes; losing it breaks new node integration.

    2. Step 2: Trace failure impact

      Existing nodes continue serving keys assigned before loss; new nodes cannot be assigned tokens properly.

    3. Final Answer:

      Option A -> Option A

    4. Quick Check:

      Token metadata loss blocks new node assignment; existing nodes unaffected [check]
    Hint: Token metadata loss blocks new nodes, existing nodes serve [check]
    Common Mistakes:
    5. At 10x scale, which component in a distributed lock system using ZooKeeper is most likely to fail first and why?
    hard
    A. ZooKeeper ensemble leader, due to increased write load and session management overhead
    B. Clients, because they cannot handle the increased lock acquisition requests
    C. Network routers, because of increased traffic causing packet loss
    D. Storage backend, because ephemeral nodes consume excessive disk space

    Solution

    1. Step 1: Identify ZooKeeper leader role

      The leader handles all write requests and session heartbeats, becoming a bottleneck under high scale.

    2. Step 2: Evaluate other components

      Clients (A) can be scaled horizontally; network routers (C) are less likely the first bottleneck; ephemeral nodes (D) are lightweight and cleaned up automatically.

    3. Final Answer:

      Option A -> Option A

    4. Quick Check:

      Leader bottleneck due to write and session overhead [check]
    Hint: ZooKeeper leader handles all writes -> first bottleneck [check]
    Common Mistakes:
    Recommended

    Ready to prove it?

    Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

    10 questions~5 mineasy hard