Bird
Raised Fist0

Message Queue Design - Producers, Consumers, Brokers, Partitions

HLDmediumInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Message Queue Design - Producers, Consumers, Brokers, Partitions
Design a scalable, durable, and low-latency distributed message queue system supporting high throughput producers and consumers with partitioned brokers.
mediumMessage Queues and Event StreamingFreq: high
Prerequisites:System Design FundamentalsCAP TheoremDistributed Systems BasicsData Partitioning and Replication
⚙️
Functional Requirements
  • Support multiple producers publishing messages concurrently to topics.
  • Support multiple consumers subscribing to topics with at-least-once delivery semantics.
  • Partition topics to enable horizontal scalability and parallel consumption.
  • Ensure message ordering within partitions.
  • Provide message durability with replication across brokers.
  • Allow consumers to track and commit offsets for message processing.
📊
Non-Functional Requirements
  • Support 2 billion daily active users (DAU).
  • Handle peak write throughput of approximately 925,000 messages per second.
  • Ensure 99.99% availability for producers and consumers.
  • Achieve end-to-end message delivery latency under 100ms P99.
  • Support message retention of 7 days with replication factor 3.
❓
Clarifying Questions to Ask
  • What is the expected average message size?
  • Are messages ordered globally or only within partitions?
  • What consistency guarantees are required for message delivery?
  • What is the expected message retention period?
  • Should the system support both push and pull consumer models?
  • Is exactly-once delivery required or is at-least-once sufficient?
Out of Scope
✗ Designing the client SDKs for producers and consumers.
✗ Implementing complex stream processing or transformation logic.
✗ Security features like encryption and authentication.
✗ Cross-data center geo-replication.
📌
Assumptions
  • 2 billion DAU
  • 40 messages per user per day
  • Average message size 100 bytes
  • Replication factor 3
⚡
QPS Calculation
write qps2B × 40 / 86400 ≈ 925,926 msg/sec
peak write qpsAssuming 2× peak factor → 1,851,852 msg/sec
read qpsAssuming each message consumed once → 925,926 msg/sec
💾
Storage Estimate
daily storage925,926 msg/sec × 100 bytes × 86400 sec ≈ 8 TB/day
retention storage8 TB/day × 7 days = 56 TB
with replication56 TB × 3 = 168 TB total storage
🖥️
Server Estimate
broker serversAssuming each broker handles 50K msg/sec → 1,851,852 / 50,000 ≈ 38 brokers
storage nodesAssuming each node stores 5 TB → 168 TB / 5 TB = 34 storage nodes
🏗️
Architecture Diagram
Architecture Diagram
          +------------+
          |  Producers |
          +-----+------+
                |
                v
          +-----+------+
          |   Brokers  |
          | (Kafka-like|
          |  cluster)  |
          +-----+------+
          /      |      \
         /       |       \
   +----+--+ +---+----+ +--+----+
   |Partition| |Partition| |Partition|
   |  0      | |  1      | |  N      |
   +----+---+ +----+----+ +----+----+
         |          |          |
         v          v          v
   +-----+----+ +---+-----+ +--+-----+
   |Consumers | |Consumers| |Consumers|
   +----------+ +---------+ +---------+
🧩
Components
Producers
Clients that publish messages to topics asynchronously.
Why thisProducers push messages to brokers using a high-throughput TCP protocol (e.g., Kafka Producer API) optimized for batching and compression.
Why notPolling-based producers increase latency and reduce throughput, unsuitable for 1.8M msg/sec peak.
Brokers
Cluster nodes that receive, store, replicate, and serve messages partitioned by topic.
Why thisApache Kafka brokers provide partitioned logs with durable storage on disk, replication, and fault tolerance at scale.
Why notRabbitMQ uses in-memory queues limiting durability and throughput at 925K+ msg/sec scale.
Partitions
Logical shards of a topic enabling parallelism and ordering guarantees within each partition.
Why thisPartitioning allows horizontal scaling of throughput and storage; ordering is guaranteed per partition.
Why notSingle log per topic limits throughput and causes hot partitions under skewed keys.
Consumers
Clients that subscribe to partitions and consume messages, committing offsets to track progress.
Why thisKafka Consumer API supports offset management and group coordination for load balancing and fault tolerance.
Why notPolling consumers without offset commit risk duplicate processing and lack fault recovery.
ZooKeeper / Metadata Service
Manages broker metadata, partition leader election, and cluster membership.
Why thisZooKeeper provides consistent coordination and leader election critical for broker failover and partition assignment.
Why notUsing a custom metadata store risks split-brain and inconsistent cluster state at scale.
🔄
Data Flow
1
Producer → Broker
Producer sends message to broker partition based on partition key hashing.
Ensures messages with same key go to same partition for ordering.
2
Broker → Broker Replicas
Broker appends message to local log and replicates to follower brokers.
Replication factor 3 ensures durability and fault tolerance.
3
Broker → Producer
Broker acknowledges message write after leader commit.
Provides durability guarantee before ack.
4
Consumer → Broker
Consumer fetches messages from assigned partitions using offset.
Allows parallel consumption and load balancing.
5
Consumer → Broker
Consumer commits offset after processing messages.
Enables at-least-once delivery and recovery after failure.

Practice

(1/5)
1. A system must efficiently handle millions of concurrent calendar availability updates from hosts worldwide, ensuring low latency and eventual consistency. Which component is best suited to buffer and process these updates asynchronously?
easy
A. Use a relational database with strong ACID transactions for immediate consistency
B. Use a cache like Redis with TTL to store availability states temporarily
C. Use direct synchronous API calls to update availability in the main database
D. Use a distributed message queue like Kafka to decouple update ingestion from processing

Solution

  1. Step 1: Identify the need for asynchronous buffering

    Millions of concurrent updates require decoupling ingestion from processing to avoid bottlenecks.

  2. Step 2: Evaluate component suitability

    Relational DB (A) causes high contention and latency; Redis (C) is volatile and not durable; synchronous API calls (D) block and reduce throughput. Kafka (B) provides durable, ordered, and scalable buffering.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Kafka buffers high-throughput writes asynchronously with durability [check]
Hint: Buffer high-write updates asynchronously with durable queues [check]
Common Mistakes:
2. In a graph database request flow, at step 4 the query engine evaluates the traversal depth limit. What happens if the depth limit is reached before finding the target node?
easy
A. The traversal stops immediately and returns partial results up to that depth
B. The query engine throws an error and aborts the entire query
C. The query engine restarts traversal from the root node with a higher limit
D. The query engine continues traversing all remaining edges ignoring the limit

Solution

  1. Step 1: Understand traversal depth limit role

    The depth limit caps how far the traversal explores to avoid excessive computation.

  2. Step 2: Behavior at limit

    When the limit is reached, traversal stops and returns partial results found so far, not an error or restart.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Traversal depth limit stops traversal and returns partial results [check]
Hint: Traversal stops at depth limit, returns partial results [check]
Common Mistakes:
3. If the event ingestion pipeline fails to persist incoming metric data, what is the exact failure observed in the alerting system?
medium
A. Threshold and anomaly detectors receive incomplete data, causing missed or false alerts
B. Alerts are generated with delayed timestamps but eventually delivered
C. Composite alerts continue to function normally using cached inputs
D. The system switches to a backup data source automatically without alert impact

Solution

  1. Step 1: Role of event ingestion pipeline

    This pipeline persists raw metric data for downstream alerting components.

  2. Step 2: Consequence of pipeline failure

    Without persistence, threshold and anomaly detectors get incomplete or missing data, leading to missed alerts or false positives.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Missing data -> detectors miss or misfire alerts [check]
Hint: No data persistence -> detectors get incomplete data, miss alerts [check]
Common Mistakes:
4. You need to design an Elasticsearch cluster optimized for low-latency search queries on a read-heavy workload with occasional node failures. Which architecture choice best balances query speed and fault tolerance?
medium
A. Use multiple smaller primary shards with replica shards distributed across nodes
B. Use only replica shards for queries and keep primary shards offline
C. Disable replicas to reduce write overhead and rely on snapshot backups
D. Use a single large shard per index to minimize cross-shard coordination

Solution

  1. Step 1: Evaluate shard size and replication

    Multiple smaller shards allow parallel query execution; replicas provide fault tolerance and load balancing.

  2. Step 2: Eliminate other options

    Single large shard causes query bottleneck; disabling replicas risks data loss and no failover; using only replicas offline is invalid as primaries handle writes.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Multiple shards + replicas balance speed and fault tolerance [check]
Hint: Multiple shards + replicas optimize read speed and fault tolerance [check]
Common Mistakes:
5. At 10x scale of incidents, which post-mortem component is most likely to fail first and why?
hard
A. The blameless culture erodes first due to increased pressure and finger-pointing
B. The action item tracking system becomes overwhelmed, causing delays and lost tasks
C. The root cause analysis team becomes too large, causing coordination overhead
D. The timeline tool crashes due to excessive event volume

Solution

  1. Step 1: Analyze scaling bottlenecks in post-mortem components

    Action item tracking systems must handle many tasks; at 10x scale, manual or poorly automated systems get overwhelmed.

  2. Step 2: Compare other components

    Blameless culture erosion is a risk but not a technical failure. RCA team size affects coordination but can be mitigated. Timeline tools usually scale better with storage solutions.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Tracking overload causes lost/delayed fixes first at scale [check]
Hint: Tracking overload breaks fixes first at scale [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard