Bird
Raised Fist0

Design a Distributed Message Queue - Kafka-Like Architecture End to End

HLDhardInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Design a Distributed Message Queue - Kafka-Like Architecture End to End
Design a scalable, fault-tolerant distributed message queue system inspired by Kafka for high throughput and durability.
hardInfrastructure and Platform SystemsFreq: high
Prerequisites:Message QueuesPub-Sub Pattern
⚙️
Functional Requirements
  • Publish messages to topics with partitioning for parallelism and scalability.
  • Subscribe consumers to topics with support for consumer groups and offset management.
  • Durable message storage with configurable retention policies (time and size based).
  • Support for message ordering within partitions.
  • High throughput ingestion and delivery with low latency (sub-100ms P99).
  • Support for exactly-once or at-least-once delivery semantics.
📊
Non-Functional Requirements
  • System must support 2 billion daily active users generating 40 messages per user per day.
  • End-to-end latency P99 for message delivery should be under 100ms.
  • Availability SLA of 99.99% (less than ~52 minutes downtime per year).
  • Durability with replication factor of at least 3 to tolerate node failures.
  • Scalable to handle peak write QPS of approximately 925,000 messages per second.
❓
Clarifying Questions to Ask
  • What is the expected message size distribution and max message size?
  • Are message ordering guarantees required globally or only within partitions?
  • What delivery semantics are mandatory: at-least-once, at-most-once, or exactly-once?
  • What is the expected retention period for messages (hours, days, weeks)?
  • Should the system support dynamic topic creation and partition scaling?
  • Are consumers expected to be push-based or pull-based for message delivery?
Out of Scope
✗ Message content transformation or enrichment (e.g., filtering, routing logic).
✗ Complex query or analytics on message content (beyond simple offset-based consumption).
✗ Security features like encryption at rest or in transit (assumed handled externally).
✗ Integration with external systems (e.g., connectors, stream processing frameworks).
📌
Assumptions
  • 2 billion daily active users (DAU).
  • Each user produces 40 messages per day on average.
  • Average message size is 100 bytes.
  • Replication factor is 3 for durability.
⚡
QPS Calculation
write qps2,000,000,000 × 40 / 86,400 ≈ 925,926 messages/sec
peak write qpsAssuming 2× peak factor: 925,926 × 2 ≈ 1,851,852 messages/sec
read qpsAssuming each message is consumed once by one consumer group: ~925,926 messages/sec
💾
Storage Estimate
daily storage925,926 msg/sec × 100 bytes × 86,400 sec ≈ 8 TB/day uncompressed
retention days7
total storage8 TB/day × 7 days = 56 TB raw
with replication56 TB × 3 = 168 TB total storage needed
🖥️
Server Estimate
broker nodesAssuming each broker handles 10,000 writes/sec, need ~185 brokers
zookeeper nodes3-5 nodes for metadata management and leader election
🏗️
Architecture Diagram
Architecture Diagram
          +-------------+
          |  Producers  |
          +------+------+
                 |
                 v
          +------+-------+
          |   Load       |
          | Balancer     |
          +------+-------+
                 |
                 v
          +------+-------+
          | Kafka Brokers|
          | (Partitioned |
          |  Topics)     |
          +------+-------+
                 |
        +--------+---------+
        |                  |
  +-----v-----+      +-----v-----+
  | Zookeeper |      | Consumers |
  | (Metadata |      | (Consumer |
  |  Mgmt)    |      |  Groups)  |
  +-----------+      +-----------+
🧩
Components
Kafka Brokers
Store messages durably in partitioned logs, handle writes and reads, replicate data for fault tolerance.
Why thisKafka brokers use append-only logs with segment files enabling high-throughput sequential disk writes and efficient replication.
Why notTraditional relational databases like PostgreSQL require complex manual sharding and do not scale linearly to 925K writes/sec; B-tree writes cause random I/O bottlenecks.
Zookeeper
Manage cluster metadata, leader election for partitions, and track consumer group offsets.
Why thisZookeeper provides consistent distributed coordination with strong consistency guarantees needed for partition leader election and offset management.
Why notEtcd or Consul offer similar features but Kafka’s ecosystem and tooling are tightly integrated with Zookeeper; also, Etcd’s performance at this scale is less proven in Kafka-like workloads.
Producers
Publish messages to topics with partitioning logic to distribute load evenly across brokers.
Why thisKafka producer clients support configurable partitioners and batching to optimize throughput and latency.
Why notSimple HTTP APIs or REST-based producers lack native batching and partition awareness, causing higher latency and lower throughput.
Consumers
Subscribe to topics, fetch messages from brokers, manage offsets, and support consumer groups for load balancing.
Why thisKafka consumers support pull-based consumption with offset commits, enabling fault-tolerant and scalable consumption patterns.
Why notPush-based consumers like MQTT brokers do not scale well for large consumer groups and lack offset management for replay.
Load Balancer
Distribute incoming producer requests evenly across Kafka brokers to prevent hotspots.
Why thisA Layer 4 TCP load balancer (e.g., HAProxy or Envoy) efficiently balances connections without adding latency or complexity.
Why notDNS round-robin lacks health checks and can cause uneven load distribution; application-level routing adds unnecessary complexity.
🔄
Data Flow
1
Producer → Load Balancer
Send message publish request with topic and key
Producer applies partitioner to determine partition key
2
Load Balancer → Kafka Broker
Forward request to appropriate broker based on partition
Load balancer balances connections across brokers
3
Kafka Broker → Local Disk + Replicas
Append message to partition log segment and replicate to follower brokers
Replication ensures durability and fault tolerance
4
Consumer → Kafka Broker
Fetch messages from assigned partitions using offset
Consumers pull messages and commit offsets for fault tolerance
🗄️
Messages
PK: topic_partition
topic_partition STRINGoffset BIGINTtimestamp TIMESTAMPkey STRINGvalue BLOB
Stores messages per topic partition with offset ordering.
🗄️
ConsumerOffsets
PK: consumer_group
consumer_group STRINGtopic_partition STRINGoffset BIGINTcommit_timestamp TIMESTAMP
Tracks committed offsets per consumer group and partition.
🔌
PublishMessage
Request
{
  "topic": "string",
  "key": "string",
  "value": "bytes"
}
Response
{
  "status": "string",
  "offset": "long"
}
Producers call this to publish messages to a topic.
🔌
FetchMessages
Request
{
  "topic": "string",
  "partition": "int",
  "offset": "long",
  "max_bytes": "int"
}
Response
{
  "messages": "list of messages",
  "next_offset": "long"
}
Consumers call this to fetch messages starting from an offset.
🔌
CommitOffset
Request
{
  "consumer_group": "string",
  "topic": "string",
  "partition": "int",
  "offset": "long"
}
Response
{
  "status": "string"
}
Consumers call this to commit their read offsets.

Practice

(1/5)
1. In the timeline generation flow, at step 4 the Delivery Worker reads from Kafka and checks the follower's timeline cache TTL. What happens if the cache TTL has expired for a follower at this step?
easy
A. The Delivery Worker triggers a timeline rebuild for that follower before delivering
B. The Delivery Worker retries reading the cache until TTL refresh completes
C. The Delivery Worker writes the tweet directly to the follower's timeline DB without cache refresh
D. The Delivery Worker skips the follower and marks the message as delivered

Solution

  1. Step 1: Understand Delivery Worker cache TTL check

    If TTL expired, the cached timeline is stale and must be rebuilt to ensure fresh data.

  2. Step 2: Evaluate actions on expired TTL

    Skipping (A) loses data; direct DB write (C) bypasses cache rebuild; retrying cache read (D) causes blocking and delays.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Expired TTL triggers timeline rebuild to maintain freshness [check]
Hint: Expired cache TTL triggers timeline rebuild [check]
Common Mistakes:
2. In an Elasticsearch query flow, at Step 4 the Coordinating Node receives partial results from all shards. What does the Coordinating Node do at this step before returning the final response?
easy
A. It merges and sorts the partial results from shards to produce a unified response
B. It forwards the query to the Master Node for final aggregation
C. It writes the query results to a temporary index for caching
D. It directly queries the replica shards again to verify consistency

Solution

  1. Step 1: Understand Coordinating Node role at Step 4

    The Coordinating Node collects partial results from primary and replica shards and merges them.

  2. Step 2: Eliminate incorrect actions

    Master Node does not aggregate query results; caching is handled elsewhere; querying replicas again is unnecessary for consistency at this stage.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Coordinating Node merges and sorts shard results [check]
Hint: Coordinating Node merges shard results before response [check]
Common Mistakes:
3. In a system where a cache is checked before querying the database, at Step 3 the Cache Layer receives a read request. What happens if the cache misses at this step?
easy
A. The Cache Layer forwards the request to the Database and waits for the response
B. The Cache Layer returns a null response immediately to the client
C. The Cache Layer retries the cache read multiple times before failing
D. The Cache Layer writes a placeholder value to the cache and returns it

Solution

  1. Step 1: Trace Cache Layer behavior on miss

    On a cache miss, the Cache Layer must fetch data from the Database to serve the request.

  2. Step 2: Understand why other options fail

    Returning null or placeholders breaks correctness; retrying cache reads wastes time without new data.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Cache miss triggers DB fetch before responding [check]
Hint: Cache miss -> fetch from DB before reply [check]
Common Mistakes:
4. If the artifact registry becomes unavailable during deployment, what is the most likely failure mode in the CI/CD pipeline?
medium
A. Deployment proceeds with cached artifacts, risking inconsistent versions
B. Deployment triggers rollback immediately without attempting deployment
C. Deployment blocks waiting for artifact registry recovery, causing pipeline timeout
D. Deployment skips artifact validation and proceeds, risking corrupted deployments

Solution

  1. Step 1: Trace deployment behavior on artifact registry unavailability

    Deployment cannot validate or fetch artifacts, so it aborts and triggers rollback to maintain safety.

  2. Step 2: Eliminate other failure modes

    Proceeding with cached or skipping validation risks corrupted deployments; blocking indefinitely causes pipeline stalls but rollback is preferred.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Unavailable registry -> deployment aborts and triggers rollback [check]
Hint: No artifact registry -> abort deployment, rollback triggered [check]
Common Mistakes:
5. If the prefix hash map component fails and becomes unavailable, what is the immediate impact on the autocomplete system's behavior?
medium
A. The system returns no results for prefix queries until the prefix hash map recovers
B. The Trie component automatically handles prefix queries without any degradation in performance
C. Autocomplete queries fall back to Elasticsearch but with increased latency and less precise prefix matching
D. User input is cached locally on the client until the prefix hash map is restored

Solution

  1. Step 1: Understand component roles

    The prefix hash map is an optimization for O(1) prefix lookups, but the Trie is the primary data structure for prefix matching.

  2. Step 2: Trace failure impact

    If the prefix hash map fails, the system falls back to the Trie for prefix queries, which still works but may have slightly higher latency.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Trie fallback ensures prefix queries continue despite prefix hash map failure [check]
Hint: Trie fallback keeps prefix search alive if prefix hash fails [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard