Bird
Raised Fist0

Message Queue Design - Producers, Consumers, Brokers, Partitions

HLDmediumInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
▶
Request Trace
Step 1/5
Producers

Send message to Broker

Producer sends a message over WebSocket to Broker

Protocol:WebSocket
BeforeMessage exists only in Producer's local buffer
AfterMessage received by Broker, ready for routing
Brokers

Determine partition and forward message

Broker uses metadata from ZooKeeper to select partition

Protocol:Kafka Protocol
BeforeBroker has message but no assigned partition
AfterMessage forwarded to selected Partition for storage
Partitions

Persist message in partition log

Partition appends message to its durable log

Protocol:Internal Broker Protocol
BeforePartition log does not contain the new message
AfterMessage durably stored in Partition log
Consumers

Fetch message from partition

Consumer polls Partition and retrieves new message

Protocol:ACK Protocol
BeforeConsumer has not yet received the new message
AfterConsumer has message and sends ACK to Broker
ZooKeeper / Metadata Service

Maintain cluster metadata and partition leadership

ZooKeeper updates Broker with partition leader info

Protocol:ZooKeeper Protocol
BeforeBroker may have stale partition leadership info
AfterBroker has up-to-date metadata for routing decisions
🏗️
Architecture Diagram
Architecture Diagram
┌───────────────┐
│   Producers   │
└───────┬───────┘
        │ WebSocket
        ▼
┌───────────────┐
│    Brokers    │
│ (Msg Hub)     │
└───────┬───────┘
        │ Kafka Protocol
        ▼
┌───────────────┐
│  Partitions   │
│ (Msg Queues)  │
└───────┬───────┘
        │ Internal Broker Protocol
        ▼
┌───────────────┐
│   Consumers   │
└───────┬───────┘
        │ ACK Protocol
        ▼
┌───────────────────────────────┐
│ ZooKeeper / Metadata Service   │
│ (Cluster Coord. & State)       │
└───────────────────────────────┘
Design Rationale

This architecture diagram illustrates the vertical flow of a message from Producers through Brokers and Partitions to Consumers, with ZooKeeper coordinating cluster metadata. Producers send messages over WebSocket to Brokers, which use ZooKeeper metadata to route messages to the correct Partition. Partitions store messages durably in ordered logs. Consumers poll Partitions to fetch messages and acknowledge receipt. ZooKeeper ensures Brokers have current partition leadership and cluster state, enabling consistent routing and fault tolerance. This design highlights the separation of concerns: Producers and Consumers as clients, Brokers as message routers, Partitions as durable queues, and ZooKeeper as the metadata coordinator.

Primary Focus

✓

Practice

(1/5)
1. A messaging system must guarantee ordered processing of events per user session while supporting high throughput. Which Kafka feature ensures this ordering?
easy
A. Partitioning messages by user session key
B. Setting a short retention policy to reduce storage
C. Using multiple consumer groups to parallelize consumption
D. Using Kafka Streams for message transformation

Solution

  1. Step 1: Understand parallelism vs ordering

    Using multiple consumer groups increases parallelism but does not guarantee ordering per session.

  2. Step 2: Identify ordering guarantee

    Partitioning messages by user session key ensures ordering within partitions.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Multiple consumer groups increase throughput but ordering per session requires partitioning [check]
Hint: Multiple consumer groups increase throughput [OK]
Common Mistakes:
2. For a system requiring low-latency approximate frequency queries on a data stream with heavy skew, which approach is better and why?
medium
A. Use Count-Min Sketch with conservative update to reduce overestimation bias
B. Use a single large hash table with exact counts to avoid approximation errors
C. Use a Bloom filter combined with exact counters for heavy hitters
D. Use a HyperLogLog sketch to estimate frequencies with lower memory

Solution

  1. Step 1: Evaluate exact counts vs approximation

    Exact counts require large memory and high latency for heavy skew data streams.

  2. Step 2: Recognize Count-Min Sketch with conservative update

    Conservative update reduces overestimation bias, improving accuracy for skewed data.

  3. Step 3: Eliminate Bloom filter and HyperLogLog

    Bloom filters don't count frequencies; HyperLogLog estimates cardinality, not frequency.

  4. Final Answer:

    Option A -> Option A

  5. Quick Check:

    Conservative update Count-Min Sketch balances accuracy and memory for skewed streams [check]
Hint: Conservative update reduces bias in Count-Min Sketch for skewed data [check]
Common Mistakes:
3. For a global API rate limiter with strict per-user limits, which architecture best balances consistency and latency?
medium
A. Use client-side counters with periodic server reconciliation
B. Use a single centralized Redis instance for all counters
C. Use a sharded Redis cluster with consistent hashing per user ID
D. Use a distributed SQL database with strong consistency guarantees

Solution

  1. Step 1: Analyze consistency and latency needs

    Strict per-user limits require consistent counters with low latency.

  2. Step 2: Evaluate architectures

    Centralized Redis (A) is a bottleneck; client-side counters (C) are eventually consistent and unreliable; distributed SQL (D) has high latency. Sharded Redis with consistent hashing (B) balances load and maintains per-user consistency.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Sharded Redis cluster scales with consistent hashing per user [check]
Hint: Shard counters by user ID for scalable consistency [check]
Common Mistakes:
4. If the health monitoring component of the GSLB fails silently, what is the most likely failure observed by end users?
medium
A. Users are routed to unhealthy data centers causing request failures or timeouts
B. Traffic is evenly distributed regardless of data center health, causing overload
C. DNS queries fail to resolve, causing complete service outage
D. Users experience increased latency due to rerouting through backup data centers

Solution

  1. Step 1: Understand health monitor role

    The health monitor informs GSLB which data centers are healthy; without it, GSLB cannot avoid unhealthy endpoints.

  2. Step 2: Analyze failure impact

    Without health info, GSLB routes users to unhealthy data centers causing failures (Users are routed to unhealthy data centers causing request failures or timeouts). Increased latency (Users experience increased latency due to rerouting through backup data centers) or DNS failures (DNS queries fail to resolve, causing complete service outage) are indirect or unlikely. Even traffic distribution (Traffic is evenly distributed regardless of data center health, causing overload) ignores health but is less precise than routing to unhealthy nodes.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Silent health monitor failure -> routing to unhealthy data centers [check]
Hint: No health data -> users sent to unhealthy data centers [check]
Common Mistakes:
5. At 10x traffic scale, which API Gateway component is most likely to become the bottleneck first, and why?
hard
A. Rate limiting store, because centralized counters cause contention and latency spikes
B. Authentication service, because token validation is CPU-intensive and stateful
C. API Gateway routing logic, because routing tables grow exponentially with services
D. Backend microservices, because they cannot horizontally scale as easily as the gateway

Solution

  1. Step 1: Identify scaling challenges

    Rate limiting often relies on centralized counters or stores (e.g., Redis) that become contention points under high request rates.

  2. Step 2: Eliminate other options

    Backend microservices, because they cannot horizontally scale as easily as the gateway is incorrect as backend services typically scale horizontally. API Gateway routing logic, because routing tables grow exponentially with services is unlikely as routing tables are cached and optimized. Authentication service, because token validation is CPU-intensive and stateful is less likely as token validation is stateless and can be cached.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Centralized rate limiting store bottlenecks first at scale [check]
Hint: Centralized counters bottleneck rate limiting at scale [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard