Bird
Raised Fist0

Design a Distributed Message Queue - Kafka-Like Architecture End to End

HLDhardInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
▶
Request Trace
Step 1/5
Producers

Send message

Producer creates message and sends to Load Balancer via WebSocket

Protocol:WebSocket
BeforeMessage is plaintext on Producer device only
AfterMessage is in-flight over WebSocket to Load Balancer
Load Balancer

Route message

Load Balancer forwards message to a Kafka Broker over TCP

Protocol:TCP
BeforeMessage received from Producer via WebSocket
AfterMessage is queued for Kafka Broker processing
Kafka Brokers

Persist message

Kafka Broker appends message to partition log and replicates

Protocol:Kafka protocol
BeforeMessage queued for persistence
AfterMessage durably stored and replicated in Kafka partition
Zookeeper

Coordinate cluster

Zookeeper manages broker metadata and leader election

Protocol:Zookeeper protocol
BeforeKafka Brokers operating with current metadata
AfterCluster metadata updated and consistent
Consumers

Fetch message

Consumer polls Kafka Broker and receives new message

Protocol:Kafka protocol
BeforeConsumer offset behind latest message
AfterConsumer offset advanced; message delivered to client
🏗️
Architecture Diagram
Architecture Diagram
┌───────────────┐
│   Producers   │
└───────┬───────┘
        │ WebSocket
        ▼
┌───────────────┐
│ Load Balancer │
└───────┬───────┘
        │ TCP
        ▼
┌───────────────┐
│ Kafka Brokers │
│  (Partitioned)│
└───────┬───────┘
        │ Zookeeper coord.
        ▼
┌───────────────┐
│  Zookeeper    │
└───────────────┘
        ▲
        │ Kafka protocol
┌───────┴───────┐
│   Consumers   │
└───────────────┘
Design Rationale

This Kafka-like distributed message queue architecture shows how Producers send messages through a Load Balancer to Kafka Brokers, which partition and durably store messages. Zookeeper coordinates cluster metadata and leader election to ensure consistency and availability. Consumers poll Kafka Brokers to fetch messages, advancing their offsets. The vertical flow emphasizes the asynchronous, decoupled nature of message delivery, with Load Balancer distributing load and Kafka Brokers ensuring replication and fault tolerance. This design supports high throughput and fault tolerance by separating concerns and using Zookeeper for coordination.

Primary Focus

✓

Practice

(1/5)
1. In the request flow of a real-time ML inference system, at Step 4 the Model Router receives a prediction request. What happens if the system is currently in shadow mode with a new model deployed?
easy
A. The Model Router sends the request to both the production and shadow models but returns only the production model's prediction.
B. The Model Router forwards the request only to the new shadow model and returns its prediction.
C. The Model Router queues the request until the shadow model finishes training before forwarding.
D. The Model Router rejects the request to avoid inconsistent predictions during shadow testing.

Solution

  1. Step 1: Understand shadow mode request routing

    Shadow mode runs new models in parallel without affecting live traffic. So, requests are sent to both models.

  2. Step 2: Identify response behavior

    Only the production model's prediction is returned to the client; shadow model predictions are logged for evaluation.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Shadow mode duplicates requests but returns production output [check]
Hint: Shadow mode duplicates requests; only production output returned [OK]
Common Mistakes:
2. In a system where a client writes a user profile update to DynamoDB, then a cache invalidation message is sent to Redis at step 4, what happens if the cache invalidation fails at step 4?
easy
A. DynamoDB write is rolled back automatically to maintain consistency
B. Redis automatically retries invalidation and blocks client until success
C. The cache remains stale until TTL expires or manual refresh
D. Client receives an error and retries the entire update operation

Solution

  1. Step 1: Trace cache invalidation at step 4

    Cache invalidation is asynchronous and best-effort; failure does not rollback DB writes.

  2. Step 2: Understand failure impact on cache state

    If invalidation fails, Redis cache remains stale until TTL expiry or next explicit refresh.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Failed cache invalidation -> stale cache until TTL or manual refresh [check]
Hint: Failed cache invalidation -> stale cache, DB write persists [check]
Common Mistakes:
3. A public API receives unpredictable bursts of traffic that can overwhelm backend services. Which component is best suited to protect the system from overload?
easy
A. A distributed cache like Redis to store frequent requests for faster responses
B. A rate limiter that enforces request quotas per client IP or API key
C. A load balancer that evenly distributes requests across servers
D. A message queue to buffer incoming requests before processing

Solution

  1. Step 1: Identify the protection mechanism against traffic bursts

    Distributed cache (A) improves response time but doesn't limit request rate; load balancer (C) distributes load but can't prevent overload from bursts; message queue (D) buffers but can be overwhelmed if bursts are too large.

  2. Step 2: Recognize rate limiter role

    Rate limiter (B) enforces quotas per client, directly preventing backend overload by rejecting or delaying excess requests.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Rate limiter enforces quotas to prevent overload [check]
Hint: Rate limiter controls request bursts to protect backend [check]
Common Mistakes:
4. Without a rate limiter in place, what exactly happens when a sudden traffic spike occurs on a backend service?
medium
A. The backend service crashes or becomes unresponsive, causing client requests to time out or fail
B. The backend service rejects requests gracefully with error codes indicating overload
C. The backend service automatically scales up instantly to handle the spike
D. The backend service queues all requests, causing increased latency but no failures

Solution

  1. Step 1: Understand backend behavior without rate limiting

    Without rate limiting, traffic spikes overwhelm backend, causing resource exhaustion and crashes or unresponsiveness.

  2. Step 2: Eliminate incorrect failure modes

    Automatic instant scaling (C) is unrealistic; graceful rejection (D) requires rate limiting; queuing all requests (A) leads to resource exhaustion and eventual failure.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Spike without limiter -> backend crash or unresponsiveness [check]
Hint: No limiter -> spike crashes backend, causing timeouts [check]
Common Mistakes:
5. At 10x scale, which component in a microservices architecture is most likely to become the first bottleneck, and why?
hard
A. The shared database, because it cannot be sharded easily without complex coordination
B. The individual microservices, because their stateful WebSocket connections limit horizontal scaling
C. The service discovery system, because it must track exponentially more services and instances
D. The API Gateway, because it handles all incoming requests and becomes a centralized choke point

Solution

  1. Step 1: Identify stateful components at risk

    Stateful microservices holding WebSocket connections cannot easily shard or move connections, limiting horizontal scaling.

  2. Step 2: Compare other components' scaling characteristics

    Shared DB and service discovery can be sharded or cached; API Gateway can be scaled horizontally with load balancers.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Stateful WebSocket connections limit microservices horizontal scaling [check]
Hint: Stateful WebSocket limits microservices scaling first [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard