Bird
Raised Fist0

Pub-Sub Pattern - Topics, Subscriptions, Fan-Out Design

HLDmediumInterview freq: common
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
▶
Request Trace
Step 1/5
Producers

Publish message

Producer sends message to Kafka topic 'news'

Protocol:Kafka Produce
BeforeMessage exists only on Producer's local buffer
AfterMessage is sent to Kafka Brokers for durable storage
Kafka Brokers

Store message

Kafka appends message to topic partition log

Protocol:Kafka internal
BeforeMessage not yet persisted in Kafka topic
AfterMessage durably stored in Kafka topic partition
Subscription Manager

Consume message

Reads message from Kafka topic to identify subscribers

Protocol:Kafka Consume
BeforeMessage available in Kafka topic, not yet routed
AfterSubscription Manager knows which subscribers to notify
Delivery Workers

Fan-out message

Delivery Workers receive message and fan-out to subscribers

Protocol:Internal RPC
BeforeDelivery Workers idle, waiting for messages
AfterDelivery Workers have message ready to send to subscribers
Subscribers

Receive message

Subscribers get message via WebSocket or push notification

Protocol:WebSocket / Push
BeforeSubscribers have no new message
AfterSubscribers display new message to end users
🏗️
Architecture Diagram
Architecture Diagram
┌─────────────┐
│  Producers  │
└──────┬──────┘
       │  Kafka Produce
       ▼
┌─────────────┐
│ Kafka       │
│ Brokers     │
└──────┬──────┘
       │ Kafka Consume
       ▼
┌─────────────────────┐
│ Subscr. Manager     │
└──────┬──────────────┘
       │ Fan-Out Dispatch
       ▼
┌─────────────────┐
│ Delivery Workers│
└──────┬──────────┘
       │ WebSocket / Push
       ▼
┌─────────────┐
│ Subscribers │
└─────────────┘
Design Rationale

This architecture uses Kafka Brokers as a durable, partitioned log to store messages published by Producers. The Subscription Manager consumes messages from Kafka and determines which Subscribers are registered for each topic. Delivery Workers then fan-out the messages to all Subscribers using WebSocket or push protocols. This design decouples message production from delivery, enabling scalable fan-out and reliable message persistence. The vertical flow from Producers through Kafka, Subscription Manager, Delivery Workers, to Subscribers illustrates the clear separation of concerns and asynchronous message handling typical in pub-sub systems.

Primary Focus

✓

Practice

(1/5)
1. A distributed messaging system processes millions of events per second. Some messages fail processing due to transient errors and must be retried without blocking the main queue. Which component best handles isolating these failed messages for later inspection and retry?
easy
A. A Dead Letter Queue (DLQ) configured to capture failed messages separately
B. A cache layer like Redis to temporarily store failed messages for retry
C. A high-throughput relational database with retry logic in application code
D. A load balancer that redirects failed messages to a backup server

Solution

  1. Step 1: Identify the role of Dead Letter Queues

    DLQs isolate failed messages from the main queue, enabling retries without blocking normal processing.

  2. Step 2: Evaluate alternatives

    Relational DBs (A) add latency and complexity; Redis (C) lacks durability guarantees; load balancers (D) do not handle message state or retries.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    DLQ isolates failures for retry without blocking main queue [check]
Hint: Dead Letter Queues isolate failures, not caches or DBs [OK]
Common Mistakes:
2. For an API Gateway that must support both REST and gRPC clients, which architecture best balances protocol support and transformation complexity?
medium
A. Use gRPC-only API Gateway and require REST clients to upgrade to gRPC
B. Use separate API Gateways for REST and gRPC, each optimized for its protocol
C. Use REST-only API Gateway and require gRPC clients to use a proxy converting gRPC to REST
D. Use a single API Gateway that performs protocol translation and transformation internally

Solution

  1. Step 1: Analyze protocol support requirements

    Supporting both REST and gRPC clients in one gateway reduces operational complexity and centralizes transformation logic.

  2. Step 2: Evaluate alternatives

    Use separate API Gateways for REST and gRPC, each optimized for its protocol increases deployment and maintenance overhead. Use REST-only API Gateway and require gRPC clients to use a proxy converting gRPC to REST adds latency and complexity on client side. Use gRPC-only API Gateway and require REST clients to upgrade to gRPC is impractical as REST clients are widespread.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Single gateway with protocol translation balances support and complexity [check]
Hint: One gateway handles both protocols with internal translation [check]
Common Mistakes:
3. Without a distributed consensus system (like Paxos or Raft) coordinating leader election, what exactly breaks in a system designed for 99.999% availability?
medium
A. Clients receive stale data but system continues operating normally
B. Multiple nodes may believe they're leader simultaneously, causing conflicting writes and data corruption
C. The system automatically fails over to a backup leader without downtime
D. Leader election happens instantly with no impact on availability

Solution

  1. Step 1: Role of distributed consensus in leader election

    Consensus protocols ensure exactly one leader at a time, preventing split-brain scenarios.

  2. Step 2: Consequences of missing consensus

    Without consensus, multiple nodes may act as leader simultaneously, causing conflicting writes and data corruption, breaking availability guarantees.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    No consensus -> split-brain -> conflicting writes -> data corruption [check]
Hint: No consensus -> split-brain -> conflicting writes break availability [check]
Common Mistakes:
4. For a distributed task scheduler requiring high availability and exactly-once execution, which architecture is preferable and why?
medium
A. Use a centralized master node to assign tasks to workers synchronously
B. Use independent cron jobs on each worker node without coordination
C. Use a distributed priority queue with idempotent workers and lease-based locks
D. Use a replicated database with polling workers and no task locking

Solution

  1. Step 1: Evaluate centralized master node (A)

    Centralized master is a single point of failure and bottleneck, reducing availability.

  2. Step 2: Evaluate distributed priority queue with leases (B)

    Distributed queue with lease-based locks prevents duplicate execution, supports failover, and scales horizontally.

  3. Step 3: Evaluate cron jobs (C) and polling without locks (D)

    Cron jobs lack coordination causing duplicates; polling without locks risks multiple workers executing the same task.

  4. Final Answer:

    Option C -> Option C

  5. Quick Check:

    Distributed queue + leases enable exactly-once and high availability [check]
Hint: Leases + distributed queue = exactly-once and HA [check]
Common Mistakes:
5. At 10x scale, which component in the proximity service is most likely to fail first and why?
hard
A. The Quadtree service because its recursive traversal becomes a CPU bottleneck with many concurrent queries
B. The database because it cannot handle the increased write throughput for location updates
C. The Geohash prefix index because its fixed-length buckets cause uneven load and hotspotting
D. The API gateway because it cannot maintain the increased number of open WebSocket connections

Solution

  1. Step 1: Analyze component scalability

    Databases (The database because it cannot handle the increased write throughput for location updates) can be sharded and scaled horizontally; API gateways (The API gateway because it cannot maintain the increased number of open WebSocket connections) typically scale with load balancers.

  2. Step 2: Identify Quadtree bottleneck

    Quadtree's recursive traversal is CPU-intensive and stateful, making it hard to parallelize and shard, causing CPU bottlenecks under high query concurrency.

  3. Step 3: Evaluate Geohash index load

    Geohash buckets (The Geohash prefix index because its fixed-length buckets cause uneven load and hotspotting) can cause hotspots but are easier to shard and cache than Quadtree traversal.

  4. Final Answer:

    Option A -> Option A

  5. Quick Check:

    Quadtree traversal CPU bottleneck limits scale first [check]
Hint: Quadtree traversal CPU bottleneck limits scale first [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard