Bird
Raised Fist0

Pub-Sub Pattern - Topics, Subscriptions, Fan-Out Design

HLDmediumInterview freq: common
Choose your preparation mode4 modes available
🏗️
Pub-Sub Pattern - Topics, Subscriptions, Fan-Out Design
Design a scalable pub-sub system supporting topics, subscriptions, and fan-out at billions of messages per day
mediumMessage Queues and Event StreamingFreq: common
⚙️
Functional Requirements
  • Support creation of topics and multiple subscriptions per topic
  • Publish messages to topics with fan-out to all active subscriptions
  • Guarantee at-least-once delivery semantics for subscribers
  • Support message retention configurable per topic (e.g., 7 days)
  • Allow subscribers to acknowledge messages and support replay from offset
  • Provide APIs for topic management, subscription management, and message consumption
📊
Non-Functional Requirements
  • Handle 2 billion daily active users publishing 40 messages per user per day
  • Support peak write throughput of approximately 925,000 messages per second
  • Ensure 99.99% system availability with automatic failover
  • Maintain end-to-end message delivery latency under 100ms P99
  • Support message retention of 7 days with replication factor of 3
❓
Clarifying Questions to Ask
  • What delivery semantics are required: at-least-once, at-most-once, or exactly-once?
  • Are subscriptions push-based (server pushes messages) or pull-based (clients poll)?
  • What is the expected message size distribution and max message size?
  • Is message ordering required within topics or subscriptions?
  • What is the expected number of subscriptions per topic on average and max?
  • Are there any constraints on message retention duration or storage cost?
Out of Scope
✗ Message content transformation or filtering (e.g., content-based routing)
✗ End-to-end encryption or message-level security
✗ Subscriber-side processing logic or complex event processing
✗ Cross-datacenter geo-replication beyond single-region design
📌
Assumptions
  • 2 billion daily active users (DAU)
  • Each user publishes 40 messages per day on average
  • Average message size is 100 bytes
  • Replication factor for durability is 3
⚡
QPS Calculation
write qps2B users × 40 msg/user/day ÷ 86400 sec/day ≈ 925,926 msg/sec
peak write qpsAssuming 2× peak factor: 925,926 × 2 ≈ 1.85M msg/sec
read qpsAssuming each message fan-out to 5 subscriptions on average: 925,926 × 5 ≈ 4.63M msg/sec
💾
Storage Estimate
daily storage2B × 40 × 100 bytes = 8 TB/day uncompressed
retention 7 days8 TB/day × 7 days = 56 TB
with replication56 TB × 3 replicas = 168 TB total storage
🖥️
Server Estimate
broker nodesAssuming each Kafka broker handles 20K msg/sec write throughput, need 1.85M ÷ 20K ≈ 93 brokers
storage nodesAssuming each storage node supports 2 TB usable storage, need 168 TB ÷ 2 TB = 84 storage nodes
🏗️
Architecture Diagram
Architecture Diagram
       +-------------+
       |  Producers  |
       +------+------+
              |
              v
       +------+-------+
       |   Kafka      |
       |  Brokers     |
       +------+-------+
              |
              v
       +------+-------+
       | Subscription |
       |  Manager    |
       +------+-------+
              |
              v
       +------+-------+
       | Delivery    |
       | Workers     |
       +------+-------+
              |
              v
       +------+-------+
       | Subscribers |
       +-------------+
🧩
Components
Kafka Brokers
Durable, partitioned log storage for topics supporting high-throughput writes and reads
Why thisKafka supports partitioned logs with high throughput (millions msg/sec per cluster), built-in replication, and consumer offset tracking
Why notRabbitMQ lacks Kafka’s horizontal scalability and partitioning; Redis Streams cannot handle 1.85M writes/sec with durability
Subscription Manager
Manages topic subscriptions metadata, tracks active subscribers and their offsets
Why thisA metadata service backed by Apache Cassandra provides scalable, highly available storage for subscription state with fast lookups
Why notRelational DB like PostgreSQL would require complex manual sharding and cannot handle 925K writes/sec metadata updates
Delivery Workers
Consume messages from Kafka partitions and fan-out to subscribers via push or pull mechanisms
Why thisStateless microservices deployed in Kubernetes allow elastic scaling to handle variable fan-out load and maintain <100ms latency
Why notMonolithic delivery service would be a bottleneck; serverless functions have cold start latency unsuitable for low P99
Producers
Clients or upstream services publishing messages to topics via Kafka producer APIs
Why thisKafka producer clients provide efficient batching, compression, and retries to optimize throughput and durability
Why notDirect writes to storage or DB would not scale to 1.85M msg/sec and lack partitioning benefits
Subscribers
Clients consuming messages from subscriptions, acknowledging processed messages for offset commits
Why thisPull-based subscribers allow backpressure control and replay from offsets; push-based can be supported optionally
Why notPush-only subscribers risk message loss or overload; polling provides better flow control at scale
🔄
Data Flow
1
Producers → Kafka Brokers
Publish messages to topic partitions
Kafka partitions messages by key for parallelism and durability
2
Kafka Brokers → Subscription Manager
Notify subscription manager of new messages
Subscription manager tracks offsets per subscription
3
Subscription Manager → Delivery Workers
Assign messages to delivery workers for fan-out
Workers fetch subscriber lists and offsets
4
Delivery Workers → Subscribers
Push or make messages available for pull
Delivery respects subscriber offsets and backpressure
5
Subscribers → Subscription Manager
Acknowledge messages and commit offsets
Ensures at-least-once delivery and replay capability

Practice

(1/5)
1. A system must handle 1 million writes per second for incoming user events and decouple ingestion from processing to avoid backpressure. Which component is best suited to absorb this write load reliably?
easy
A. Use a distributed log system like Kafka with partitioning by user ID
B. Use a relational database with batch inserts every second
C. Use an in-memory cache like Redis without persistence
D. Use a REST API server that writes directly to disk synchronously

Solution

  1. Step 1: Identify the need for decoupling ingestion and processing

    Relational DB with batch inserts (A) cannot handle 1M writes/sec reliably due to disk I/O and transaction overhead. Redis (C) lacks durability and persistence, risking data loss. REST API with synchronous disk writes (D) blocks on I/O, causing backpressure.
  2. Step 2: Recognize Kafka's role as a distributed log with partitioning

    Kafka partitions by user ID to distribute load and provides durable, ordered logs that decouple ingestion from processing, handling high write throughput reliably.
  3. Final Answer:

    Option A -> Option A
  4. Quick Check:

    Kafka partitions absorb high write load with durability and ordering [check]
Hint: Distributed log with partitioning handles high write load [OK]
Common Mistakes:
2. A microservices system needs to ensure reliable event publishing without losing messages during local transaction commits. Which component best guarantees atomicity between database writes and event emission?
easy
A. Use a distributed transaction coordinator (2PC) across services
B. Implement the Outbox Pattern with a local transaction and event relay
C. Rely on eventual consistency with periodic reconciliation jobs
D. Use a message queue with at-least-once delivery without local persistence

Solution

  1. Step 1: Identify atomicity requirement

    The system must atomically write business data and produce events without distributed transactions.
  2. Step 2: Evaluate Outbox Pattern

    Outbox Pattern writes events in the same local DB transaction, then asynchronously relays them, ensuring no message loss.
  3. Final Answer:

    Option B -> Option B
  4. Quick Check:

    Outbox Pattern ensures atomic DB write + event emission [check]
Hint: Outbox Pattern couples DB write and event emission atomically [check]
Common Mistakes:
3. A system needs to estimate the number of unique users visiting a website daily, with minimal memory usage and acceptable error margin. Which component is best suited to handle this requirement?
easy
A. A Redis cache storing all user IDs explicitly
B. A relational database with unique user ID indexing
C. A HyperLogLog data structure for cardinality estimation
D. A Bloom filter to check if a user has visited before

Solution

  1. Step 1: Understand memory constraints and error tolerance

    Storing all user IDs explicitly (Redis or relational DB) is memory-heavy and not scalable for large user bases.
  2. Step 2: Identify HyperLogLog's role

    HyperLogLog provides approximate unique counts with fixed small memory and tunable error, ideal for large-scale cardinality estimation.
  3. Final Answer:

    Option C -> Option C
  4. Quick Check:

    HyperLogLog trades accuracy for memory efficiency [check]
Hint: Approximate unique counts with tiny memory -> HyperLogLog [check]
Common Mistakes:
4. If the circuit breaker component is removed from a microservice architecture, what is the most likely failure sequence when the downstream service becomes slow or unresponsive?
medium
A. The system automatically queues requests and processes them later without blocking callers
B. The upstream service immediately fails requests without retries, causing data loss
C. The system switches to a backup service transparently without any delay
D. The upstream service blocks on calls until timeout, causing thread exhaustion and cascading failures

Solution

  1. Step 1: Understand circuit breaker removal impact

    Without circuit breaker, upstream calls block waiting for slow downstream responses.
  2. Step 2: Trace failure propagation

    Blocking calls exhaust thread pools, leading to cascading failures (B). Queuing (A) or immediate failover (D) require explicit mechanisms. Immediate fail without retries (C) is unlikely without circuit breaker logic.
  3. Final Answer:

    Option D -> Option D
  4. Quick Check:

    Blocking calls cause thread exhaustion and cascade without circuit breaker [check]
Hint: No circuit breaker -> blocking calls exhaust threads -> cascade [OK]
Common Mistakes:
5. For timeline generation at scale, which architecture better handles users with millions of followers and why?
medium
A. Pure pull-based: generate timelines on-demand by querying follow graph and tweets
B. Push-based fan-out: write each tweet to all followers' timelines synchronously
C. Hybrid approach: push fan-out for users with few followers, pull on read for celebrities
D. Store timelines in a global sorted set updated asynchronously for all users

Solution

  1. Step 1: Analyze push vs pull trade-offs

    Push fan-out is efficient for low-follower users but expensive for celebrities; pull is costly on read latency.
  2. Step 2: Evaluate hybrid approach

    Hybrid uses push for most users, pull for celebrities, balancing latency and write amplification.
  3. Step 3: Eliminate other options

    Pure push (A) causes huge write amplification; pure pull (C) causes high read latency; global sorted set (D) is hard to scale and update asynchronously.
  4. Final Answer:

    Option C -> Option C
  5. Quick Check:

    Hybrid push-pull balances write cost and read latency [check]
Hint: Hybrid push-pull fits follower count extremes [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard