Bird
Raised Fist0

Pub-Sub Pattern - Topics, Subscriptions, Fan-Out Design

HLDmediumInterview freq: common
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Pub-Sub Pattern - Topics, Subscriptions, Fan-Out Design
45 min · senior
senior

Design a Pub-Sub messaging system for a startup with 50 million daily active users (DAU). The system must support topics and subscriptions with efficient fan-out to millions of subscribers. Messages must be encrypted end-to-end. Offline subscribers should receive messages upon reconnection. The system must guarantee P99 latency under 200ms for message delivery. The business requires exactly-once delivery semantics and horizontal scalability to handle peak loads.

Constraints
  • End-to-end encryption of messages in transit and at rest
  • Offline delivery support with message persistence and replay
  • P99 message delivery latency under 200ms at peak load
  • Exactly-once delivery semantics per subscriber
  • Support millions of topics and subscriptions with dynamic fan-out
📋
Guided Steps
1/5
Step 1
Choosing the Messaging Backbone
Which messaging infrastructure will you use to handle high-throughput topic-based fan-out with ordering guarantees?
💡 Hint
Consider partitioning by topic to maintain order and scale horizontally.
Key Decision
Use Apache Kafka with topic as the partition key to ensure ordered, scalable message storage and fan-out.
Interviewer Expectation
Mid-level: Names Kafka for pub-sub. Senior: Explains topic-based partitioning to maintain order and enable parallelism, rejects RabbitMQ due to limited horizontal scalability at 50M DAU.
⚠️Common Wrong Answer
Using RabbitMQ or Redis Pub/Sub which cannot scale to millions of topics or guarantee ordering at this scale, causing message loss or latency spikes.
Step 2
Ensuring End-to-End Encryption
How will you implement end-to-end encryption for messages in transit and at rest without sacrificing performance?
💡 Hint
Separate transport layer encryption from application-level encryption keys managed per subscriber.
Key Decision
Use TLS for transport encryption and client-side encryption with AWS KMS-managed keys per topic for message payloads.
Interviewer Expectation
Mid-level: Mentions TLS for transport security. Senior: Details client-side encryption with key rotation and KMS integration, rejects relying solely on TLS which exposes plaintext in brokers.
⚠️Common Wrong Answer
Relying only on TLS encryption, exposing plaintext messages inside brokers, violating end-to-end encryption requirements.
Step 3
Offline Delivery and Message Persistence
How will you support offline subscribers to receive missed messages upon reconnection?
💡 Hint
Use durable storage with cursor-based replay and retention policies.
Key Decision
Leverage Kafka’s log retention with consumer offsets stored in Apache Cassandra for durable, scalable cursor tracking and replay.
Interviewer Expectation
Mid-level: Mentions message retention in Kafka. Senior: Explains Cassandra’s partition key as subscriber_id for efficient offset storage and fast cursor lookups, rejects using Kafka offsets alone due to multi-region replication complexity.
⚠️Common Wrong Answer
Not persisting messages or relying on ephemeral queues causing message loss for offline subscribers.
Step 4
Guaranteeing Exactly-Once Delivery
How will you ensure exactly-once delivery semantics to subscribers in a distributed environment?
💡 Hint
Combine idempotent producers, transactional writes, and offset commits.
Key Decision
Use Kafka’s idempotent producers and transactions combined with atomic offset commits stored in Cassandra to guarantee exactly-once delivery.
Interviewer Expectation
Mid-level: Mentions at-least-once delivery with retries. Senior: Explains Kafka transactions and atomic offset commits to avoid duplicates, rejects simple at-least-once which causes duplicates under failures.
⚠️Common Wrong Answer
Implementing only at-least-once delivery causing duplicate messages and inconsistent subscriber state.
Step 5
API Design for Publish, Subscribe, and Acknowledge
Design the three core API endpoints for publishing messages, subscribing to topics, and acknowledging message receipt.
💡 Hint
Consider REST or gRPC with authentication, payload encryption, and offset management.
Key Decision
Define Publish(topic, encrypted_message), Subscribe(topic, subscriber_id), and Acknowledge(topic, subscriber_id, message_id) endpoints using gRPC with mutual TLS authentication.
Interviewer Expectation
Mid-level: Defines basic endpoints. Senior: Details encryption handling, offset management in Acknowledge, and scalable subscription registration rejecting polling-based APIs due to load.
⚠️Common Wrong Answer
Using HTTP polling for subscription causing excessive load and high latency at 50M DAU.
🔌
API Design Challenge
Sketch 3 core API endpoints for the Pub-Sub system: publishing messages, subscribing to topics, and acknowledging message receipt.
Hints
  • How does a client publish a message to a topic securely?
  • How does a subscriber register and receive messages efficiently?
  • How does the system track and handle message acknowledgments for exactly-once delivery?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Use Kafka with topic partitioning for scalable ordered messaging
  • Implement end-to-end encryption with client-side encryption and TLS
  • Support offline delivery with durable storage and cursor-based replay
  • Guarantee exactly-once delivery using Kafka transactions and atomic offset commits
Good to Have - 30%
  • Design gRPC APIs with mutual TLS for secure, efficient client communication
  • Use Cassandra for scalable offset storage and fast cursor lookups
  • Explain fan-out worker design for parallel delivery maintaining latency SLAs
Bonus - 10%
  • Discuss key rotation and KMS integration for encryption key management
  • Describe multi-region replication challenges and solutions
Anti-patterns to Avoid
Using RabbitMQ or Redis Pub/Sub which cannot scale to millions of topics causing message loss and latency spikes, resulting in system instability and poor user experience.
Relying solely on TLS without client-side encryption exposing plaintext inside brokers violating security, which risks data breaches and non-compliance with privacy requirements.
Not persisting messages leading to message loss for offline subscribers, causing inconsistent state and user dissatisfaction.
Implementing only at-least-once delivery causing duplicate messages and inconsistent subscriber state, leading to data corruption and complex client-side deduplication.
Using HTTP polling for subscriptions causing excessive load and system collapse at scale due to 50 million reconnects per minute just to check for messages.
Storing offsets only in Kafka without external durable storage complicating multi-region failover and risking offset loss during broker failures.
✅
Model Answer
▼ Reveal

This design uses Apache Kafka as the messaging backbone with topic-based partitioning to ensure ordered, scalable message delivery. End-to-end encryption is achieved by combining TLS transport security with client-side encryption using AWS KMS-managed keys per topic. Offline delivery is supported by leveraging Kafka's log retention and storing consumer offsets in Apache Cassandra for durable, scalable cursor tracking and replay. Exactly-once delivery semantics are guaranteed using Kafka's idempotent producers, transactions, and atomic offset commits stored in Cassandra. The system exposes gRPC APIs secured with mutual TLS for publishing messages, subscribing to topics, and acknowledging message receipt, supporting encrypted payloads and efficient offset management.

Key Decisions
  • Use Apache Kafka with topic as the partition key to maintain message order and enable horizontal scaling.
  • Implement TLS for transport encryption and client-side encryption with AWS KMS-managed keys per topic to ensure end-to-end encryption.
  • Store consumer offsets in Apache Cassandra keyed by subscriber_id for durable, scalable cursor tracking and offline message replay.
  • Leverage Kafka's idempotent producers and transactional writes combined with atomic offset commits in Cassandra to guarantee exactly-once delivery.
  • Design gRPC APIs for Publish(topic, encrypted_message), Subscribe(topic, subscriber_id), and Acknowledge(topic, subscriber_id, message_id) with mutual TLS authentication to ensure secure and efficient client communication.
  • Apache Kafka with topic-based partitioning for ordered, scalable message storage and fan-out
  • TLS transport encryption combined with client-side encryption using AWS KMS for end-to-end security
  • Apache Cassandra for storing subscriber offsets and enabling durable offline message replay
  • Kafka idempotent producers and transactions with atomic offset commits for exactly-once delivery
  • gRPC APIs with mutual TLS for Publish, Subscribe, and Acknowledge endpoints supporting encrypted payloads
  • Cursor-based message replay using Cassandra offsets to support offline subscribers efficiently
  • Reject RabbitMQ and Redis Pub/Sub due to limited scalability and ordering guarantees at 50M DAU
  • Avoid HTTP polling for subscriptions to prevent system collapse under massive reconnect load
  • Use key rotation and KMS integration to manage encryption keys securely without performance degradation
  • Design fan-out workers consuming Kafka partitions to parallelize delivery and maintain P99 < 200ms latency

Practice

(1/5)
1. A system must handle millions of short video uploads per day, ensuring fast ingestion and metadata extraction without blocking the user. Which component is best suited to decouple upload ingestion from processing?
easy
A. Use a distributed message queue like Kafka to buffer uploads for asynchronous processing
B. Use a relational database to store uploads synchronously during user upload
C. Use a CDN edge cache to store uploads temporarily before processing
D. Use a REST API gateway that directly triggers video transcoding synchronously

Solution

  1. Step 1: Identify ingestion decoupling need

    Relational DB (A) blocks user upload due to synchronous writes; CDN (C) caches for delivery, not ingestion; REST API (D) triggers blocking transcoding.

  2. Step 2: Choose buffering component

    Kafka (B) buffers uploads asynchronously, enabling fast user response and reliable processing.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Kafka buffers uploads asynchronously for scalable ingestion [check]
Hint: Buffer uploads asynchronously with Kafka queue [OK]
Common Mistakes:
2. A distributed system needs to generate unique IDs for millions of events per second without coordination delays. Which component is best suited to generate these IDs efficiently?
easy
A. A decentralized ID generator using timestamp and node ID bits (e.g., Snowflake)
B. A centralized relational database with auto-increment primary keys
C. A UUID v4 generator relying on random number generation
D. A distributed cache like Redis with INCR commands for ID generation

Solution

  1. Step 1: Identify coordination bottlenecks in centralized DB

    Auto-increment keys require coordination, causing latency and contention at high scale.

  2. Step 2: Evaluate decentralized timestamp-based ID generators

    Snowflake-style IDs combine timestamp and node ID, enabling scalable, ordered unique IDs without coordination.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Decentralized timestamp + node ID -> scalable unique ID generation [check]
Hint: Decentralized timestamp + node ID avoids coordination delays [check]
Common Mistakes:
3. Without the bulkhead pattern implemented, what exactly happens when one microservice's database connection pool saturates under load?
medium
A. The entire system queues requests indefinitely, causing high latency everywhere
B. Requests to the saturated service block and cause thread starvation, cascading failures to other services
C. The service automatically rejects requests, preventing impact on other services
D. Load balancers detect saturation and reroute traffic to healthy instances instantly

Solution

  1. Step 1: Understand absence of bulkhead isolation

    Without bulkheads, saturation causes blocking (B) and thread starvation, but automatic rejection (C) is a bulkhead feature. Queuing indefinitely (A) is unlikely without bulkheads. Load balancers (D) don't detect internal resource saturation.

  2. Step 2: Trace failure impact

    Without bulkheads, blocking threads cause cascading failures; with bulkheads, requests are rejected to contain blast radius.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Without bulkheads, blocking causes cascading failures [check]
Hint: No bulkheads -> blocking threads cascade failures [check]
Common Mistakes:
4. Without the segment manifest (playlist) update mechanism, what exactly breaks in the adaptive bitrate HLS streaming workflow?
medium
A. Clients will download all bitrate variants simultaneously, wasting bandwidth
B. Clients will fail to discover new segments, causing playback stalls or infinite buffering
C. Clients will continue playing old segments indefinitely without switching bitrates
D. Clients will switch bitrates too frequently, causing playback instability

Solution

  1. Step 1: Role of manifest updates in HLS

    Manifest updates inform clients about new segments and available bitrates, enabling selective fetching.

  2. Step 2: Consequence of missing manifest updates

    Without updates, clients cannot know which segments to fetch next, leading them to request all variants to avoid stalls, wasting bandwidth.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Missing manifest updates cause clients to fetch all variants, wasting bandwidth [check]
Hint: No manifest updates -> clients fetch all bitrates, wasting bandwidth [check]
Common Mistakes:
5. In a Paxos-based system, what exactly breaks if the leader (proposer) crashes before receiving majority acceptances?
medium
A. Previously accepted values remain safe, but no new values are chosen until recovery
B. Proposed values may never be chosen, causing client requests to block indefinitely
C. The system immediately elects a new leader and continues without delay
D. Followers start accepting conflicting values leading to data corruption

Solution

  1. Step 1: Understand Paxos proposer failure impact

    If the proposer crashes before majority acceptance, no new values are chosen but previously accepted values remain safe.

  2. Step 2: Consequence on system progress

    The system cannot make progress until a new proposer initiates a new round, but no data corruption occurs.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Leader crash before majority -> no new values chosen, safety preserved [check]
Hint: Leader crash before majority acceptance -> safety preserved, progress stalled [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard