Pub/sub pattern in HLD - System Design Guide

Problem Statement

When multiple components need to communicate, tightly coupling them causes failures to cascade and makes scaling difficult. Direct communication means every sender must know every receiver, leading to complex dependencies and fragile systems.

Solution

The pub/sub pattern decouples senders and receivers by introducing a message broker. Publishers send messages to topics without knowing who subscribes. Subscribers listen to topics and receive messages asynchronously, enabling flexible, scalable communication.

Architecture

Publisher

↓

Broker

(Message

↓

Subscriber 1

This diagram shows publishers sending messages to a broker, which then forwards messages to multiple subscribers asynchronously.

Trade-offs

✓ Pros

→

Decouples components, reducing dependencies and improving system flexibility.

→

Supports asynchronous communication, improving responsiveness and scalability.

→

Enables multiple subscribers to receive the same message without extra load on the publisher.

→

Simplifies adding or removing subscribers without changing publishers.

✗ Cons

→

Introduces a broker component, adding operational complexity and a potential single point of failure.

→

Message delivery latency can increase due to asynchronous processing.

→

Requires careful handling of message ordering and duplication depending on broker capabilities.

Use when multiple independent components need to react to events or data changes asynchronously, especially at scale above thousands of messages per second or when decoupling is critical.

Avoid when communication is simple point-to-point with low message volume under 1000 messages per second or when strict synchronous request-response is required.

Real World Examples

Netflix

Uses pub/sub to decouple microservices for event-driven workflows, allowing independent scaling and failure isolation.

Uber

Employs pub/sub for real-time location updates and notifications to multiple services without tight coupling.

Amazon

Uses pub/sub in AWS SNS to fan out notifications to multiple endpoints, enabling scalable alerting and messaging.

Alternatives

Direct point-to-point messaging

Senders communicate directly with receivers without an intermediary broker.

Use when: Use when the number of components is small and tightly coupled communication is acceptable.

Message queue (point-to-point)

Messages are sent to a queue consumed by a single receiver, unlike pub/sub which supports multiple subscribers.

Use when: Choose when tasks need to be processed by only one consumer to balance load.

Event streaming

Stores and streams ordered event logs for replay and processing, unlike pub/sub which focuses on message delivery.

Use when: Use when event history and replayability are important.

Summary

Pub/sub decouples senders and receivers by using a broker to route messages asynchronously.

It enables scalable, flexible communication where multiple subscribers can receive the same message.

This pattern is ideal for event-driven systems requiring loose coupling and asynchronous processing.

Practice

1. What is the main purpose of the Pub/Sub pattern in system design?

easy

A. To decouple message senders from receivers for flexible communication

B. To tightly couple components for faster data exchange

C. To store data permanently in a database

D. To encrypt messages for security

Solution

1. Step 1: Understand the role of Pub/Sub

   Pub/Sub pattern separates the sender (publisher) and receiver (subscriber) so they do not depend on each other directly.

2. Step 2: Identify the main benefit

   This separation allows flexible communication and easier scaling of systems.

3. Final Answer:

   To decouple message senders from receivers for flexible communication -> Option A

4. Quick Check:

   Decoupling = A [OK]

Hint: Pub/Sub means senders and receivers don't know each other [OK]

Common Mistakes:

• Thinking Pub/Sub stores data permanently
• Confusing Pub/Sub with encryption
• Assuming tight coupling between components

2. Which of the following is the correct component in a Pub/Sub system?

easy

A. Sender, Receiver, Firewall

B. Client, Server, Database

C. Publisher, Subscriber, Message Broker

D. Producer, Consumer, Cache

Solution

1. Step 1: Recall Pub/Sub components

   Pub/Sub systems have publishers (senders), subscribers (receivers), and a message broker that routes messages.

2. Step 2: Match components to options

   Only Publisher, Subscriber, Message Broker lists these three correctly.

3. Final Answer:

   Publisher, Subscriber, Message Broker -> Option C

4. Quick Check:

   Pub/Sub components = D [OK]

Hint: Remember the broker routes messages between publishers and subscribers [OK]

Common Mistakes:

• Confusing Pub/Sub with client-server terms
• Mixing cache or firewall as core components
• Using generic terms like producer/consumer without broker

3. Consider a Pub/Sub system where a publisher sends messages to a topic. If three subscribers are subscribed to that topic, how many copies of each message does the broker send?

medium

A. One copy shared among all subscribers

B. Three copies, one for each subscriber

C. Zero copies, subscribers pull messages themselves

D. Depends on the message size

Solution

1. Step 1: Understand message delivery in Pub/Sub

   Each subscriber receives its own copy of the message from the broker to ensure independent processing.

2. Step 2: Calculate copies sent

   With three subscribers, the broker sends three copies, one per subscriber.

3. Final Answer:

   Three copies, one for each subscriber -> Option B

4. Quick Check:

   Copies = Subscribers count = B [OK]

Hint: Broker sends one message per subscriber, not shared [OK]

Common Mistakes:

• Assuming one copy shared by all subscribers
• Thinking subscribers pull messages without broker sending
• Believing message size affects copy count

4. A developer implemented a Pub/Sub system but notices subscribers never receive messages. Which is the most likely cause?

medium

A. Subscribers are using the wrong database

B. Publisher is sending messages too frequently

C. Message broker is encrypting messages

D. Subscribers are not subscribed to the correct topic

Solution

1. Step 1: Analyze why subscribers get no messages

   If subscribers never receive messages, a common cause is they are not subscribed to the topic where messages are published.

2. Step 2: Evaluate other options

   Sending frequency or encryption does not prevent delivery; wrong database is unrelated.

3. Final Answer:

   Subscribers are not subscribed to the correct topic -> Option D

4. Quick Check:

   Subscription mismatch = A [OK]

Hint: Check subscriber topic matches publisher topic first [OK]

Common Mistakes:

• Blaming message frequency for delivery failure
• Assuming encryption blocks delivery
• Confusing database usage with Pub/Sub messaging

5. You need to design a scalable notification system using Pub/Sub. Which approach best ensures messages are delivered even if some subscribers are temporarily offline?

hard

A. Use a message broker with persistent storage and durable subscriptions

B. Send messages directly from publisher to subscribers without broker

C. Use a broker that deletes messages immediately after sending

D. Require subscribers to poll the publisher continuously

Solution

1. Step 1: Identify requirements for offline subscriber support

   To handle offline subscribers, messages must be stored until subscribers can receive them.

2. Step 2: Choose the right Pub/Sub feature

   Persistent storage and durable subscriptions in the broker ensure messages are saved and delivered later.

3. Step 3: Evaluate other options

   Direct sending or immediate deletion loses messages; polling is inefficient and not scalable.

4. Final Answer:

   Use a message broker with persistent storage and durable subscriptions -> Option A

5. Quick Check:

   Durable subscriptions = reliable delivery = C [OK]

Hint: Durable subscriptions keep messages for offline subscribers [OK]

Common Mistakes:

• Skipping message persistence in broker
• Trying direct publisher-to-subscriber without broker
• Relying on polling instead of push delivery