Pub/sub pattern in HLD - Deep Dive

Overview - Pub/sub pattern

What is it?

The Pub/sub pattern is a way for different parts of a system to communicate by sending and receiving messages without knowing about each other directly. Publishers send messages to a topic or channel, and subscribers listen to those topics to get the messages. This helps systems stay loosely connected and scalable.

Why it matters

Without the Pub/sub pattern, systems would need to know exactly who to talk to, making them tightly linked and hard to change or grow. Pub/sub allows many parts to work independently and react to events in real time, which is essential for modern apps like chat, notifications, or live updates.

Where it fits

Before learning Pub/sub, you should understand basic messaging and client-server communication. After this, you can explore event-driven architectures, message queues, and real-time data streaming systems.

Mental Model

Core Idea

Pub/sub lets senders broadcast messages to many receivers without knowing who they are, enabling flexible and scalable communication.

Think of it like...

Imagine a radio station (publisher) broadcasting music over the airwaves (topic). Anyone with a radio (subscriber) tuned to that station can listen, but the station doesn't know who is listening or how many people are tuned in.

Publisher(s) ──▶ Topic/Channel ──▶ Subscriber(s)

  [Publisher]       [Message Broker]       [Subscriber]
      │                   │                    │
      └── Sends message ──▶│                    │
                          │  Distributes message ──▶
                          │                    │
                          └────────────────────▶

Build-Up - 7 Steps

1

Foundation

Understanding basic messaging

Concept: Learn what messages are and how systems send and receive them.

A message is a piece of information sent from one part of a system to another. In simple systems, one sender talks directly to one receiver. For example, a chat app sends your message directly to your friend's device.

Result

You understand that messages carry data and that communication can be direct between two parties.

Knowing what a message is and how it flows is the foundation for understanding more complex communication patterns like Pub/sub.

2

Foundation

Direct communication limits

Concept: Explore why direct sender-to-receiver communication can be limiting.

When a sender must know exactly who to send messages to, the system becomes tightly linked. If you add more receivers or change them, the sender must also change. This makes the system hard to grow or maintain.

Result

You see that direct communication creates dependencies that reduce flexibility.

Understanding these limits shows why a pattern like Pub/sub is needed to decouple parts of a system.

3

Intermediate

Introducing the Pub/sub pattern

🤔Before reading on: do you think publishers know who their subscribers are in Pub/sub? Commit to your answer.

Concept: Learn how Pub/sub decouples senders and receivers using topics or channels.

In Pub/sub, publishers send messages to a topic without knowing who listens. Subscribers express interest in topics and receive messages sent to them. A message broker or event bus manages this distribution.

Result

You understand that publishers and subscribers do not know about each other, enabling loose coupling.

Knowing that Pub/sub decouples communication helps you design systems that are easier to scale and change.

4

Intermediate

Message brokers and delivery

🤔Before reading on: do you think messages in Pub/sub are always delivered instantly and reliably? Commit to your answer.

Concept: Explore the role of message brokers and how messages are delivered to subscribers.

A message broker receives messages from publishers and forwards them to subscribers. Brokers can store messages temporarily, retry delivery, and handle many subscribers. Delivery can be at-most-once, at-least-once, or exactly-once depending on the system.

Result

You see that brokers manage message flow and reliability, which affects system behavior.

Understanding broker roles and delivery guarantees helps you choose the right Pub/sub system for your needs.

5

Intermediate

Filtering and topic hierarchies

Concept: Learn how subscribers can choose which messages to receive using filters and topic structures.

Topics can be organized hierarchically, like sports/football or sports/basketball. Subscribers can listen to specific topics or use patterns to get groups of messages. Filters help reduce unnecessary message processing.

Result

You understand how to control message flow to subscribers efficiently.

Knowing filtering and topic hierarchies improves system performance and subscriber experience.

6

Advanced

Scaling Pub/sub systems

🤔Before reading on: do you think a single message broker can handle millions of messages and subscribers easily? Commit to your answer.

Concept: Explore how Pub/sub systems scale to handle large loads and many clients.

To scale, Pub/sub systems use multiple brokers, partition topics, and replicate data. Load balancing and fault tolerance ensure messages reach subscribers even if parts fail. Systems like Kafka or Google Pub/Sub use these techniques.

Result

You see how Pub/sub can support large, distributed systems with high message volumes.

Understanding scaling techniques prepares you to design robust, high-performance Pub/sub architectures.

7

Expert

Handling ordering and consistency

🤔Before reading on: do you think Pub/sub guarantees message order and consistency by default? Commit to your answer.

Concept: Learn about challenges with message order, duplication, and consistency in Pub/sub systems.

Pub/sub systems may deliver messages out of order or more than once. Handling this requires design choices like partition keys, sequence numbers, or idempotent processing. Tradeoffs exist between performance and strict ordering or exactly-once delivery.

Result

You understand the complexity behind reliable and ordered message delivery in Pub/sub.

Knowing these challenges helps you build systems that handle real-world data consistency needs.

Under the Hood

Underneath, a Pub/sub system uses a message broker that acts as a middleman. Publishers send messages to the broker, which stores them temporarily and forwards them to all subscribers interested in the topic. The broker manages connections, message queues, and delivery retries. Internally, it may partition topics and replicate data for scalability and fault tolerance.

Why designed this way?

Pub/sub was designed to decouple senders and receivers to allow independent scaling and evolution. Early systems were tightly coupled and hard to maintain. Using a broker centralizes message management, enabling features like filtering, retries, and load balancing. Alternatives like direct messaging were less flexible and scalable.

┌─────────────┐       ┌───────────────┐       ┌─────────────┐
│  Publisher  │──────▶│ Message Broker │──────▶│ Subscriber  │
└─────────────┘       └───────────────┘       └─────────────┘
       │                     │                      │
       │  Sends message       │  Stores & forwards   │  Receives message
       │────────────────────▶│─────────────────────▶│
       │                     │                      │

Myth Busters - 4 Common Misconceptions

Quick: Do publishers know who their subscribers are in Pub/sub? Commit to yes or no.

Common Belief: Publishers must know exactly who their subscribers are to send messages.

Reality: Publishers send messages to topics without knowing any subscribers; the broker handles delivery.

Why it matters: Believing otherwise leads to tightly coupled systems that lose the flexibility and scalability benefits of Pub/sub.

Quick: Does Pub/sub guarantee messages arrive in the order they were sent? Commit to yes or no.

Common Belief: Pub/sub always delivers messages in the exact order they were published.

Reality: Message order is not guaranteed by default; it depends on the system and configuration.

Why it matters: Assuming order can cause bugs in systems that rely on strict sequencing, leading to inconsistent states.

Quick: Are messages in Pub/sub always delivered exactly once? Commit to yes or no.

Common Belief: Pub/sub systems guarantee exactly-once message delivery.

Reality: Many Pub/sub systems provide at-least-once or at-most-once delivery; exactly-once is hard and requires extra design.

Why it matters: Ignoring this can cause duplicate processing or lost messages, affecting data accuracy.

Quick: Can a single message broker easily handle millions of subscribers and messages? Commit to yes or no.

Common Belief: One message broker can handle unlimited scale without issues.

Reality: Single brokers have limits; scaling requires partitioning, replication, and multiple brokers.

Why it matters: Overloading a single broker causes delays, message loss, or system crashes.

Expert Zone

1

Some Pub/sub systems allow subscribers to replay past messages, enabling event sourcing and debugging.

2

The choice between push and pull delivery models affects latency, load, and complexity in Pub/sub.

3

Message filtering can be done at the broker or subscriber side, impacting performance and network usage.

When NOT to use

Pub/sub is not ideal when strict request-response communication is needed or when message order and exactly-once delivery are critical without complex handling. Alternatives include direct RPC calls, REST APIs, or transactional message queues.

Production Patterns

In production, Pub/sub is used for event-driven microservices, real-time notifications, log aggregation, and IoT data streaming. Systems often combine Pub/sub with message queues and databases to ensure reliability and consistency.

Connections

Event-driven architecture

Pub/sub is a core building block of event-driven systems.

Understanding Pub/sub helps grasp how events trigger actions across loosely coupled services.

Observer pattern (software design)

Pub/sub is a distributed, scalable form of the Observer pattern.

Knowing the Observer pattern clarifies how Pub/sub manages subscriptions and notifications.

Broadcasting in wireless networks

Both involve sending information to many receivers without direct connections.

Recognizing this connection shows how communication principles apply across technology and physical systems.

Common Pitfalls

#1 Assuming message order is guaranteed and designing logic based on that.

Wrong approach:

Process messages as they arrive without checking sequence numbers or timestamps.

Correct approach:

Include sequence checks or use ordered partitions to ensure correct processing order.

Root cause:

Misunderstanding that Pub/sub systems do not always preserve message order.

#2 Ignoring message duplication and processing messages multiple times.

Wrong approach:

Treat every received message as unique without idempotency.

Correct approach:

Design subscribers to handle duplicate messages safely using idempotent operations or deduplication.

Root cause:

Belief that Pub/sub guarantees exactly-once delivery.

#3 Overloading a single message broker without scaling strategies.

Wrong approach:

Deploy one broker instance for all topics and subscribers regardless of load.

Correct approach:

Partition topics, replicate brokers, and use load balancing to distribute traffic.

Root cause:

Underestimating the scale and resource needs of Pub/sub systems.

Key Takeaways

Pub/sub decouples senders and receivers by using topics and a message broker, enabling scalable and flexible communication.

Publishers do not know who subscribes; subscribers choose which topics to listen to, allowing independent evolution.

Message brokers manage delivery, retries, and filtering but do not always guarantee order or exactly-once delivery.

Scaling Pub/sub requires partitioning, replication, and careful design to handle large volumes and many clients.

Understanding the limits and tradeoffs of Pub/sub helps design reliable, maintainable, and efficient distributed systems.