Pub/sub vs streams comparison in Redis - Trade-offs & Expert Analysis

Overview - Pub/sub vs streams comparison

What is it?

Pub/sub and streams are two ways Redis lets programs send and receive messages. Pub/sub is like a live chat where messages go to all listeners instantly but disappear after. Streams are like a message log that keeps all messages so listeners can read them anytime, even later. Both help different programs talk to each other but work in different ways.

Why it matters

Without pub/sub or streams, programs would struggle to share information quickly or reliably. Pub/sub solves the need for instant communication, like live updates, while streams solve the need to keep messages for later review or replay. Without these, apps would lose real-time features or risk missing important data.

Where it fits

Before learning this, you should know basic Redis commands and data types. After this, you can explore advanced messaging patterns, event-driven design, or distributed systems using Redis features.

Mental Model

Core Idea

Pub/sub sends messages instantly to all current listeners without storing them, while streams save messages so listeners can read them anytime, even after they were sent.

Think of it like...

Pub/sub is like shouting in a room where everyone hears you live, but if someone enters late, they miss what was said. Streams are like writing notes on a bulletin board that anyone can read anytime, even if they come later.

┌───────────────┐       ┌───────────────┐
│   Publisher   │       │   Publisher   │
└──────┬────────┘       └──────┬────────┘
       │                        │
       ▼                        ▼
┌───────────────┐       ┌───────────────┐
│   Pub/Sub     │       │    Streams    │
│ (no storage)  │       │ (stores data) │
└──────┬────────┘       └──────┬────────┘
       │                        │
       ▼                        ▼
┌───────────────┐       ┌───────────────┐
│ Subscribers   │       │ Consumers     │
│ (live only)   │       │ (can read old │
│               │       │  messages)    │
└───────────────┘       └───────────────┘

Build-Up - 7 Steps

1

Foundation

Understanding Redis Pub/Sub Basics

Concept: Pub/sub is a messaging pattern where messages are sent to all current subscribers instantly without storing them.

In Redis pub/sub, a publisher sends a message to a channel. All clients subscribed to that channel receive the message immediately. If a client is not subscribed at that moment, it misses the message. There is no history or storage of messages.

Result

Subscribers get messages only while they are connected and subscribed. Messages are not saved.

Understanding that pub/sub delivers messages live only helps grasp why it is fast but not reliable for message history.

2

Foundation

Understanding Redis Streams Basics

Concept: Streams store messages in a log so consumers can read them anytime, even after messages are sent.

Redis streams keep a list of messages with unique IDs. Producers add messages to the stream. Consumers can read messages from any point, including past messages. This allows replaying or processing messages at different speeds.

Result

Messages persist in the stream until deleted, allowing consumers to catch up or reprocess data.

Knowing streams store messages changes how you think about message delivery and reliability.

3

Intermediate

Comparing Message Delivery Guarantees

🤔Before reading on: do you think pub/sub guarantees message delivery if a subscriber disconnects? Commit to yes or no.

Concept: Pub/sub delivers messages only to connected subscribers, while streams guarantee message availability until consumed.

Pub/sub does not store messages, so if a subscriber disconnects, it loses messages sent during that time. Streams keep messages until consumers acknowledge them, ensuring no message loss even if consumers are offline.

Result

Streams provide reliable message delivery; pub/sub does not.

Understanding delivery guarantees helps choose the right tool for real-time vs reliable messaging needs.

4

Intermediate

Exploring Consumer Groups in Streams

🤔Before reading on: do you think pub/sub supports multiple consumers sharing message load like streams? Commit to yes or no.

Concept: Streams support consumer groups that allow multiple consumers to share processing messages, unlike pub/sub.

Redis streams have consumer groups where messages are distributed among consumers, enabling load balancing and fault tolerance. Pub/sub sends every message to all subscribers, so no message sharing or tracking happens.

Result

Streams enable scalable message processing; pub/sub is simpler but less flexible.

Knowing consumer groups unlocks advanced stream use cases like parallel processing.

5

Intermediate

Latency and Performance Differences

Concept: Pub/sub is faster with lower latency because it does not store messages, while streams add overhead for storage and tracking.

Pub/sub sends messages directly to subscribers with minimal delay. Streams write messages to disk or memory and track consumer states, which adds latency but improves reliability. The choice depends on whether speed or durability matters more.

Result

Pub/sub suits instant notifications; streams suit durable event processing.

Balancing latency and reliability is key to picking pub/sub or streams.

6

Advanced

Handling Message Loss and Recovery

🤔Before reading on: do you think pub/sub can recover lost messages after a subscriber reconnects? Commit to yes or no.

Concept: Streams allow consumers to recover and process missed messages; pub/sub cannot recover lost messages.

In streams, consumers track their last read message ID and can request messages from that point after reconnecting. Pub/sub has no message history, so missed messages are lost forever. This affects system design for fault tolerance.

Result

Streams support robust recovery; pub/sub requires always-on subscribers.

Understanding recovery mechanisms guides building resilient messaging systems.

7

Expert

Scaling and Resource Implications

🤔Before reading on: do you think streams always use more memory than pub/sub? Commit to yes or no.

Concept: Streams consume more memory and storage due to message retention, requiring careful management; pub/sub uses minimal resources but offers no persistence.

Streams store messages until explicitly deleted, which can grow memory and disk usage. Proper trimming and retention policies are needed. Pub/sub uses less memory since messages are transient. Choosing between them involves trade-offs in resource use and functionality.

Result

Streams need resource planning; pub/sub is lightweight but less feature-rich.

Knowing resource trade-offs helps optimize system performance and cost.

Under the Hood

Pub/sub in Redis uses a simple publish command that sends messages directly to all subscribed clients without storing them. Redis keeps a list of subscribers per channel and pushes messages immediately. Streams use a log data structure where each message has a unique ID and is appended to the stream. Consumers read messages by ID and track their position. Redis manages consumer groups by assigning messages and tracking acknowledgments to ensure reliable delivery.

Why designed this way?

Pub/sub was designed for speed and simplicity, enabling real-time messaging without overhead. Streams were introduced later to solve the need for message durability, replay, and complex consumer patterns. The two coexist to serve different use cases: instant notifications vs reliable event processing.

┌───────────────┐
│   Publisher   │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│   Pub/Sub     │
│ (No Storage)  │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Subscribers   │
│ (Live Only)   │
└───────────────┘


┌───────────────┐
│   Publisher   │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│   Streams     │
│ (Message Log) │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Consumer Group│
│ (Track State) │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Consumers     │
│ (Read Anytime)│
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does Redis pub/sub store messages for offline subscribers? Commit yes or no.

Common Belief: Pub/sub stores messages so subscribers can get them later if they were offline.

Reality: Pub/sub does not store messages; offline subscribers miss all messages sent while disconnected.

Why it matters: Assuming pub/sub stores messages can cause data loss in applications expecting reliable delivery.

Quick: Can Redis streams deliver messages to multiple consumers without duplication? Commit yes or no.

Common Belief: Streams send every message to all consumers like pub/sub does.

Reality: Streams distribute messages among consumers in a group so each message is processed once, avoiding duplication.

Why it matters: Misunderstanding this leads to inefficient processing or duplicate work in consumer applications.

Quick: Is pub/sub always faster than streams? Commit yes or no.

Common Belief: Pub/sub is always faster because it doesn't store messages.

Reality: Pub/sub is faster for live delivery, but streams can be optimized and sometimes perform well enough for many use cases despite storage overhead.

Why it matters: Assuming pub/sub is always better for speed may cause wrong choices in systems needing durability.

Quick: Do streams automatically delete old messages? Commit yes or no.

Common Belief: Streams automatically remove old messages to save space.

Reality: Streams keep messages until explicitly trimmed or deleted by the user or policy.

Why it matters: Not managing stream size can cause memory bloat and performance issues.

Expert Zone

1

Streams allow consumers to claim pending messages from failed consumers, enabling fault-tolerant processing.

2

Pub/sub channels do not support pattern matching for message history, but streams can be filtered by message content.

3

Streams support blocking reads with timeouts, allowing efficient waiting for new messages without busy polling.

When NOT to use

Avoid pub/sub when message durability or replay is needed; use streams instead. Avoid streams if ultra-low latency with no persistence is critical; use pub/sub. For complex workflows needing transactions or multi-step processing, consider combining streams with other Redis data types or external systems.

Production Patterns

Use pub/sub for real-time notifications like chat messages or live updates. Use streams for event sourcing, audit logs, or task queues with consumer groups. Combine streams with trimming policies and consumer group management for scalable, reliable processing pipelines.

Connections

Message Queue

Streams in Redis act like message queues with persistence and consumer groups.

Understanding Redis streams helps grasp how message queues ensure reliable delivery and load balancing.

Event Sourcing

Streams provide a log of events that can be replayed, similar to event sourcing in software design.

Knowing streams supports event sourcing clarifies how systems can rebuild state from stored events.

Broadcast Radio

Pub/sub is like broadcast radio where messages are sent live to all listeners without recording.

Recognizing this connection helps understand why pub/sub is fast but ephemeral.

Common Pitfalls

#1 Expecting pub/sub to deliver messages to subscribers who connect late.

Wrong approach:

SUBSCRIBE channel
// Subscriber connects after messages sent
// Misses all previous messages

Correct approach:

Use streams with consumer groups to read messages from any point:
XREADGROUP GROUP group1 consumer1 STREAMS mystream >

Root cause:

Misunderstanding that pub/sub does not store messages for late subscribers.

#2 Not trimming streams causing memory to grow indefinitely.

Wrong approach:

XADD mystream * field1 value1
// No trimming, stream grows forever

Correct approach:

XADD mystream MAXLEN ~ 1000 * field1 value1
// Keeps stream length around 1000 messages

Root cause:

Forgetting to manage stream size leads to resource exhaustion.

#3 Using pub/sub for critical message delivery requiring guaranteed processing.

Wrong approach:

PUBLISH channel message
// No tracking if message was received or processed

Correct approach:

Use streams with consumer groups and acknowledgments:
XREADGROUP GROUP group1 consumer1 STREAMS mystream >

Root cause:

Ignoring pub/sub's lack of delivery guarantees causes lost messages.

Key Takeaways

Redis pub/sub is a fast, live messaging system without message storage, suitable for real-time notifications.

Redis streams store messages persistently, allowing consumers to read and replay messages anytime, supporting reliable processing.

Pub/sub delivers messages only to connected subscribers, while streams guarantee message availability until acknowledged.

Streams support consumer groups for load balancing and fault tolerance, unlike pub/sub which broadcasts to all subscribers.

Choosing between pub/sub and streams depends on the need for speed, reliability, message history, and resource management.