Pub/sub limitations (no persistence) in Redis - Deep Dive

Overview - Pub/sub limitations (no persistence)

What is it?

Pub/sub is a messaging pattern where publishers send messages to channels, and subscribers listen to those channels to receive messages instantly. In Redis, this system allows real-time communication but does not save messages. If a subscriber is not connected when a message is sent, it misses that message forever. This means messages are transient and only delivered live.

Why it matters

Without persistence, messages can be lost if subscribers are offline or disconnected. This limits reliability for critical data delivery. Imagine missing an important alert because you were not connected at that moment. Persistence solves this by storing messages until subscribers can receive them, ensuring no data loss.

Where it fits

Before learning this, you should understand basic Redis commands and the pub/sub messaging concept. After this, you can explore Redis Streams or other message queue systems that provide persistence and message durability.

Mental Model

Core Idea

Pub/sub without persistence means messages are like live TV broadcasts: if you miss the show, you cannot watch it later.

Think of it like...

It's like shouting a message in a crowded room: only people listening at that exact moment hear it; if they come late, they miss it.

┌─────────────┐       ┌─────────────┐       ┌─────────────┐
│ Publisher   │──────▶│ Channel     │──────▶│ Subscriber  │
│ (sends msg) │       │ (no storage)│       │ (must listen│
└─────────────┘       └─────────────┘       │ live only)  │
                                            └─────────────┘

Build-Up - 6 Steps

1

Foundation

Understanding Pub/Sub Basics

Concept: Introduce the basic pub/sub pattern where messages are sent and received live.

In Redis, a publisher sends messages to a named channel. Subscribers listen to that channel and get messages instantly. If no one is subscribed, messages disappear immediately.

Result

Messages are delivered only to currently connected subscribers.

Understanding that pub/sub delivers messages live without storing them is key to grasping its limitations.

2

Foundation

No Message Storage Explained

Concept: Explain that Redis pub/sub does not save messages for later delivery.

When a message is published, Redis sends it to all subscribers connected at that moment. If a subscriber connects after the message is sent, it will not receive that message. Redis does not keep a history or queue of messages.

Result

Subscribers only get messages sent while they are connected.

Knowing that messages vanish if no subscriber is listening helps understand why pub/sub is not reliable for all use cases.

3

Intermediate

Impact of Subscriber Disconnections

🤔Before reading on: do you think a subscriber who disconnects briefly will receive missed messages when reconnecting? Commit to yes or no.

Concept: Explore how disconnections cause message loss in pub/sub.

If a subscriber disconnects even for a short time, any messages published during that period are lost to it. Redis does not queue or replay messages. This can cause missed data or alerts.

Result

Disconnected subscribers permanently miss messages sent during their absence.

Understanding this limitation is crucial for designing systems that require guaranteed message delivery.

4

Intermediate

No Delivery Guarantees in Pub/Sub

🤔Before reading on: do you think Redis pub/sub guarantees every message reaches all subscribers? Commit to yes or no.

Concept: Clarify that pub/sub offers no delivery guarantees or acknowledgments.

Redis pub/sub does not confirm if subscribers received messages. It simply broadcasts messages to connected clients. Network issues or client crashes can cause message loss without detection.

Result

Message delivery is best-effort and can silently fail.

Knowing the lack of delivery guarantees helps decide when pub/sub is appropriate or when more robust messaging is needed.

5

Advanced

Alternatives for Persistence in Redis

🤔Before reading on: do you think Redis pub/sub can be extended to store messages for later? Commit to yes or no.

Concept: Introduce Redis Streams as a persistent alternative to pub/sub.

Redis Streams store messages with IDs, allowing consumers to read messages at their own pace, even after disconnecting. This solves the no-persistence problem of pub/sub by keeping message history.

Result

Using Redis Streams enables reliable message delivery with persistence and replay.

Understanding Redis Streams provides a path to overcome pub/sub's limitations in real applications.

6

Expert

Hidden Pitfalls in High-Load Pub/Sub

🤔Before reading on: do you think pub/sub scales infinitely without issues? Commit to yes or no.

Concept: Discuss performance and message loss risks under heavy load.

Under high message rates or many subscribers, Redis pub/sub can drop messages if clients cannot keep up. Since there is no buffering, slow subscribers cause message loss. This is a subtle but critical limitation in production.

Result

Message loss can occur silently under load, affecting system reliability.

Knowing this helps design systems with proper load handling or choose persistent queues for critical data.

Under the Hood

Redis pub/sub uses an in-memory publish-subscribe system where messages are pushed directly to all connected subscribers of a channel. There is no internal queue or storage; messages are sent over TCP sockets immediately. If a subscriber is not connected, Redis simply discards the message for that client. This design prioritizes speed and simplicity over durability.

Why designed this way?

Redis pub/sub was designed for low-latency, real-time messaging where speed is critical and message history is not needed. Storing messages would add complexity, memory use, and latency. For use cases needing persistence, Redis Streams or external message brokers are recommended.

┌─────────────┐
│ Publisher   │
└─────┬───────┘
      │ Publish message
      ▼
┌─────────────┐
│ Redis Server│
│ Pub/Sub     │
│ (no storage)│
└─────┬───────┘
      │ Broadcast
      ▼
┌─────────────┐   ┌─────────────┐
│ Subscriber1 │   │ Subscriber2 │
│ (connected) │   │ (disconnected)│
└─────────────┘   └─────────────┘
      ▲                 X
      │                 │
      └─────────Message lost─────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does Redis pub/sub store messages for subscribers who are offline? Commit to yes or no.

Common Belief: Redis pub/sub keeps messages until subscribers connect, so no messages are lost.

Reality: Redis pub/sub does not store messages; offline subscribers miss all messages sent while disconnected.

Why it matters: Believing messages are stored can lead to data loss in critical systems when subscribers disconnect.

Quick: Does Redis pub/sub guarantee every subscriber receives every message? Commit to yes or no.

Common Belief: Redis pub/sub guarantees message delivery to all subscribers.

Reality: Redis pub/sub offers no delivery guarantees; messages can be lost due to network issues or slow clients.

Why it matters: Assuming guaranteed delivery can cause silent failures and missed important data.

Quick: Can Redis pub/sub handle very high message rates without dropping messages? Commit to yes or no.

Common Belief: Redis pub/sub scales perfectly and never loses messages under load.

Reality: Under heavy load, slow subscribers can cause message drops since there is no buffering or backpressure.

Why it matters: Ignoring this can cause unexpected message loss in production environments.

Quick: Can Redis pub/sub be used as a persistent message queue? Commit to yes or no.

Common Belief: Redis pub/sub can be used as a persistent queue by default.

Reality: Redis pub/sub is not persistent; Redis Streams or other queues are needed for persistence.

Why it matters: Using pub/sub for persistence leads to lost messages and unreliable systems.

Expert Zone

1

Pub/sub message loss is silent; Redis does not notify if a subscriber missed messages, making debugging tricky.

2

Network partitions or client slowdowns can cause message drops without affecting Redis server health metrics.

3

Redis pub/sub channels are ephemeral; they exist only while publishers or subscribers are connected, affecting dynamic system design.

When NOT to use

Do not use Redis pub/sub when message durability, guaranteed delivery, or replay is required. Instead, use Redis Streams, Kafka, RabbitMQ, or other persistent message brokers.

Production Patterns

In production, Redis pub/sub is used for ephemeral notifications like cache invalidation or live updates where losing messages is acceptable. For critical workflows, systems combine pub/sub with persistent queues or logs.

Connections

Event-driven architecture

Pub/sub is a core pattern in event-driven systems for decoupling components.

Understanding pub/sub limitations helps design event-driven systems that balance real-time updates with reliability.

Distributed systems

Pub/sub messaging is a communication method in distributed systems but lacks persistence and delivery guarantees.

Knowing pub/sub limits clarifies why distributed systems often layer persistent queues for fault tolerance.

Live broadcasting

Pub/sub without persistence is like live broadcasting where viewers miss content if not tuned in.

Recognizing this similarity helps grasp why buffering or recording is needed for reliable message delivery.

Common Pitfalls

#1 Assuming subscribers receive all messages even if disconnected.

Wrong approach:

SUBSCRIBE channel
// Subscriber disconnects
PUBLISH channel 'message'
// Subscriber reconnects and expects the message

Correct approach:

Use Redis Streams to store messages:
XADD mystream * message 'message'
XREAD ... to consume messages reliably

Root cause:

Misunderstanding that Redis pub/sub does not store or queue messages for offline clients.

#2 Using pub/sub for critical data delivery without backup.

Wrong approach:

Rely solely on PUBLISH/SUBSCRIBE for order processing notifications.

Correct approach:

Combine pub/sub for real-time alerts with persistent queues or databases for guaranteed processing.

Root cause:

Ignoring the lack of delivery guarantees and persistence in pub/sub.

#3 Ignoring message loss under high load.

Wrong approach:

Assuming Redis pub/sub handles all message rates without tuning or monitoring.

Correct approach:

Monitor subscriber performance and use persistent queues or rate limiting to avoid message drops.

Root cause:

Overestimating pub/sub scalability and ignoring backpressure effects.

Key Takeaways

Redis pub/sub delivers messages only to subscribers connected at the time of publishing, with no message storage.

Subscribers disconnected during message publishing permanently miss those messages, making pub/sub unreliable for critical data.

Pub/sub offers no delivery guarantees or acknowledgments, so message loss can occur silently.

For persistence and reliable delivery, Redis Streams or other message queue systems should be used instead of pub/sub.

Understanding these limitations helps design messaging systems that balance real-time needs with reliability.