Kafka vs Redis Pub/Sub: Key Differences and When to Use Each

Both Kafka and Redis Pub/Sub are messaging systems but serve different purposes. Kafka is designed for durable, scalable event streaming with message persistence, while Redis Pub/Sub offers fast, in-memory message broadcasting without persistence or message replay.

Quick Comparison

This table summarizes the main differences between Kafka and Redis Pub/Sub across key factors.

Factor	Kafka	Redis Pub/Sub
Architecture	Distributed, partitioned log	In-memory, single-node or clustered
Message Durability	Persistent on disk	No persistence, messages lost if subscriber offline
Scalability	Highly scalable with partitions and brokers	Limited by Redis instance size
Message Ordering	Guaranteed within partitions	No ordering guarantees
Use Case	Event streaming, data pipelines, durable messaging	Real-time notifications, ephemeral messaging
Latency	Higher latency due to disk writes	Very low latency, in-memory delivery

Key Differences

Kafka is built as a distributed commit log that stores messages on disk, allowing consumers to read messages at their own pace and replay them if needed. It supports partitioning for scalability and guarantees message ordering within each partition. This makes Kafka ideal for event sourcing, stream processing, and reliable data pipelines.

In contrast, Redis Pub/Sub is an in-memory messaging system that broadcasts messages to all connected subscribers instantly. It does not store messages, so if a subscriber is offline, it misses those messages. Redis Pub/Sub is simpler and faster but lacks durability and replay capabilities, making it suitable for real-time notifications or ephemeral events.

Kafka requires more setup and resources but provides strong durability and scalability. Redis Pub/Sub is lightweight and easy to use but is limited to transient messaging scenarios without guaranteed delivery.

Code Comparison

Here is a simple example of publishing and subscribing to messages using Kafka in Python with the confluent-kafka library.

python

from confluent_kafka import Producer, Consumer, KafkaError

# Kafka producer configuration
producer_conf = {'bootstrap.servers': 'localhost:9092'}
producer = Producer(producer_conf)

def delivery_report(err, msg):
    if err is not None:
        print(f'Message delivery failed: {err}')
    else:
        print(f'Message delivered to {msg.topic()} [{msg.partition()}]')

# Produce a message
producer.produce('test-topic', key='key1', value='Hello Kafka!', callback=delivery_report)
producer.flush()

# Kafka consumer configuration
consumer_conf = {
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'my-group',
    'auto.offset.reset': 'earliest'
}
consumer = Consumer(consumer_conf)
consumer.subscribe(['test-topic'])

# Consume one message
msg = consumer.poll(1.0)
if msg is None:
    print('No message received')
elif msg.error():
    print(f'Error: {msg.error()}')
else:
    print(f'Received message: {msg.value().decode("utf-8")}')
consumer.close()

Output

Message delivered to test-topic [0]
Received message: Hello Kafka!

Redis Pub/Sub Equivalent

Here is the equivalent example using Redis Pub/Sub in Python with the redis library.

python

import redis
import threading
import time

r = redis.Redis(host='localhost', port=6379)

# Subscriber function

def subscriber():
    pubsub = r.pubsub()
    pubsub.subscribe('test-channel')
    for message in pubsub.listen():
        if message['type'] == 'message':
            print(f"Received message: {message['data'].decode('utf-8')}")

# Start subscriber in a separate thread
threading.Thread(target=subscriber, daemon=True).start()

# Give subscriber time to connect
time.sleep(1)

# Publish a message
r.publish('test-channel', 'Hello Redis Pub/Sub!')

# Wait to receive message
time.sleep(1)

Output

Received message: Hello Redis Pub/Sub!

When to Use Which

Choose Kafka when you need a reliable, durable messaging system that supports message replay, ordering, and high scalability for event streaming or data pipelines. Kafka is best for systems where message loss is unacceptable and consumers may process data at different speeds.

Choose Redis Pub/Sub when you want very fast, simple, real-time message broadcasting without the need for persistence or replay. It is ideal for ephemeral notifications, chat messages, or live updates where losing messages is acceptable.

Key Takeaways

Kafka provides durable, scalable, and ordered messaging with message replay support.

Redis Pub/Sub offers fast, in-memory message broadcasting without persistence.

Use Kafka for event streaming and reliable data pipelines.

Use Redis Pub/Sub for real-time notifications and ephemeral messaging.

Kafka requires more setup but ensures no message loss; Redis is simpler but less reliable.