Kafka vs RabbitMQ vs Redis Pub/Sub - Hands-On Comparison

Kafka vs RabbitMQ vs Redis Pub/Sub Comparison

📖 Scenario: You are working on a messaging system for a small online store. You want to understand how three popular messaging tools work: Kafka, RabbitMQ, and Redis Pub/Sub. Each tool helps different parts of your store talk to each other by sending messages.

🎯 Goal: Build simple examples to see how Kafka, RabbitMQ, and Redis Pub/Sub send and receive messages. This will help you understand their differences and when to use each one.

📋 What You'll Learn

Create a Kafka topic and send a message

Set up a RabbitMQ queue and send a message

Use Redis Pub/Sub to publish and subscribe to a message

Print the received messages from each system

💡 Why This Matters

🌍 Real World

Messaging systems help different parts of an application talk to each other smoothly, like sending orders from a website to a warehouse system.

💼 Career

Understanding Kafka, RabbitMQ, and Redis Pub/Sub is useful for backend developers, system architects, and anyone working with real-time data or distributed systems.

1

Set up Kafka topic and send a message

Create a Kafka producer that sends the message 'Hello Kafka' to the topic 'store-messages'. Use the variable producer for the Kafka producer and topic for the topic name.

Kafka

# Create Kafka producer and send 'Hello Kafka' to 'store-messages'
# Your code here

Hint

Use KafkaProducer from the kafka library. The message must be bytes, so prefix the string with b.

2

Set up RabbitMQ queue and send a message

Create a RabbitMQ connection and channel. Declare a queue named 'store-queue' and send the message 'Hello RabbitMQ' to this queue. Use variables connection, channel, and queue_name.

Kafka

from kafka import KafkaProducer
import pika

producer = KafkaProducer(bootstrap_servers='localhost:9092')
topic = 'store-messages'

producer.send(topic, b'Hello Kafka')
producer.flush()

# Create RabbitMQ connection and send 'Hello RabbitMQ' to 'store-queue'
# Your code here

Hint

Use pika.BlockingConnection to connect to RabbitMQ. Declare the queue before sending the message.

3

Use Redis Pub/Sub to publish and subscribe

Create a Redis client using redis.Redis(). Subscribe to the channel 'store-channel' and publish the message 'Hello Redis' to it. Use variables redis_client and channel_name.

Kafka

from kafka import KafkaProducer
import pika
import redis

producer = KafkaProducer(bootstrap_servers='localhost:9092')
topic = 'store-messages'

producer.send(topic, b'Hello Kafka')
producer.flush()

connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()
queue_name = 'store-queue'
channel.queue_declare(queue=queue_name)
channel.basic_publish(exchange='', routing_key=queue_name, body='Hello RabbitMQ')
connection.close()

# Create Redis client, subscribe to 'store-channel', and publish 'Hello Redis'
# Your code here

Hint

Use redis.Redis() to connect. Use pubsub.subscribe() to listen and publish() to send messages.

4

Print received messages from Kafka, RabbitMQ, and Redis

Print the messages received from Kafka topic 'store-messages', RabbitMQ queue 'store-queue', and Redis channel 'store-channel'. Use variables producer, connection, channel, and redis_client as needed. Show the exact messages: Hello Kafka, Hello RabbitMQ, and Hello Redis.

Kafka

from kafka import KafkaProducer
import pika
import redis

producer = KafkaProducer(bootstrap_servers='localhost:9092')
topic = 'store-messages'

producer.send(topic, b'Hello Kafka')
producer.flush()

connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()
queue_name = 'store-queue'
channel.queue_declare(queue=queue_name)
channel.basic_publish(exchange='', routing_key=queue_name, body='Hello RabbitMQ')
connection.close()

redis_client = redis.Redis()
channel_name = 'store-channel'

pubsub = redis_client.pubsub()
pubsub.subscribe(channel_name)
redis_client.publish(channel_name, 'Hello Redis')

# Print the messages received from Kafka, RabbitMQ, and Redis
# Your code here

Hint

Use KafkaConsumer to read from Kafka. Use channel.basic_get to get RabbitMQ message. Use pubsub.listen() to get Redis message. Decode bytes to strings before printing.

Practice

1. Which messaging system is best suited for storing message history and handling large data streams?

easy

A. None of the above

B. RabbitMQ

C. Redis Pub/Sub

D. Kafka

Solution

1. Step 1: Understand Kafka's purpose

   Kafka is designed to handle large data streams and keeps a durable log of messages, allowing message history to be stored.

2. Step 2: Compare with RabbitMQ and Redis Pub/Sub

   RabbitMQ focuses on reliable delivery and routing, while Redis Pub/Sub is fast but does not store messages.

3. Final Answer:

   Kafka -> Option D

4. Quick Check:

   Message history and big data = Kafka [OK]

Hint: Remember: Kafka stores history; Redis Pub/Sub does not [OK]

Common Mistakes:

• Confusing Redis Pub/Sub as storing messages
• Thinking RabbitMQ stores message history
• Assuming all systems keep message logs

2. Which of the following is the correct statement about Redis Pub/Sub?

easy

A. Redis Pub/Sub guarantees message delivery even if subscribers are offline.

B. Redis Pub/Sub does not store messages; it only delivers messages to active subscribers.

C. Redis Pub/Sub supports complex message routing like RabbitMQ.

D. Redis Pub/Sub is designed for batch processing of messages.

Solution

1. Step 1: Analyze Redis Pub/Sub behavior

   Redis Pub/Sub delivers messages only to subscribers currently connected; it does not store messages for offline clients.

2. Step 2: Check other options

   Redis Pub/Sub guarantees message delivery even if subscribers are offline. is false because Redis Pub/Sub does not guarantee delivery if subscribers are offline. Redis Pub/Sub supports complex message routing like RabbitMQ. is false because Redis Pub/Sub lacks complex routing. Redis Pub/Sub is designed for batch processing of messages. is false because Redis Pub/Sub is for real-time messaging, not batch processing.

3. Final Answer:

   Redis Pub/Sub does not store messages; it only delivers messages to active subscribers. -> Option B

4. Quick Check:

   Redis Pub/Sub delivers only to active subscribers [OK]

Hint: Redis Pub/Sub = no message storage, only live delivery [OK]

Common Mistakes:

• Assuming Redis Pub/Sub stores messages
• Confusing Redis with RabbitMQ routing features
• Thinking Redis Pub/Sub supports offline message delivery

3. Given the following scenario: You need a messaging system that supports flexible routing and guarantees message delivery even if consumers are temporarily offline. Which system fits best?

medium

A. Kafka

B. Redis Pub/Sub

C. RabbitMQ

D. None of the above

Solution

1. Step 1: Understand RabbitMQ features

   RabbitMQ supports flexible routing through exchanges and queues and guarantees message delivery with acknowledgments and persistence.

2. Step 2: Compare with Kafka and Redis Pub/Sub

   Kafka stores messages but is less flexible in routing. Redis Pub/Sub is fast but does not guarantee delivery if consumers are offline.

3. Final Answer:

   RabbitMQ -> Option C

4. Quick Check:

   Flexible routing + guaranteed delivery = RabbitMQ [OK]

Hint: Flexible routing and delivery guarantees point to RabbitMQ [OK]

Common Mistakes:

• Choosing Kafka for flexible routing
• Assuming Redis Pub/Sub guarantees delivery
• Ignoring RabbitMQ's message acknowledgment feature

4. You have a system using Redis Pub/Sub, but messages are lost when subscribers disconnect. What is the best way to fix this issue?

medium

A. Switch to Kafka to store messages and allow offline consumption.

B. Increase Redis server memory to store more messages.

C. Use Redis Pub/Sub with message persistence enabled.

D. Configure Redis to retry sending messages to offline subscribers.

Solution

1. Step 1: Identify Redis Pub/Sub limitation

   Redis Pub/Sub does not store messages; if subscribers disconnect, messages are lost.

2. Step 2: Choose a system that stores messages

   Kafka stores messages durably, allowing consumers to read messages even after being offline.

3. Final Answer:

   Switch to Kafka to store messages and allow offline consumption. -> Option A

4. Quick Check:

   Redis Pub/Sub no storage; Kafka stores messages [OK]

Hint: Redis Pub/Sub can't store messages; Kafka can [OK]

Common Mistakes:

• Trying to enable persistence on Redis Pub/Sub (not supported)
• Assuming Redis memory increase solves message loss
• Expecting Redis to retry offline delivery automatically

5. You want to build a real-time analytics system that processes large volumes of data and keeps a full history of all events for auditing. Which combination of messaging systems is the best approach?

hard

A. Use Kafka for event storage and Redis Pub/Sub for real-time notifications.

B. Use RabbitMQ for event storage and Redis Pub/Sub for message routing.

C. Use Redis Pub/Sub alone for both storage and real-time processing.

D. Use RabbitMQ alone for storing large data streams and real-time analytics.

Solution

1. Step 1: Identify requirements

   The system needs to handle large data volumes, keep full event history, and provide real-time notifications.

2. Step 2: Match systems to requirements

   Kafka is ideal for storing large data streams and keeping history. Redis Pub/Sub is fast for real-time notifications but does not store data.

3. Step 3: Evaluate other options

   RabbitMQ is good for routing and delivery but less suited for big data storage. Redis Pub/Sub alone cannot store messages. RabbitMQ alone is not optimized for large data streams.

4. Final Answer:

   Use Kafka for event storage and Redis Pub/Sub for real-time notifications. -> Option A

5. Quick Check:

   Kafka stores history + Redis Pub/Sub fast notifications [OK]

Hint: Combine Kafka storage with Redis Pub/Sub speed [OK]

Common Mistakes:

• Using Redis Pub/Sub alone for storage
• Assuming RabbitMQ handles big data storage well
• Ignoring Kafka's strength in event history