Kafka vs RabbitMQ vs Redis Pub/Sub - CLI Comparison

Introduction

When you want to send messages between different parts of your system, you need a tool that can handle this smoothly. Kafka, RabbitMQ, and Redis Pub/Sub are three popular tools that help with sending and receiving messages, but they work in different ways and are good for different jobs.

When you need to process a large stream of data reliably and keep a history of messages, like tracking user activity on a website.

When you want to send messages between different services in your app with guaranteed delivery and complex routing, like order processing in an online store.

When you want a simple way to send real-time notifications or updates to many users quickly, like chat messages or live scores.

When you need to handle high message throughput with fault tolerance, such as logging events from many servers.

When you want a lightweight, fast messaging system for simple pub/sub without message persistence.

Commands

This command creates a Kafka topic named 'example-topic' with 3 partitions and a replication factor of 1. Topics are where Kafka stores messages.

Terminal

kafka-topics.sh --create --topic example-topic --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1

Expected Output

Created topic example-topic.

→

--topic - Name of the topic to create

→

--partitions - Number of partitions for parallelism

→

--replication-factor - Number of copies of data for fault tolerance

This command creates a virtual host in RabbitMQ named 'example_vhost' to isolate messaging environments.

Terminal

rabbitmqctl add_vhost example_vhost

Expected Output

No output (command runs silently)

This command sends a message 'Hello subscribers!' to all clients subscribed to 'example-channel' in Redis Pub/Sub.

Terminal

redis-cli PUBLISH example-channel "Hello subscribers!"

Expected Output

(integer) 1

This command reads one message from the beginning of the 'example-topic' in Kafka to verify messages are stored and retrievable.

Terminal

kafka-console-consumer.sh --topic example-topic --bootstrap-server localhost:9092 --from-beginning --max-messages 1

Expected Output

Hello Kafka

Processed a total of 1 messages

→

--from-beginning - Read messages from the start of the topic

→

--max-messages - Limit the number of messages to consume

This command lists all queues in the 'example_vhost' virtual host in RabbitMQ to check message queues.

Terminal

rabbitmqctl list_queues -p example_vhost

Expected Output

Queue	Messages

my-queue	0

→

-p - Specify the virtual host

Key Concept

If you remember nothing else, remember: Kafka is best for high-throughput, durable streaming; RabbitMQ is best for complex routing and guaranteed delivery; Redis Pub/Sub is best for simple, fast, real-time messaging without persistence.

Common Mistakes

Using Redis Pub/Sub when message durability is needed

Redis Pub/Sub does not store messages, so if a subscriber is offline, it misses messages.

Use Kafka or RabbitMQ when you need to keep messages until consumers process them.

Creating too few partitions in Kafka for high load

Too few partitions limit parallel processing and throughput.

Create enough partitions to allow multiple consumers to read in parallel.

Not setting up virtual hosts in RabbitMQ for environment separation

Without virtual hosts, different apps share the same queues and exchanges, causing conflicts.

Use virtual hosts to isolate messaging environments.

Summary

Kafka stores messages durably and handles large data streams with partitions and replication.

RabbitMQ supports complex message routing and guarantees delivery with queues and exchanges.

Redis Pub/Sub offers fast, simple messaging but does not keep messages for offline subscribers.

Practice

1. Which messaging system is best suited for storing message history and handling large data streams?

easy

A. None of the above

B. RabbitMQ

C. Redis Pub/Sub

D. Kafka

Solution

1. Step 1: Understand Kafka's purpose

   Kafka is designed to handle large data streams and keeps a durable log of messages, allowing message history to be stored.

2. Step 2: Compare with RabbitMQ and Redis Pub/Sub

   RabbitMQ focuses on reliable delivery and routing, while Redis Pub/Sub is fast but does not store messages.

3. Final Answer:

   Kafka -> Option D

4. Quick Check:

   Message history and big data = Kafka [OK]

Hint: Remember: Kafka stores history; Redis Pub/Sub does not [OK]

Common Mistakes:

• Confusing Redis Pub/Sub as storing messages
• Thinking RabbitMQ stores message history
• Assuming all systems keep message logs

2. Which of the following is the correct statement about Redis Pub/Sub?

easy

A. Redis Pub/Sub guarantees message delivery even if subscribers are offline.

B. Redis Pub/Sub does not store messages; it only delivers messages to active subscribers.

C. Redis Pub/Sub supports complex message routing like RabbitMQ.

D. Redis Pub/Sub is designed for batch processing of messages.

Solution

1. Step 1: Analyze Redis Pub/Sub behavior

   Redis Pub/Sub delivers messages only to subscribers currently connected; it does not store messages for offline clients.

2. Step 2: Check other options

   Redis Pub/Sub guarantees message delivery even if subscribers are offline. is false because Redis Pub/Sub does not guarantee delivery if subscribers are offline. Redis Pub/Sub supports complex message routing like RabbitMQ. is false because Redis Pub/Sub lacks complex routing. Redis Pub/Sub is designed for batch processing of messages. is false because Redis Pub/Sub is for real-time messaging, not batch processing.

3. Final Answer:

   Redis Pub/Sub does not store messages; it only delivers messages to active subscribers. -> Option B

4. Quick Check:

   Redis Pub/Sub delivers only to active subscribers [OK]

Hint: Redis Pub/Sub = no message storage, only live delivery [OK]

Common Mistakes:

• Assuming Redis Pub/Sub stores messages
• Confusing Redis with RabbitMQ routing features
• Thinking Redis Pub/Sub supports offline message delivery

3. Given the following scenario: You need a messaging system that supports flexible routing and guarantees message delivery even if consumers are temporarily offline. Which system fits best?

medium

A. Kafka

B. Redis Pub/Sub

C. RabbitMQ

D. None of the above

Solution

1. Step 1: Understand RabbitMQ features

   RabbitMQ supports flexible routing through exchanges and queues and guarantees message delivery with acknowledgments and persistence.

2. Step 2: Compare with Kafka and Redis Pub/Sub

   Kafka stores messages but is less flexible in routing. Redis Pub/Sub is fast but does not guarantee delivery if consumers are offline.

3. Final Answer:

   RabbitMQ -> Option C

4. Quick Check:

   Flexible routing + guaranteed delivery = RabbitMQ [OK]

Hint: Flexible routing and delivery guarantees point to RabbitMQ [OK]

Common Mistakes:

• Choosing Kafka for flexible routing
• Assuming Redis Pub/Sub guarantees delivery
• Ignoring RabbitMQ's message acknowledgment feature

4. You have a system using Redis Pub/Sub, but messages are lost when subscribers disconnect. What is the best way to fix this issue?

medium

A. Switch to Kafka to store messages and allow offline consumption.

B. Increase Redis server memory to store more messages.

C. Use Redis Pub/Sub with message persistence enabled.

D. Configure Redis to retry sending messages to offline subscribers.

Solution

1. Step 1: Identify Redis Pub/Sub limitation

   Redis Pub/Sub does not store messages; if subscribers disconnect, messages are lost.

2. Step 2: Choose a system that stores messages

   Kafka stores messages durably, allowing consumers to read messages even after being offline.

3. Final Answer:

   Switch to Kafka to store messages and allow offline consumption. -> Option A

4. Quick Check:

   Redis Pub/Sub no storage; Kafka stores messages [OK]

Hint: Redis Pub/Sub can't store messages; Kafka can [OK]

Common Mistakes:

• Trying to enable persistence on Redis Pub/Sub (not supported)
• Assuming Redis memory increase solves message loss
• Expecting Redis to retry offline delivery automatically

5. You want to build a real-time analytics system that processes large volumes of data and keeps a full history of all events for auditing. Which combination of messaging systems is the best approach?

hard

A. Use Kafka for event storage and Redis Pub/Sub for real-time notifications.

B. Use RabbitMQ for event storage and Redis Pub/Sub for message routing.

C. Use Redis Pub/Sub alone for both storage and real-time processing.

D. Use RabbitMQ alone for storing large data streams and real-time analytics.

Solution

1. Step 1: Identify requirements

   The system needs to handle large data volumes, keep full event history, and provide real-time notifications.

2. Step 2: Match systems to requirements

   Kafka is ideal for storing large data streams and keeping history. Redis Pub/Sub is fast for real-time notifications but does not store data.

3. Step 3: Evaluate other options

   RabbitMQ is good for routing and delivery but less suited for big data storage. Redis Pub/Sub alone cannot store messages. RabbitMQ alone is not optimized for large data streams.

4. Final Answer:

   Use Kafka for event storage and Redis Pub/Sub for real-time notifications. -> Option A

5. Quick Check:

   Kafka stores history + Redis Pub/Sub fast notifications [OK]

Hint: Combine Kafka storage with Redis Pub/Sub speed [OK]

Common Mistakes:

• Using Redis Pub/Sub alone for storage
• Assuming RabbitMQ handles big data storage well
• Ignoring Kafka's strength in event history