Kafka vs RabbitMQ vs SQS in HLD - Design Approaches Compared

Design: Message Queue System Comparison

Compare three popular message queue systems: Apache Kafka, RabbitMQ, and AWS SQS. Focus on their architecture, use cases, strengths, and limitations. Out of scope: detailed installation or configuration steps.

Functional Requirements

FR1: Support asynchronous communication between distributed components

FR2: Handle high throughput of messages

FR3: Ensure message durability and reliability

FR4: Support message ordering where needed

FR5: Provide scalability to handle increasing load

FR6: Allow multiple consumers to process messages

FR7: Support different delivery guarantees (at least once, exactly once, at most once)

Non-Functional Requirements

NFR1: Latency for message delivery should be under 100ms for typical use cases

NFR2: System should handle at least 100,000 messages per second

NFR3: Availability target of 99.9% uptime

NFR4: Support integration with cloud and on-premise environments

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

❓ Question 6

❓ Question 7

Key Components

Message broker or cluster

Producers and consumers

Message storage and retention

Consumer groups and load balancing

Delivery guarantees and acknowledgments

Scaling mechanisms (partitioning, sharding)

Monitoring and management tools

Design Patterns

Publish-subscribe pattern

Message queue pattern

Event streaming

Load balancing with consumer groups

Dead letter queues for failed messages

Reference Architecture

 +----------------+       +----------------+       +----------------+
 |   Producers    |       |   Message      |       |   Consumers    |
 | (Kafka, RabbitMQ|----->|   Broker       |-----> | (Kafka, RabbitMQ|
 |  or SQS clients)|       | (Kafka, RabbitMQ|       |  or SQS clients)|
 +----------------+       |  or SQS)       |       +----------------+
                          +----------------+

Kafka: Distributed log with partitions and brokers
RabbitMQ: Broker with queues and exchanges
SQS: Managed queue service with polling consumers

Components

Apache Kafka Cluster

Kafka

Distributed log storage with partitions for high throughput and scalability

RabbitMQ Broker

RabbitMQ

Message broker supporting flexible routing with exchanges and queues

AWS SQS Service

Amazon SQS

Fully managed message queue service with simple API and automatic scaling

Producers

Kafka Producer API, RabbitMQ Producer, SQS SDK

Send messages to the broker or queue

Consumers

Kafka Consumer API, RabbitMQ Consumer, SQS SDK

Receive and process messages from the broker or queue

Request Flow

1. Producer sends message to the broker or queue.

2. Broker stores message persistently (Kafka in partitions, RabbitMQ in queues, SQS in managed storage).

3. Consumers poll or subscribe to receive messages.

4. Broker delivers messages according to configured routing and delivery guarantees.

5. Consumers acknowledge message processing (depending on system).

6. Failed messages can be routed to dead letter queues or retried.

Database Schema

Not applicable as these are messaging systems. Key concepts include topics (Kafka), queues (RabbitMQ, SQS), partitions (Kafka), exchanges (RabbitMQ), and message metadata (offsets, timestamps, delivery status).

Scaling Discussion

Bottlenecks

Broker becoming a single point of failure or performance bottleneck

Message storage limits causing slowdowns

Consumer processing speed limiting throughput

Network bandwidth constraints

Managing message ordering with many partitions or queues

Solutions

Kafka uses partitioning and replication to scale horizontally and provide fault tolerance

RabbitMQ supports clustering and federation for scaling and high availability

SQS automatically scales with demand as a managed service

Use consumer groups to parallelize message processing

Implement backpressure and rate limiting to avoid overload

Use dead letter queues to handle poison messages without blocking

Interview Tips

Time: Spend 10 minutes explaining each system's architecture and use cases, 10 minutes comparing their strengths and weaknesses, and 5 minutes discussing scaling and operational considerations.

Kafka is best for high-throughput event streaming with strong ordering guarantees.

RabbitMQ excels in flexible routing and supports complex messaging patterns.

SQS is a simple, fully managed queue service ideal for cloud-native applications.

Discuss delivery guarantees: Kafka supports exactly-once with extra setup, RabbitMQ supports at-least-once, SQS supports at-least-once with FIFO queues for ordering.

Highlight scaling approaches: Kafka partitions, RabbitMQ clustering, SQS managed scaling.

Mention operational complexity: Kafka and RabbitMQ require management, SQS is managed by AWS.

Practice

1. Which messaging system is best suited for handling high-speed, large-scale streaming with multiple independent consumers?

easy

A. ActiveMQ

B. RabbitMQ

C. Kafka

D. SQS

Solution

1. Step 1: Understand Kafka's design for streaming

   Kafka is designed for high-throughput, distributed streaming where multiple consumers can read independently from the same stream.

2. Step 2: Compare with RabbitMQ and SQS

   RabbitMQ focuses on complex routing and guaranteed delivery with message removal, while SQS is a simple managed queue service without streaming capabilities.

3. Final Answer:

   Kafka -> Option C

4. Quick Check:

   High-speed streaming = Kafka [OK]

Hint: High-speed streaming with multiple readers? Think Kafka [OK]

Common Mistakes:

• Confusing RabbitMQ's routing with Kafka's streaming
• Assuming SQS supports multiple independent consumers reading the same message
• Choosing a generic queue like ActiveMQ without streaming focus

2. Which of the following correctly describes a key feature of RabbitMQ?

easy

A. It supports only one consumer per queue

B. It stores messages indefinitely for replay

C. It is a fully managed cloud service

D. It removes messages after they are consumed

Solution

1. Step 1: Recall RabbitMQ message handling

   RabbitMQ removes messages from the queue once they are consumed to ensure guaranteed delivery and avoid duplicate processing.

2. Step 2: Eliminate incorrect options

   Kafka stores messages for replay, SQS is managed cloud service, and RabbitMQ supports multiple consumers with routing.

3. Final Answer:

   It removes messages after they are consumed -> Option D

4. Quick Check:

   RabbitMQ message removal = It removes messages after they are consumed [OK]

Hint: RabbitMQ deletes messages after consumption [OK]

Common Mistakes:

• Thinking RabbitMQ stores messages indefinitely like Kafka
• Confusing SQS managed service with RabbitMQ
• Assuming RabbitMQ supports only one consumer

3. Given a system requiring simple, reliable message queuing without managing servers, which service would output the most scalable solution?

medium

A. RabbitMQ on self-hosted servers

B. AWS SQS managed queue

C. Kafka cluster managed by user

D. Local in-memory queue

Solution

1. Step 1: Identify managed services for scalability

   AWS SQS is a fully managed queue service that scales automatically without server management.

2. Step 2: Compare with other options

   Kafka and RabbitMQ require managing clusters or servers, while local queues lack scalability and reliability.

3. Final Answer:

   AWS SQS managed queue -> Option B

4. Quick Check:

   Simple, scalable, no server management = SQS [OK]

Hint: Managed, scalable queue without servers? Choose SQS [OK]

Common Mistakes:

• Choosing Kafka or RabbitMQ without considering management overhead
• Assuming local queues scale well
• Confusing managed service with self-hosted

4. A developer tries to use RabbitMQ for a use case requiring message replay by multiple consumers independently. What is the main issue with this approach?

medium

A. RabbitMQ removes messages after consumption, so replay is not possible

B. RabbitMQ does not support multiple consumers

C. RabbitMQ cannot guarantee message delivery

D. RabbitMQ is not scalable for multiple consumers

Solution

1. Step 1: Understand RabbitMQ message lifecycle

   RabbitMQ deletes messages once consumed, preventing replay by multiple consumers independently.

2. Step 2: Validate other options

   RabbitMQ supports multiple consumers and guarantees delivery; scalability depends on setup but is not the main issue here.

3. Final Answer:

   RabbitMQ removes messages after consumption, so replay is not possible -> Option A

4. Quick Check:

   Message replay issue = RabbitMQ message removal [OK]

Hint: RabbitMQ deletes messages; no replay for multiple consumers [OK]

Common Mistakes:

• Assuming RabbitMQ supports message replay like Kafka
• Thinking RabbitMQ lacks multiple consumer support
• Confusing delivery guarantees with replay capability

5. You need to design a system that processes millions of events per second with multiple independent consumers, requires message retention for replay, and must handle complex routing. Which combination of messaging systems is the best architectural choice?

hard

A. Kafka for streaming and replay, RabbitMQ for routing

B. SQS for streaming, RabbitMQ for routing

C. RabbitMQ for streaming and replay, Kafka for routing

D. SQS for routing, Kafka for message retention

Solution

1. Step 1: Match requirements to system strengths

   Kafka excels at high-throughput streaming with message retention and replay for multiple consumers.

2. Step 2: Address complex routing needs

   RabbitMQ provides advanced routing capabilities that Kafka lacks.

3. Step 3: Combine systems for best fit

   Use Kafka for event streaming and replay, RabbitMQ for routing messages to appropriate consumers.

4. Final Answer:

   Kafka for streaming and replay, RabbitMQ for routing -> Option A

5. Quick Check:

   Streaming + replay + routing = Kafka + RabbitMQ [OK]

Hint: Use Kafka for streaming, RabbitMQ for routing [OK]

Common Mistakes:

• Using SQS for streaming or replay which it doesn't support
• Assuming RabbitMQ can handle large-scale streaming with replay
• Mixing roles of Kafka and RabbitMQ incorrectly