Exactly-once processing strategies in RabbitMQ - Step-by-Step Execution

Process Flow - Exactly-once processing strategies

Message Produced

↓

Message Sent to Queue

↓

Consumer Receives Message

↓

Process Message

↓

Acknowledge Message

↓

Commit Processing Result

↓

Message Removed from Queue

↓

Exactly-once Achieved

This flow shows how a message moves from producer to consumer with processing and acknowledgment to ensure it is handled exactly once.

Execution Sample

RabbitMQ

channel.basicConsume(queue, false, consumer);
// process message
channel.basicAck(deliveryTag, false);

Consumer receives a message, processes it, then sends an acknowledgment to RabbitMQ to confirm successful processing.

Process Table

Step	Action	Message State	Acknowledgment Sent	Queue State	Result
1	Producer sends message	In Queue	No	Message in queue	Message ready for consumption
2	Consumer receives message	Being Processed	No	Message reserved, not removed	Message locked for consumer
3	Consumer processes message	Being Processed	No	Message reserved	Processing in progress
4	Consumer sends ack	Processed	Yes	Message removed	Message removed from queue
5	Message processing committed	Processed	Yes	No message	Exactly-once processing ensured
6	Consumer crashes before ack	Being Processed	No	Message requeued	Message redelivered to another consumer
7	Consumer reprocesses message	Being Processed	No	Message reserved	Duplicate processing avoided by idempotency

💡 Message is removed from queue only after acknowledgment, ensuring exactly-once processing.

Status Tracker

Variable	Start	After Step 2	After Step 4	After Step 6	After Step 7
Message State	In Queue	Being Processed	Processed	Being Processed	Being Processed
Acknowledgment Sent	No	No	Yes	No	No
Queue State	Message in queue	Message reserved	Message removed	Message requeued	Message reserved

Key Moments - 3 Insights

Why doesn't RabbitMQ remove the message immediately when the consumer receives it?

Because the message is only removed after the consumer sends an acknowledgment (see Step 4 in execution_table). This prevents message loss if the consumer crashes during processing.

What happens if the consumer crashes before sending an acknowledgment?

The message is requeued and delivered again to another consumer (see Step 6). This ensures the message is not lost but may cause duplicate processing if not handled carefully.

How do we avoid processing the same message twice when it is redelivered?

By making the processing idempotent or using unique message IDs to detect duplicates (refer to Step 7). This is key to exactly-once processing.

Visual Quiz - 3 Questions

Look at the execution_table, at which step is the message removed from the queue?

A Step 4

B Step 2

C Step 6

D Step 1

💡 Hint

Check the 'Queue State' column for when the message changes from reserved to removed.

According to variable_tracker, what is the acknowledgment state after Step 6?

A Yes

B Unknown

C No

D Pending

💡 Hint

Look at the 'Acknowledgment Sent' row under 'After Step 6' column.

If the consumer never sends an acknowledgment, what happens to the message according to the execution_table?

A Message stays in queue forever

B Message is requeued and redelivered

C Message is lost

D Message is deleted automatically

💡 Hint

Refer to Step 6 where the consumer crashes before ack.

Concept Snapshot

Exactly-once processing in RabbitMQ means each message is processed one time only.
Producer sends message to queue.
Consumer receives and processes message but does NOT remove it immediately.
Consumer sends acknowledgment after successful processing.
Message is removed only after ack, preventing loss or duplicates.
Idempotent processing helps handle redelivered messages safely.

Full Transcript

Exactly-once processing strategies in RabbitMQ ensure that each message is processed a single time without loss or duplication. The producer sends a message to the queue. The consumer receives the message and processes it but the message remains in the queue until the consumer sends an acknowledgment. Only after the acknowledgment does RabbitMQ remove the message from the queue. If the consumer crashes before sending the acknowledgment, the message is requeued and delivered again to another consumer. To avoid duplicate processing, consumers should implement idempotent processing or use unique message IDs. This flow guarantees that messages are processed exactly once, even in failure scenarios.

Practice

1. What does "exactly-once processing" mean in RabbitMQ?

easy

A. Each message is processed only one time without duplicates

B. Messages are processed at least once, possibly multiple times

C. Messages are processed only if the queue is empty

D. Messages are processed without any acknowledgments

Solution

1. Step 1: Understand message processing guarantees

   Exactly-once means no message is lost or processed more than once.

2. Step 2: Compare options with definition

   Only Each message is processed only one time without duplicates matches the definition of exactly-once processing.

3. Final Answer:

   Each message is processed only one time without duplicates -> Option A

4. Quick Check:

   Exactly-once = single processing [OK]

Hint: Exactly-once means no repeats or losses in processing [OK]

Common Mistakes:

• Confusing exactly-once with at-least-once processing
• Ignoring the role of acknowledgments
• Thinking messages process only when queue is empty

2. Which RabbitMQ feature helps ensure messages are not lost during processing?

easy

A. Automatic message deletion

B. Disabling publisher confirms

C. Unlimited queue length

D. Manual message acknowledgments

Solution

1. Step 1: Identify message loss prevention methods

   Manual acknowledgments let consumers confirm message processing before removal.

2. Step 2: Evaluate options for loss prevention

   Only manual acknowledgments ensure messages aren't lost if consumer fails.

3. Final Answer:

   Manual message acknowledgments -> Option D

4. Quick Check:

   Manual ack = message safe until confirmed [OK]

Hint: Manual ack confirms message processed safely [OK]

Common Mistakes:

• Assuming automatic deletion prevents loss
• Thinking unlimited queue length affects message safety
• Disabling confirms helps reliability (it doesn't)

3. Given this consumer code snippet, what happens if the consumer crashes before ack?

channel.basic_consume(queue='task_queue', on_message_callback=callback)

def callback(ch, method, properties, body):
    process_message(body)
    # no ack sent here

medium

A. Message is lost and not requeued

B. Message remains unacknowledged and will be redelivered

C. Message is acknowledged automatically

D. Queue deletes the message immediately

Solution

1. Step 1: Understand manual ack behavior

   If no ack is sent, RabbitMQ keeps message unacknowledged.

2. Step 2: Effect of consumer crash before ack

   RabbitMQ will redeliver unacknowledged messages to another consumer.

3. Final Answer:

   Message remains unacknowledged and will be redelivered -> Option B

4. Quick Check:

   No ack = redelivery on crash [OK]

Hint: No ack means message redelivery on failure [OK]

Common Mistakes:

• Assuming message is lost without ack
• Thinking messages auto-acknowledge by default
• Believing queue deletes messages immediately

4. You notice duplicate message processing in your RabbitMQ consumer. Which fix helps ensure exactly-once processing?

medium

A. Implement idempotent message handling and manual acks

B. Disable publisher confirms

C. Use automatic acknowledgments

D. Increase queue prefetch count

Solution

1. Step 1: Identify cause of duplicates

   Duplicates often occur if messages are reprocessed after failure or redelivery.

2. Step 2: Apply exactly-once strategies

   Idempotent handling plus manual acknowledgments prevent duplicate side effects.

3. Final Answer:

   Implement idempotent message handling and manual acks -> Option A

4. Quick Check:

   Idempotent + manual ack = exactly-once [OK]

Hint: Idempotent code + manual ack stops duplicates [OK]

Common Mistakes:

• Using automatic ack causes duplicates
• Disabling confirms reduces reliability
• Increasing prefetch doesn't fix duplicates

5. To achieve exactly-once processing in RabbitMQ, which combination is best practice?

hard

A. Automatic acknowledgments with no idempotency

B. Using only transactions without manual acknowledgments

C. Manual acknowledgments, idempotent consumer code, and publisher confirms

D. Disabling transactions and using auto-delete queues

Solution

1. Step 1: Review exactly-once requirements

   Exactly-once needs safe message delivery, no duplicates, and confirmed processing.

2. Step 2: Match features to requirements

   Manual acks confirm processing, idempotent code avoids duplicates, publisher confirms ensure message sent.

3. Step 3: Eliminate incorrect options

   Automatic ack or disabling transactions risk message loss or duplicates.

4. Final Answer:

   Manual acknowledgments, idempotent consumer code, and publisher confirms -> Option C

5. Quick Check:

   Manual ack + idempotent + confirms = exactly-once [OK]

Hint: Combine manual ack, idempotency, and confirms for safety [OK]

Common Mistakes:

• Relying on automatic acknowledgments alone
• Ignoring idempotent consumer design
• Assuming transactions alone guarantee exactly-once