Fair dispatch with prefetch in RabbitMQ - Step-by-Step Execution

Process Flow - Fair dispatch with prefetch

Consumer connects to queue

↓

Set prefetch count = N

↓

Receive up to N messages

↓

Process message

↓

Send ack to server

↓

Server sends next message

↩Back to Receive

The consumer sets a limit (prefetch count) on how many messages it can handle at once. The server sends only that many messages before waiting for acknowledgments, ensuring fair load distribution.

Execution Sample

RabbitMQ

channel.basic_qos(prefetch_count=1)
channel.basic_consume(queue='task_queue', on_message_callback=callback)
channel.start_consuming()

This code sets the prefetch count to 1, so the consumer gets one message at a time and acknowledges it before receiving the next.

Process Table

Step	Prefetch Count	Messages Sent by Server	Messages Processed by Consumer	Acknowledgments Sent	Server Waits?
1	1	1	0	0	No
2	1	1	1	1	No
3	1	1	1	1	Yes
4	1	2	2	2	No
5	1	2	2	2	Yes
6	1	3	3	3	No
7	1	3	3	3	Yes
8	1	4	4	4	No
9	1	4	4	4	Yes
10	1	5	5	5	No
Exit	1	5	5	5	Yes - no more messages

💡 All messages processed and acknowledged; server has no more messages to send.

Status Tracker

Variable	Start	After 1	After 2	After 3	After 4	After 5	Final
Messages Sent by Server	0	1	1	2	2	3	5
Messages Processed by Consumer	0	0	1	1	2	3	5
Acknowledgments Sent	0	0	1	1	2	3	5

Key Moments - 3 Insights

Why does the server send only one message at a time even if more are waiting?

Because the prefetch count is set to 1, the server waits for an acknowledgment before sending the next message, ensuring fair dispatch as shown in steps 1 and 2 of the execution_table.

What happens if the consumer does not acknowledge a message?

The server will not send more messages beyond the prefetch count, causing it to wait indefinitely as the acknowledgment is required to proceed, demonstrated by the 'Server Waits?' column.

Why is fair dispatch important in message processing?

It prevents one consumer from being overloaded by limiting unacknowledged messages, ensuring all consumers get a fair share of work, as seen by the controlled message flow in the execution_table.

Visual Quiz - 3 Questions

Look at the execution_table at step 4, how many messages has the consumer processed?

A 2

B 1

C 3

D 0

💡 Hint

Check the 'Messages Processed by Consumer' column at step 4 in the execution_table.

At which step does the server stop sending messages because all are processed?

A Step 10

B Exit

C Step 8

D Step 5

💡 Hint

Look at the 'Messages Sent by Server' column and the exit_note in the execution_table.

If prefetch_count was set to 2, how would the 'Messages Sent by Server' change at step 1?

A It would be 1

B It would be 0

C It would be 2

D It would be 3

💡 Hint

Prefetch count controls how many messages the server sends before waiting for ack; see concept_flow.

Concept Snapshot

Fair dispatch with prefetch:
- Set prefetch_count to limit unacknowledged messages.
- Server sends only up to prefetch_count messages.
- Consumer processes and acknowledges each message.
- Server waits for ack before sending more.
- Ensures balanced load among consumers.

Full Transcript

Fair dispatch with prefetch in RabbitMQ means the consumer tells the server how many messages it can handle at once by setting prefetch_count. The server then sends only that many messages before waiting for acknowledgments. This prevents one consumer from getting overloaded and ensures messages are fairly distributed. The example code sets prefetch_count to 1, so the consumer gets one message at a time. The execution table shows the server sending one message, the consumer processing and acknowledging it, then the server sending the next. This cycle repeats until all messages are processed. If the consumer does not acknowledge, the server stops sending new messages. This method balances work fairly among consumers.

Practice

1. What is the main purpose of setting prefetch_count=1 in RabbitMQ fair dispatch?

easy

A. To ensure each worker receives only one task at a time

B. To allow workers to receive unlimited tasks simultaneously

C. To disable message acknowledgments

D. To increase the queue size automatically

Solution

1. Step 1: Understand prefetch_count meaning

   Prefetch_count controls how many messages a worker can receive before acknowledging previous ones.

2. Step 2: Effect of setting prefetch_count=1

   Setting it to 1 means the worker gets one task at a time, preventing overload and balancing workload.

3. Final Answer:

   To ensure each worker receives only one task at a time -> Option A

4. Quick Check:

   prefetch_count=1 means one task per worker [OK]

Hint: Prefetch 1 means one task per worker at a time [OK]

Common Mistakes:

• Thinking prefetch_count controls queue size
• Believing it disables acknowledgments
• Assuming workers get unlimited tasks

2. Which of the following is the correct way to set prefetch count to 1 using RabbitMQ's channel in Python?

easy

A. channel.set_prefetch(1)

B. channel.qos(prefetch=1)

C. channel.prefetch(1)

D. channel.basic_qos(prefetch_count=1)

Solution

1. Step 1: Recall RabbitMQ Python API method

   The method to set prefetch count is basic_qos with prefetch_count argument.

2. Step 2: Match correct syntax

   Only channel.basic_qos(prefetch_count=1) matches the official API syntax.

3. Final Answer:

   channel.basic_qos(prefetch_count=1) -> Option D

4. Quick Check:

   basic_qos with prefetch_count sets prefetch [OK]

Hint: Use basic_qos(prefetch_count=1) to set prefetch in Python [OK]

Common Mistakes:

• Using incorrect method names like set_prefetch
• Passing wrong argument names
• Confusing prefetch with qos

3. Given the following Python code snippet using RabbitMQ, what will be the output behavior when multiple workers consume from the same queue with prefetch_count=1?

channel.basic_qos(prefetch_count=1)
channel.basic_consume(queue='task_queue', on_message_callback=callback, auto_ack=False)
channel.start_consuming()

medium

A. Messages are processed only by the first worker

B. All workers receive all messages simultaneously

C. Each worker processes one message at a time, distributing tasks evenly

D. Workers process messages without waiting for acknowledgments

Solution

1. Step 1: Understand prefetch_count=1 effect

   With prefetch_count=1, each worker gets one message and must ack before next.

2. Step 2: Effect on multiple workers

   This causes RabbitMQ to distribute messages fairly, so tasks are balanced among workers.

3. Final Answer:

   Each worker processes one message at a time, distributing tasks evenly -> Option C

4. Quick Check:

   prefetch_count=1 balances tasks fairly [OK]

Hint: Prefetch 1 means tasks spread evenly, one per worker [OK]

Common Mistakes:

• Thinking all workers get all messages
• Assuming one worker gets all tasks
• Ignoring manual acknowledgments

4. You set prefetch_count=1 but notice one worker is still overloaded while others are idle. What is the most likely cause?

medium

A. The worker is not sending message acknowledgments properly

B. Prefetch count is set too high

C. The queue is empty

D. Multiple workers have the same consumer tag

Solution

1. Step 1: Analyze prefetch_count=1 behavior

   Prefetch_count=1 limits unacknowledged messages to one per worker.

2. Step 2: Identify cause of overload

   If a worker does not ack messages, RabbitMQ won't send new ones, causing that worker to hold tasks and appear overloaded.

3. Final Answer:

   The worker is not sending message acknowledgments properly -> Option A

4. Quick Check:

   Missing ack blocks new messages [OK]

Hint: Check if worker sends acknowledgments to avoid overload [OK]

Common Mistakes:

• Assuming prefetch count is too high
• Ignoring message acknowledgments
• Confusing consumer tags with load balancing

5. You want to optimize a RabbitMQ system where some workers are slower. How should you configure prefetch to ensure fair dispatch and prevent slow workers from blocking others?

hard

A. Set prefetch_count to a high number to maximize throughput

B. Set prefetch_count=1 so each worker gets one task at a time

C. Disable acknowledgments to speed up processing

D. Use multiple queues instead of prefetch settings

Solution

1. Step 1: Understand slow worker impact

   Slow workers holding many tasks block others if prefetch is high.

2. Step 2: Apply prefetch_count=1 for fair dispatch

   Setting prefetch_count=1 ensures each worker processes one task at a time, preventing blocking.

3. Step 3: Evaluate other options

   High prefetch or disabling acks causes imbalance; multiple queues add complexity.

4. Final Answer:

   Set prefetch_count=1 so each worker gets one task at a time -> Option B

5. Quick Check:

   Prefetch 1 prevents slow worker blocking [OK]

Hint: Use prefetch_count=1 to balance slow and fast workers [OK]

Common Mistakes:

• Setting prefetch too high causing imbalance
• Disabling acknowledgments causing message loss
• Ignoring prefetch and using multiple queues unnecessarily