Fair dispatch with prefetch in RabbitMQ - Commands & Configuration

Introduction

When multiple workers consume messages from a queue, some may get overloaded while others stay idle. Fair dispatch with prefetch helps balance the load by limiting how many messages a worker can receive before acknowledging them.

When you have several workers processing tasks from the same queue and want to avoid one worker getting too many tasks at once.

When tasks vary in processing time and you want to prevent slow workers from blocking others.

When you want to improve resource usage by distributing work evenly across workers.

When you want to ensure that a worker only gets a new message after finishing the previous one.

When you want to avoid message loss by ensuring messages are acknowledged before sending more.

Commands

Check existing queues to confirm the queue you want to use exists.

Terminal

rabbitmqctl list_queues

Expected Output

Listing queues ...

my-task-queue	0

Create a durable queue named 'my-task-queue' to hold tasks persistently.

Terminal

rabbitmqadmin declare queue name=my-task-queue durable=true

Expected Output

Queue 'my-task-queue' declared

Send a task message 'Task 1' to the queue for workers to process.

Terminal

rabbitmqadmin publish routing_key=my-task-queue payload='Task 1'

Expected Output

Published message to queue 'my-task-queue'

Set prefetch count to 1 so each worker receives only one unacknowledged message at a time, enabling fair dispatch.

Terminal

rabbitmqctl set_prefetch_count 1

Expected Output

No output (command runs silently)

→

set_prefetch_count - Limits the number of unacknowledged messages per consumer

Verify that consumers are connected and prefetch settings are applied.

Terminal

rabbitmqctl list_consumers

Expected Output

Listing consumers ...

consumer_tag	my-task-queue	channel_details

Key Concept

If you remember nothing else from this pattern, remember: setting prefetch to 1 ensures each worker gets one message at a time, balancing the load fairly.

Common Mistakes

Not setting prefetch count, so workers get many messages at once.

This causes some workers to be overloaded while others are idle, leading to inefficient processing.

Always set prefetch count to 1 for fair dispatch when tasks vary in processing time.

Not acknowledging messages after processing.

Unacknowledged messages stay assigned to a worker, blocking new messages and causing delays.

Make sure each worker sends an acknowledgment after finishing a task.

Summary

Create a durable queue to hold tasks persistently.

Publish messages to the queue for workers to process.

Set prefetch count to 1 to ensure fair dispatch of messages.

Verify consumers and prefetch settings to confirm balanced load.

Practice

1. What is the main purpose of setting prefetch_count=1 in RabbitMQ fair dispatch?

easy

A. To ensure each worker receives only one task at a time

B. To allow workers to receive unlimited tasks simultaneously

C. To disable message acknowledgments

D. To increase the queue size automatically

Solution

1. Step 1: Understand prefetch_count meaning

   Prefetch_count controls how many messages a worker can receive before acknowledging previous ones.

2. Step 2: Effect of setting prefetch_count=1

   Setting it to 1 means the worker gets one task at a time, preventing overload and balancing workload.

3. Final Answer:

   To ensure each worker receives only one task at a time -> Option A

4. Quick Check:

   prefetch_count=1 means one task per worker [OK]

Hint: Prefetch 1 means one task per worker at a time [OK]

Common Mistakes:

• Thinking prefetch_count controls queue size
• Believing it disables acknowledgments
• Assuming workers get unlimited tasks

2. Which of the following is the correct way to set prefetch count to 1 using RabbitMQ's channel in Python?

easy

A. channel.set_prefetch(1)

B. channel.qos(prefetch=1)

C. channel.prefetch(1)

D. channel.basic_qos(prefetch_count=1)

Solution

1. Step 1: Recall RabbitMQ Python API method

   The method to set prefetch count is basic_qos with prefetch_count argument.

2. Step 2: Match correct syntax

   Only channel.basic_qos(prefetch_count=1) matches the official API syntax.

3. Final Answer:

   channel.basic_qos(prefetch_count=1) -> Option D

4. Quick Check:

   basic_qos with prefetch_count sets prefetch [OK]

Hint: Use basic_qos(prefetch_count=1) to set prefetch in Python [OK]

Common Mistakes:

• Using incorrect method names like set_prefetch
• Passing wrong argument names
• Confusing prefetch with qos

3. Given the following Python code snippet using RabbitMQ, what will be the output behavior when multiple workers consume from the same queue with prefetch_count=1?

channel.basic_qos(prefetch_count=1)
channel.basic_consume(queue='task_queue', on_message_callback=callback, auto_ack=False)
channel.start_consuming()

medium

A. Messages are processed only by the first worker

B. All workers receive all messages simultaneously

C. Each worker processes one message at a time, distributing tasks evenly

D. Workers process messages without waiting for acknowledgments

Solution

1. Step 1: Understand prefetch_count=1 effect

   With prefetch_count=1, each worker gets one message and must ack before next.

2. Step 2: Effect on multiple workers

   This causes RabbitMQ to distribute messages fairly, so tasks are balanced among workers.

3. Final Answer:

   Each worker processes one message at a time, distributing tasks evenly -> Option C

4. Quick Check:

   prefetch_count=1 balances tasks fairly [OK]

Hint: Prefetch 1 means tasks spread evenly, one per worker [OK]

Common Mistakes:

• Thinking all workers get all messages
• Assuming one worker gets all tasks
• Ignoring manual acknowledgments

4. You set prefetch_count=1 but notice one worker is still overloaded while others are idle. What is the most likely cause?

medium

A. The worker is not sending message acknowledgments properly

B. Prefetch count is set too high

C. The queue is empty

D. Multiple workers have the same consumer tag

Solution

1. Step 1: Analyze prefetch_count=1 behavior

   Prefetch_count=1 limits unacknowledged messages to one per worker.

2. Step 2: Identify cause of overload

   If a worker does not ack messages, RabbitMQ won't send new ones, causing that worker to hold tasks and appear overloaded.

3. Final Answer:

   The worker is not sending message acknowledgments properly -> Option A

4. Quick Check:

   Missing ack blocks new messages [OK]

Hint: Check if worker sends acknowledgments to avoid overload [OK]

Common Mistakes:

• Assuming prefetch count is too high
• Ignoring message acknowledgments
• Confusing consumer tags with load balancing

5. You want to optimize a RabbitMQ system where some workers are slower. How should you configure prefetch to ensure fair dispatch and prevent slow workers from blocking others?

hard

A. Set prefetch_count to a high number to maximize throughput

B. Set prefetch_count=1 so each worker gets one task at a time

C. Disable acknowledgments to speed up processing

D. Use multiple queues instead of prefetch settings

Solution

1. Step 1: Understand slow worker impact

   Slow workers holding many tasks block others if prefetch is high.

2. Step 2: Apply prefetch_count=1 for fair dispatch

   Setting prefetch_count=1 ensures each worker processes one task at a time, preventing blocking.

3. Step 3: Evaluate other options

   High prefetch or disabling acks causes imbalance; multiple queues add complexity.

4. Final Answer:

   Set prefetch_count=1 so each worker gets one task at a time -> Option B

5. Quick Check:

   Prefetch 1 prevents slow worker blocking [OK]

Hint: Use prefetch_count=1 to balance slow and fast workers [OK]

Common Mistakes:

• Setting prefetch too high causing imbalance
• Disabling acknowledgments causing message loss
• Ignoring prefetch and using multiple queues unnecessarily