Fair dispatch with prefetch in RabbitMQ - Mini Project: Build & Apply

Fair dispatch with prefetch in RabbitMQ

📖 Scenario: You are setting up a RabbitMQ message queue system where multiple workers consume tasks. To avoid one worker getting overloaded with many messages while others are idle, you want to use fair dispatch with prefetch.This means each worker will only get one message at a time until it acknowledges the previous one, ensuring balanced work distribution.

🎯 Goal: Configure a RabbitMQ consumer to use fair dispatch by setting the prefetch count to 1, so that the server sends only one unacknowledged message at a time to the worker.

📋 What You'll Learn

Create a queue named task_queue with durability enabled

Set the prefetch count to 1 for fair dispatch

Consume messages from task_queue with manual acknowledgments

Print the received message body in the consumer

💡 Why This Matters

🌍 Real World

In real systems, fair dispatch prevents one worker from being overwhelmed while others are idle, improving resource use and responsiveness.

💼 Career

Understanding prefetch and manual acknowledgments is essential for roles involving message queue management, backend development, and system reliability.

1

Create a durable queue named task_queue

Write code to declare a queue named task_queue with durability enabled using channel.queue_declare.

RabbitMQ

import pika

connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()
# Declare a durable queue named 'task_queue'
# Your code here

Hint

Use channel.queue_declare with queue='task_queue' and durable=True.

2

Set the prefetch count to 1 for fair dispatch

Add code to set the prefetch count to 1 using channel.basic_qos to enable fair dispatch.

RabbitMQ

import pika

connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()
channel.queue_declare(queue='task_queue', durable=True)
# Set prefetch count to 1 for fair dispatch
# Your code here

Hint

Use channel.basic_qos(prefetch_count=1) to limit unacknowledged messages to one.

3

Consume messages with manual acknowledgments

Write a callback function named callback that takes ch, method, properties, and body as parameters. Inside it, print the message body decoded as UTF-8. Then acknowledge the message using ch.basic_ack with delivery_tag=method.delivery_tag. Finally, start consuming from task_queue with auto_ack=False and the callback function.

RabbitMQ

import pika

connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()
channel.queue_declare(queue='task_queue', durable=True)
channel.basic_qos(prefetch_count=1)
# Define callback to print message and acknowledge it
# Start consuming from 'task_queue' with auto_ack=False
# Your code here

Hint

Define callback to print the decoded message and acknowledge it. Use channel.basic_consume with auto_ack=False.

4

Start consuming and print output

Add code to start consuming messages using channel.start_consuming().

RabbitMQ

import pika

connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()
channel.queue_declare(queue='task_queue', durable=True)
channel.basic_qos(prefetch_count=1)

def callback(ch, method, properties, body):
    print(f"Received {body.decode()}")
    ch.basic_ack(delivery_tag=method.delivery_tag)

channel.basic_consume(queue='task_queue', on_message_callback=callback, auto_ack=False)
# Start consuming messages
# Your code here

Hint

Call channel.start_consuming() to begin receiving messages.

Practice

1. What is the main purpose of setting prefetch_count=1 in RabbitMQ fair dispatch?

easy

A. To ensure each worker receives only one task at a time

B. To allow workers to receive unlimited tasks simultaneously

C. To disable message acknowledgments

D. To increase the queue size automatically

Solution

1. Step 1: Understand prefetch_count meaning

   Prefetch_count controls how many messages a worker can receive before acknowledging previous ones.

2. Step 2: Effect of setting prefetch_count=1

   Setting it to 1 means the worker gets one task at a time, preventing overload and balancing workload.

3. Final Answer:

   To ensure each worker receives only one task at a time -> Option A

4. Quick Check:

   prefetch_count=1 means one task per worker [OK]

Hint: Prefetch 1 means one task per worker at a time [OK]

Common Mistakes:

• Thinking prefetch_count controls queue size
• Believing it disables acknowledgments
• Assuming workers get unlimited tasks

2. Which of the following is the correct way to set prefetch count to 1 using RabbitMQ's channel in Python?

easy

A. channel.set_prefetch(1)

B. channel.qos(prefetch=1)

C. channel.prefetch(1)

D. channel.basic_qos(prefetch_count=1)

Solution

1. Step 1: Recall RabbitMQ Python API method

   The method to set prefetch count is basic_qos with prefetch_count argument.

2. Step 2: Match correct syntax

   Only channel.basic_qos(prefetch_count=1) matches the official API syntax.

3. Final Answer:

   channel.basic_qos(prefetch_count=1) -> Option D

4. Quick Check:

   basic_qos with prefetch_count sets prefetch [OK]

Hint: Use basic_qos(prefetch_count=1) to set prefetch in Python [OK]

Common Mistakes:

• Using incorrect method names like set_prefetch
• Passing wrong argument names
• Confusing prefetch with qos

3. Given the following Python code snippet using RabbitMQ, what will be the output behavior when multiple workers consume from the same queue with prefetch_count=1?

channel.basic_qos(prefetch_count=1)
channel.basic_consume(queue='task_queue', on_message_callback=callback, auto_ack=False)
channel.start_consuming()

medium

A. Messages are processed only by the first worker

B. All workers receive all messages simultaneously

C. Each worker processes one message at a time, distributing tasks evenly

D. Workers process messages without waiting for acknowledgments

Solution

1. Step 1: Understand prefetch_count=1 effect

   With prefetch_count=1, each worker gets one message and must ack before next.

2. Step 2: Effect on multiple workers

   This causes RabbitMQ to distribute messages fairly, so tasks are balanced among workers.

3. Final Answer:

   Each worker processes one message at a time, distributing tasks evenly -> Option C

4. Quick Check:

   prefetch_count=1 balances tasks fairly [OK]

Hint: Prefetch 1 means tasks spread evenly, one per worker [OK]

Common Mistakes:

• Thinking all workers get all messages
• Assuming one worker gets all tasks
• Ignoring manual acknowledgments

4. You set prefetch_count=1 but notice one worker is still overloaded while others are idle. What is the most likely cause?

medium

A. The worker is not sending message acknowledgments properly

B. Prefetch count is set too high

C. The queue is empty

D. Multiple workers have the same consumer tag

Solution

1. Step 1: Analyze prefetch_count=1 behavior

   Prefetch_count=1 limits unacknowledged messages to one per worker.

2. Step 2: Identify cause of overload

   If a worker does not ack messages, RabbitMQ won't send new ones, causing that worker to hold tasks and appear overloaded.

3. Final Answer:

   The worker is not sending message acknowledgments properly -> Option A

4. Quick Check:

   Missing ack blocks new messages [OK]

Hint: Check if worker sends acknowledgments to avoid overload [OK]

Common Mistakes:

• Assuming prefetch count is too high
• Ignoring message acknowledgments
• Confusing consumer tags with load balancing

5. You want to optimize a RabbitMQ system where some workers are slower. How should you configure prefetch to ensure fair dispatch and prevent slow workers from blocking others?

hard

A. Set prefetch_count to a high number to maximize throughput

B. Set prefetch_count=1 so each worker gets one task at a time

C. Disable acknowledgments to speed up processing

D. Use multiple queues instead of prefetch settings

Solution

1. Step 1: Understand slow worker impact

   Slow workers holding many tasks block others if prefetch is high.

2. Step 2: Apply prefetch_count=1 for fair dispatch

   Setting prefetch_count=1 ensures each worker processes one task at a time, preventing blocking.

3. Step 3: Evaluate other options

   High prefetch or disabling acks causes imbalance; multiple queues add complexity.

4. Final Answer:

   Set prefetch_count=1 so each worker gets one task at a time -> Option B

5. Quick Check:

   Prefetch 1 prevents slow worker blocking [OK]

Hint: Use prefetch_count=1 to balance slow and fast workers [OK]

Common Mistakes:

• Setting prefetch too high causing imbalance
• Disabling acknowledgments causing message loss
• Ignoring prefetch and using multiple queues unnecessarily