How to Use Prefetch Count in RabbitMQ for Efficient Message Processing

Use the prefetch_count setting in RabbitMQ to limit how many messages a consumer receives before acknowledging previous ones. This controls message flow and helps balance load by preventing consumers from being overwhelmed.

Syntax

The prefetch_count is set on the channel or consumer level using the basic_qos method. It controls how many messages can be sent to a consumer without receiving an acknowledgment.

• prefetch_count: Number of messages to prefetch.
• global: Boolean to apply setting to the entire channel (true) or just the consumer (false).

python

channel.basic_qos(prefetch_count=10, global=False)

Example

This example shows how to set prefetch_count in a Python RabbitMQ consumer using the pika library. It limits the consumer to process only 5 messages at a time before sending acknowledgments.

python

import pika

def callback(ch, method, properties, body):
    print(f"Received {body.decode()}")
    # Simulate work
    import time
    time.sleep(1)
    ch.basic_ack(delivery_tag=method.delivery_tag)

connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()

channel.queue_declare(queue='task_queue', durable=True)

# Set prefetch count to 5
channel.basic_qos(prefetch_count=5)

channel.basic_consume(queue='task_queue', on_message_callback=callback)

print('Waiting for messages. To exit press CTRL+C')
channel.start_consuming()

Output

Waiting for messages. To exit press CTRL+C
Received message1
Received message2
... (up to 5 messages processed concurrently)

Common Pitfalls

• Not setting prefetch_count: Consumers may get flooded with messages, causing slow processing or memory issues.
• Setting too high: Prefetching too many messages can overload the consumer and delay acknowledgments.
• Using global=True unintentionally: This applies the limit to the whole channel, affecting all consumers on it, which might not be desired.

python

channel.basic_qos(prefetch_count=0)  # Wrong: no limit, can flood consumer
channel.basic_qos(prefetch_count=10, global=True)  # Applies to all consumers on channel

# Correct usage:
channel.basic_qos(prefetch_count=5, global=False)  # Limits per consumer

Quick Reference

Remember these key points when using prefetch_count:

• Set prefetch_count to control how many messages a consumer can handle at once.
• Use global=False to apply limits per consumer, not the whole channel.
• Adjust the count based on your consumer's processing speed and resource limits.

Key Takeaways

Set prefetch_count to limit unacknowledged messages per consumer for better load control.

Use basic_qos(prefetch_count=N, global=False) to apply limits per consumer channel.

Avoid setting prefetch_count too high to prevent consumer overload.

Not setting prefetch_count can cause message flooding and slow processing.

Adjust prefetch_count based on your consumer's processing capacity.