Consumer prefetch optimization in RabbitMQ - Commands & Configuration

Introduction

When multiple messages are sent to a consumer, it can get overwhelmed if it tries to process all at once. Consumer prefetch optimization controls how many messages a consumer receives before acknowledging them, helping balance load and improve performance.

When a consumer is slow and you want to prevent it from being flooded with too many messages at once

When you want to improve message processing efficiency by controlling how many messages are handled concurrently

When you want to avoid message re-delivery caused by unacknowledged messages piling up

When running multiple consumers and you want to distribute messages evenly among them

When you want to reduce memory usage on the consumer by limiting unprocessed messages

Commands

This command sets a policy named 'my_policy' that applies a prefetch count of 10 messages to all consumers, limiting how many messages each consumer can receive before acknowledging.

Terminal

rabbitmqctl set_policy my_policy "^" '{"prefetch-count":10}' --apply-to consumers

Expected Output

Setting policy "my_policy" for pattern "^" to "{\"prefetch-count\":10}" for consumers ...

→

--apply-to consumers - Specifies that the policy applies only to consumers

This command lists all queues with the number of messages ready to be delivered and messages that are unacknowledged, helping verify the effect of prefetch settings.

Terminal

rabbitmqctl list_queues name messages_ready messages_unacknowledged

Expected Output

name	messages_ready	messages_unacknowledged

my-queue	5	3

another-queue	0	1

This command removes the previously set prefetch policy, returning consumers to default behavior.

Terminal

rabbitmqctl clear_policy my_policy

Expected Output

Clearing policy "my_policy" ...

Key Concept

If you remember nothing else from this pattern, remember: setting a prefetch count limits how many messages a consumer can handle at once, preventing overload and improving processing flow.

Common Mistakes

Setting prefetch count too high or too low without testing

Too high causes consumer overload; too low reduces throughput and wastes resources

Start with a moderate prefetch count like 10 and adjust based on consumer speed and workload

Not applying the policy to the correct target (e.g., applying to queues instead of consumers)

Prefetch count only affects consumers, so applying to queues has no effect

Always use --apply-to consumers when setting prefetch policies

Forgetting to monitor queue message states after changing prefetch

Without monitoring, you can't tell if the change improved or worsened message flow

Use rabbitmqctl list_queues to check messages_ready and messages_unacknowledged after changes

Summary

Set a prefetch count policy to limit how many messages a consumer receives before acknowledging.

Check queue message states to verify the impact of prefetch settings.

Remove or adjust the policy as needed to optimize consumer performance.