Work queue for task distribution in RabbitMQ - Commands & Configuration

Introduction

When you have many tasks to do and want to share them fairly among workers, a work queue helps by holding tasks until workers are ready. This way, no worker is overloaded and tasks get done smoothly.

When you want to split a big job into smaller tasks and have multiple workers process them.

When you need to make sure tasks are not lost if a worker crashes.

When you want to balance the load so no single worker is overwhelmed.

When tasks come in faster than workers can handle and need to wait in line.

When you want to add or remove workers without stopping the system.

Config File - work_queue_producer.py

work_queue_producer.py

import pika

connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()

channel.queue_declare(queue='task_queue', durable=True)

message = ' '.join(['Hello'] * 5)
channel.basic_publish(
    exchange='',
    routing_key='task_queue',
    body=message,
    properties=pika.BasicProperties(
        delivery_mode=2  # make message persistent
    ))
print(" [x] Sent %r" % message)
connection.close()

This Python script connects to RabbitMQ on localhost and declares a durable queue named task_queue. It sends a persistent message to this queue. Durable queues and persistent messages ensure tasks are not lost if RabbitMQ restarts.

Commands

Run the producer script to send a task message to the work queue.

Terminal

python work_queue_producer.py

Expected Output

 [x] Sent 'Hello Hello Hello Hello Hello'

Check the list of queues on RabbitMQ server to confirm the queue exists and see message counts.

Terminal

rabbitmqctl list_queues

Expected Output

Listing queues ...

task_queue	1

...done.

Run the consumer script to receive and process tasks from the work queue.

Terminal

python work_queue_consumer.py

Expected Output

 [*] Waiting for messages. To exit press CTRL+C

 [x] Received Hello Hello Hello Hello Hello

 [x] Done

Key Concept

If you remember nothing else, remember: durable queues and persistent messages keep tasks safe and fairly distributed among workers.

Common Mistakes

Not declaring the queue as durable in both producer and consumer.

If the queue is not durable, tasks will be lost if RabbitMQ restarts.

Always declare the queue with durable=True in both producer and consumer scripts.

Not setting message delivery_mode to 2 (persistent) when publishing.

Messages will not be saved to disk and can be lost on server failure.

Set delivery_mode=2 in message properties to make messages persistent.

Not acknowledging messages in the consumer.

RabbitMQ will not know when a task is done and may resend or lose track of tasks.

Use manual message acknowledgments in the consumer to confirm task completion.

Summary

Declare a durable queue to hold tasks safely.

Send persistent messages so tasks survive server restarts.

Run consumers that receive tasks and acknowledge them after processing.