Shovel and Federation for multi-DC in RabbitMQ - Step-by-Step Execution

Process Flow - Shovel and Federation for multi-DC

Start: Message in DC1 Queue

↓

Shovel or Federation picks message

↓

Connect to DC2 Broker

↓

Publish message to DC2 Queue

↓

Message available in DC2

↓

End

Messages flow from one data center's queue to another using Shovel or Federation, connecting brokers across data centers.

Execution Sample

RabbitMQ

rabbitmqctl set_parameter shovel myshovel '{"src-uri":"amqp://dc1","src-queue":"task_queue","dest-uri":"amqp://dc2","dest-queue":"task_queue"}'

This command sets up a Shovel to move messages from 'task_queue' in DC1 to 'task_queue' in DC2.

Process Table

Step	Action	Source Queue State	Shovel/Federation State	Destination Queue State	Result
1	Message arrives in DC1 'task_queue'	1 message	Idle	0 messages	Message queued in DC1
2	Shovel connects to DC1 and DC2 brokers	1 message	Connected	0 messages	Shovel ready to transfer
3	Shovel fetches message from DC1 queue	0 messages	Fetching message	0 messages	Message taken from DC1
4	Shovel publishes message to DC2 queue	0 messages	Publishing message	1 message	Message delivered to DC2
5	Shovel confirms delivery and idles	0 messages	Idle	1 message	Message ready for consumers in DC2
6	No more messages, Shovel waits	0 messages	Idle	1 message	Waiting for new messages

💡 Shovel waits idle when no messages remain in source queue

Status Tracker

Variable	Start	After Step 1	After Step 3	After Step 4	Final
DC1 Queue Messages	0	1	0	0	0
Shovel State	Idle	Idle	Fetching message	Publishing message	Idle
DC2 Queue Messages	0	0	0	1	1

Key Moments - 3 Insights

Why does the DC1 queue message count drop to zero before the message appears in DC2?

Because the Shovel removes the message from DC1 queue first (Step 3) before publishing it to DC2 (Step 4), ensuring no duplicates.

What happens if the connection to DC2 broker fails during publishing?

The Shovel will retry publishing and keep the message until successful, preventing message loss (not shown in table but important).

How is Federation different from Shovel in this flow?

Federation links exchanges or queues to forward messages continuously, while Shovel moves messages one by one actively; both connect DCs but differ in setup and behavior.

Visual Quiz - 3 Questions

Look at the execution table, at which step does the DC1 queue become empty?

A Step 4

B Step 2

C Step 3

D Step 5

💡 Hint

Check the 'Source Queue State' column in the execution table rows

According to the variable tracker, what is the Shovel state after Step 4?

A Publishing message

B Idle

C Fetching message

D Disconnected

💡 Hint

Look at the 'Shovel State' row and the 'After Step 4' column in variable_tracker

If a new message arrives in DC1 after Step 6, what will the Shovel state likely be?

A Idle

B Fetching message

C Disconnected

D Publishing message

💡 Hint

Refer to the flow where Shovel fetches messages when available (see execution_table Step 3)

Concept Snapshot

Shovel and Federation connect RabbitMQ brokers across data centers.
Shovel actively moves messages from source queue to destination queue.
Federation links exchanges or queues to forward messages continuously.
Use Shovel for reliable, controlled message transfer.
Use Federation for continuous message sharing.
Both enable multi-DC message replication.

Full Transcript

This visual execution shows how RabbitMQ Shovel moves messages between two data centers (DC1 and DC2). First, a message arrives in DC1's queue. The Shovel connects to both DC1 and DC2 brokers, then fetches the message from DC1 queue, removing it there. Next, it publishes the message to DC2 queue. After confirming delivery, the Shovel idles waiting for new messages. Variables like queue message counts and Shovel state change step-by-step. Key points include that messages are removed from source before publishing to destination, and Shovel retries on failures. Federation differs by linking queues or exchanges for continuous forwarding. The quiz checks understanding of message flow and Shovel states. This helps beginners see how multi-DC message replication works in RabbitMQ.

Practice

1. What is the main purpose of RabbitMQ Shovel in a multi-data center (multi-DC) setup?

easy

A. To encrypt messages within a single RabbitMQ server

B. To share user credentials across multiple RabbitMQ servers

C. To automatically scale RabbitMQ nodes in a cluster

D. To copy messages between queues on different RabbitMQ servers

Solution

1. Step 1: Understand Shovel's role

   Shovel is designed to move or copy messages from one queue to another, even across different RabbitMQ servers.

2. Step 2: Compare options

   Options B, C, and D describe unrelated features like credential sharing, scaling, or encryption, which are not Shovel's purpose.

3. Final Answer:

   To copy messages between queues on different RabbitMQ servers -> Option D

4. Quick Check:

   Shovel copies messages = A [OK]

Hint: Shovel moves messages between queues on different servers [OK]

Common Mistakes:

• Confusing Shovel with Federation
• Thinking Shovel manages user credentials
• Assuming Shovel handles encryption

2. Which of the following is the correct minimal configuration snippet to define a Shovel in RabbitMQ's configuration file?

easy

A. listeners:
amqp:
port: 5672

B. shovels:
my_shovel:
source-uri: amqp://
source-queue: source_queue
destination-uri: amqp://remote
destination-queue: dest_queue

C. clusters:
nodes: [rabbit@node1, rabbit@node2]

D. federation:
upstreams:
my_upstream:
uri: amqp://remote
exchanges: ["my_exchange"]

Solution

1. Step 1: Identify Shovel configuration format

   Shovel config uses 'shovels' key with shovel name and source/destination URIs and queues.

2. Step 2: Match options to Shovel syntax

   shovels: my_shovel: source-uri: amqp:// source-queue: source_queue destination-uri: amqp://remote destination-queue: dest_queue matches the correct syntax for defining a shovel. Options A, B, and C relate to listeners, federation, and clustering, not shovel.

3. Final Answer:

   shovels: my_shovel: source-uri: amqp:// source-queue: source_queue destination-uri: amqp://remote destination-queue: dest_queue -> Option B

4. Quick Check:

   Shovel config starts with 'shovels:' = D [OK]

Hint: Shovel config uses 'shovels:' with source and destination URIs [OK]

Common Mistakes:

• Confusing federation config with shovel config
• Using cluster or listener config for shovel
• Missing source or destination queue keys

3. Given a federation upstream configured to share an exchange named logs from a remote RabbitMQ server, what will happen if a message is published to the local logs exchange?

medium

A. The message will stay local and not be sent to the remote server

B. The message will be forwarded to the remote exchange automatically

C. The message will be received from the remote exchange and appear locally

D. The message will be duplicated on both local and remote exchanges

Solution

1. Step 1: Understand Federation direction

   Federation pulls messages from remote to local, not the other way around.

2. Step 2: Analyze message flow

   Publishing locally to the exchange does not send messages upstream; messages flow from remote to local only.

3. Final Answer:

   The message will stay local and not be sent to the remote server -> Option A

4. Quick Check:

   Federation pulls from remote; local publish stays local = B [OK]

Hint: Federation pulls messages from remote to local, not vice versa [OK]

Common Mistakes:

• Thinking federation pushes messages upstream
• Assuming messages duplicate automatically
• Confusing federation with shovel direction

4. You configured a Shovel to move messages from a local queue to a remote queue, but no messages appear on the remote side. Which is the most likely cause?

medium

A. The Shovel's destination URI is incorrect or unreachable

B. The local queue is empty, so no messages to move

C. The Federation plugin is not enabled on the local server

D. The RabbitMQ server is running in standalone mode

Solution

1. Step 1: Check Shovel connectivity

   If the destination URI is wrong or unreachable, messages cannot be delivered remotely.

2. Step 2: Eliminate other causes

   Local queue being empty means no messages to move, but question implies messages exist. Federation plugin is unrelated to Shovel. Standalone mode does not prevent Shovel operation.

3. Final Answer:

   The Shovel's destination URI is incorrect or unreachable -> Option A

4. Quick Check:

   Shovel needs correct destination URI = A [OK]

Hint: Check destination URI if shovel moves no messages [OK]

Common Mistakes:

• Confusing Federation plugin with Shovel
• Ignoring network or URI errors
• Assuming local queue always has messages

5. In a multi-DC RabbitMQ setup, you want to ensure messages published in any data center are available in all others with minimal delay and no duplicates. Which combination is best?

hard

A. Use only Shovel to copy messages between queues in data centers

B. Use only Federation to share queues between data centers

C. Use Federation to share exchanges and Shovel to copy queues between data centers

D. Use clustering across data centers with mirrored queues

Solution

1. Step 1: Understand Federation and Shovel roles

   Federation shares exchanges or queues, enabling message sharing with minimal delay. Shovel copies messages between queues reliably.

2. Step 2: Combine for best multi-DC setup

   Using Federation for exchange sharing ensures messages are available across DCs. Shovel ensures queues are synchronized, preventing duplicates and delays.

3. Final Answer:

   Use Federation to share exchanges and Shovel to copy queues between data centers -> Option C

4. Quick Check:

   Federation + Shovel = best multi-DC sync [OK]

Hint: Combine Federation for sharing and Shovel for copying queues [OK]

Common Mistakes:

• Relying on clustering across data centers (not recommended)
• Using only one method causing delays or duplicates
• Confusing queue sharing with exchange sharing