Network partitions and split-brain in RabbitMQ - Step-by-Step Execution

Process Flow - Network partitions and split-brain

Network Partition Occurs

↓

Cluster Nodes Split

↓

Partition A

↓

Both Think They Are Master

↓

Split-Brain

↓

Data Diverges / Conflicts

↓

Manual or Automatic Resolution

This flow shows how a network partition splits a cluster into parts, causing each to think it is the master, leading to split-brain and data conflicts until resolved.

Execution Sample

RabbitMQ

rabbitmqctl cluster_status
# Simulate network partition
# Observe node states
# Resolve partition
rabbitmqctl cluster_status

This sequence shows checking cluster status, simulating a network partition causing split-brain, then resolving it and checking status again.

Process Table

Step	Action	Node A State	Node B State	Cluster Status	Notes
1	Initial cluster check	Running, connected	Running, connected	Healthy cluster	All nodes connected and synced
2	Network partition occurs	Running, isolated	Running, isolated	Partitioned cluster	Nodes cannot communicate
3	Each node assumes master	Master	Master	Split-brain detected	Both nodes think they control cluster
4	Data changes on Node A	Master with new data	Master old data	Diverged data	Data conflict begins
5	Data changes on Node B	Master with new data	Master with different new data	Data conflict worsens	Split-brain causes inconsistency
6	Manual resolution starts	Paused	Paused	Cluster paused	Prevent further changes
7	Partition fixed, nodes reconnect	Running, synced	Running, synced	Cluster healthy	Data merged or reconciled
8	Final cluster check	Running, connected	Running, connected	Healthy cluster	Split-brain resolved

💡 Partition fixed and cluster reconciled, nodes connected and data consistent

Status Tracker

Variable	Start	After Step 2	After Step 3	After Step 5	After Step 7	Final
Node A State	Running, connected	Running, isolated	Master	Master with new data	Running, synced	Running, connected
Node B State	Running, connected	Running, isolated	Master	Master with different new data	Running, synced	Running, connected
Cluster Status	Healthy cluster	Partitioned cluster	Split-brain detected	Data conflict worsens	Cluster healthy	Healthy cluster

Key Moments - 3 Insights

Why do both nodes think they are the master during a network partition?

Because the network partition isolates nodes, each loses communication with the other and assumes control to keep operating, as shown in step 3 of the execution table.

What causes data conflicts in a split-brain scenario?

When both nodes accept changes independently during partition (steps 4 and 5), their data diverges causing conflicts.

How is split-brain resolved in RabbitMQ clusters?

By pausing nodes, fixing the network, reconnecting nodes, and reconciling data as shown in steps 6 and 7.

Visual Quiz - 3 Questions

Look at the execution table, at which step do both nodes start thinking they are masters?

A Step 2

B Step 4

C Step 3

D Step 5

💡 Hint

Check the 'Cluster Status' and 'Node State' columns in step 3 where split-brain is detected.

According to the variable tracker, what is the state of Node B after step 5?

A Master with different new data

B Running, connected

C Running, isolated

D Paused

💡 Hint

Look at the 'Node B State' row under 'After Step 5' in the variable tracker.

If the network partition is not fixed, what will happen to the cluster status?

A Cluster becomes healthy

B Split-brain persists

C Cluster pauses automatically

D Nodes shut down

💡 Hint

Refer to the execution table rows 2 to 5 where partition causes split-brain and data conflicts.

Concept Snapshot

Network partitions split cluster nodes into isolated groups.
Each group may assume master role causing split-brain.
Split-brain leads to data conflicts and inconsistency.
Resolution requires fixing network and reconciling data.
RabbitMQ admins must monitor and resolve partitions quickly.

Full Transcript

Network partitions happen when cluster nodes lose communication. This splits the cluster into isolated parts. Each part thinks it is the master, causing split-brain. Both nodes accept changes independently, leading to data conflicts. To fix this, administrators pause nodes, restore network connectivity, and reconcile data. After resolution, the cluster returns to a healthy state with consistent data.

Practice

1. What is a network partition in a RabbitMQ cluster?

easy

A. A message queue overflow

B. A single node failure in the cluster

C. A configuration error in RabbitMQ settings

D. A split in the cluster where nodes cannot communicate with each other

Solution

1. Step 1: Understand network partition meaning

   A network partition occurs when parts of a RabbitMQ cluster lose communication with each other.

2. Step 2: Identify the effect on cluster nodes

   Nodes separated by the partition cannot exchange messages or cluster state, causing split behavior.

3. Final Answer:

   A split in the cluster where nodes cannot communicate with each other -> Option D

4. Quick Check:

   Network partition = nodes disconnected [OK]

Hint: Network partition means cluster nodes lose communication [OK]

Common Mistakes:

• Confusing partition with single node failure
• Thinking partition is a config error
• Assuming partition means message overflow

2. Which RabbitMQ command is used to set how the cluster handles network partitions?

easy

A. rabbitmqctl set_cluster_partition_handling

B. rabbitmqctl set_partition_mode

C. rabbitmqctl configure_network_partition

D. rabbitmqctl cluster_partition_policy

Solution

1. Step 1: Recall RabbitMQ partition handling command

   The correct command to control partition handling is set_cluster_partition_handling.

2. Step 2: Verify command syntax

   Other options are invalid or do not exist in RabbitMQ CLI.

3. Final Answer:

   rabbitmqctl set_cluster_partition_handling -> Option A

4. Quick Check:

   Partition handling command = set_cluster_partition_handling [OK]

Hint: Remember exact command: set_cluster_partition_handling [OK]

Common Mistakes:

• Using incorrect command names
• Confusing with cluster or network config commands
• Misspelling the command

3. Given a RabbitMQ cluster split into two partitions, what happens if the partition handling is set to ignore?

medium

A. Both partitions continue operating independently, risking split-brain

B. The cluster automatically heals and merges partitions

C. The cluster shuts down all nodes until partition resolves

D. Messages are dropped until the partition is fixed

Solution

1. Step 1: Understand 'ignore' partition handling

   When set to 'ignore', RabbitMQ nodes do not take action on partitions and continue working independently.

2. Step 2: Identify consequence of ignoring partitions

   This causes split-brain where multiple cluster parts act as main cluster, risking data conflicts.

3. Final Answer:

   Both partitions continue operating independently, risking split-brain -> Option A

4. Quick Check:

   Ignore mode = independent operation, split-brain risk [OK]

Hint: 'ignore' means no action, partitions run separately [OK]

Common Mistakes:

• Thinking cluster auto-heals with 'ignore'
• Assuming cluster shuts down nodes
• Believing messages are dropped automatically

4. You set RabbitMQ partition handling to pause_minority but the cluster does not pause any nodes during a partition. What is the likely cause?

medium

A. RabbitMQ does not support pause_minority mode

B. The command syntax was incorrect

C. The minority partition is not correctly identified

D. The cluster has only one node

Solution

1. Step 1: Understand pause_minority behavior

   In pause_minority mode, RabbitMQ pauses nodes in the smaller partition to avoid split-brain.

2. Step 2: Identify why no nodes paused

   If no nodes pause, likely the cluster did not detect which partition is minority, so no action taken.

3. Final Answer:

   The minority partition is not correctly identified -> Option C

4. Quick Check:

   Pause minority requires correct minority detection [OK]

Hint: Pause minority needs correct minority partition detection [OK]

Common Mistakes:

• Assuming syntax error without checking logs
• Believing pause_minority is unsupported
• Ignoring cluster size and partition count

5. In a 5-node RabbitMQ cluster, a network partition splits the cluster into two parts: one with 3 nodes and one with 2 nodes. If the partition handling is set to autoheal, what will happen after the partition resolves?

hard

A. Both partitions will continue independently causing split-brain

B. The smaller partition nodes will be stopped and the cluster will heal automatically

C. The cluster will require manual intervention to merge

D. All nodes will restart automatically regardless of partition size

Solution

1. Step 1: Understand autoheal partition handling

   Autoheal mode stops nodes in minority partitions and merges cluster state automatically after partition resolves.

2. Step 2: Apply to given cluster split

   With 3 nodes vs 2 nodes, the 2-node partition is minority and will be stopped to avoid split-brain.

3. Final Answer:

   The smaller partition nodes will be stopped and the cluster will heal automatically -> Option B

4. Quick Check:

   Autoheal stops minority, merges cluster [OK]

Hint: Autoheal stops minority partition nodes after split [OK]

Common Mistakes:

• Thinking autoheal allows split-brain
• Assuming manual merge is always needed
• Believing all nodes restart regardless