Network partitions and split-brain in RabbitMQ - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to enable automatic network partition handling in RabbitMQ configuration.

RabbitMQ

cluster_partition_handling = [1]

Drag options to blanks, or click blank then click option'

Aautoheal

Bstop

Cpause_minority

Dignore

Attempts:

3 left

💡 Hint

Common Mistakes

Choosing 'ignore' which causes split-brain issues.

Choosing 'stop' which halts nodes instead of healing.

The autoheal setting allows RabbitMQ to automatically recover from network partitions by healing the cluster.

2fill in blank

medium

Complete the command to check the current cluster status in RabbitMQ.

RabbitMQ

rabbitmqctl [1]

Drag options to blanks, or click blank then click option'

Alist_nodes

Bcluster_status

Cstatus

Dlist_queues

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'status' which only shows local node info.

Using 'list_queues' which lists queues, not cluster status.

The cluster_status command shows the current status of the RabbitMQ cluster including nodes and partitions.

3fill in blank

hard

Fix the error in the command to force a node to forget a partitioned node.

RabbitMQ

rabbitmqctl forget_[1] rabbit@node2

Drag options to blanks, or click blank then click option'

Acluster_node

Bpartition

Ccluster

Dnode_name

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'forget_partition' which is not a valid command.

Using 'forget_cluster' which does not exist.

The correct command is rabbitmqctl forget_cluster_node rabbit@node2 to remove a partitioned node from the cluster.

4fill in blank

hard

Fill both blanks to configure RabbitMQ to pause minority nodes during network partitions.

RabbitMQ

cluster_partition_handling = [1]
ha-mode = [2]

Drag options to blanks, or click blank then click option'

Apause_minority

Ball

Cexactly

Dignore

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'ignore' for partition handling which causes split-brain.

Using 'exactly' for ha-mode which is incomplete without a number.

Setting cluster_partition_handling to pause_minority pauses nodes in the minority partition. Setting ha-mode to all replicates queues to all nodes.

5fill in blank

hard

Fill all three blanks to create a policy that automatically syncs mirrored queues after a network partition.

RabbitMQ

rabbitmqctl set_policy [1] "^" '{{"ha-mode": "[2]", "ha-sync-mode": "[3]"}}' --apply-to queues

Drag options to blanks, or click blank then click option'

Async_all

Ball

Cautomatic

Dpause_minority

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'pause_minority' as ha-sync-mode which is invalid.

Using 'manual' instead of 'automatic' for ha-sync-mode.

The policy name is sync_all. The ha-mode is set to all to mirror queues to all nodes. The ha-sync-mode is set to automatic to sync queues automatically after partitions heal.

Practice

1. What is a network partition in a RabbitMQ cluster?

easy

A. A message queue overflow

B. A single node failure in the cluster

C. A configuration error in RabbitMQ settings

D. A split in the cluster where nodes cannot communicate with each other

Solution

1. Step 1: Understand network partition meaning

   A network partition occurs when parts of a RabbitMQ cluster lose communication with each other.

2. Step 2: Identify the effect on cluster nodes

   Nodes separated by the partition cannot exchange messages or cluster state, causing split behavior.

3. Final Answer:

   A split in the cluster where nodes cannot communicate with each other -> Option D

4. Quick Check:

   Network partition = nodes disconnected [OK]

Hint: Network partition means cluster nodes lose communication [OK]

Common Mistakes:

• Confusing partition with single node failure
• Thinking partition is a config error
• Assuming partition means message overflow

2. Which RabbitMQ command is used to set how the cluster handles network partitions?

easy

A. rabbitmqctl set_cluster_partition_handling

B. rabbitmqctl set_partition_mode

C. rabbitmqctl configure_network_partition

D. rabbitmqctl cluster_partition_policy

Solution

1. Step 1: Recall RabbitMQ partition handling command

   The correct command to control partition handling is set_cluster_partition_handling.

2. Step 2: Verify command syntax

   Other options are invalid or do not exist in RabbitMQ CLI.

3. Final Answer:

   rabbitmqctl set_cluster_partition_handling -> Option A

4. Quick Check:

   Partition handling command = set_cluster_partition_handling [OK]

Hint: Remember exact command: set_cluster_partition_handling [OK]

Common Mistakes:

• Using incorrect command names
• Confusing with cluster or network config commands
• Misspelling the command

3. Given a RabbitMQ cluster split into two partitions, what happens if the partition handling is set to ignore?

medium

A. Both partitions continue operating independently, risking split-brain

B. The cluster automatically heals and merges partitions

C. The cluster shuts down all nodes until partition resolves

D. Messages are dropped until the partition is fixed

Solution

1. Step 1: Understand 'ignore' partition handling

   When set to 'ignore', RabbitMQ nodes do not take action on partitions and continue working independently.

2. Step 2: Identify consequence of ignoring partitions

   This causes split-brain where multiple cluster parts act as main cluster, risking data conflicts.

3. Final Answer:

   Both partitions continue operating independently, risking split-brain -> Option A

4. Quick Check:

   Ignore mode = independent operation, split-brain risk [OK]

Hint: 'ignore' means no action, partitions run separately [OK]

Common Mistakes:

• Thinking cluster auto-heals with 'ignore'
• Assuming cluster shuts down nodes
• Believing messages are dropped automatically

4. You set RabbitMQ partition handling to pause_minority but the cluster does not pause any nodes during a partition. What is the likely cause?

medium

A. RabbitMQ does not support pause_minority mode

B. The command syntax was incorrect

C. The minority partition is not correctly identified

D. The cluster has only one node

Solution

1. Step 1: Understand pause_minority behavior

   In pause_minority mode, RabbitMQ pauses nodes in the smaller partition to avoid split-brain.

2. Step 2: Identify why no nodes paused

   If no nodes pause, likely the cluster did not detect which partition is minority, so no action taken.

3. Final Answer:

   The minority partition is not correctly identified -> Option C

4. Quick Check:

   Pause minority requires correct minority detection [OK]

Hint: Pause minority needs correct minority partition detection [OK]

Common Mistakes:

• Assuming syntax error without checking logs
• Believing pause_minority is unsupported
• Ignoring cluster size and partition count

5. In a 5-node RabbitMQ cluster, a network partition splits the cluster into two parts: one with 3 nodes and one with 2 nodes. If the partition handling is set to autoheal, what will happen after the partition resolves?

hard

A. Both partitions will continue independently causing split-brain

B. The smaller partition nodes will be stopped and the cluster will heal automatically

C. The cluster will require manual intervention to merge

D. All nodes will restart automatically regardless of partition size

Solution

1. Step 1: Understand autoheal partition handling

   Autoheal mode stops nodes in minority partitions and merges cluster state automatically after partition resolves.

2. Step 2: Apply to given cluster split

   With 3 nodes vs 2 nodes, the 2-node partition is minority and will be stopped to avoid split-brain.

3. Final Answer:

   The smaller partition nodes will be stopped and the cluster will heal automatically -> Option B

4. Quick Check:

   Autoheal stops minority, merges cluster [OK]

Hint: Autoheal stops minority partition nodes after split [OK]

Common Mistakes:

• Thinking autoheal allows split-brain
• Assuming manual merge is always needed
• Believing all nodes restart regardless