Rolling upgrades in Elasticsearch - Step-by-Step Execution

Concept Flow - Rolling upgrades

Start: Current Cluster Version

↓

Select Node for Upgrade

↓

Take Node Out of Cluster

↓

Upgrade Node Software

↓

Restart Node with New Version

↓

Node Rejoins Cluster

↓

Verify Cluster Health

↓

Repeat for Next Node

↓

All Nodes Upgraded

↓

Upgrade Complete

Rolling upgrades upgrade one node at a time to keep the cluster running without downtime.

Execution Sample

Elasticsearch

PUT /_cluster/settings
{
  "transient": {
    "cluster.routing.allocation.enable": "none"
  }
}

Disables shard allocation temporarily to safely upgrade a node.

Execution Table

Step	Action	Node State	Cluster State	Result
1	Select node1 for upgrade	node1 active	All nodes active	Ready to upgrade node1
2	Disable shard allocation	node1 active	Allocation disabled	Prevents shard movement
3	Stop node1	node1 stopped	Cluster yellow	Node1 offline, cluster still running
4	Upgrade node1 software	node1 stopped	Cluster yellow	Node1 upgraded
5	Start node1 with new version	node1 active (new version)	Cluster recovering	Node1 rejoins cluster
6	Enable shard allocation	node1 active	Allocation enabled	Cluster rebalances shards
7	Verify cluster health	All nodes active	Cluster green	Node1 upgrade successful
8	Repeat steps 1-7 for node2	node2 active	Cluster green	Node2 upgraded
9	Repeat steps 1-7 for node3	node3 active	Cluster green	Node3 upgraded
10	All nodes upgraded	All nodes active (new version)	Cluster green	Rolling upgrade complete

💡 All nodes upgraded one by one, cluster remains available throughout

Variable Tracker

Variable	Start	After Step 3	After Step 5	After Step 7	After Step 10
node1 state	active	stopped	active (new version)	active (new version)	active (new version)
cluster allocation	enabled	disabled	disabled	enabled	enabled
cluster health	green	yellow	yellow	green	green

Key Moments - 3 Insights

Why do we disable shard allocation before stopping a node?

Disabling shard allocation (see step 2 in execution_table) prevents shards from moving around during the node upgrade, avoiding extra load and instability.

What happens to cluster health when a node is stopped?

Cluster health changes from green to yellow (step 3) because one node is offline, but the cluster remains available.

Why upgrade nodes one at a time instead of all at once?

Upgrading one node at a time (steps 1-9) keeps the cluster running and avoids downtime, ensuring continuous availability.

Visual Quiz - 3 Questions

Look at the execution_table, what is the cluster health after stopping node1 (step 3)?

A green

B yellow

C red

D unknown

💡 Hint

Check the 'Cluster State' column at step 3 in the execution_table.

At which step is shard allocation re-enabled during the upgrade of node1?

A Step 2

B Step 4

C Step 6

D Step 8

💡 Hint

Look for 'Enable shard allocation' action in the execution_table.

If shard allocation was not disabled before stopping a node, what would likely happen?

A Shards would move causing extra load and instability

B Cluster health would remain green

C Node would upgrade faster

D Nothing changes

💡 Hint

Refer to the key_moments explanation about disabling shard allocation.

Concept Snapshot

Rolling upgrades upgrade one node at a time.
Disable shard allocation before stopping a node.
Stop, upgrade, restart node with new version.
Re-enable allocation and verify cluster health.
Repeat for each node until all upgraded.
Cluster stays available during the process.

Full Transcript

Rolling upgrades in Elasticsearch mean upgrading nodes one by one to keep the cluster running without downtime. The process starts by disabling shard allocation to prevent shards from moving during the upgrade. Then, a node is stopped, upgraded, and restarted with the new version. After the node rejoins the cluster, shard allocation is re-enabled and cluster health is checked. This process repeats for each node until all nodes are upgraded. Cluster health may temporarily degrade but remains available throughout. This method avoids downtime and keeps data safe.

Practice

1. What is the main purpose of performing a rolling upgrade in Elasticsearch?

easy

A. To disable the cluster permanently during upgrade

B. To upgrade all nodes simultaneously for faster updates

C. To upgrade nodes one by one without stopping the entire cluster

D. To delete old data before upgrading

Solution

1. Step 1: Understand rolling upgrade concept

   A rolling upgrade updates nodes one at a time to keep the cluster running.

2. Step 2: Compare options

   Only To upgrade nodes one by one without stopping the entire cluster describes upgrading nodes one by one without stopping the cluster.

3. Final Answer:

   To upgrade nodes one by one without stopping the entire cluster -> Option C

4. Quick Check:

   Rolling upgrade = upgrade nodes individually [OK]

Hint: Rolling upgrade means upgrading nodes one by one [OK]

Common Mistakes:

• Thinking all nodes upgrade at once
• Confusing rolling upgrade with cluster shutdown
• Assuming data is deleted during upgrade

2. Which command is recommended to disable shard allocation before starting a rolling upgrade?

easy

A. PUT /_cluster/settings {"persistent": {"cluster.routing.allocation.enable": "none"}}

B. POST /_cluster/disable_shards

C. GET /_cluster/settings {"allocation": "disable"}

D. DELETE /_cluster/shards

Solution

1. Step 1: Identify correct syntax to disable shard allocation

   The correct way is to update cluster settings with PUT and set allocation to "none".

2. Step 2: Check options

   Only PUT /_cluster/settings {"persistent": {"cluster.routing.allocation.enable": "none"}} uses the correct HTTP method, endpoint, and JSON body.

3. Final Answer:

   PUT /_cluster/settings {"persistent": {"cluster.routing.allocation.enable": "none"}} -> Option A

4. Quick Check:

   Disable shard allocation = PUT cluster settings with allocation none [OK]

Hint: Use PUT with cluster settings and allocation none to disable shards [OK]

Common Mistakes:

• Using wrong HTTP method like POST or GET
• Wrong endpoint or missing persistent key
• Trying to delete shards instead of disabling allocation

3. Given the following sequence during a rolling upgrade:

1. Disable shard allocation
2. Upgrade node 1
3. Upgrade node 2
4. Enable shard allocation

What is the expected cluster behavior after step 4?

medium

A. The cluster will stop accepting new data

B. The cluster will rebalance shards across all nodes

C. The cluster will delete old shards permanently

D. The cluster will remain unbalanced with shards stuck

Solution

1. Step 1: Understand shard allocation states

   Disabling shard allocation prevents shard movement during upgrade; enabling it allows rebalancing.

2. Step 2: Analyze cluster behavior after enabling allocation

   After enabling, the cluster redistributes shards to balance load.

3. Final Answer:

   The cluster will rebalance shards across all nodes -> Option B

4. Quick Check:

   Enable allocation = rebalance shards [OK]

Hint: Enabling shard allocation triggers shard rebalancing [OK]

Common Mistakes:

• Thinking cluster stops accepting data
• Assuming shards get deleted
• Believing shards remain stuck after enabling allocation

4. You ran a rolling upgrade but forgot to disable shard allocation first. What problem might occur?

medium

A. Shards may move during upgrade causing data loss or instability

B. The upgrade will fail immediately with syntax error

C. The cluster will automatically disable allocation for you

D. Nothing happens; upgrade proceeds safely

Solution

1. Step 1: Understand shard allocation role during upgrade

   Disabling allocation prevents shards from moving and keeps data safe during node restarts.

2. Step 2: Consequence of not disabling allocation

   If not disabled, shards may relocate while nodes restart, risking data loss or cluster instability.

3. Final Answer:

   Shards may move during upgrade causing data loss or instability -> Option A

4. Quick Check:

   Not disabling allocation risks shard movement and instability [OK]

Hint: Always disable shard allocation before upgrade to avoid shard moves [OK]

Common Mistakes:

• Expecting upgrade to fail with syntax error
• Assuming cluster disables allocation automatically
• Thinking upgrade is safe without disabling allocation

5. During a rolling upgrade, you want to ensure minimal downtime and data safety. Which sequence of actions is best practice?

hard

A. Upgrade all nodes simultaneously -> Disable shard allocation -> Enable shard allocation -> Restart cluster

B. Restart cluster -> Disable shard allocation -> Upgrade nodes one by one -> Enable shard allocation

C. Enable shard allocation -> Upgrade nodes one by one -> Disable shard allocation -> Verify cluster health

D. Disable shard allocation -> Upgrade nodes one by one -> Enable shard allocation -> Verify cluster health

Solution

1. Step 1: Identify correct upgrade steps

   Best practice is to disable shard allocation first to prevent shard moves, then upgrade nodes one by one.

2. Step 2: Finalize upgrade process

   After upgrading all nodes, enable shard allocation to rebalance shards and verify cluster health to confirm stability.

3. Final Answer:

   Disable shard allocation -> Upgrade nodes one by one -> Enable shard allocation -> Verify cluster health -> Option D

4. Quick Check:

   Disable allocation, upgrade nodes, enable allocation, check health [OK]

Hint: Disable allocation first, upgrade nodes, then enable allocation [OK]

Common Mistakes:

• Upgrading all nodes at once causing downtime
• Enabling allocation before upgrade completion
• Restarting cluster unnecessarily