Rolling upgrades in Elasticsearch - Deep Dive

Overview - Rolling upgrades

What is it?

Rolling upgrades are a way to update a running Elasticsearch cluster without stopping the entire system. Instead of shutting down all nodes at once, nodes are upgraded one by one. This keeps the cluster available and serving requests during the upgrade process. It helps avoid downtime and service interruptions.

Why it matters

Without rolling upgrades, upgrading Elasticsearch would require stopping the whole cluster, causing downtime and disrupting users or applications relying on search and data. Rolling upgrades solve this by allowing continuous operation, which is critical for businesses that need their data accessible 24/7. It reduces risk and improves user experience during upgrades.

Where it fits

Before learning rolling upgrades, you should understand Elasticsearch cluster basics, nodes, and how data is distributed. After mastering rolling upgrades, you can explore advanced cluster management, backup strategies, and performance tuning during upgrades.

Mental Model

Core Idea

Rolling upgrades update one node at a time in a cluster to keep the system running without downtime.

Think of it like...

Imagine replacing light bulbs in a long hallway one by one while keeping the hallway lit, instead of turning off all lights at once and walking in the dark.

Elasticsearch Cluster Upgrade Flow:

┌─────────────┐    Upgrade Node 1    ┌─────────────┐
│ Node 1 (old)│ ───────────────▶ │ Node 1 (new)│
└─────────────┘                     └─────────────┘
       │                                  │
       ▼                                  ▼
┌─────────────┐    Upgrade Node 2    ┌─────────────┐
│ Node 2 (old)│ ───────────────▶ │ Node 2 (new)│
└─────────────┘                     └─────────────┘
       │                                  │
       ▼                                  ▼
      ...                                ...

Each node is upgraded individually while others keep the cluster alive.

Build-Up - 7 Steps

1

Foundation

Understanding Elasticsearch Clusters

Concept: Learn what an Elasticsearch cluster is and how nodes work together.

An Elasticsearch cluster is a group of one or more nodes (servers) that store data and provide search capabilities. Nodes share data and coordinate to handle requests. Each node can hold parts of the data called shards. The cluster works as one system to provide fast and reliable search.

Result

You understand that a cluster is made of nodes working together to store and search data.

Knowing the cluster structure is essential because rolling upgrades affect nodes individually but impact the whole cluster.

2

Foundation

Why Upgrades Are Needed

Concept: Understand the reasons for upgrading Elasticsearch nodes.

Upgrades bring new features, security patches, and performance improvements. Without upgrading, the cluster may become outdated, insecure, or incompatible with other tools. However, upgrading must be done carefully to avoid downtime or data loss.

Result

You see why keeping Elasticsearch updated is important for reliability and security.

Recognizing the need for upgrades motivates learning how to do them safely without stopping the cluster.

3

Intermediate

What Is a Rolling Upgrade

🤔Before reading on: do you think upgrading all nodes at once is safer or upgrading one node at a time? Commit to your answer.

Concept: Introduce the rolling upgrade method where nodes are upgraded sequentially.

A rolling upgrade updates nodes one by one. First, one node is taken offline, upgraded, and restarted. Then the next node is upgraded, and so on. This way, the cluster remains mostly operational because other nodes handle requests while one is upgrading.

Result

You learn that rolling upgrades minimize downtime by upgrading nodes sequentially.

Understanding rolling upgrades helps prevent full cluster downtime and keeps services available.

4

Intermediate

Steps to Perform a Rolling Upgrade

🤔Before reading on: do you think you should upgrade data nodes first or master nodes first? Commit to your answer.

Concept: Learn the recommended order and steps for upgrading nodes safely.

The typical steps are: 1. Upgrade master-eligible nodes first, one at a time. 2. Upgrade data nodes next, one at a time. 3. Upgrade client or coordinating nodes last. Each node is stopped, upgraded, and restarted before moving to the next. This order ensures cluster stability and leadership continuity.

Result

You know the correct sequence and process to upgrade nodes without breaking the cluster.

Knowing the upgrade order prevents cluster instability and leadership loss during upgrades.

5

Intermediate

Handling Compatibility and Settings

🤔Before reading on: do you think all versions of Elasticsearch can upgrade directly to the latest version? Commit to your answer.

Concept: Understand version compatibility and configuration adjustments needed during upgrades.

Elasticsearch supports rolling upgrades only between compatible versions, usually minor version bumps (e.g., 7.10 to 7.11). Major version upgrades often require full cluster shutdown. Also, some settings or plugins may need updates to work with the new version. Checking compatibility and preparing configurations is crucial.

Result

You realize that not all upgrades can be rolling and preparation is needed.

Understanding compatibility avoids failed upgrades and data issues.

6

Advanced

Monitoring Cluster Health During Upgrade

🤔Before reading on: do you think the cluster can be unhealthy during a rolling upgrade? Commit to your answer.

Concept: Learn how to watch cluster status and react to issues during upgrades.

During rolling upgrades, the cluster may temporarily show yellow or red status if shards relocate or nodes are offline. Monitoring tools and APIs help track cluster health. If problems arise, you can pause or roll back upgrades to prevent data loss or downtime.

Result

You can keep the cluster stable by monitoring and responding during upgrades.

Knowing how to monitor prevents surprises and ensures smooth upgrades.

7

Expert

Surprises and Pitfalls in Rolling Upgrades

🤔Before reading on: do you think rolling upgrades guarantee zero downtime in all cases? Commit to your answer.

Concept: Explore edge cases and unexpected behaviors during rolling upgrades.

Rolling upgrades reduce downtime but do not guarantee zero downtime. For example, if a node holds unique shards and is offline, some queries may slow or fail temporarily. Also, network issues or incompatible plugins can cause failures. Experts plan for these by using replica shards, backups, and testing upgrades in staging environments.

Result

You understand the limits and risks of rolling upgrades and how to mitigate them.

Recognizing rolling upgrade limits helps prepare fallback plans and avoid critical failures.

Under the Hood

Elasticsearch nodes communicate via a cluster coordination protocol. During rolling upgrades, the cluster master tracks node states and shard allocations. When a node is stopped for upgrade, its shards are relocated to other nodes to maintain data availability. The cluster waits for the upgraded node to rejoin and reassigns shards back if needed. This dynamic shard movement and master coordination keep the cluster operational.

Why designed this way?

Rolling upgrades were designed to avoid full cluster downtime, which is costly and disruptive. The distributed nature of Elasticsearch allows nodes to be independent enough to upgrade one at a time. Alternatives like full shutdown were rejected because they interrupt service completely. Rolling upgrades balance availability with upgrade safety.

Cluster Upgrade Internal Flow:

┌───────────────┐
│ Master Node   │
│ - Tracks nodes│
│ - Manages     │
│   shard moves │
└──────┬────────┘
       │
       ▼
┌───────────────┐       Node 1 stops for upgrade
│ Data Node 1   │ ──────────────▶ Offline
└───────────────┘
       │
       ▼
┌───────────────┐       Shards move to other nodes
│ Data Node 2   │ ◀─────────────
└───────────────┘
       │
       ▼
┌───────────────┐       Node 1 upgraded and rejoins
│ Data Node 1   │ ◀─────────────
└───────────────┘
       │
       ▼
Master rebalances shards to original state

Myth Busters - 4 Common Misconceptions

Quick: do you think rolling upgrades mean zero downtime always? Commit yes or no.

Common Belief: Rolling upgrades guarantee zero downtime with no impact on users.

Reality: Rolling upgrades minimize downtime but some temporary slowdowns or partial unavailability can occur during shard relocation or node restarts.

Why it matters: Believing in zero downtime can lead to under-preparing for brief service impacts, causing unexpected user complaints.

Quick: can you upgrade any Elasticsearch version directly with rolling upgrades? Commit yes or no.

Common Belief: You can roll upgrade between any Elasticsearch versions without stopping the cluster.

Reality: Rolling upgrades only work between compatible minor versions. Major version upgrades usually require full cluster shutdown.

Why it matters: Trying unsupported upgrades can break the cluster and cause data loss.

Quick: do you think upgrading data nodes first is better than master nodes? Commit your answer.

Common Belief: Upgrading data nodes first is fine and does not affect cluster stability.

Reality: Master-eligible nodes should be upgraded first to maintain cluster leadership and coordination during the upgrade.

Why it matters: Upgrading data nodes first can cause cluster instability or split-brain scenarios.

Quick: do you think plugins always work after rolling upgrades? Commit yes or no.

Common Belief: All plugins continue working seamlessly after rolling upgrades.

Reality: Some plugins may be incompatible with new versions and require updates or removal before upgrading.

Why it matters: Ignoring plugin compatibility can cause cluster errors or failures post-upgrade.

Expert Zone

1

Master node upgrades must be done carefully to avoid losing cluster coordination and causing split-brain.

2

Shard relocation during upgrades can cause temporary performance degradation, so monitoring resource usage is critical.

3

Rolling upgrades require careful plugin and setting compatibility checks to avoid subtle runtime errors.

When NOT to use

Rolling upgrades are not suitable for major version jumps or when cluster state is unstable. In those cases, a full cluster shutdown upgrade or blue-green deployment is safer.

Production Patterns

In production, rolling upgrades are automated with orchestration tools that drain nodes, upgrade, and verify health before proceeding. Teams use canary upgrades on test clusters first and maintain backups to recover from failures.

Connections

Blue-Green Deployment

Alternative upgrade strategy with zero downtime by switching between two identical environments.

Understanding blue-green deployments helps appreciate rolling upgrades as a different approach to continuous availability.

Distributed Consensus Algorithms

Rolling upgrades rely on cluster coordination protocols like Raft or Zen Discovery to maintain cluster state.

Knowing consensus algorithms clarifies how cluster leadership and shard allocation remain consistent during node upgrades.

Continuous Integration/Continuous Deployment (CI/CD)

Rolling upgrades fit into CI/CD pipelines to automate safe, incremental software updates.

Seeing rolling upgrades as part of CI/CD helps integrate Elasticsearch upgrades into broader DevOps practices.

Common Pitfalls

#1 Stopping all nodes at once to upgrade causes full downtime.

Wrong approach:

Stop all Elasticsearch nodes simultaneously, upgrade, then restart.

Correct approach:

Stop and upgrade one node at a time, letting the cluster stay online with remaining nodes.

Root cause:

Misunderstanding that the cluster can only be upgraded node-by-node to avoid downtime.

#2 Upgrading data nodes before master nodes leads to cluster instability.

Wrong approach:

Upgrade data nodes first, then master nodes.

Correct approach:

Upgrade master-eligible nodes first, then data nodes.

Root cause:

Not knowing the master node's role in cluster coordination and leadership.

#3 Ignoring plugin compatibility causes errors after upgrade.

Wrong approach:

Upgrade Elasticsearch without checking or updating plugins.

Correct approach:

Verify and update plugins to compatible versions before upgrading Elasticsearch nodes.

Root cause:

Assuming plugins always work across versions without testing.

Key Takeaways

Rolling upgrades update Elasticsearch nodes one at a time to keep the cluster running without full downtime.

Master nodes should be upgraded before data nodes to maintain cluster stability and leadership.

Rolling upgrades only work between compatible versions; major upgrades require different strategies.

Monitoring cluster health during upgrades helps detect and fix issues early to avoid data loss.

Understanding rolling upgrades is essential for maintaining high availability in production Elasticsearch clusters.

Practice

1. What is the main purpose of performing a rolling upgrade in Elasticsearch?

easy

A. To disable the cluster permanently during upgrade

B. To upgrade all nodes simultaneously for faster updates

C. To upgrade nodes one by one without stopping the entire cluster

D. To delete old data before upgrading

Solution

1. Step 1: Understand rolling upgrade concept

   A rolling upgrade updates nodes one at a time to keep the cluster running.

2. Step 2: Compare options

   Only To upgrade nodes one by one without stopping the entire cluster describes upgrading nodes one by one without stopping the cluster.

3. Final Answer:

   To upgrade nodes one by one without stopping the entire cluster -> Option C

4. Quick Check:

   Rolling upgrade = upgrade nodes individually [OK]

Hint: Rolling upgrade means upgrading nodes one by one [OK]

Common Mistakes:

• Thinking all nodes upgrade at once
• Confusing rolling upgrade with cluster shutdown
• Assuming data is deleted during upgrade

2. Which command is recommended to disable shard allocation before starting a rolling upgrade?

easy

A. PUT /_cluster/settings {"persistent": {"cluster.routing.allocation.enable": "none"}}

B. POST /_cluster/disable_shards

C. GET /_cluster/settings {"allocation": "disable"}

D. DELETE /_cluster/shards

Solution

1. Step 1: Identify correct syntax to disable shard allocation

   The correct way is to update cluster settings with PUT and set allocation to "none".

2. Step 2: Check options

   Only PUT /_cluster/settings {"persistent": {"cluster.routing.allocation.enable": "none"}} uses the correct HTTP method, endpoint, and JSON body.

3. Final Answer:

   PUT /_cluster/settings {"persistent": {"cluster.routing.allocation.enable": "none"}} -> Option A

4. Quick Check:

   Disable shard allocation = PUT cluster settings with allocation none [OK]

Hint: Use PUT with cluster settings and allocation none to disable shards [OK]

Common Mistakes:

• Using wrong HTTP method like POST or GET
• Wrong endpoint or missing persistent key
• Trying to delete shards instead of disabling allocation

3. Given the following sequence during a rolling upgrade:

1. Disable shard allocation
2. Upgrade node 1
3. Upgrade node 2
4. Enable shard allocation

What is the expected cluster behavior after step 4?

medium

A. The cluster will stop accepting new data

B. The cluster will rebalance shards across all nodes

C. The cluster will delete old shards permanently

D. The cluster will remain unbalanced with shards stuck

Solution

1. Step 1: Understand shard allocation states

   Disabling shard allocation prevents shard movement during upgrade; enabling it allows rebalancing.

2. Step 2: Analyze cluster behavior after enabling allocation

   After enabling, the cluster redistributes shards to balance load.

3. Final Answer:

   The cluster will rebalance shards across all nodes -> Option B

4. Quick Check:

   Enable allocation = rebalance shards [OK]

Hint: Enabling shard allocation triggers shard rebalancing [OK]

Common Mistakes:

• Thinking cluster stops accepting data
• Assuming shards get deleted
• Believing shards remain stuck after enabling allocation

4. You ran a rolling upgrade but forgot to disable shard allocation first. What problem might occur?

medium

A. Shards may move during upgrade causing data loss or instability

B. The upgrade will fail immediately with syntax error

C. The cluster will automatically disable allocation for you

D. Nothing happens; upgrade proceeds safely

Solution

1. Step 1: Understand shard allocation role during upgrade

   Disabling allocation prevents shards from moving and keeps data safe during node restarts.

2. Step 2: Consequence of not disabling allocation

   If not disabled, shards may relocate while nodes restart, risking data loss or cluster instability.

3. Final Answer:

   Shards may move during upgrade causing data loss or instability -> Option A

4. Quick Check:

   Not disabling allocation risks shard movement and instability [OK]

Hint: Always disable shard allocation before upgrade to avoid shard moves [OK]

Common Mistakes:

• Expecting upgrade to fail with syntax error
• Assuming cluster disables allocation automatically
• Thinking upgrade is safe without disabling allocation

5. During a rolling upgrade, you want to ensure minimal downtime and data safety. Which sequence of actions is best practice?

hard

A. Upgrade all nodes simultaneously -> Disable shard allocation -> Enable shard allocation -> Restart cluster

B. Restart cluster -> Disable shard allocation -> Upgrade nodes one by one -> Enable shard allocation

C. Enable shard allocation -> Upgrade nodes one by one -> Disable shard allocation -> Verify cluster health

D. Disable shard allocation -> Upgrade nodes one by one -> Enable shard allocation -> Verify cluster health

Solution

1. Step 1: Identify correct upgrade steps

   Best practice is to disable shard allocation first to prevent shard moves, then upgrade nodes one by one.

2. Step 2: Finalize upgrade process

   After upgrading all nodes, enable shard allocation to rebalance shards and verify cluster health to confirm stability.

3. Final Answer:

   Disable shard allocation -> Upgrade nodes one by one -> Enable shard allocation -> Verify cluster health -> Option D

4. Quick Check:

   Disable allocation, upgrade nodes, enable allocation, check health [OK]

Hint: Disable allocation first, upgrade nodes, then enable allocation [OK]

Common Mistakes:

• Upgrading all nodes at once causing downtime
• Enabling allocation before upgrade completion
• Restarting cluster unnecessarily