Cluster upgrade strategies in Kubernetes - Commands & Configuration

Introduction

Upgrading a Kubernetes cluster means updating its software to a newer version. This keeps the cluster secure, stable, and able to use new features. Doing this without breaking running applications is the main challenge.

When a new Kubernetes version is released with important security fixes.

When you want to use new Kubernetes features that require a newer version.

When your current cluster version is no longer supported by your cloud provider.

When you need to fix bugs or improve performance in your cluster.

When preparing your cluster to support newer versions of your applications.

Commands

This command safely evicts all pods from node-1 so it can be upgraded without affecting running applications. It ignores daemonsets and deletes local data to avoid blocking.

Terminal

kubectl drain node-1 --ignore-daemonsets --delete-local-data

Expected Output

node/node-1 cordoned

pod/my-app-1234 evicted

pod/my-app-5678 evicted

node/node-1 drained

→

--ignore-daemonsets - Allows draining even if daemonset pods are present on the node.

→

--delete-local-data - Deletes pods with local data to avoid blocking the drain.

This command upgrades the Kubernetes control plane to version 1.27.3. It updates the master components safely.

Terminal

sudo kubeadm upgrade apply v1.27.3

Expected Output

[upgrade] Making sure the cluster is healthy:

[upgrade] Upgrading your Static Pod-hosted control plane to version "v1.27.3"

[upgrade] Successfully upgraded control plane.

This command marks node-1 as schedulable again so it can run pods after the upgrade is complete.

Terminal

kubectl uncordon node-1

Expected Output

node/node-1 uncordoned

This command lists all nodes and their status to verify the upgrade was successful and nodes are ready.

Terminal

kubectl get nodes

Expected Output

NAME      STATUS   ROLES           AGE   VERSION

node-1    Ready    control-plane   10d   v1.27.3

node-2    Ready    <none>          10d   v1.27.3

Key Concept

If you remember nothing else from this pattern, remember: drain nodes before upgrading and uncordon them after to keep your cluster stable.

Common Mistakes

Upgrading nodes without draining them first

This causes running pods to be disrupted and can lead to downtime or data loss.

Always run 'kubectl drain' on a node before upgrading it to safely move pods away.

Forgetting to uncordon nodes after upgrade

Nodes remain unschedulable and new pods will not be placed on them, reducing cluster capacity.

Run 'kubectl uncordon' after upgrading to allow pods to be scheduled again.

Not verifying node versions after upgrade

You might think the upgrade succeeded but nodes could still be running old versions.

Use 'kubectl get nodes' to check the version and status of all nodes after upgrade.

Summary

Drain nodes to safely evict pods before upgrading.

Use kubeadm to upgrade the control plane to the desired Kubernetes version.

Uncordon nodes after upgrade to resume scheduling pods.

Verify node status and versions to confirm successful upgrade.

Practice

1. What is the recommended order when upgrading a Kubernetes cluster?

easy

A. Upgrade all nodes simultaneously

B. Upgrade worker nodes first, then control plane nodes

C. Upgrade control plane nodes first, then worker nodes

D. Upgrade only the worker nodes

Solution

1. Step 1: Understand the role of control plane nodes

   Control plane nodes manage the cluster state and API server, so they must be stable first.

2. Step 2: Upgrade worker nodes after control plane

   Worker nodes run workloads and depend on the control plane, so upgrade them after control plane nodes.

3. Final Answer:

   Upgrade control plane nodes first, then worker nodes -> Option C

4. Quick Check:

   Control plane first, workers second = A [OK]

Hint: Always upgrade control plane nodes before worker nodes [OK]

Common Mistakes:

• Upgrading worker nodes before control plane
• Upgrading all nodes at once causing downtime
• Skipping control plane upgrade

2. Which command correctly drains a node before upgrading it?

easy

A. kubectl drain --ignore-daemonsets --delete-local-data

B. kubectl upgrade node

C. kubectl delete node

D. kubectl cordon --force

Solution

1. Step 1: Identify the correct drain command syntax

   The command to safely evict pods is 'kubectl drain' with flags to ignore daemonsets and delete local data.

2. Step 2: Verify other options are incorrect

   Upgrade and delete commands do not drain nodes; cordon only marks unschedulable but does not evict pods.

3. Final Answer:

   kubectl drain <node-name> --ignore-daemonsets --delete-local-data -> Option A

4. Quick Check:

   Drain command with correct flags = A [OK]

Hint: Use 'kubectl drain' with flags to safely evict pods [OK]

Common Mistakes:

• Using 'kubectl cordon' instead of 'drain'
• Deleting nodes instead of draining
• Missing flags causing pod eviction failure

3. Given this upgrade sequence, what is the expected cluster state?

1. Drain node1
2. Upgrade node1
3. Uncordon node1
4. Repeat for node2 and node3

medium

A. Control plane nodes are upgraded last

B. Cluster remains available with minimal downtime

C. Pods are deleted permanently during upgrade

D. Cluster goes down during node upgrades

Solution

1. Step 1: Analyze the upgrade steps

   Each node is drained to safely evict pods, upgraded, then uncordoned to resume scheduling.

2. Step 2: Understand impact on cluster availability

   Upgrading nodes one by one with draining keeps workloads running on other nodes, minimizing downtime.

3. Final Answer:

   Cluster remains available with minimal downtime -> Option B

4. Quick Check:

   Draining and upgrading nodes one by one = D [OK]

Hint: Upgrade nodes one at a time with drain/un-cordon for uptime [OK]

Common Mistakes:

• Assuming cluster goes down during upgrades
• Not draining nodes causing pod failures
• Upgrading all nodes simultaneously

4. You ran kubectl drain node1 but pods did not evict. What is the likely cause?

medium

A. DaemonSet pods are blocking eviction

B. Node is already uncordoned

C. Control plane node cannot be drained

D. Pods have no local storage

Solution

1. Step 1: Understand drain behavior with DaemonSets

   By default, drain blocks if DaemonSet pods are running unless --ignore-daemonsets is used.

2. Step 2: Check other options for correctness

   Uncordon status does not block eviction; control plane nodes can be drained; pods without local storage do not block drain.

3. Final Answer:

   DaemonSet pods are blocking eviction -> Option A

4. Quick Check:

   DaemonSet pods block drain without flag = C [OK]

Hint: Use --ignore-daemonsets flag to drain nodes with DaemonSet pods [OK]

Common Mistakes:

• Not using --ignore-daemonsets flag
• Confusing cordon with drain
• Assuming control plane nodes cannot be drained

5. You want to upgrade a large Kubernetes cluster with minimal downtime. Which strategy is best?

hard

A. Upgrade all control plane nodes simultaneously, then all workers simultaneously

B. Skip draining and upgrade nodes in random order

C. Drain all nodes at once, upgrade, then uncordon all nodes

D. Use cloud provider upgrade tools to upgrade control plane, then drain and upgrade workers one by one

Solution

1. Step 1: Consider cloud provider tools for control plane upgrade

   Cloud tools often automate safe control plane upgrades reducing manual errors.

2. Step 2: Upgrade worker nodes one by one with drain/un-cordon

   This approach avoids downtime by keeping workloads running on other nodes during upgrade.

3. Step 3: Evaluate other options for risks

   Upgrading all nodes simultaneously or skipping drain risks downtime and pod failures.

4. Final Answer:

   Use cloud provider upgrade tools to upgrade control plane, then drain and upgrade workers one by one -> Option D

5. Quick Check:

   Cloud tools + sequential worker upgrade = B [OK]

Hint: Use cloud tools and upgrade workers one at a time with drain [OK]

Common Mistakes:

• Upgrading all nodes simultaneously causing downtime
• Skipping drain causing pod disruption
• Ignoring cloud provider upgrade features