High availability cluster setup in Kubernetes - Deep Dive

Overview - High availability cluster setup

What is it?

A high availability cluster setup means arranging multiple computers or servers to work together so that if one fails, others keep the system running without interruption. In Kubernetes, this involves setting up multiple control plane nodes and worker nodes to ensure the system stays available and responsive. This setup prevents downtime and keeps applications accessible even during hardware or software failures. It is like having backup team members ready to take over instantly.

Why it matters

Without high availability, a single failure in the system can cause downtime, making applications unreachable and causing loss of trust or revenue. High availability clusters ensure continuous service, which is critical for businesses that rely on their applications being always online. It reduces risks and improves user experience by avoiding interruptions.

Where it fits

Before learning high availability cluster setup, you should understand basic Kubernetes architecture, including control plane and worker nodes. After mastering this, you can explore advanced topics like disaster recovery, scaling, and multi-cluster management.

Mental Model

Core Idea

High availability clusters use multiple nodes working together so that if one fails, others immediately take over to keep services running without interruption.

Think of it like...

It's like having several lifeguards watching a pool; if one lifeguard needs a break or is unavailable, others instantly cover to keep everyone safe without any gap.

┌───────────────────────────────┐
│       High Availability        │
│          Cluster Setup         │
├─────────────┬─────────────┬────┤
│ Control     │ Control     │    │
│ Plane Node 1│ Plane Node 2│ ...│
├─────────────┴─────────────┴────┤
│           Worker Nodes         │
│  Node 1  Node 2  Node 3  Node 4│
└───────────────────────────────┘

If one control plane node fails, others continue managing the cluster.
Worker nodes run applications and stay available through redundancy.

Build-Up - 7 Steps

1

Foundation

Understanding Kubernetes Cluster Basics

Concept: Learn what a Kubernetes cluster is and its main components: control plane and worker nodes.

A Kubernetes cluster consists of a control plane that manages the cluster and worker nodes that run applications. The control plane includes components like API server, scheduler, and controller manager. Worker nodes run containers and communicate with the control plane. This basic setup allows you to deploy and manage applications.

Result

You can identify the roles of control plane and worker nodes in a Kubernetes cluster.

Understanding the cluster's basic structure is essential before adding complexity like high availability.

2

Foundation

Single Control Plane Node Limitations

Concept: Recognize why having only one control plane node is risky.

A single control plane node is a single point of failure. If it crashes or becomes unreachable, the entire cluster management stops, and you cannot deploy or manage applications. This setup is simple but not reliable for production environments.

Result

You see that a single control plane node can cause downtime if it fails.

Knowing this risk motivates the need for multiple control plane nodes for high availability.

3

Intermediate

Setting Up Multiple Control Plane Nodes

🤔Before reading on: do you think multiple control plane nodes share the same data or have separate copies? Commit to your answer.

Concept: Learn how multiple control plane nodes work together by sharing cluster state data.

In a high availability setup, multiple control plane nodes run simultaneously. They share the cluster state using etcd, a distributed key-value store. This sharing ensures all control planes have the same information and can manage the cluster together. If one control plane fails, others continue without losing data.

Result

The cluster remains manageable even if one control plane node goes down.

Understanding shared state via etcd is key to grasping how control plane redundancy works.

4

Intermediate

Load Balancing Control Plane Access

🤔Before reading on: do you think clients connect directly to each control plane node or through a single entry point? Commit to your answer.

Concept: Learn why and how to use a load balancer to distribute requests to control plane nodes.

Clients and worker nodes access the control plane through a load balancer. This load balancer forwards requests to healthy control plane nodes. It hides the complexity of multiple nodes and ensures requests reach an available node. Without it, clients would need to know all control plane addresses and handle failures themselves.

Result

Requests to the control plane are reliably routed to available nodes, improving cluster stability.

Using a load balancer simplifies access and prevents single points of failure at the network level.

5

Intermediate

Ensuring etcd Cluster High Availability

Concept: Understand how etcd, the data store for Kubernetes, is made highly available.

etcd stores all cluster data and must be highly available. This is done by running multiple etcd nodes in a cluster. They use consensus algorithms to agree on data changes. Losing a majority of etcd nodes causes data loss risk, so an odd number of nodes is recommended. etcd nodes are often co-located with control plane nodes.

Result

Cluster state data remains consistent and available even if some etcd nodes fail.

Knowing etcd's role and its high availability is critical because control plane nodes depend on it.

6

Advanced

Worker Node High Availability Strategies

🤔Before reading on: do you think worker nodes need to be identical or can they differ? Commit to your answer.

Concept: Learn how worker nodes are managed for high availability and load distribution.

Worker nodes run application workloads. To ensure availability, multiple worker nodes run the same application replicas. Kubernetes schedules pods across nodes to balance load and avoid single points of failure. If a node fails, pods are rescheduled on other nodes automatically. Nodes can differ in size or capacity but must meet application requirements.

Result

Applications stay running and responsive even if some worker nodes fail.

Understanding pod replication and scheduling is essential for application-level high availability.

7

Expert

Handling Network and Storage in HA Clusters

🤔Before reading on: do you think network and storage are automatically highly available in Kubernetes? Commit to your answer.

Concept: Explore how network and storage components must be designed for high availability in Kubernetes clusters.

Network and storage are critical for cluster availability. Network failures can isolate nodes or control planes, so redundant network paths and reliable DNS are needed. Storage must be accessible from multiple nodes; solutions like distributed storage or cloud volumes with replication are used. Misconfigurations here can cause downtime despite control plane and worker node redundancy.

Result

The cluster maintains connectivity and data access even during network or storage failures.

Knowing that HA requires all infrastructure layers to be redundant prevents hidden single points of failure.

Under the Hood

High availability in Kubernetes relies on multiple control plane nodes running the same components and sharing cluster state via an etcd cluster. etcd uses a consensus algorithm called Raft to keep data consistent across nodes. A load balancer fronts the control plane nodes to distribute API requests. Worker nodes run pods scheduled by the control plane, and pod replicas ensure application availability. Network and storage layers must also be redundant to avoid isolating nodes or losing data.

Why designed this way?

Kubernetes was designed for cloud-native environments where failures are expected. Using multiple control plane nodes and etcd clusters avoids single points of failure. The Raft consensus algorithm ensures data consistency even with node failures. Load balancers simplify client access. This design balances complexity and reliability, avoiding centralized bottlenecks.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ Control Plane │       │ Control Plane │       │ Control Plane │
│ Node 1       │◄──────│ Node 2       │──────►│ Node 3       │
└──────┬────────┘       └──────┬────────┘       └──────┬────────┘
       │                       │                       │
       │                       │                       │
       ▼                       ▼                       ▼
    ┌───────────────────────────────────────────────┐
    │                   etcd Cluster                 │
    │  Node 1   Node 2   Node 3   (Consensus via Raft)│
    └───────────────────────────────────────────────┘
               ▲                       ▲
               │                       │
       ┌───────┴───────────────────────┴────────┐
       │           Load Balancer (API Server)    │
       └─────────────────────────────────────────┘
                        ▲
                        │
           ┌────────────┴─────────────┐
           │       Worker Nodes        │
           │ Node 1  Node 2  Node 3... │
           └──────────────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does adding more control plane nodes always improve cluster performance? Commit yes or no.

Common Belief: More control plane nodes always make the cluster faster and better.

Reality: Adding control plane nodes improves availability but does not necessarily improve performance; it can add coordination overhead.

Why it matters: Expecting performance gains can lead to unnecessary complexity and resource use without benefits.

Quick: Can a single etcd node be enough for production? Commit yes or no.

Common Belief: One etcd node is enough if it's reliable and backed up regularly.

Reality: A single etcd node is a single point of failure; if it goes down, cluster state is lost or inaccessible.

Why it matters: Relying on one etcd node risks total cluster failure during outages.

Quick: Are worker nodes automatically highly available just by adding more nodes? Commit yes or no.

Common Belief: Adding more worker nodes automatically makes applications highly available.

Reality: Worker node availability depends on pod replication and scheduling; just adding nodes without replicas does not ensure availability.

Why it matters: Misunderstanding this can cause downtime if pods are not replicated properly.

Quick: Is network and storage redundancy optional in HA clusters? Commit yes or no.

Common Belief: Network and storage do not need special HA setup if control plane and nodes are redundant.

Reality: Network and storage must also be highly available; otherwise, failures here can cause cluster outages despite node redundancy.

Why it matters: Ignoring these layers leads to hidden single points of failure and unexpected downtime.

Expert Zone

1

etcd quorum requires an odd number of nodes to maintain consensus and avoid split-brain scenarios.

2

Load balancer health checks must be carefully configured to avoid routing traffic to unhealthy control plane nodes.

3

Pod disruption budgets help control how many pods can be down during maintenance, balancing availability and updates.

When NOT to use

High availability clusters add complexity and resource costs; for small, non-critical projects or development environments, a single control plane may suffice. Alternatives include managed Kubernetes services that handle HA automatically or simpler orchestrators for lightweight workloads.

Production Patterns

In production, HA clusters often use dedicated etcd clusters separate from control plane nodes, cloud provider load balancers with health checks, and automated monitoring with alerting. Multi-zone or multi-region clusters increase resilience against data center failures.

Connections

Distributed Consensus Algorithms

High availability clusters use distributed consensus algorithms like Raft to keep data consistent across nodes.

Understanding consensus algorithms explains how cluster state remains reliable despite node failures.

Load Balancing in Networking

Load balancers distribute client requests across multiple servers, similar to how they distribute API requests to control plane nodes.

Knowing load balancing principles helps grasp how HA clusters avoid single points of failure at the network level.

Emergency Response Teams

Like emergency teams with backups ready to act instantly, HA clusters have redundant nodes ready to take over without delay.

This cross-domain connection highlights the importance of readiness and redundancy in critical systems.

Common Pitfalls

#1 Setting up only one control plane node and assuming the cluster is highly available.

Wrong approach:

kubeadm init --pod-network-cidr=10.244.0.0/16

Correct approach:

kubeadm init --control-plane-endpoint="LOAD_BALANCER_DNS:6443" --upload-certs --pod-network-cidr=10.244.0.0/16
# Then join multiple control plane nodes with kubeadm join --control-plane ...

Root cause:

Misunderstanding that a single control plane node cannot provide high availability.

#2 Using a single etcd node without backups or clustering.

Wrong approach:

Running etcd on only one control plane node without replication.

Correct approach:

Deploying an etcd cluster with at least three nodes distributed across control plane nodes.

Root cause:

Underestimating the critical role of etcd in cluster state and availability.

#3 Not configuring a load balancer in front of control plane nodes.

Wrong approach:

Accessing control plane nodes directly via their IPs without a load balancer.

Correct approach:

Setting up a load balancer (e.g., HAProxy, NGINX, cloud LB) to route API requests to healthy control plane nodes.

Root cause:

Ignoring the need for a single stable endpoint and failover mechanism for control plane access.

Key Takeaways

High availability clusters prevent downtime by having multiple control plane and worker nodes ready to take over if one fails.

etcd is the heart of Kubernetes state and must be run as a highly available cluster itself.

A load balancer is essential to distribute requests and hide control plane node failures from clients.

Worker nodes achieve availability through pod replication and intelligent scheduling across nodes.

Network and storage redundancy are critical layers often overlooked but necessary for true high availability.

Practice

1. What is the main purpose of setting up a high availability (HA) cluster in Kubernetes?

easy

A. To prevent downtime by having multiple master nodes

B. To reduce the number of worker nodes

C. To speed up pod creation on a single node

D. To disable load balancing between nodes

Solution

1. Step 1: Understand HA cluster purpose

   High availability clusters are designed to avoid downtime by having multiple master nodes so if one fails, others take over.

2. Step 2: Compare options

   Options B, C, and D do not relate to preventing downtime or multiple masters.

3. Final Answer:

   To prevent downtime by having multiple master nodes -> Option A

4. Quick Check:

   HA cluster = multiple masters for uptime [OK]

Hint: HA means multiple masters to avoid downtime [OK]

Common Mistakes:

• Thinking HA reduces worker nodes
• Confusing HA with pod scaling
• Ignoring the role of multiple masters

2. Which of the following is the correct syntax to initialize a Kubernetes HA cluster using kubeadm with a config file named ha-config.yaml?

easy

A. kubeadm create cluster ha-config.yaml

B. kubeadm start --config=ha-config.yaml

C. kubeadm init --config ha-config.yaml

D. kubeadm init ha-config.yaml

Solution

1. Step 1: Recall kubeadm init syntax

   The correct command to initialize a cluster with a config file is kubeadm init --config filename.

2. Step 2: Check options

   kubeadm init --config ha-config.yaml matches the correct syntax. Options A, B, and D use incorrect commands or missing flags.

3. Final Answer:

   kubeadm init --config ha-config.yaml -> Option C

4. Quick Check:

   kubeadm init + --config = correct syntax [OK]

Hint: Use 'kubeadm init --config filename' to start HA cluster [OK]

Common Mistakes:

• Using 'start' instead of 'init'
• Omitting '--config' flag
• Passing config file without flag

3. Given the following HA cluster setup snippet in ha-config.yaml:

apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
controlPlaneEndpoint: "lb.example.com:6443"
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs

What does the controlPlaneEndpoint specify in this configuration?

medium

A. The IP address of the worker node

B. The port for kubelet communication

C. The DNS name of the pod network

D. The load balancer address for master nodes

Solution

1. Step 1: Understand controlPlaneEndpoint role

   This field defines the address (usually a load balancer) that routes traffic to the master nodes in an HA setup.

2. Step 2: Analyze options

   The load balancer address for master nodes correctly identifies it as the load balancer address. Other options do not relate to controlPlaneEndpoint.

3. Final Answer:

   The load balancer address for master nodes -> Option D

4. Quick Check:

   controlPlaneEndpoint = load balancer address [OK]

Hint: controlPlaneEndpoint points to the HA load balancer [OK]

Common Mistakes:

• Confusing it with worker node IP
• Thinking it is pod network DNS
• Mixing it with kubelet port

4. You tried to join a new master node to your HA cluster using this command:

kubeadm join lb.example.com:6443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:12345

But it failed with an error about missing --control-plane flag. What is the correct fix?

medium

A. Remove the token from the command

B. Add the --control-plane flag to the join command

C. Use kubeadm init instead of join

D. Change the port number to 8080

Solution

1. Step 1: Identify the error cause

   Joining a master node requires the --control-plane flag to indicate it is a control plane node.

2. Step 2: Apply the fix

   Add --control-plane to the join command to fix the error.

3. Final Answer:

   Add the --control-plane flag to the join command -> Option B

4. Quick Check:

   Joining master needs --control-plane flag [OK]

Hint: Joining master nodes requires --control-plane flag [OK]

Common Mistakes:

• Removing token breaks authentication
• Using init instead of join for adding nodes
• Changing port to wrong value

5. You want to set up a Kubernetes HA cluster with 3 master nodes behind a load balancer. Which of the following steps is the correct order to achieve this?

hard

A. Set up load balancer -> Initialize first master with kubeadm and config -> Join other masters with --control-plane -> Join worker nodes

B. Initialize all masters separately -> Set up load balancer -> Join worker nodes

C. Join worker nodes -> Initialize first master -> Set up load balancer -> Join other masters

D. Set up load balancer -> Join worker nodes -> Initialize all masters

Solution

1. Step 1: Set up load balancer first

   The load balancer must be ready to route traffic to masters before initializing the cluster.

2. Step 2: Initialize first master with kubeadm and config

   This creates the cluster control plane and configures the controlPlaneEndpoint.

3. Step 3: Join other masters with --control-plane flag

   Other masters join as control plane nodes to form HA.

4. Step 4: Join worker nodes

   Finally, worker nodes join the cluster to run workloads.

5. Final Answer:

   Set up load balancer -> Initialize first master with kubeadm and config -> Join other masters with --control-plane -> Join worker nodes -> Option A

6. Quick Check:

   Load balancer first, then masters, then workers [OK]

Hint: Load balancer first, then init masters, then join workers [OK]

Common Mistakes:

• Initializing all masters before load balancer
• Joining workers before masters
• Skipping --control-plane flag on masters