Desired state vs actual state reconciliation in Kubernetes - Hands-On Comparison

Desired State vs Actual State Reconciliation in Kubernetes

📖 Scenario: You are managing a small web application running on Kubernetes. You want to ensure that the number of running pods always matches the number you specify in your deployment configuration. This is called reconciliation, where Kubernetes compares the desired state you set with the actual state running in the cluster and makes changes to match them.

🎯 Goal: Learn how to define a Kubernetes Deployment manifest with a specific number of replicas (desired state), check the actual number of running pods, and understand how Kubernetes reconciles the difference automatically.

📋 What You'll Learn

Create a Kubernetes Deployment manifest with exactly 3 replicas

Add a label selector to the Deployment to identify pods

Use kubectl commands to check the actual number of running pods

Observe how Kubernetes reconciles the actual state to match the desired state

💡 Why This Matters

🌍 Real World

Kubernetes operators and DevOps engineers use reconciliation to keep applications running as expected, automatically fixing differences between desired and actual states.

💼 Career

Understanding reconciliation is essential for managing Kubernetes workloads reliably and troubleshooting deployment issues in real-world cloud environments.

1

Create a Deployment manifest with 3 replicas

Create a YAML file named deployment.yaml with a Deployment resource. Set replicas to 3. Use app: mywebapp as the label under metadata.labels and spec.selector.matchLabels. Use the container image nginx:latest.

Kubernetes

# Create deployment.yaml with Deployment, replicas: 3, labels: app: mywebapp, image: nginx:latest
# Your code here

Hint

Make sure the replicas field is set to 3 and the labels match exactly app: mywebapp.

2

Add a label selector to the Deployment

Ensure the Deployment manifest has a spec.selector.matchLabels field with app: mywebapp to select pods correctly. This helps Kubernetes know which pods belong to this Deployment.

Kubernetes

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mywebapp-deployment
  labels:
    app: mywebapp
spec:
  replicas: 3
  # Add selector.matchLabels: app: mywebapp below
  selector:
    matchLabels:
      app: mywebapp
  template:
    metadata:
      labels:
        app: mywebapp
    spec:
      containers:
      - name: nginx-container
        image: nginx:latest

Hint

The selector.matchLabels must exactly match the pod template labels.

3

Check the actual number of running pods

After applying the Deployment with kubectl apply -f deployment.yaml, run the command kubectl get pods -l app=mywebapp to list pods with the label app=mywebapp. Count how many pods are in the Running state.

Kubernetes

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mywebapp-deployment
  labels:
    app: mywebapp
spec:
  replicas: 3
  selector:
    matchLabels:
      app: mywebapp
  template:
    metadata:
      labels:
        app: mywebapp
    spec:
      containers:
      - name: nginx-container
        image: nginx:latest

# Run kubectl get pods -l app=mywebapp to check running pods
# Your code here

Hint

Use the label selector -l app=mywebapp to filter pods belonging to your Deployment.

4

Observe reconciliation by scaling replicas

Run kubectl scale deployment mywebapp-deployment --replicas=5 to change the desired state to 5 replicas. Then run kubectl get pods -l app=mywebapp --no-headers | wc -l to count how many pods are running. Kubernetes will create new pods to match the desired state.

Kubernetes

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mywebapp-deployment
  labels:
    app: mywebapp
spec:
  replicas: 3
  selector:
    matchLabels:
      app: mywebapp
  template:
    metadata:
      labels:
        app: mywebapp
    spec:
      containers:
      - name: nginx-container
        image: nginx:latest

# Scale deployment to 5 replicas
kubectl scale deployment mywebapp-deployment --replicas=5
# Count running pods
kubectl get pods -l app=mywebapp --no-headers | wc -l
# Your code here

Hint

Scaling changes the desired state. Kubernetes will create or delete pods to match it.

Practice

1. What does Kubernetes do when the actual state of a pod differs from the desired state?

easy

A. It changes the actual state to match the desired state automatically.

B. It deletes the pod without recreating it.

C. It ignores the difference and does nothing.

D. It sends an alert but does not change anything.

Solution

1. Step 1: Understand desired vs actual state

   The desired state is what you want Kubernetes to have running, and the actual state is what is currently running.

2. Step 2: Kubernetes reconciliation process

   Kubernetes continuously compares actual state to desired state and makes changes to fix any differences automatically.

3. Final Answer:

   It changes the actual state to match the desired state automatically. -> Option A

4. Quick Check:

   Reconciliation means fixing actual state to desired state [OK]

Hint: Kubernetes fixes differences automatically [OK]

Common Mistakes:

• Thinking Kubernetes ignores differences
• Believing Kubernetes only alerts without fixing
• Assuming Kubernetes deletes pods without recreating

2. Which Kubernetes component is responsible for ensuring the actual state matches the desired state?

easy

A. kube-apiserver

B. kube-scheduler

C. etcd

D. kube-controller-manager

Solution

1. Step 1: Identify the role of kube-controller-manager

   The kube-controller-manager runs controllers that watch the desired state and make changes to match the actual state.

2. Step 2: Differentiate other components

   kube-apiserver handles API requests, kube-scheduler assigns pods to nodes, and etcd stores cluster data but does not reconcile states.

3. Final Answer:

   kube-controller-manager -> Option D

4. Quick Check:

   Controllers fix state differences [OK]

Hint: Controllers fix state differences [OK]

Common Mistakes:

• Confusing scheduler with controller
• Thinking apiserver reconciles state
• Assuming etcd manages reconciliation

3. Given this YAML snippet for a Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: web
        image: nginx:1.19

If one pod crashes and stops running, what will Kubernetes do?

medium

A. Create a new pod to maintain 3 replicas.

B. Do nothing and keep 2 pods running.

C. Delete all pods and restart the deployment.

D. Scale down the deployment to 2 replicas.

Solution

1. Step 1: Understand the desired replicas

   The Deployment specifies 3 replicas, so Kubernetes wants 3 pods running.

2. Step 2: Reconciliation after pod crash

   If one pod crashes, Kubernetes detects the actual state has only 2 pods and creates a new pod to restore 3 replicas.

3. Final Answer:

   Create a new pod to maintain 3 replicas. -> Option A

4. Quick Check:

   Pods count fixed to desired replicas [OK]

Hint: Pods auto-recreated to match replicas [OK]

Common Mistakes:

• Thinking Kubernetes ignores crashed pods
• Assuming deployment restarts all pods
• Believing deployment scales down automatically

4. You notice a Deployment is stuck with fewer pods than desired. Which of these is a likely cause preventing reconciliation?

medium

A. The kube-apiserver is running normally.

B. The image name in the container spec is correct.

C. The Deployment's selector labels do not match pod labels.

D. The desired replicas count is set to 3.

Solution

1. Step 1: Check label selectors for matching pods

   If the Deployment's selector labels do not match pod labels, Kubernetes cannot track pods correctly, so reconciliation fails.

2. Step 2: Understand other options

   Correct image name, running apiserver, and desired replicas do not prevent reconciliation by themselves.

3. Final Answer:

   The Deployment's selector labels do not match pod labels. -> Option C

4. Quick Check:

   Label mismatch blocks reconciliation [OK]

Hint: Selector and pod labels must match [OK]

Common Mistakes:

• Ignoring label selector importance
• Blaming image name for pod count issues
• Assuming apiserver status causes pod count

5. You want to update a Deployment's container image from nginx:1.19 to nginx:1.21. After applying the change, you see some pods still run the old image. What explains this behavior in terms of desired vs actual state reconciliation?

hard

A. Kubernetes ignores image changes unless manually restarted.

B. Kubernetes gradually updates pods to match desired state using rolling updates.

C. Kubernetes cannot update pods without deleting the Deployment.

D. Kubernetes immediately deletes all old pods and creates new ones.

Solution

1. Step 1: Understand rolling update strategy

   Kubernetes uses rolling updates to gradually replace old pods with new ones to avoid downtime.

2. Step 2: Explain why old pods remain temporarily

   During rolling update, some pods run old image until replaced, so actual state temporarily differs from desired state.

3. Final Answer:

   Kubernetes gradually updates pods to match desired state using rolling updates. -> Option B

4. Quick Check:

   Rolling update means gradual pod replacement [OK]

Hint: Rolling updates replace pods gradually [OK]

Common Mistakes:

• Expecting instant pod replacement
• Thinking Deployment must be deleted to update
• Believing Kubernetes ignores image changes