Probe failure and container restart behavior in Kubernetes - Mini Project: Build & Apply

Probe failure and container restart behavior

📖 Scenario: You are managing a Kubernetes cluster running a simple web application. You want to understand how Kubernetes handles container health checks and restarts when probes fail.

🎯 Goal: Learn how to configure liveness and readiness probes in a Kubernetes Pod manifest and observe container restart behavior when probes fail.

📋 What You'll Learn

Create a Pod manifest with a container running the nginx image

Add a liveness probe that checks the HTTP endpoint /healthz on port 80

Add a readiness probe that checks the HTTP endpoint /ready on port 80

Simulate probe failure by using incorrect probe paths

Observe and print the Pod status to see container restart behavior

💡 Why This Matters

🌍 Real World

Kubernetes uses probes to check if containers are healthy and ready to serve traffic. Understanding probe failures helps keep applications stable.

💼 Career

DevOps engineers and site reliability engineers configure probes to automate container health checks and manage restarts, improving uptime.

1

Create a basic Pod manifest with nginx container

Create a Kubernetes Pod manifest named nginx-pod.yaml with a single container named nginx using the image nginx:latest. The container should expose port 80.

Kubernetes

# Write the Pod manifest YAML with apiVersion, kind, metadata, and spec including the nginx container

Hint

Use apiVersion: v1 and kind: Pod. Define metadata.name as nginx-pod. Under spec.containers, add one container named nginx with image nginx:latest and expose port 80.

2

Add liveness and readiness probes with incorrect paths

Add a livenessProbe and a readinessProbe to the nginx container. Both probes should use httpGet on port 80. Set the livenessProbe path to /healthz and the readinessProbe path to /ready. Use default probe settings for initial delay and period.

Kubernetes

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
spec:
  containers:
  - name: nginx
    image: nginx:latest
    ports:
    - containerPort: 80
    # Add livenessProbe and readinessProbe below

Hint

Under the container spec, add livenessProbe and readinessProbe sections. Use httpGet with path and port fields. Use paths /healthz and /ready respectively.

3

Apply the Pod manifest and observe container restart behavior

Apply the Pod manifest using kubectl apply -f nginx-pod.yaml. Then use kubectl get pods nginx-pod -o jsonpath='{.status.containerStatuses[0].restartCount}' to check the container restart count. Write a command to get the restart count.

Kubernetes

# Apply the Pod manifest and write the command to get the restart count

Hint

Use kubectl apply -f nginx-pod.yaml to create the Pod. Then use kubectl get pods nginx-pod -o jsonpath='{.status.containerStatuses[0].restartCount}' to see how many times the container restarted.

4

Fix probe paths and verify no restarts

Edit the Pod manifest to change the livenessProbe and readinessProbe paths to / (root path). Apply the updated manifest with kubectl apply -f nginx-pod.yaml. Then print the restart count again using kubectl get pods nginx-pod -o jsonpath='{.status.containerStatuses[0].restartCount}'. Write the command to print the restart count.

Kubernetes

# Change probe paths to '/' and apply the manifest
# Then print the restart count again

Hint

Change both probe paths to / which is a valid path for nginx. Apply the manifest again and check the restart count. It should not increase if probes succeed.

Practice

1. What happens when a Kubernetes liveness probe fails for a container?

easy

A. Traffic to the container is stopped but it is not restarted.

B. The container is restarted by Kubernetes.

C. The container continues running without changes.

D. The container is removed permanently.

Solution

1. Step 1: Understand liveness probe purpose

   Liveness probes check if a container is alive and healthy.

2. Step 2: Effect of liveness probe failure

   If the liveness probe fails, Kubernetes restarts the container to fix the issue.

3. Final Answer:

   The container is restarted by Kubernetes. -> Option B

4. Quick Check:

   Liveness probe failure = container restart [OK]

Hint: Liveness probe failure always triggers container restart [OK]

Common Mistakes:

• Confusing liveness with readiness probe effects
• Thinking container is deleted permanently
• Assuming no action on failure

2. Which of the following is the correct YAML snippet to define a liveness probe using HTTP GET on path /health at port 8080?

easy

A. livenessProbe:
httpGet:
path: health
port: 8080
initialDelaySeconds: 5
periodSeconds: 10

B. livenessProbe:
exec:
command: ["curl", "/health"]
initialDelaySeconds: 5
periodSeconds: 10

C. livenessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 5
periodSeconds: 10

D. livenessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 5
periodSeconds: 10

Solution

1. Step 1: Check HTTP GET probe syntax

   The correct syntax uses httpGet with path starting with a slash and port as a number.

2. Step 2: Validate each option

   livenessProbe: httpGet: path: /health port: 8080 initialDelaySeconds: 5 periodSeconds: 10 correctly uses /health and port 8080. livenessProbe: exec: command: ["curl", "/health"] initialDelaySeconds: 5 periodSeconds: 10 uses exec with curl incorrectly. livenessProbe: tcpSocket: port: 8080 initialDelaySeconds: 5 periodSeconds: 10 uses tcpSocket which is different. livenessProbe: httpGet: path: health port: 8080 initialDelaySeconds: 5 periodSeconds: 10 misses the leading slash in path.

3. Final Answer:

   livenessProbe: httpGet: path: /health port: 8080 initialDelaySeconds: 5 periodSeconds: 10 -> Option D

4. Quick Check:

   HTTP GET probe with correct path and port = livenessProbe: httpGet: path: /health port: 8080 initialDelaySeconds: 5 periodSeconds: 10 [OK]

Hint: HTTP path must start with '/' and port is a number [OK]

Common Mistakes:

• Missing leading slash in path
• Using exec command incorrectly
• Confusing tcpSocket with httpGet

3. Given this pod spec snippet, what happens if the readiness probe fails but the liveness probe passes?

readinessProbe:
  httpGet:
    path: /ready
    port: 80
  initialDelaySeconds: 5
  periodSeconds: 10
livenessProbe:
  httpGet:
    path: /live
    port: 80
  initialDelaySeconds: 5
  periodSeconds: 10

medium

A. The container is restarted immediately.

B. The container is deleted and recreated.

C. The container is marked not ready and removed from service endpoints.

D. Nothing happens; container continues serving traffic.

Solution

1. Step 1: Understand readiness probe role

   Readiness probe controls if the container receives traffic by marking it ready or not.

2. Step 2: Effect of readiness probe failure

   If readiness probe fails but liveness probe passes, container stays running but is removed from service endpoints, so no traffic is sent to it.

3. Final Answer:

   The container is marked not ready and removed from service endpoints. -> Option C

4. Quick Check:

   Readiness fail = no traffic, container runs [OK]

Hint: Readiness failure stops traffic, liveness failure restarts container [OK]

Common Mistakes:

• Confusing readiness failure with container restart
• Thinking container is deleted
• Assuming no effect on traffic

4. You notice your container keeps restarting frequently. The liveness probe is configured with initialDelaySeconds: 5 and periodSeconds: 10. What is a likely cause and fix?

medium

A. The initial delay is too short; increase initialDelaySeconds to allow startup time.

B. The period is too long; decrease periodSeconds to check faster.

C. The probe path is incorrect; change it to /healthz.

D. The container image is corrupted; rebuild the image.

Solution

1. Step 1: Analyze probe timing settings

   If initialDelaySeconds is too short, the probe runs before the app is ready, causing false failures.

2. Step 2: Adjust initial delay

   Increasing initialDelaySeconds gives the container time to start before probing, preventing unnecessary restarts.

3. Final Answer:

   The initial delay is too short; increase initialDelaySeconds to allow startup time. -> Option A

4. Quick Check:

   Short initialDelay causes early probe failure [OK]

Hint: Increase initialDelaySeconds if container restarts too soon [OK]

Common Mistakes:

• Changing periodSeconds instead of initialDelaySeconds
• Assuming probe path is always wrong
• Blaming container image without checking probes

5. You have a container with both readiness and liveness probes. The readiness probe fails intermittently, but the liveness probe always passes. How should Kubernetes behave and what is the best way to improve user experience?

hard

A. Kubernetes removes the pod from service during readiness failure but does not restart it; improve by tuning readiness probe thresholds.

B. Kubernetes restarts the container on readiness failure; improve by disabling readiness probe.

C. Kubernetes ignores readiness probe failures if liveness passes; no action needed.

D. Kubernetes deletes the pod on readiness failure; improve by increasing liveness probe timeout.

Solution

1. Step 1: Understand probe roles on intermittent readiness failure

   Readiness failure removes pod from service endpoints but does not restart container if liveness passes.

2. Step 2: Improve user experience by tuning readiness probe

   Tuning thresholds like failureThreshold or periodSeconds reduces false negatives and prevents unnecessary traffic disruption.

3. Final Answer:

   Kubernetes removes the pod from service during readiness failure but does not restart it; improve by tuning readiness probe thresholds. -> Option A

4. Quick Check:

   Readiness fail = no traffic; tune probe to reduce false failures [OK]

Hint: Tune readiness probe to avoid false failures and traffic loss [OK]

Common Mistakes:

• Thinking readiness failure restarts container
• Disabling readiness probe instead of tuning
• Ignoring intermittent failures