Probe failure and container restart behavior in Kubernetes - Commands & Configuration

Introduction

Sometimes containers in Kubernetes stop working properly. Probes help check if a container is healthy. If a probe fails, Kubernetes can restart the container to fix problems automatically.

When you want to automatically restart a web server if it stops responding.

When you need to check if a database container is ready before sending traffic.

When you want to detect if a container is stuck and recover it without manual intervention.

When you want to avoid sending requests to a container that is not ready to serve.

When you want to monitor container health and improve application reliability.

Config File - pod.yaml

pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: example-pod
spec:
  containers:
  - name: example-container
    image: nginx:1.23
    livenessProbe:
      httpGet:
        path: /healthz
        port: 80
      initialDelaySeconds: 5
      periodSeconds: 5
      failureThreshold: 3
    readinessProbe:
      httpGet:
        path: /ready
        port: 80
      initialDelaySeconds: 3
      periodSeconds: 5
      failureThreshold: 3

This pod runs an nginx container with two probes:

• livenessProbe: Checks if the container is alive by requesting /healthz on port 80. If it fails 3 times, Kubernetes restarts the container.
• readinessProbe: Checks if the container is ready to accept traffic by requesting /ready on port 80. If it fails, the container is removed from service endpoints.
• initialDelaySeconds: Wait time before starting probes after container starts.
• periodSeconds: How often to run the probe.
• failureThreshold: Number of failures before action.

Commands

This command creates the pod with the configured liveness and readiness probes. Kubernetes starts the container and begins health checks.

Terminal

kubectl apply -f pod.yaml

Expected Output

pod/example-pod created

Check the pod status to see if it is running and ready after probes start working.

Terminal

kubectl get pods example-pod

Expected Output

NAME          READY   STATUS    RESTARTS   AGE

example-pod   1/1     Running   0          10s

View detailed pod information including probe results and container restarts if any probe failed.

Terminal

kubectl describe pod example-pod

Expected Output

Name:         example-pod

Namespace:    default

Status:       Running

Containers:

  example-container:

    Container ID:   docker://abcdef123456

    Image:          nginx:1.23

    State:          Running

    Ready:          True

    Restart Count:  0

    Liveness:       http-get http://:80/healthz delay=5s timeout=1s period=5s #success=1 #failure=3

    Readiness:      http-get http://:80/ready delay=3s timeout=1s period=5s #success=1 #failure=3

Events:

  Type    Reason     Age   From               Message

  ----    ------     ----  ----               -------

  Normal  Scheduled  1m    default-scheduler  Successfully assigned default/example-pod to node-1

Key Concept

If a liveness probe fails repeatedly, Kubernetes restarts the container to fix it automatically.

Common Mistakes

Not setting initialDelaySeconds causing probes to run before the container is ready.

Probes may fail immediately and cause unnecessary restarts.

Set initialDelaySeconds to allow the container time to start before probes begin.

Using the same probe for both readiness and liveness checks.

Readiness and liveness have different purposes; mixing them can cause traffic routing issues or restarts.

Configure separate readiness and liveness probes tailored to their specific checks.

Setting failureThreshold too low causing frequent restarts on temporary glitches.

Containers may restart unnecessarily, causing instability.

Choose a reasonable failureThreshold to tolerate brief failures before restarting.

Summary

Create a pod with liveness and readiness probes to monitor container health.

Use kubectl apply to deploy the pod and kubectl get pods to check status.

Describe the pod to see probe results and container restart counts.