Probe failure and container restart behavior in Kubernetes - Deep Dive

Overview - Probe failure and container restart behavior

What is it?

In Kubernetes, probes are checks that monitor the health and readiness of containers. There are three main types: liveness, readiness, and startup probes. When a probe fails, Kubernetes can restart the container or stop sending traffic to it, depending on the probe type. This helps keep applications running smoothly by detecting and handling problems automatically.

Why it matters

Without probes, Kubernetes would not know if a container is healthy or ready to serve users. This could lead to broken services, slow responses, or crashes going unnoticed. Probes ensure that unhealthy containers are restarted or removed from service, improving reliability and user experience.

Where it fits

Before learning about probe failures and restarts, you should understand basic Kubernetes concepts like pods, containers, and deployments. After this, you can explore advanced topics like custom health checks, pod lifecycle management, and automated recovery strategies.

Mental Model

Core Idea

Probes are automatic health checks that tell Kubernetes when to restart or stop sending traffic to a container based on its health status.

Think of it like...

It's like a smoke detector in your home: if it senses smoke (a problem), it triggers an alarm (restart or stop traffic) to prevent bigger damage.

┌───────────────┐
│   Kubernetes  │
│   Pod with    │
│   Container   │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│   Probes      │
│ ┌───────────┐ │
│ │Liveness   │ │
│ │Readiness  │ │
│ │Startup    │ │
│ └───────────┘ │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Actions on    │
│ Probe Failure │
│ - Restart     │
│ - Stop Traffic│
└───────────────┘

Build-Up - 7 Steps

1

Foundation

Understanding Kubernetes Probes Basics

Concept: Introduce what probes are and their types in Kubernetes.

Kubernetes uses probes to check container health. There are three types: - Liveness probe: checks if the container is alive. If it fails, Kubernetes restarts the container. - Readiness probe: checks if the container is ready to accept traffic. If it fails, Kubernetes stops sending traffic. - Startup probe: checks if the container has started properly. It helps avoid premature restarts during slow startups.

Result

Learners understand the purpose and types of probes in Kubernetes.

Knowing the different probes and their roles is key to managing container health and traffic flow effectively.

2

Foundation

How Kubernetes Handles Probe Failures

Concept: Explain what happens when each probe fails.

When a liveness probe fails, Kubernetes restarts the container to try to fix the problem. When a readiness probe fails, Kubernetes removes the container from the service endpoints, so it stops receiving traffic but is not restarted. When a startup probe fails repeatedly, Kubernetes restarts the container, but it waits longer before deciding to restart compared to liveness probes.

Result

Learners see the direct consequences of probe failures on container lifecycle and traffic.

Understanding failure responses helps prevent downtime and ensures smooth user experience.

3

Intermediate

Configuring Probe Failure Thresholds

🤔Before reading on: do you think increasing failure thresholds makes containers restart faster or slower? Commit to your answer.

Concept: Introduce probe configuration options that control failure sensitivity.

Probes have settings like: - failureThreshold: how many times a probe can fail before action is taken. - periodSeconds: how often the probe runs. - initialDelaySeconds: delay before the first probe runs. Increasing failureThreshold or initialDelaySeconds makes Kubernetes wait longer before restarting or removing containers.

Result

Learners can tune probe sensitivity to avoid false restarts or traffic removal.

Knowing how to configure thresholds prevents unnecessary restarts and improves stability.

4

Intermediate

Differences Between Liveness and Readiness Probes

🤔Before reading on: do you think a failing readiness probe causes a container restart? Commit to yes or no.

Concept: Clarify the distinct roles and effects of liveness vs readiness probes.

Liveness probes detect if a container is dead or stuck and trigger restarts. Readiness probes detect if a container can serve traffic and control load balancing. A failing readiness probe does NOT restart the container; it just stops traffic to it. This separation allows graceful handling of temporary issues without downtime.

Result

Learners understand how Kubernetes balances availability and recovery.

Distinguishing these probes helps design resilient applications that recover smoothly.

5

Advanced

Startup Probe Role in Slow-Starting Containers

🤔Before reading on: do you think startup probes replace liveness probes or work alongside them? Commit to your answer.

Concept: Explain how startup probes prevent premature restarts during container startup.

Startup probes run during container startup to check if the app is initializing properly. They disable liveness probes until startup succeeds. This avoids restarts caused by slow startups that liveness probes might misinterpret as failure. Once startup probe passes, liveness and readiness probes take over.

Result

Learners can handle slow-starting containers without false restarts.

Understanding startup probes prevents common issues with complex or heavy initialization.

6

Expert

Probe Failure Impact on Stateful Applications

🤔Before reading on: do you think restarting a stateful container always fixes issues? Commit to yes or no.

Concept: Explore how probe failures and restarts affect stateful workloads and data consistency.

Stateful applications like databases may lose data or corrupt state if restarted abruptly. Probe failures triggering restarts can cause data loss or downtime. Experts design probes carefully, sometimes disabling liveness probes or using readiness probes only. They also implement graceful shutdown hooks and persistent storage to protect data.

Result

Learners appreciate the complexity of probe use in stateful systems.

Knowing probe impact on stateful apps guides safer production configurations.

7

Expert

Advanced Probe Failure Patterns and Recovery

🤔Before reading on: do you think combining multiple probes can improve reliability? Commit to your answer.

Concept: Discuss complex probe strategies and recovery mechanisms in production.

Experts combine probes with custom scripts or HTTP checks for precise health signals. They use different thresholds for liveness and readiness probes. They integrate probes with Kubernetes events and alerts for faster incident response. Some use external monitoring to complement probes. This layered approach improves fault detection and recovery.

Result

Learners see how to build robust health monitoring and recovery in Kubernetes.

Understanding advanced patterns helps build resilient, self-healing systems.

Under the Hood

Kubernetes runs probe commands or HTTP requests inside or against containers at configured intervals. The kubelet process on each node manages these probes. When a probe fails consecutively beyond the failureThreshold, kubelet triggers actions: restarting the container for liveness or removing it from service endpoints for readiness. Startup probes temporarily disable liveness probes until success. This mechanism relies on periodic health checks and kubelet's control loop to maintain container health.

Why designed this way?

Probes were designed to automate container health management without manual intervention. Separating liveness and readiness allows Kubernetes to distinguish between a container that is dead and one that is temporarily unable to serve traffic. Startup probes address the problem of slow-starting containers causing premature restarts. This design balances availability, reliability, and recovery, avoiding unnecessary restarts while ensuring unhealthy containers are fixed.

┌───────────────┐
│   kubelet     │
│  (node agent) │
└──────┬────────┘
       │ runs probes
       ▼
┌───────────────┐
│ Container     │
│ ┌───────────┐ │
│ │ Liveness  │ │
│ │ Readiness │ │
│ │ Startup   │ │
│ └───────────┘ │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Probe Results │
│ - Success     │
│ - Failure     │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ kubelet Acts  │
│ - Restart     │
│ - Remove from │
│   Service     │
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does a failing readiness probe cause Kubernetes to restart the container? Commit yes or no.

Common Belief: A failing readiness probe causes the container to restart immediately.

Reality: Readiness probe failures only stop traffic to the container; they do not trigger restarts.

Why it matters: Misunderstanding this can lead to incorrect troubleshooting and unnecessary configuration changes.

Quick: Do startup probes replace liveness probes after startup? Commit yes or no.

Common Belief: Startup probes permanently replace liveness probes once the container starts.

Reality: Startup probes only run during startup; after success, liveness probes resume monitoring.

Why it matters: Confusing this can cause gaps in health monitoring after startup.

Quick: Does increasing failureThreshold always improve stability? Commit yes or no.

Common Belief: Increasing failureThreshold always makes the system more stable by avoiding restarts.

Reality: Too high failureThreshold delays detection of real failures, causing longer downtime.

Why it matters: Balancing thresholds is critical; too high or too low harms availability.

Quick: Can restarting a stateful container always fix probe failures? Commit yes or no.

Common Belief: Restarting a stateful container always resolves health issues.

Reality: Restarting stateful containers can cause data loss or corruption if not handled carefully.

Why it matters: Ignoring this can lead to serious production outages and data integrity problems.

Expert Zone

1

Liveness probes should be designed to detect only unrecoverable failures to avoid unnecessary restarts.

2

Readiness probes can be used to implement rolling updates by controlling traffic flow during deployment.

3

Startup probes help avoid the 'crash loop backoff' problem common in containers with heavy initialization.

When NOT to use

Avoid using liveness probes that restart stateful containers without graceful shutdown logic; instead, rely on readiness probes and external monitoring. For very simple or short-lived containers, probes might be unnecessary and add overhead.

Production Patterns

In production, teams often combine HTTP GET readiness probes with custom command liveness probes. They tune failureThreshold and periodSeconds based on app behavior. They integrate probe events with alerting systems and use readiness probes to manage traffic during deployments and scaling.

Connections

Load Balancing

Probes control traffic routing by signaling readiness, which directly affects load balancing decisions.

Understanding probe readiness helps grasp how Kubernetes ensures traffic only goes to healthy containers, improving system reliability.

Fault Tolerance in Distributed Systems

Probe failure handling is a form of automated fault detection and recovery, a core principle in fault-tolerant systems.

Knowing how Kubernetes probes work deepens understanding of how distributed systems maintain availability despite failures.

Medical Diagnostics

Probes are like medical tests that check patient health and decide treatment steps based on results.

This connection highlights the importance of timely and accurate health checks to prevent worsening conditions, similar to container health management.

Common Pitfalls

#1 Using a liveness probe that restarts a container too aggressively during normal temporary delays.

Wrong approach:

livenessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
failureThreshold: 1

Correct approach:

livenessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 30
periodSeconds: 10
failureThreshold: 3

Root cause:

Misunderstanding container startup time and probe sensitivity causes premature restarts.

#2 Configuring readiness probe to restart container on failure.

Wrong approach:

readinessProbe:
exec:
command: ["/bin/check_ready.sh"]
failureThreshold: 1
restartPolicy: Always

Correct approach:

readinessProbe:
exec:
command: ["/bin/check_ready.sh"]
failureThreshold: 3
# No restartPolicy here; readiness probe does not trigger restarts

Root cause:

Confusing readiness probe behavior with liveness probe leads to wrong assumptions about restarts.

#3 Not using startup probe for slow-starting containers, causing crash loops.

Wrong approach:

livenessProbe:
tcpSocket:
port: 3306
initialDelaySeconds: 10
periodSeconds: 5
failureThreshold: 3

Correct approach:

startupProbe:
tcpSocket:
port: 3306
failureThreshold: 30
periodSeconds: 10
livenessProbe:
tcpSocket:
port: 3306
initialDelaySeconds: 0
periodSeconds: 10
failureThreshold: 3

Root cause:

Ignoring startup probe leads to liveness probe restarting container before it finishes starting.

Key Takeaways

Kubernetes probes are essential tools that monitor container health and control restarts and traffic flow.

Liveness probes trigger container restarts on failure, while readiness probes control traffic without restarting.

Startup probes prevent premature restarts during slow container startups by temporarily disabling liveness probes.

Proper configuration of probe thresholds and timing is critical to avoid false positives and unnecessary restarts.

In stateful applications, probe failures and restarts must be handled carefully to prevent data loss or corruption.

Practice

1. What happens when a Kubernetes liveness probe fails for a container?

easy

A. Traffic to the container is stopped but it is not restarted.

B. The container is restarted by Kubernetes.

C. The container continues running without changes.

D. The container is removed permanently.

Solution

1. Step 1: Understand liveness probe purpose

   Liveness probes check if a container is alive and healthy.

2. Step 2: Effect of liveness probe failure

   If the liveness probe fails, Kubernetes restarts the container to fix the issue.

3. Final Answer:

   The container is restarted by Kubernetes. -> Option B

4. Quick Check:

   Liveness probe failure = container restart [OK]

Hint: Liveness probe failure always triggers container restart [OK]

Common Mistakes:

• Confusing liveness with readiness probe effects
• Thinking container is deleted permanently
• Assuming no action on failure

2. Which of the following is the correct YAML snippet to define a liveness probe using HTTP GET on path /health at port 8080?

easy

A. livenessProbe:
httpGet:
path: health
port: 8080
initialDelaySeconds: 5
periodSeconds: 10

B. livenessProbe:
exec:
command: ["curl", "/health"]
initialDelaySeconds: 5
periodSeconds: 10

C. livenessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 5
periodSeconds: 10

D. livenessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 5
periodSeconds: 10

Solution

1. Step 1: Check HTTP GET probe syntax

   The correct syntax uses httpGet with path starting with a slash and port as a number.

2. Step 2: Validate each option

   livenessProbe: httpGet: path: /health port: 8080 initialDelaySeconds: 5 periodSeconds: 10 correctly uses /health and port 8080. livenessProbe: exec: command: ["curl", "/health"] initialDelaySeconds: 5 periodSeconds: 10 uses exec with curl incorrectly. livenessProbe: tcpSocket: port: 8080 initialDelaySeconds: 5 periodSeconds: 10 uses tcpSocket which is different. livenessProbe: httpGet: path: health port: 8080 initialDelaySeconds: 5 periodSeconds: 10 misses the leading slash in path.

3. Final Answer:

   livenessProbe: httpGet: path: /health port: 8080 initialDelaySeconds: 5 periodSeconds: 10 -> Option D

4. Quick Check:

   HTTP GET probe with correct path and port = livenessProbe: httpGet: path: /health port: 8080 initialDelaySeconds: 5 periodSeconds: 10 [OK]

Hint: HTTP path must start with '/' and port is a number [OK]

Common Mistakes:

• Missing leading slash in path
• Using exec command incorrectly
• Confusing tcpSocket with httpGet

3. Given this pod spec snippet, what happens if the readiness probe fails but the liveness probe passes?

readinessProbe:
  httpGet:
    path: /ready
    port: 80
  initialDelaySeconds: 5
  periodSeconds: 10
livenessProbe:
  httpGet:
    path: /live
    port: 80
  initialDelaySeconds: 5
  periodSeconds: 10

medium

A. The container is restarted immediately.

B. The container is deleted and recreated.

C. The container is marked not ready and removed from service endpoints.

D. Nothing happens; container continues serving traffic.

Solution

1. Step 1: Understand readiness probe role

   Readiness probe controls if the container receives traffic by marking it ready or not.

2. Step 2: Effect of readiness probe failure

   If readiness probe fails but liveness probe passes, container stays running but is removed from service endpoints, so no traffic is sent to it.

3. Final Answer:

   The container is marked not ready and removed from service endpoints. -> Option C

4. Quick Check:

   Readiness fail = no traffic, container runs [OK]

Hint: Readiness failure stops traffic, liveness failure restarts container [OK]

Common Mistakes:

• Confusing readiness failure with container restart
• Thinking container is deleted
• Assuming no effect on traffic

4. You notice your container keeps restarting frequently. The liveness probe is configured with initialDelaySeconds: 5 and periodSeconds: 10. What is a likely cause and fix?

medium

A. The initial delay is too short; increase initialDelaySeconds to allow startup time.

B. The period is too long; decrease periodSeconds to check faster.

C. The probe path is incorrect; change it to /healthz.

D. The container image is corrupted; rebuild the image.

Solution

1. Step 1: Analyze probe timing settings

   If initialDelaySeconds is too short, the probe runs before the app is ready, causing false failures.

2. Step 2: Adjust initial delay

   Increasing initialDelaySeconds gives the container time to start before probing, preventing unnecessary restarts.

3. Final Answer:

   The initial delay is too short; increase initialDelaySeconds to allow startup time. -> Option A

4. Quick Check:

   Short initialDelay causes early probe failure [OK]

Hint: Increase initialDelaySeconds if container restarts too soon [OK]

Common Mistakes:

• Changing periodSeconds instead of initialDelaySeconds
• Assuming probe path is always wrong
• Blaming container image without checking probes

5. You have a container with both readiness and liveness probes. The readiness probe fails intermittently, but the liveness probe always passes. How should Kubernetes behave and what is the best way to improve user experience?

hard

A. Kubernetes removes the pod from service during readiness failure but does not restart it; improve by tuning readiness probe thresholds.

B. Kubernetes restarts the container on readiness failure; improve by disabling readiness probe.

C. Kubernetes ignores readiness probe failures if liveness passes; no action needed.

D. Kubernetes deletes the pod on readiness failure; improve by increasing liveness probe timeout.

Solution

1. Step 1: Understand probe roles on intermittent readiness failure

   Readiness failure removes pod from service endpoints but does not restart container if liveness passes.

2. Step 2: Improve user experience by tuning readiness probe

   Tuning thresholds like failureThreshold or periodSeconds reduces false negatives and prevents unnecessary traffic disruption.

3. Final Answer:

   Kubernetes removes the pod from service during readiness failure but does not restart it; improve by tuning readiness probe thresholds. -> Option A

4. Quick Check:

   Readiness fail = no traffic; tune probe to reduce false failures [OK]

Hint: Tune readiness probe to avoid false failures and traffic loss [OK]

Common Mistakes:

• Thinking readiness failure restarts container
• Disabling readiness probe instead of tuning
• Ignoring intermittent failures