Exec probe configuration in Kubernetes - Deep Dive

Overview - Exec probe configuration

What is it?

Exec probe configuration in Kubernetes is a way to check the health of a container by running a command inside it. Kubernetes runs this command periodically to see if the container is working properly. If the command succeeds, the container is considered healthy; if it fails, Kubernetes can restart or stop the container. This helps keep applications running smoothly without manual checks.

Why it matters

Without exec probes, Kubernetes would not know if a container is stuck or broken inside. This could lead to bad user experiences or downtime because unhealthy containers keep running. Exec probes automate health checks, so Kubernetes can fix problems quickly by restarting containers. This makes applications more reliable and reduces manual work for operators.

Where it fits

Before learning exec probes, you should understand basic Kubernetes concepts like pods and containers. After this, you can learn about other probe types like HTTP and TCP probes, and then explore advanced health check strategies and custom readiness checks.

Mental Model

Core Idea

An exec probe runs a command inside a container to check if it is healthy and responsive.

Think of it like...

It's like a doctor checking your heartbeat by listening inside your chest; if the heartbeat is normal, you are healthy, otherwise you need attention.

┌─────────────────────────────┐
│ Kubernetes Node             │
│ ┌─────────────────────────┐ │
│ │ Pod                     │ │
│ │ ┌─────────────────────┐ │ │
│ │ │ Container           │ │ │
│ │ │ ┌───────────────┐  │ │ │
│ │ │ │ Exec Probe    │──┼─┼─┤ Runs command inside container
│ │ │ └───────────────┘  │ │ │
│ │ └─────────────────────┘ │ │
│ └─────────────────────────┘ │
└─────────────────────────────┘

Build-Up - 7 Steps

1

Foundation

What is an Exec Probe

Concept: Exec probe is a Kubernetes feature that runs a command inside a container to check its health.

In Kubernetes, an exec probe runs a specified command inside a container. If the command exits with status 0, the container is healthy. If it exits with any other status, the container is unhealthy. This check happens regularly based on configured intervals.

Result

Kubernetes knows if the container is healthy or not by the command's exit status.

Understanding that Kubernetes can run commands inside containers to check health is key to grasping how exec probes work.

2

Foundation

Basic Exec Probe Configuration Syntax

Concept: How to write the exec probe section in a pod's YAML manifest.

An exec probe is defined under the 'livenessProbe' or 'readinessProbe' field in the container spec. It uses the 'exec' key with a 'command' array. Example: livenessProbe: exec: command: - cat - /tmp/healthy initialDelaySeconds: 5 periodSeconds: 10

Result

Kubernetes will run 'cat /tmp/healthy' inside the container every 10 seconds after an initial delay of 5 seconds.

Knowing the YAML structure lets you configure exec probes correctly to monitor container health.

3

Intermediate

Difference Between Liveness and Readiness Probes

🤔Before reading on: do you think liveness and readiness probes do the same thing? Commit to your answer.

Concept: Liveness probes check if a container should be restarted; readiness probes check if a container is ready to serve traffic.

Liveness probes detect if a container is stuck or dead. If it fails, Kubernetes restarts the container. Readiness probes check if the container is ready to accept requests. If it fails, Kubernetes stops sending traffic to it but does not restart it.

Result

You can control container lifecycle and traffic flow separately using exec probes.

Understanding the different roles of liveness and readiness probes helps you design better health checks.

4

Intermediate

Configuring Timing Parameters for Exec Probes

🤔Before reading on: do you think setting very short probe intervals is always better? Commit to your answer.

Concept: Exec probes have timing settings like initial delay, timeout, period, and failure threshold to control how often and when they run.

Key timing parameters: - initialDelaySeconds: wait before first probe - periodSeconds: how often to run - timeoutSeconds: max time to wait for command - failureThreshold: how many failures before action Example: livenessProbe: exec: command: ["/bin/check.sh"] initialDelaySeconds: 10 periodSeconds: 5 timeoutSeconds: 2 failureThreshold: 3

Result

Kubernetes runs the probe every 5 seconds after waiting 10 seconds initially, and restarts container after 3 failures.

Knowing how to tune timing prevents false positives and ensures timely detection of issues.

5

Intermediate

Common Commands Used in Exec Probes

Concept: Typical commands used in exec probes check for files, processes, or custom scripts inside containers.

Examples: - cat /tmp/healthy (checks if file exists) - pgrep myprocess (checks if process runs) - /bin/check.sh (custom script returning 0 or 1) Commands must be quick and reliable to avoid probe timeouts.

Result

Exec probes can be customized to check any condition inside the container.

Choosing the right command is critical for accurate health detection without slowing down the container.

6

Advanced

Handling Exec Probe Failures Gracefully

🤔Before reading on: do you think Kubernetes immediately kills a container on first exec probe failure? Commit to your answer.

Concept: Kubernetes uses failureThreshold and grace periods to avoid restarting containers on transient failures.

If an exec probe fails, Kubernetes waits until failureThreshold is reached before restarting. This avoids restarts from temporary glitches. You can also combine exec probes with readiness probes to control traffic flow during recovery.

Result

Containers are not restarted unnecessarily, improving stability.

Understanding failure thresholds helps prevent flapping and improves application uptime.

7

Expert

Exec Probe Limitations and Security Considerations

🤔Before reading on: do you think exec probes can run any command without restrictions? Commit to your answer.

Concept: Exec probes run commands inside containers but have limitations and security risks if misused.

Exec probes require the container to have the command available and permissions to run it. Running complex or slow commands can cause probe failures. Also, exec probes expose an attack surface if commands are not secure. Best practice is to keep commands simple and safe. Some environments restrict exec probes for security reasons.

Result

Exec probes must be designed carefully to avoid security and reliability issues.

Knowing exec probe limits and risks helps design secure and robust health checks.

Under the Hood

Kubernetes kubelet runs the exec probe command inside the container's namespace using container runtime APIs. It waits for the command to finish and checks the exit code. Exit code 0 means success; any other code means failure. The kubelet uses this result to update the container's health status and trigger restarts or traffic routing changes.

Why designed this way?

Exec probes were designed to allow flexible, custom health checks inside containers without requiring network access. Running commands inside the container gives direct insight into its state. Alternatives like HTTP probes require the container to expose endpoints, which is not always possible or desired.

┌───────────────┐
│ Kubernetes    │
│ kubelet       │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Container     │
│ Runtime API   │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Exec Probe    │
│ Command Runs  │
│ Inside Container│
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Exit Code     │
│ 0 = Healthy  │
│ !=0 = Failure│
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: does a failing exec probe always mean the container is broken? Commit yes or no.

Common Belief: If an exec probe fails, the container is definitely broken and must be restarted immediately.

Reality: Exec probe failures can be temporary or caused by slow commands; Kubernetes waits for multiple failures before restarting.

Why it matters: Restarting containers on transient failures causes unnecessary downtime and instability.

Quick: can exec probes check external services? Commit yes or no.

Common Belief: Exec probes can check if external services are reachable by running network commands inside the container.

Reality: Exec probes only check inside the container; they cannot directly verify external dependencies' health.

Why it matters: Relying on exec probes for external checks can give false health signals and hide real issues.

Quick: do exec probes run commands as root by default? Commit yes or no.

Common Belief: Exec probes always run commands with root privileges inside the container.

Reality: Exec probes run commands with the container's user permissions, which may be non-root.

Why it matters: Assuming root privileges can cause probe failures if commands require higher permissions.

Quick: does exec probe output affect container logs? Commit yes or no.

Common Belief: The output of exec probe commands is logged and visible in container logs.

Reality: Exec probe command output is discarded by default and does not appear in container logs.

Why it matters: Debugging probe failures requires separate logging or command design since output is not captured.

Expert Zone

1

Exec probes can cause performance overhead if commands are slow or resource-intensive, impacting container responsiveness.

2

Combining exec probes with readiness probes allows graceful traffic shifting during container startup or recovery phases.

3

Some container runtimes or security policies restrict exec probe commands, requiring fallback to HTTP or TCP probes.

When NOT to use

Avoid exec probes when containers do not have the required commands or when security policies forbid exec commands. Use HTTP or TCP probes instead for network-accessible services.

Production Patterns

In production, exec probes often run lightweight scripts checking file existence or process status. They are combined with readiness probes to manage traffic and liveness probes to trigger restarts. Timing parameters are tuned to balance sensitivity and stability.

Connections

HTTP health probes

Alternative probe type with network checks instead of command execution

Understanding exec probes clarifies why HTTP probes are preferred for web services but exec probes are needed for non-networked health checks.

Container security policies

Exec probes run commands inside containers, which can be restricted by security settings

Knowing exec probe internals helps understand how security policies impact container health monitoring.

Medical diagnostics

Both exec probes and medical tests run internal checks to assess health status

Seeing exec probes as diagnostic tests helps appreciate the importance of timing, accuracy, and false positives in health monitoring.

Common Pitfalls

#1 Using a slow or blocking command in exec probe causing probe timeouts.

Wrong approach:

livenessProbe:
exec:
command: ["sleep", "10"]
timeoutSeconds: 1
periodSeconds: 5

Correct approach:

livenessProbe:
exec:
command: ["/bin/check_quick.sh"]
timeoutSeconds: 1
periodSeconds: 5

Root cause:

Misunderstanding that exec probe commands must complete quickly within timeout to avoid false failures.

#2 Configuring exec probe without initialDelaySeconds causing immediate failures on container start.

Wrong approach:

livenessProbe:
exec:
command: ["cat", "/tmp/healthy"]
periodSeconds: 10

Correct approach:

livenessProbe:
exec:
command: ["cat", "/tmp/healthy"]
initialDelaySeconds: 10
periodSeconds: 10

Root cause:

Not allowing container startup time before health checks leads to premature failure detection.

#3 Assuming exec probe output is logged and using it for debugging probe failures.

Wrong approach:

Relying on container logs to see exec probe command output for troubleshooting.

Correct approach:

Add logging inside the exec command script or use sidecar containers for detailed health check logs.

Root cause:

Not knowing exec probe output is discarded by default, making debugging harder.

Key Takeaways

Exec probes run commands inside containers to check their health by exit status.

They help Kubernetes decide when to restart containers or stop sending traffic.

Proper timing and command choice prevent false failures and improve stability.

Exec probes have security and performance limits that must be considered.

Combining exec probes with other probe types creates robust health monitoring.