Container health checks in Docker - Deep Dive

Overview - Container health checks

What is it?

Container health checks are a way to tell if a running container is working properly. They are commands or scripts that Docker runs inside the container to check its status. If the check fails, Docker knows the container might have a problem and can act accordingly. This helps keep applications reliable and stable.

Why it matters

Without health checks, Docker only knows if a container is running or stopped, but not if the app inside is actually working. This can cause broken services to stay running unnoticed, leading to bad user experiences or system failures. Health checks help detect problems early and allow automatic recovery, making systems more robust and trustworthy.

Where it fits

Before learning container health checks, you should understand basic Docker concepts like containers, images, and running containers. After this, you can learn about Docker orchestration tools like Docker Compose and Kubernetes, which use health checks to manage container lifecycles automatically.

Mental Model

Core Idea

A container health check is like a doctor’s quick exam that regularly tests if the container’s app is healthy and working as expected.

Think of it like...

Imagine a car dashboard light that turns on if something is wrong with the engine. The health check is like that light, signaling if the container needs attention.

┌─────────────────────────────┐
│       Docker Container       │
│  ┌───────────────────────┐  │
│  │  Application Process   │  │
│  └───────────────────────┘  │
│  ┌───────────────────────┐  │
│  │  Health Check Command  │──┼─> Docker monitors result
│  └───────────────────────┘  │
└─────────────────────────────┘

Build-Up - 7 Steps

1

Foundation

What is a container health check

Concept: Introduce the basic idea of health checks as commands run inside containers to test their health.

A container health check is a command or script that Docker runs inside a container at regular intervals. It checks if the application inside the container is working correctly. If the command returns success, Docker marks the container as healthy. If it fails, Docker marks it as unhealthy.

Result

Docker knows the container’s health status: healthy or unhealthy.

Understanding that health checks give Docker a way to know more than just running or stopped status is key to managing container reliability.

2

Foundation

How to define health checks in Dockerfile

Concept: Learn the syntax to add health checks inside a Dockerfile using the HEALTHCHECK instruction.

In a Dockerfile, you add a HEALTHCHECK instruction followed by a command. For example: HEALTHCHECK --interval=30s --timeout=5s CMD curl -f http://localhost/ || exit 1 This tells Docker to run the curl command every 30 seconds and wait up to 5 seconds for a response. If curl fails, the container is unhealthy.

Result

The built image includes a health check that Docker runs automatically when containers start.

Knowing how to embed health checks in the Dockerfile makes health monitoring part of the container’s definition and portable.

3

Intermediate

Health check command exit codes

🤔Before reading on: do you think a health check command must print output to be considered healthy, or is the exit code enough? Commit to your answer.

Concept: Understand that Docker uses the command’s exit code to decide health, not its output.

Docker runs the health check command and looks only at its exit code: - Exit code 0 means healthy - Exit code 1 means unhealthy - Exit code 2 means starting (used for initialization) The command does not need to print anything; only the exit code matters.

Result

You can write simple scripts that exit with the right code to signal health status.

Knowing that exit codes control health status helps you write efficient and clear health check commands without worrying about output parsing.

4

Intermediate

Using health checks with Docker Compose

🤔Before reading on: do you think Docker Compose automatically restarts unhealthy containers by default? Commit to your answer.

Concept: Learn how to define health checks in Docker Compose files and how Compose handles container health.

In a docker-compose.yml file, you add health checks under the service like this: services: web: image: myapp healthcheck: test: ["CMD", "curl", "-f", "http://localhost/"] interval: 30s timeout: 5s retries: 3 Docker Compose monitors health but does not restart containers automatically unless you configure restart policies.

Result

Compose shows container health status and can be combined with restart policies for resilience.

Understanding Compose’s health check support helps you build multi-container apps that self-monitor and recover.

5

Advanced

Health check retries and thresholds

🤔Before reading on: do you think a single failed health check marks the container unhealthy immediately? Commit to your answer.

Concept: Explore how Docker uses retries and thresholds to avoid false alarms in health checks.

Docker does not mark a container unhealthy after one failed check. It waits for a number of retries (default 3) before changing status. This avoids marking containers unhealthy due to temporary glitches. You can configure retries, interval, and timeout in the HEALTHCHECK instruction.

Result

Health status changes only after consistent failures, reducing false positives.

Knowing how retries work helps you tune health checks for stability and avoid unnecessary container restarts.

6

Advanced

How Docker reacts to unhealthy containers

🤔Before reading on: do you think Docker automatically restarts unhealthy containers by default? Commit to your answer.

Concept: Understand Docker’s default behavior and options when a container becomes unhealthy.

By default, Docker marks the container as unhealthy but does not restart it automatically. You can combine health checks with restart policies (like restart: always) to have Docker restart unhealthy containers. Or external orchestrators like Kubernetes can use health status to replace containers.

Result

You control how to respond to unhealthy containers, enabling flexible recovery strategies.

Knowing Docker’s default limits and how to combine health checks with restart policies or orchestrators is key to building resilient systems.

7

Expert

Limitations and pitfalls of container health checks

🤔Before reading on: do you think health checks can detect all types of application failures inside containers? Commit to your answer.

Concept: Explore what health checks cannot do and common mistakes in their use.

Health checks run simple commands and can only test what you script. They cannot detect complex internal app bugs or performance issues unless explicitly checked. Overly frequent or heavy health checks can cause performance overhead. Also, health checks that depend on network or external services may give false negatives if those services are down.

Result

Health checks are a useful but limited tool that must be designed carefully and combined with other monitoring.

Understanding health check limits prevents overreliance and encourages complementary monitoring and alerting.

Under the Hood

Docker runs the health check command inside the container’s namespace using the container’s default user and environment. It captures the command’s exit code to update the container’s health status in Docker’s internal state. This status is exposed via Docker APIs and CLI. Docker schedules health checks at configured intervals and tracks retries to avoid flapping status.

Why designed this way?

Health checks were designed to be simple and lightweight so they can run frequently without impacting container performance. Using exit codes leverages standard Unix conventions for success/failure. Running checks inside the container ensures they test the actual app environment, not just external connectivity.

┌───────────────┐       ┌─────────────────────┐
│ Docker Engine │──────▶│ Run health check cmd │
└──────┬────────┘       └──────────┬──────────┘
       │                           │
       │                           ▼
       │                  ┌─────────────────┐
       │                  │ Container shell │
       │                  └────────┬────────┘
       │                           │
       │                   Execute command
       │                           │
       │                           ▼
       │                  ┌─────────────────┐
       │                  │ App environment  │
       │                  └─────────────────┘
       │                           │
       │                  Return exit code
       │                           │
       │◀──────────────────────────┘
       │
       ▼
Update container health status

Myth Busters - 4 Common Misconceptions

Quick: does Docker restart unhealthy containers automatically by default? Commit to yes or no.

Common Belief: Docker automatically restarts containers as soon as they become unhealthy.

Reality: Docker marks containers as unhealthy but does not restart them unless a restart policy is set or an orchestrator manages it.

Why it matters: Assuming automatic restarts can lead to unnoticed downtime if containers stay unhealthy without recovery.

Quick: do health checks rely on command output or exit code? Commit to your answer.

Common Belief: Health checks depend on the command’s output to decide health.

Reality: Docker uses only the command’s exit code to determine health status, ignoring output.

Why it matters: Misunderstanding this can cause health checks to fail silently or behave unpredictably.

Quick: can health checks detect all application problems inside containers? Commit yes or no.

Common Belief: Health checks can detect every kind of failure inside a container.

Reality: Health checks can only detect what the check command tests; complex bugs or performance issues may go unnoticed.

Why it matters: Overreliance on health checks alone can cause missed failures and system instability.

Quick: does a single failed health check mark the container unhealthy immediately? Commit yes or no.

Common Belief: One failed health check instantly marks the container unhealthy.

Reality: Docker waits for multiple failed retries before marking unhealthy to avoid false alarms.

Why it matters: Not knowing this can lead to confusion about container health status changes.

Expert Zone

1

Health checks run inside the container’s namespace, so environment variables and permissions affect their behavior subtly.

2

The timing of health checks can cause race conditions if the app is still starting but the check runs too early.

3

Combining health checks with orchestrator probes (like Kubernetes readiness and liveness probes) requires careful alignment to avoid conflicting signals.

When NOT to use

Avoid using container health checks as the sole monitoring tool for complex applications. Use them alongside external monitoring, logging, and alerting systems. For very simple containers or short-lived jobs, health checks may add unnecessary overhead.

Production Patterns

In production, health checks are combined with restart policies and orchestrator probes to automate recovery. Teams often write custom scripts that check multiple app aspects (database connectivity, API responsiveness) for robust health signals. Health check results feed into dashboards and alerting systems for proactive maintenance.

Connections

Kubernetes Probes

Builds-on

Understanding Docker health checks helps grasp Kubernetes readiness and liveness probes, which extend the same idea to orchestrated container clusters.

System Monitoring

Similar pattern

Container health checks are like system monitoring checks that verify service health, showing how health signals are a universal concept in IT reliability.

Medical Diagnostics

Analogous process

Health checks in containers mirror medical diagnostics where simple tests indicate overall health, illustrating how complex systems use simple signals to detect problems early.

Common Pitfalls

#1 Writing health check commands that always succeed regardless of app state.

Wrong approach:

HEALTHCHECK CMD echo "always healthy"

Correct approach:

HEALTHCHECK CMD curl -f http://localhost/ || exit 1

Root cause:

Misunderstanding that the command’s exit code controls health status, not just running any command.

#2 Setting health check intervals too short causing performance issues.

Wrong approach:

HEALTHCHECK --interval=1s CMD curl -f http://localhost/ || exit 1

Correct approach:

HEALTHCHECK --interval=30s CMD curl -f http://localhost/ || exit 1

Root cause:

Not considering the resource cost of frequent health checks inside containers.

#3 Assuming Docker restarts unhealthy containers without restart policy.

Wrong approach:

No restart policy set, relying on health check alone.

Correct approach:

Add restart policy: restart: always in docker-compose.yml or --restart=always in docker run.

Root cause:

Confusing health status reporting with automatic recovery behavior.

Key Takeaways

Container health checks let Docker know if the app inside a container is working, not just if the container is running.

Health checks run commands inside the container and use the command’s exit code to decide health status.

Docker waits for multiple failed health checks before marking a container unhealthy to avoid false alarms.

By default, Docker does not restart unhealthy containers unless combined with restart policies or orchestrators.

Health checks are a simple but limited tool and should be combined with other monitoring and recovery strategies.