Health check pattern in Microservices - Deep Dive

Overview - Health check pattern

What is it?

The health check pattern is a way to monitor if a service or system is working properly. It involves regularly checking the status of components to ensure they are alive and responsive. This helps detect problems early and maintain system reliability. Health checks can be simple pings or detailed tests of functionality.

Why it matters

Without health checks, failures in services can go unnoticed until they cause bigger problems, like downtime or data loss. This can frustrate users and damage trust. Health checks allow systems to detect issues quickly and recover or alert teams before users are affected. They are essential for keeping complex systems stable and available.

Where it fits

Before learning health checks, you should understand basic microservices architecture and service communication. After this, you can explore advanced monitoring, alerting, and self-healing systems that build on health checks to automate recovery and improve resilience.

Mental Model

Core Idea

A health check is a regular test that tells if a service is alive and working as expected.

Think of it like...

It's like a doctor checking your vital signs regularly to make sure you are healthy and catch problems early.

┌─────────────┐   periodic check   ┌─────────────┐
│  Monitoring │───────────────────▶│  Service    │
│   System    │                    │  Instance   │
└─────────────┘                    └─────────────┘
       ▲                                  │
       │                                  │
       │          health status           │
       └──────────────────────────────────┘

Build-Up - 7 Steps

1

Foundation

What is a health check

Concept: Introduce the basic idea of checking if a service is alive.

A health check is a simple test to see if a service is running. It can be as basic as sending a ping or requesting a small response. If the service replies correctly, it is considered healthy.

Result

You understand that health checks confirm if a service is reachable and responsive.

Understanding that health checks are the first step to knowing if a service is working prevents blind spots in system monitoring.

2

Foundation

Types of health checks

Concept: Learn the difference between basic and detailed health checks.

There are two main types: liveness checks and readiness checks. Liveness checks confirm if the service is alive. Readiness checks confirm if the service is ready to handle requests, including dependencies like databases.

Result

You can distinguish between a service being alive and being ready to serve traffic.

Knowing these types helps avoid sending traffic to services that are alive but not ready, improving user experience.

3

Intermediate

Implementing health endpoints

🤔Before reading on: do you think health checks should be part of the main service or a separate system? Commit to your answer.

Concept: Learn how services expose health check endpoints for monitoring systems to query.

Services usually expose a special URL like /health or /status that returns health information. This endpoint can return simple status codes or detailed JSON with component statuses. Monitoring tools call this endpoint regularly.

Result

You know how to add a health check endpoint to a service for external monitoring.

Understanding that health endpoints are part of the service itself simplifies integration with monitoring and reduces external dependencies.

4

Intermediate

Health checks in load balancers

🤔Before reading on: do you think load balancers rely on health checks to route traffic? Commit to yes or no.

Concept: Learn how load balancers use health checks to decide where to send user requests.

Load balancers regularly call health check endpoints on service instances. If an instance fails, the load balancer stops sending traffic to it until it recovers. This prevents users from hitting broken services.

Result

You understand how health checks improve traffic routing and system reliability.

Knowing that health checks directly influence traffic flow helps design systems that gracefully handle failures.

5

Intermediate

Health checks for dependencies

🤔Before reading on: do you think a service is healthy if it can respond but its database is down? Commit to yes or no.

Concept: Learn why health checks should verify critical dependencies, not just the service itself.

A service might be running but unable to serve requests properly if its database or other dependencies are down. Health checks can include tests for these dependencies to give a true picture of service health.

Result

You realize that health checks must cover all parts needed for correct service operation.

Understanding this prevents false positives where a service appears healthy but cannot fulfill its purpose.

6

Advanced

Designing scalable health check systems

🤔Before reading on: do you think checking every service instance every second scales well? Commit to yes or no.

Concept: Learn how to design health check systems that scale with many services and instances.

In large systems, health checks can create heavy load if done too frequently or without coordination. Techniques like caching results, staggering checks, and hierarchical health monitoring reduce overhead and improve scalability.

Result

You can design health check strategies that work efficiently in large distributed systems.

Knowing how to scale health checks prevents monitoring from becoming a bottleneck or causing failures.

7

Expert

Advanced health check patterns and pitfalls

🤔Before reading on: do you think a service that passes health checks but returns errors to users is truly healthy? Commit to yes or no.

Concept: Explore complex cases where health checks can be misleading and how to improve them.

Sometimes services pass health checks but have degraded performance or errors. Advanced patterns include multi-level health checks, synthetic transactions, and anomaly detection. Also, beware of health check endpoints that are too simple or cause side effects.

Result

You understand the limits of basic health checks and how to build more reliable monitoring.

Recognizing health check limitations helps avoid blind spots and improves system resilience.

Under the Hood

Health checks work by exposing a dedicated interface, usually an HTTP endpoint, that monitoring systems query periodically. The service runs internal checks on its components and dependencies, then returns a status code and optional details. The monitoring system interprets these results to decide if the service is healthy. Load balancers and orchestrators use this information to manage traffic and service lifecycle.

Why designed this way?

Health checks were designed to provide a simple, standardized way to detect service failures quickly. Early systems lacked automated failure detection, causing long downtimes. The pattern balances simplicity and effectiveness by using lightweight checks that services can implement themselves, avoiding complex external probes that might not reflect real service health.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ Monitoring    │──────▶│ Health Check  │──────▶│ Service       │
│ System        │       │ Endpoint      │       │ Components    │
└───────────────┘       └───────────────┘       └───────────────┘
       ▲                      │                        │
       │                      │                        │
       │                      ▼                        ▼
       │               ┌───────────────┐        ┌───────────────┐
       │               │ Dependency 1  │        │ Dependency 2  │
       │               └───────────────┘        └───────────────┘
       │                      │                        │
       └──────────────────────┴────────────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: does a passing health check always mean the service is fully functional? Commit yes or no.

Common Belief: If a service passes its health check, it is fully healthy and serving users correctly.

Reality: A service can pass simple health checks but still have errors, slow responses, or partial failures affecting users.

Why it matters: Relying only on basic health checks can cause unnoticed user impact and delayed incident response.

Quick: should health checks be very frequent, like every second, for all services? Commit yes or no.

Common Belief: More frequent health checks always improve system reliability.

Reality: Too frequent health checks can overload services and networks, causing performance degradation or false failures.

Why it matters: Improper health check frequency can ironically cause system instability instead of preventing it.

Quick: is it enough to check only the service process for health? Commit yes or no.

Common Belief: Checking if the service process is running is enough to declare it healthy.

Reality: A service process can run but be unable to connect to databases or other dependencies, making it effectively unhealthy.

Why it matters: Ignoring dependencies in health checks leads to false positives and poor user experience.

Quick: can health check endpoints cause side effects or change service state? Commit yes or no.

Common Belief: Health check endpoints are always safe and have no side effects.

Reality: Poorly designed health checks can trigger expensive operations or change data, causing unintended consequences.

Why it matters: Side effects in health checks can degrade performance or corrupt data, harming system stability.

Expert Zone

1

Health checks should balance thoroughness and speed; too detailed checks slow responses and can cause false negatives.

2

In container orchestration, readiness and liveness probes serve different roles and must be configured carefully to avoid restart loops.

3

Caching health check results for a short time can reduce load but risks delayed failure detection; tuning is critical.

When NOT to use

Health checks are not a substitute for full monitoring or alerting systems. For complex failure detection, use synthetic transactions, tracing, and anomaly detection. Avoid health checks as the only signal for system health in highly dynamic or stateful systems.

Production Patterns

In production, health checks integrate with load balancers, service meshes, and orchestration platforms like Kubernetes. Teams use multi-level health checks combining liveness, readiness, and dependency checks. Alerts trigger on health check failures, and automated recovery actions like restarts or traffic shifting are common.

Connections

Circuit Breaker Pattern

Builds-on

Health checks provide the status signals that circuit breakers use to stop sending requests to failing services, preventing cascading failures.

Synthetic Monitoring

Complementary

While health checks test internal service status, synthetic monitoring simulates real user actions to detect issues health checks might miss.

Human Health Monitoring

Analogous

Just like doctors monitor vital signs to detect illness early, health checks monitor system vitals to catch failures before they impact users.

Common Pitfalls

#1 Health check endpoint performs heavy database queries causing slow responses.

Wrong approach:

GET /health endpoint runs full data aggregation queries to check database health.

Correct approach:

GET /health endpoint performs lightweight database ping or simple query to verify connectivity.

Root cause:

Misunderstanding that health checks must be fast and lightweight to avoid adding load.

#2 Using the same health check for liveness and readiness without distinction.

Wrong approach:

Single /health endpoint returns 'healthy' if service process is running, ignoring readiness state.

Correct approach:

Separate /live and /ready endpoints; liveness checks process, readiness checks dependencies and readiness.

Root cause:

Confusing liveness and readiness concepts leads to improper traffic routing and restarts.

#3 Health checks are too frequent causing network congestion and false alarms.

Wrong approach:

Monitoring system polls health endpoints every second for all services.

Correct approach:

Poll health endpoints at reasonable intervals (e.g., 10-30 seconds) and stagger checks across instances.

Root cause:

Assuming more frequent checks always improve reliability without considering system load.

Key Takeaways

Health checks are essential tools that regularly verify if services are alive and ready to serve requests.

Distinguishing between liveness and readiness checks prevents sending traffic to services that cannot handle it.

Health checks must include critical dependencies to reflect true service health and avoid false positives.

Designing scalable health check systems requires balancing check frequency and thoroughness to avoid overload.

Advanced health check patterns and monitoring complement basic checks to detect subtle failures and maintain resilience.

Practice

1. What is the main purpose of the health check pattern in microservices?

easy

A. To regularly verify if a service is running and responsive

B. To increase the size of the service database

C. To encrypt communication between services

D. To deploy new versions of the service automatically

Solution

1. Step 1: Understand the health check pattern purpose

   The health check pattern is designed to monitor if a microservice is alive and functioning properly by sending simple requests to it.

2. Step 2: Identify the correct purpose among options

   Only To regularly verify if a service is running and responsive describes this monitoring function, while others describe unrelated tasks like database size, encryption, or deployment.

3. Final Answer:

   To regularly verify if a service is running and responsive -> Option A

4. Quick Check:

   Health check = verify service status [OK]

Hint: Health check means checking if service is alive and working [OK]

Common Mistakes:

• Confusing health check with deployment or encryption
• Thinking health check changes service data
• Assuming health check increases service capacity

2. Which of the following is the correct way to implement a health check endpoint in a microservice?

easy

A. Create an endpoint like /health that returns status 200 if service is healthy

B. Create an endpoint that deletes all data when called

C. Create an endpoint that returns service logs only

D. Create an endpoint that restarts the service automatically

Solution

1. Step 1: Identify typical health check endpoint behavior

   A health check endpoint usually responds with a simple status code like 200 OK to indicate the service is healthy.

2. Step 2: Match this behavior with the options

   Create an endpoint like /health that returns status 200 if service is healthy matches this by returning status 200 on /health. Other options perform unrelated or harmful actions.

3. Final Answer:

   Create an endpoint like /health that returns status 200 if service is healthy -> Option A

4. Quick Check:

   Health endpoint = status 200 OK [OK]

Hint: Health endpoint returns 200 OK if service is fine [OK]

Common Mistakes:

• Confusing health check with data deletion
• Expecting health check to return logs
• Assuming health check restarts service

3. Consider this pseudocode for a health check endpoint:

function healthCheck() {
  if (database.isConnected() && cache.isAvailable()) {
    return { status: 200, message: 'Healthy' };
  } else {
    return { status: 503, message: 'Unhealthy' };
  }
}

What will the endpoint return if the database is connected but the cache is down?

medium

A. An error is thrown

B. { status: 200, message: 'Healthy' }

C. { status: 404, message: 'Not Found' }

D. { status: 503, message: 'Unhealthy' }

Solution

1. Step 1: Analyze the condition in the healthCheck function

   The function returns healthy only if both database.isConnected() and cache.isAvailable() are true.

2. Step 2: Evaluate the given scenario

   Database is connected (true), cache is down (false), so the condition is false and the function returns status 503 with 'Unhealthy'.

3. Final Answer:

   { status: 503, message: 'Unhealthy' } -> Option D

4. Quick Check:

   Both checks true = 200, else 503 [OK]

Hint: Both dependencies must be healthy for 200 OK [OK]

Common Mistakes:

• Assuming partial health returns 200 OK
• Confusing 503 with 404 status
• Expecting an error instead of status response

4. A microservice health check endpoint is implemented as follows:

GET /health
Response: { "status": "ok" }

But monitoring tools report the service as unhealthy. What is the likely problem?

medium

A. The service is actually down and not responding

B. The endpoint does not return an HTTP status code 200

C. The endpoint URL should be /status instead of /health

D. The response body is missing the word 'healthy'

Solution

1. Step 1: Understand monitoring tool expectations

   Most monitoring tools expect the health check endpoint to return HTTP status code 200 to mark service healthy.

2. Step 2: Identify the issue with the current implementation

   The response body contains status 'ok' but if the HTTP status code is not 200, tools may mark it unhealthy.

3. Final Answer:

   The endpoint does not return an HTTP status code 200 -> Option B

4. Quick Check:

   Health check needs HTTP 200 status [OK]

Hint: Health check must return HTTP 200 status code [OK]

Common Mistakes:

• Thinking response body text controls health status
• Assuming endpoint URL name matters
• Ignoring actual service availability

5. You design a microservice system with multiple services. To improve reliability, you want to implement health checks that also verify database and cache connectivity. Which approach best follows the health check pattern?

hard

A. Each service exposes a /health endpoint that always returns 200 regardless of dependency status

B. Only one central service exposes a health check endpoint for all services combined

C. Each service exposes a /health endpoint that returns 200 only if all dependencies (database, cache) are reachable; otherwise returns 503

D. Health checks are done by querying the database directly without service endpoints

Solution

1. Step 1: Understand health check pattern for dependencies

   Health checks should verify the service and its critical dependencies like database and cache to ensure full functionality.

2. Step 2: Evaluate the options for best practice

   Each service exposes a /health endpoint that returns 200 only if all dependencies (database, cache) are reachable; otherwise returns 503 correctly implements a health endpoint that returns 200 only if all dependencies are healthy, otherwise 503. This supports automatic recovery and monitoring.

3. Final Answer:

   Each service exposes a /health endpoint that returns 200 only if all dependencies (database, cache) are reachable; otherwise returns 503 -> Option C

4. Quick Check:

   Health check includes dependencies, returns 200 or 503 [OK]

Hint: Health check must verify all critical dependencies [OK]

Common Mistakes:

• Ignoring dependency status in health check
• Relying on a single central health endpoint
• Checking dependencies outside service endpoints