Health check endpoints in HLD - System Design Guide

Problem Statement

When a service or server fails silently or becomes unresponsive, the system cannot detect the failure quickly. This leads to prolonged downtime, poor user experience, and difficulty in automated recovery or scaling decisions.

Solution

Health check endpoints provide a simple URL or API that external systems can call to verify if a service is running correctly. These endpoints return a status indicating the health of the service, enabling monitoring tools and load balancers to detect failures and take action automatically.

Architecture

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│   Client /    │       │   Load        │       │   Service     │
│ Monitoring    │──────▶│   Balancer    │──────▶│   Instance    │
│ System        │       │               │       │               │
│ (Health       │       │               │       │  ┌─────────┐  │
│  Checker)     │       │               │       │  │Health   │  │
└───────────────┘       └───────────────┘       │  │Check    │  │
                                                │  │Endpoint │  │
                                                │  └─────────┘  │
                                                └───────────────┘

This diagram shows a monitoring system or load balancer querying the health check endpoint of a service instance to verify its status before routing traffic.

Trade-offs

✓ Pros

→

Enables quick detection of service failures for automated recovery.

→

Improves system reliability by allowing load balancers to avoid unhealthy instances.

→

Simplifies monitoring by providing a standard way to check service health.

→

Can include detailed checks for dependencies, improving fault diagnosis.

✗ Cons

→

Requires additional development and maintenance effort for health check logic.

→

If health checks are too simple, they may not detect deeper issues.

→

Overly complex health checks can increase response time and resource usage.

Use when running multiple service instances behind load balancers or when automated monitoring and recovery are needed, especially at scales above tens of instances.

Avoid if the system is a single instance without automated monitoring or if the overhead of health checks outweighs benefits in very small or simple deployments.

Real World Examples

Netflix

Netflix uses health check endpoints to ensure that streaming servers are responsive before routing user requests, preventing buffering and downtime.

Amazon

Amazon employs health checks in its Elastic Load Balancer to detect unhealthy EC2 instances and reroute traffic to healthy ones automatically.

Google

Google Cloud Platform uses health check endpoints to monitor VM instances and managed services, enabling auto-healing and scaling.

Alternatives

Heartbeat mechanism

Instead of a request-response endpoint, services send periodic signals to a monitoring system to indicate health.

Use when: Choose when you want push-based health reporting rather than pull-based checks, especially in systems with limited incoming request capability.

External monitoring agents

Use separate agents installed on servers to monitor service health rather than built-in endpoints.

Use when: Choose when you want to monitor multiple aspects of the host environment beyond just the service.

Summary

Health check endpoints help detect service failures quickly by providing a simple status URL.

They enable automated systems like load balancers and monitors to maintain system reliability.

Properly designed health checks balance thoroughness with performance to avoid overhead.

Practice

1. What is the primary purpose of a health check endpoint in a system?

easy

A. To verify if the service is alive and functioning

B. To store user data securely

C. To handle user authentication

D. To serve static website content

Solution

1. Step 1: Understand the role of health check endpoints

   Health check endpoints are designed to tell if a service is alive and working properly.

2. Step 2: Identify the correct purpose among options

   Only To verify if the service is alive and functioning describes checking the service's health, while others describe unrelated functions.

3. Final Answer:

   To verify if the service is alive and functioning -> Option A

4. Quick Check:

   Health check = Verify service alive [OK]

Hint: Health checks confirm service status, not data or auth [OK]

Common Mistakes:

• Confusing health check with authentication
• Thinking health check stores data
• Assuming health check serves website content

2. Which URL path is commonly used for a readiness health check endpoint?

easy

A. /ready

B. /health

C. /status

D. /check

Solution

1. Step 1: Recall standard health check paths

   Readiness endpoints typically use the path /ready to indicate if the service is ready to accept traffic.

2. Step 2: Match the correct path

   Among the options, only /ready is the standard path for readiness checks.

3. Final Answer:

   /ready -> Option A

4. Quick Check:

   Readiness path = /ready [OK]

Hint: Readiness = /ready, Liveness = /health [OK]

Common Mistakes:

• Using /health for readiness instead of liveness
• Confusing /status or /check as standard paths
• Mixing readiness and liveness endpoints

3. Consider this pseudocode for a health check endpoint:

if database.isConnected() and cache.isAvailable():
    return 200
else:
    return 503

What HTTP status code will the endpoint return if the database is connected but the cache is down?

medium

A. 200

B. 503

C. 404

D. 500

Solution

1. Step 1: Analyze the condition logic

   The endpoint returns 200 only if both database.isConnected() and cache.isAvailable() are true.

2. Step 2: Evaluate the given scenario

   Database is connected (true), cache is down (false), so condition fails and returns 503.

3. Final Answer:

   503 -> Option B

4. Quick Check:

   Both must be true for 200; else 503 [OK]

Hint: Both services must be healthy for 200 OK [OK]

Common Mistakes:

• Assuming 200 if only one service is healthy
• Confusing 503 with 404 or 500
• Ignoring the AND condition in the code

4. A health check endpoint code snippet:

def health_check():
    if db.connected() or cache.available():
        return 200
    else:
        return 503

What is the main problem with this health check logic?

medium

A. It does not check any services

B. It returns 503 when both services are healthy

C. It returns 200 even if only one service is healthy

D. It uses incorrect HTTP status codes

Solution

1. Step 1: Understand the logical operator used

   The code uses OR, so it returns 200 if either db.connected() or cache.available() is true.

2. Step 2: Identify the issue with health check accuracy

   This means it reports healthy even if one service is down, which can mislead monitoring systems.

3. Final Answer:

   It returns 200 even if only one service is healthy -> Option C

4. Quick Check:

   OR allows partial health, which is incorrect [OK]

Hint: Use AND to ensure all services are healthy [OK]

Common Mistakes:

• Using OR instead of AND in health checks
• Ignoring partial failures in health status
• Misinterpreting HTTP status codes

5. You design a system with a /health endpoint for liveness and a /ready endpoint for readiness. Which of the following best describes when the /ready endpoint should return HTTP 503?

hard

A. When the service is completely down and unreachable

B. When the service is alive and ready to accept traffic

C. When the service is overloaded but still responding

D. When the service is alive but not ready to accept traffic due to missing dependencies

Solution

1. Step 1: Understand readiness vs liveness

   Liveness (/health) checks if service is alive; readiness (/ready) checks if it can handle requests safely.

2. Step 2: Identify readiness failure condition

   Readiness returns 503 if dependencies (like database) are missing, so service should not receive traffic yet.

3. Final Answer:

   When the service is alive but not ready to accept traffic due to missing dependencies -> Option D

4. Quick Check:

   Readiness 503 = alive but not ready [OK]

Hint: Ready means all dependencies OK; else 503 [OK]

Common Mistakes:

• Confusing liveness failure with readiness failure
• Returning 503 when service is fully down (should be liveness)
• Assuming overload means readiness failure