Health checks in HLD - System Design Guide

Problem Statement

When a service or server fails silently or becomes unresponsive, the system continues to send traffic to it, causing slow responses or complete outages. Without a way to detect unhealthy components, the system cannot reroute traffic or trigger recovery actions, leading to poor user experience and downtime.

Solution

Health checks periodically test if a service or server is working correctly by sending simple requests or probes. If a component fails the health check, it is marked unhealthy and removed from the pool of active servers until it recovers. This ensures traffic only goes to healthy components, improving reliability and availability.

Architecture

┌─────────────┐       ┌─────────────┐       ┌─────────────┐
│ Load Balancer│──────▶│ Health Check│──────▶│   Service   │
└─────────────┘       └─────────────┘       └─────────────┘
       │                    ▲                     │
       │                    │                     │
       │                    │                     │
       └────────────────────┴─────────────────────┘

This diagram shows a load balancer sending traffic to services only after health checks confirm they are healthy. The health check component probes the service and reports status back to the load balancer.

Trade-offs

✓ Pros

→

Improves system reliability by routing traffic only to healthy services.

→

Enables automatic detection and removal of failed components.

→

Supports proactive recovery actions like auto-scaling or alerts.

→

Simple to implement and integrates with load balancers and orchestrators.

✗ Cons

→

Adds extra network traffic and processing overhead for frequent checks.

→

Incorrect health check design can cause false positives or negatives.

→

Requires careful tuning of check frequency and timeout values.

Use health checks in any distributed system with multiple services or servers, especially when uptime and availability are critical and traffic needs to be routed dynamically.

Avoid health checks in very simple, single-server systems where failure detection is trivial or unnecessary, or when the overhead of checks outweighs benefits at very low scale.

Real World Examples

Netflix

Netflix uses health checks to monitor microservices and edge servers, ensuring traffic is only routed to healthy instances to maintain uninterrupted streaming.

Amazon

Amazon employs health checks in its load balancers to detect unhealthy EC2 instances and automatically reroute traffic to healthy ones, improving e-commerce site availability.

Google

Google Cloud Platform uses health checks to monitor VM instances and containers, enabling automatic failover and scaling decisions.

Alternatives

Heartbeat Monitoring

Instead of active probes, components send periodic 'heartbeat' signals to indicate health.

Use when: Choose heartbeat monitoring when components can push status updates and you want to reduce probe traffic.

Passive Health Checks

Health is inferred from actual traffic success/failure rather than active probes.

Use when: Choose passive checks when you want to avoid extra probe traffic and rely on real request outcomes.

Summary

Health checks detect unhealthy services to prevent routing traffic to failed components.

They improve system availability by enabling automatic failover and recovery.

Proper design and tuning of health checks are essential to avoid false positives and overhead.

Practice

1. What is the main purpose of a health check in system design?

easy

A. To verify if system components are working properly

B. To increase the system's storage capacity

C. To speed up the system's processing power

D. To change the system's user interface

Solution

1. Step 1: Understand health check purpose

   Health checks are used to confirm if parts of a system are functioning correctly.

2. Step 2: Compare options with purpose

   Only To verify if system components are working properly matches this purpose; others relate to unrelated system improvements.

3. Final Answer:

   To verify if system components are working properly -> Option A

4. Quick Check:

   Health check = Verify system health [OK]

Hint: Health checks confirm system parts work, not improve features [OK]

Common Mistakes:

• Confusing health checks with performance tuning
• Thinking health checks change system features
• Assuming health checks increase capacity

2. Which of the following is a common way to implement a health check endpoint?

easy

A. A complex database query

B. A user login form

C. A file upload service

D. A simple HTTP GET request returning status

Solution

1. Step 1: Identify typical health check method

   Health checks often use simple HTTP GET requests to return system status.

2. Step 2: Eliminate unrelated options

   Database queries, login forms, and file uploads are not standard health check methods.

3. Final Answer:

   A simple HTTP GET request returning status -> Option D

4. Quick Check:

   Health check = Simple HTTP GET [OK]

Hint: Health checks usually use simple HTTP GET endpoints [OK]

Common Mistakes:

• Choosing complex operations for health checks
• Confusing health checks with user features
• Selecting unrelated services like file upload

3. Consider this pseudo-code for a health check endpoint:

function healthCheck() {
  if (database.isConnected() && cache.isAvailable()) {
    return 200;
  } else {
    return 503;
  }
}

What HTTP status code will be returned if the database is connected but the cache is down?

medium

A. 503

B. 500

C. 404

D. 200

Solution

1. Step 1: Analyze condition in code

   The code returns 200 only if both database and cache are healthy.

2. Step 2: Apply given scenario

   Database is connected (true), cache is down (false), so condition fails and returns 503.

3. Final Answer:

   503 -> Option A

4. Quick Check:

   Partial failure returns 503 [OK]

Hint: Both must be healthy for 200; else 503 [OK]

Common Mistakes:

• Assuming 200 if any component is healthy
• Confusing 503 with 404 or 500
• Ignoring the AND condition in code

4. A health check endpoint is returning 200 OK even when the database is down. What is the most likely cause?

medium

A. The HTTP status code 200 means failure

B. The database is actually up

C. The health check does not verify the database connection

D. The cache is down instead

Solution

1. Step 1: Understand health check logic

   If the endpoint returns 200 even when the database is down, it likely does not check the database.

2. Step 2: Evaluate other options

   Database being up contradicts the problem; 200 means success, not failure; cache status irrelevant here.

3. Final Answer:

   The health check does not verify the database connection -> Option C

4. Quick Check:

   Missing DB check causes false 200 [OK]

Hint: If failure shows 200, check what health check verifies [OK]

Common Mistakes:

• Misinterpreting HTTP 200 as failure
• Assuming database is up without proof
• Ignoring health check code logic

5. You design a health check for a distributed system with multiple services. Which approach best ensures accurate overall health status?

hard

A. Check only the main service's status

B. Aggregate health from all critical services before responding

C. Return 200 if any one service is healthy

D. Skip health checks to reduce network load

Solution

1. Step 1: Understand distributed system health

   Overall health depends on all critical services working well, not just one.

2. Step 2: Evaluate options for accuracy

   Aggregating health from all critical services gives a true picture; checking only main or any one service is incomplete; skipping checks loses visibility.

3. Final Answer:

   Aggregate health from all critical services before responding -> Option B

4. Quick Check:

   Aggregate all critical services for true health [OK]

Hint: Combine all critical services' health for accurate status [OK]

Common Mistakes:

• Checking only one service in distributed system
• Assuming any healthy service means overall health
• Skipping health checks to save resources