Health check endpoints in HLD - System Design Exercise

Design: Health Check Endpoints

Design health check endpoints for a microservice or web service. Out of scope: full monitoring system, alerting, or remediation automation.

Functional Requirements

FR1: Provide a simple HTTP endpoint to report the health status of the service

FR2: Support basic liveness check to confirm the service is running

FR3: Support readiness check to confirm the service is ready to handle requests

FR4: Include dependency checks such as database connectivity and external API availability

FR5: Return clear status codes and messages for monitoring systems

FR6: Allow configurable thresholds for health criteria

FR7: Ensure minimal performance impact on the main service

FR8: Support integration with container orchestration systems like Kubernetes

Non-Functional Requirements

NFR1: Must respond within 100ms under normal load

NFR2: Availability target of 99.9% uptime for health endpoints

NFR3: Handle up to 1000 health check requests per second

NFR4: Endpoints must be secure and not expose sensitive information

NFR5: Health checks should not cause side effects or modify data

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

❓ Question 6

Key Components

HTTP server to expose health endpoints

Dependency check modules (database, cache, external APIs)

Configuration management for thresholds and checks

Logging and metrics for health check calls

Security layer for endpoint access control

Design Patterns

Circuit breaker pattern for dependency health

Timeout and retry strategies for dependency checks

Caching health check results to reduce load

Separation of liveness and readiness endpoints

Use of standard health check response formats (e.g., JSON)

Reference Architecture

  +-------------------+       +---------------------+
  |                   |       |                     |
  |   Client/Monitor   | <---> |  Health Check API    |
  |                   |       |  (Liveness & Ready)  |
  +-------------------+       +----------+----------+
                                         |
                                         v
                          +-----------------------------+
                          | Dependency Check Modules     |
                          | - Database connectivity      |
                          | - External API availability  |
                          | - Cache health               |
                          +-----------------------------+
                                         |
                                         v
                          +-----------------------------+
                          | Configuration & Thresholds  |
                          +-----------------------------+

Components

Health Check API

HTTP REST endpoint

Expose /health/liveness and /health/readiness endpoints

Dependency Check Modules

Custom code or libraries

Check status of database, cache, external services

Configuration Manager

Config files or environment variables

Manage thresholds and enable/disable checks

Security Layer

API Gateway or middleware

Control access to health endpoints

Logging and Metrics

Logging framework and metrics system

Record health check calls and results for monitoring

Request Flow

1. Client or monitoring system sends HTTP GET request to /health/liveness or /health/readiness

2. Health Check API receives request and triggers dependency checks

3. Dependency Check Modules verify connectivity and status of each dependency

4. Configuration Manager provides thresholds and check settings

5. Health Check API aggregates results and determines overall health status

6. API returns HTTP 200 OK with JSON payload if healthy, or HTTP 503 Service Unavailable if unhealthy

7. Logging and Metrics record the request and outcome for analysis

Database Schema

Not applicable as health check endpoints typically do not require persistent storage.

Scaling Discussion

Bottlenecks

High frequency of health check requests causing CPU or network load

Dependency checks causing latency if they are slow or blocking

Health check endpoints exposing sensitive information if not secured

Configuration changes requiring service restarts if not dynamic

Solutions

Cache health check results for short intervals to reduce repeated checks

Use asynchronous or non-blocking calls for dependency checks with timeouts

Implement access control or IP whitelisting for health endpoints

Support dynamic configuration reload without restarting the service

Interview Tips

Time: Spend 10 minutes understanding requirements and clarifying scope, 20 minutes designing components and data flow, 10 minutes discussing scaling and security considerations, 5 minutes summarizing.

Explain difference between liveness and readiness checks

Discuss importance of dependency checks and thresholds

Highlight performance and security considerations

Describe how health checks integrate with orchestration tools

Mention caching and asynchronous checks to improve scalability

Practice

1. What is the primary purpose of a health check endpoint in a system?

easy

A. To verify if the service is alive and functioning

B. To store user data securely

C. To handle user authentication

D. To serve static website content

Solution

1. Step 1: Understand the role of health check endpoints

   Health check endpoints are designed to tell if a service is alive and working properly.

2. Step 2: Identify the correct purpose among options

   Only To verify if the service is alive and functioning describes checking the service's health, while others describe unrelated functions.

3. Final Answer:

   To verify if the service is alive and functioning -> Option A

4. Quick Check:

   Health check = Verify service alive [OK]

Hint: Health checks confirm service status, not data or auth [OK]

Common Mistakes:

• Confusing health check with authentication
• Thinking health check stores data
• Assuming health check serves website content

2. Which URL path is commonly used for a readiness health check endpoint?

easy

A. /ready

B. /health

C. /status

D. /check

Solution

1. Step 1: Recall standard health check paths

   Readiness endpoints typically use the path /ready to indicate if the service is ready to accept traffic.

2. Step 2: Match the correct path

   Among the options, only /ready is the standard path for readiness checks.

3. Final Answer:

   /ready -> Option A

4. Quick Check:

   Readiness path = /ready [OK]

Hint: Readiness = /ready, Liveness = /health [OK]

Common Mistakes:

• Using /health for readiness instead of liveness
• Confusing /status or /check as standard paths
• Mixing readiness and liveness endpoints

3. Consider this pseudocode for a health check endpoint:

if database.isConnected() and cache.isAvailable():
    return 200
else:
    return 503

What HTTP status code will the endpoint return if the database is connected but the cache is down?

medium

A. 200

B. 503

C. 404

D. 500

Solution

1. Step 1: Analyze the condition logic

   The endpoint returns 200 only if both database.isConnected() and cache.isAvailable() are true.

2. Step 2: Evaluate the given scenario

   Database is connected (true), cache is down (false), so condition fails and returns 503.

3. Final Answer:

   503 -> Option B

4. Quick Check:

   Both must be true for 200; else 503 [OK]

Hint: Both services must be healthy for 200 OK [OK]

Common Mistakes:

• Assuming 200 if only one service is healthy
• Confusing 503 with 404 or 500
• Ignoring the AND condition in the code

4. A health check endpoint code snippet:

def health_check():
    if db.connected() or cache.available():
        return 200
    else:
        return 503

What is the main problem with this health check logic?

medium

A. It does not check any services

B. It returns 503 when both services are healthy

C. It returns 200 even if only one service is healthy

D. It uses incorrect HTTP status codes

Solution

1. Step 1: Understand the logical operator used

   The code uses OR, so it returns 200 if either db.connected() or cache.available() is true.

2. Step 2: Identify the issue with health check accuracy

   This means it reports healthy even if one service is down, which can mislead monitoring systems.

3. Final Answer:

   It returns 200 even if only one service is healthy -> Option C

4. Quick Check:

   OR allows partial health, which is incorrect [OK]

Hint: Use AND to ensure all services are healthy [OK]

Common Mistakes:

• Using OR instead of AND in health checks
• Ignoring partial failures in health status
• Misinterpreting HTTP status codes

5. You design a system with a /health endpoint for liveness and a /ready endpoint for readiness. Which of the following best describes when the /ready endpoint should return HTTP 503?

hard

A. When the service is completely down and unreachable

B. When the service is alive and ready to accept traffic

C. When the service is overloaded but still responding

D. When the service is alive but not ready to accept traffic due to missing dependencies

Solution

1. Step 1: Understand readiness vs liveness

   Liveness (/health) checks if service is alive; readiness (/ready) checks if it can handle requests safely.

2. Step 2: Identify readiness failure condition

   Readiness returns 503 if dependencies (like database) are missing, so service should not receive traffic yet.

3. Final Answer:

   When the service is alive but not ready to accept traffic due to missing dependencies -> Option D

4. Quick Check:

   Readiness 503 = alive but not ready [OK]

Hint: Ready means all dependencies OK; else 503 [OK]

Common Mistakes:

• Confusing liveness failure with readiness failure
• Returning 503 when service is fully down (should be liveness)
• Assuming overload means readiness failure