Health check endpoints in HLD - Architecture Diagram

System Overview - Health check endpoints

This system provides health check endpoints to monitor the status of various components in a distributed application. It ensures that services, databases, and caches are operational and responsive, allowing automated systems to detect failures quickly and trigger alerts or recovery actions.

Architecture Diagram

          +------------+
          |   Client   |
          +-----+------+ 
                |
                v
       +--------+---------+
       |   Load Balancer  |
       +--------+---------+
                |
                v
       +--------+---------+
       |  API Gateway     |
       +--------+---------+
                |
       +--------+---------+-------------------+
       |                  |                   |
       v                  v                   v
+------+-----+     +------+-----+       +-----+------+
| Health     |     | Service A  |       | Service B  |
| Check      |     +------------+       +------------+
| Endpoint   |           |                    |
+------------+           |                    |
                         v                    v
                  +------+-----+        +-----+------+
                  | Database   |        | Cache      |
                  +------------+        +------------+

Components

Client

user

Sends health check requests to monitor system status

Load Balancer

load_balancer

Distributes incoming requests evenly across API Gateway instances

API Gateway

api_gateway

Routes health check requests to the health check endpoint or services

Health Check Endpoint

service

Responds to health check requests with status of system components

Service A

service

Main application service that depends on database and cache

Service B

service

Secondary service that uses cache for faster responses

Database

database

Stores persistent application data

Cache

cache

Stores frequently accessed data for quick retrieval

Request Flow - 8 Hops

Client → Load Balancer

Load Balancer → API Gateway

API Gateway → Health Check Endpoint

Health Check Endpoint → Database

Health Check Endpoint → Cache

Health Check Endpoint → API Gateway

API Gateway → Load Balancer

Load Balancer → Client

Failure Scenario

Component Fails:Database

Impact:Health check endpoint reports database as unhealthy; services depending on database may fail writes or reads

Mitigation:Use database replication for failover; health check alerts trigger automated recovery or manual intervention

Architecture Quiz - 3 Questions

Which component first receives the health check request from the client?

A Health Check Endpoint

B Load Balancer

C API Gateway

D Database

Design Principle

This architecture uses a dedicated health check endpoint accessed through a load balancer and API gateway to monitor the health of critical components like database and cache. It ensures quick detection of failures and supports automated recovery, improving system reliability.

Practice

1. What is the primary purpose of a health check endpoint in a system?

easy

A. To verify if the service is alive and functioning

B. To store user data securely

C. To handle user authentication

D. To serve static website content

Solution

1. Step 1: Understand the role of health check endpoints

   Health check endpoints are designed to tell if a service is alive and working properly.

2. Step 2: Identify the correct purpose among options

   Only To verify if the service is alive and functioning describes checking the service's health, while others describe unrelated functions.

3. Final Answer:

   To verify if the service is alive and functioning -> Option A

4. Quick Check:

   Health check = Verify service alive [OK]

Hint: Health checks confirm service status, not data or auth [OK]

Common Mistakes:

• Confusing health check with authentication
• Thinking health check stores data
• Assuming health check serves website content

2. Which URL path is commonly used for a readiness health check endpoint?

easy

A. /ready

B. /health

C. /status

D. /check

Solution

1. Step 1: Recall standard health check paths

   Readiness endpoints typically use the path /ready to indicate if the service is ready to accept traffic.

2. Step 2: Match the correct path

   Among the options, only /ready is the standard path for readiness checks.

3. Final Answer:

   /ready -> Option A

4. Quick Check:

   Readiness path = /ready [OK]

Hint: Readiness = /ready, Liveness = /health [OK]

Common Mistakes:

• Using /health for readiness instead of liveness
• Confusing /status or /check as standard paths
• Mixing readiness and liveness endpoints

3. Consider this pseudocode for a health check endpoint:

if database.isConnected() and cache.isAvailable():
    return 200
else:
    return 503

What HTTP status code will the endpoint return if the database is connected but the cache is down?

medium

A. 200

B. 503

C. 404

D. 500

Solution

1. Step 1: Analyze the condition logic

   The endpoint returns 200 only if both database.isConnected() and cache.isAvailable() are true.

2. Step 2: Evaluate the given scenario

   Database is connected (true), cache is down (false), so condition fails and returns 503.

3. Final Answer:

   503 -> Option B

4. Quick Check:

   Both must be true for 200; else 503 [OK]

Hint: Both services must be healthy for 200 OK [OK]

Common Mistakes:

• Assuming 200 if only one service is healthy
• Confusing 503 with 404 or 500
• Ignoring the AND condition in the code

4. A health check endpoint code snippet:

def health_check():
    if db.connected() or cache.available():
        return 200
    else:
        return 503

What is the main problem with this health check logic?

medium

A. It does not check any services

B. It returns 503 when both services are healthy

C. It returns 200 even if only one service is healthy

D. It uses incorrect HTTP status codes

Solution

1. Step 1: Understand the logical operator used

   The code uses OR, so it returns 200 if either db.connected() or cache.available() is true.

2. Step 2: Identify the issue with health check accuracy

   This means it reports healthy even if one service is down, which can mislead monitoring systems.

3. Final Answer:

   It returns 200 even if only one service is healthy -> Option C

4. Quick Check:

   OR allows partial health, which is incorrect [OK]

Hint: Use AND to ensure all services are healthy [OK]

Common Mistakes:

• Using OR instead of AND in health checks
• Ignoring partial failures in health status
• Misinterpreting HTTP status codes

5. You design a system with a /health endpoint for liveness and a /ready endpoint for readiness. Which of the following best describes when the /ready endpoint should return HTTP 503?

hard

A. When the service is completely down and unreachable

B. When the service is alive and ready to accept traffic

C. When the service is overloaded but still responding

D. When the service is alive but not ready to accept traffic due to missing dependencies

Solution

1. Step 1: Understand readiness vs liveness

   Liveness (/health) checks if service is alive; readiness (/ready) checks if it can handle requests safely.

2. Step 2: Identify readiness failure condition

   Readiness returns 503 if dependencies (like database) are missing, so service should not receive traffic yet.

3. Final Answer:

   When the service is alive but not ready to accept traffic due to missing dependencies -> Option D

4. Quick Check:

   Readiness 503 = alive but not ready [OK]

Hint: Ready means all dependencies OK; else 503 [OK]

Common Mistakes:

• Confusing liveness failure with readiness failure
• Returning 503 when service is fully down (should be liveness)
• Assuming overload means readiness failure