Health checks in HLD - Architecture Diagram

System Overview - Health checks

This system monitors the health of various services and components in a distributed application. It regularly checks if services are running properly and reports their status to ensure reliability and quick detection of failures.

Architecture Diagram

          +-------------+          +----------------+
          |             |          |                |
User --->| Load        |--------->| API Gateway    |
          | Balancer   |          |                |
          +-------------+          +----------------+
                                         |
                                         v
                                +----------------+
                                | Health Check   |
                                | Service       |
                                +----------------+
                                         |
                  +----------------------+---------------------+
                  |                                            |
          +---------------+                            +----------------+
          | Application   |                            | Monitoring     |
          | Service      |                            | Dashboard      |
          +---------------+                            +----------------+
                  |
                  v
          +---------------+
          | Database      |
          +---------------+

Components

User

user

Initiates requests to the system

Load Balancer

load_balancer

Distributes incoming requests evenly to API Gateway instances

API Gateway

api_gateway

Receives client requests and routes them to appropriate services

Health Check Service

service

Periodically checks the health status of application services

Application Service

service

Main business logic service whose health is monitored

Database

database

Stores application data and health check results

Monitoring Dashboard

monitoring

Displays health status and alerts for system operators

Request Flow - 10 Hops

User → Load Balancer

Load Balancer → API Gateway

API Gateway → Health Check Service

Health Check Service → Application Service

Application Service → Health Check Service

Health Check Service → Database

Health Check Service → API Gateway

API Gateway → Load Balancer

Load Balancer → User

Monitoring Dashboard → Database

Failure Scenario

Component Fails:Health Check Service

Impact:Health status updates stop; monitoring dashboard shows stale or no data; system operators unaware of service failures

Mitigation:Use redundant health check service instances with failover; alert operators on health check service failure; fallback to cached health data

Architecture Quiz - 3 Questions

Which component is responsible for distributing incoming user requests?

A API Gateway

B Load Balancer

C Health Check Service

D Database

Design Principle

This architecture uses a dedicated Health Check Service to monitor application health, separating monitoring logic from business logic. It ensures reliability by storing health data in a database and providing a monitoring dashboard for operators. Load balancers and API gateways manage traffic flow and isolate components, improving scalability and fault tolerance.

Practice

1. What is the main purpose of a health check in system design?

easy

A. To verify if system components are working properly

B. To increase the system's storage capacity

C. To speed up the system's processing power

D. To change the system's user interface

Solution

1. Step 1: Understand health check purpose

   Health checks are used to confirm if parts of a system are functioning correctly.

2. Step 2: Compare options with purpose

   Only To verify if system components are working properly matches this purpose; others relate to unrelated system improvements.

3. Final Answer:

   To verify if system components are working properly -> Option A

4. Quick Check:

   Health check = Verify system health [OK]

Hint: Health checks confirm system parts work, not improve features [OK]

Common Mistakes:

• Confusing health checks with performance tuning
• Thinking health checks change system features
• Assuming health checks increase capacity

2. Which of the following is a common way to implement a health check endpoint?

easy

A. A complex database query

B. A user login form

C. A file upload service

D. A simple HTTP GET request returning status

Solution

1. Step 1: Identify typical health check method

   Health checks often use simple HTTP GET requests to return system status.

2. Step 2: Eliminate unrelated options

   Database queries, login forms, and file uploads are not standard health check methods.

3. Final Answer:

   A simple HTTP GET request returning status -> Option D

4. Quick Check:

   Health check = Simple HTTP GET [OK]

Hint: Health checks usually use simple HTTP GET endpoints [OK]

Common Mistakes:

• Choosing complex operations for health checks
• Confusing health checks with user features
• Selecting unrelated services like file upload

3. Consider this pseudo-code for a health check endpoint:

function healthCheck() {
  if (database.isConnected() && cache.isAvailable()) {
    return 200;
  } else {
    return 503;
  }
}

What HTTP status code will be returned if the database is connected but the cache is down?

medium

A. 503

B. 500

C. 404

D. 200

Solution

1. Step 1: Analyze condition in code

   The code returns 200 only if both database and cache are healthy.

2. Step 2: Apply given scenario

   Database is connected (true), cache is down (false), so condition fails and returns 503.

3. Final Answer:

   503 -> Option A

4. Quick Check:

   Partial failure returns 503 [OK]

Hint: Both must be healthy for 200; else 503 [OK]

Common Mistakes:

• Assuming 200 if any component is healthy
• Confusing 503 with 404 or 500
• Ignoring the AND condition in code

4. A health check endpoint is returning 200 OK even when the database is down. What is the most likely cause?

medium

A. The HTTP status code 200 means failure

B. The database is actually up

C. The health check does not verify the database connection

D. The cache is down instead

Solution

1. Step 1: Understand health check logic

   If the endpoint returns 200 even when the database is down, it likely does not check the database.

2. Step 2: Evaluate other options

   Database being up contradicts the problem; 200 means success, not failure; cache status irrelevant here.

3. Final Answer:

   The health check does not verify the database connection -> Option C

4. Quick Check:

   Missing DB check causes false 200 [OK]

Hint: If failure shows 200, check what health check verifies [OK]

Common Mistakes:

• Misinterpreting HTTP 200 as failure
• Assuming database is up without proof
• Ignoring health check code logic

5. You design a health check for a distributed system with multiple services. Which approach best ensures accurate overall health status?

hard

A. Check only the main service's status

B. Aggregate health from all critical services before responding

C. Return 200 if any one service is healthy

D. Skip health checks to reduce network load

Solution

1. Step 1: Understand distributed system health

   Overall health depends on all critical services working well, not just one.

2. Step 2: Evaluate options for accuracy

   Aggregating health from all critical services gives a true picture; checking only main or any one service is incomplete; skipping checks loses visibility.

3. Final Answer:

   Aggregate health from all critical services before responding -> Option B

4. Quick Check:

   Aggregate all critical services for true health [OK]

Hint: Combine all critical services' health for accurate status [OK]

Common Mistakes:

• Checking only one service in distributed system
• Assuming any healthy service means overall health
• Skipping health checks to save resources