Health checks in HLD - System Design Exercise

Design: Health Check System for Distributed Services

In scope: Designing the health check system architecture, data flow, and scaling. Out of scope: Implementation details of individual service health checks or alerting system internals.

Functional Requirements

FR1: Periodically verify the status of multiple services in a distributed system

FR2: Detect if a service is up, down, or degraded

FR3: Provide a dashboard or API to show current health status of all services

FR4: Send alerts when a service becomes unhealthy

FR5: Support different types of health checks: simple ping, HTTP status, and custom checks

FR6: Allow configuration of check frequency and timeout per service

Non-Functional Requirements

NFR1: Must handle monitoring at least 1000 services concurrently

NFR2: Health check latency should be under 1 second per check

NFR3: System availability target: 99.9% uptime

NFR4: Minimal impact on monitored services (lightweight checks)

NFR5: Scalable to add more services without major redesign

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

❓ Question 6

Key Components

Health check scheduler to trigger checks at configured intervals

Health check executor to perform actual checks

Data store to save health status and history

API or dashboard to display health information

Alerting mechanism for unhealthy services

Load balancer or queue to distribute health check tasks

Design Patterns

Polling pattern for periodic checks

Circuit breaker pattern to avoid repeated checks on failing services

Caching to reduce load and improve response times

Bulkhead pattern to isolate failures

Event-driven architecture for alerting

Reference Architecture

                +---------------------+
                |   Health Check UI    |
                +----------+----------+
                           |
                           v
                +---------------------+
                |  Health Check API    |
                +----------+----------+
                           |
           +---------------+----------------+
           |                                |
+----------v----------+           +---------v----------+
| Health Check        |           | Alerting Service   |
| Scheduler           |           +--------------------+
+----------+----------+
           |
           v
+---------------------+
| Health Check        |
| Executor Pool       |
+----------+----------+
           |
           v
+---------------------+
| Monitored Services  |
+---------------------+

Data Store (e.g. time-series DB) connected to API and Scheduler for storing health status

Components

Health Check Scheduler

Custom service or cron-based scheduler

Triggers health checks at configured intervals for each service

Health Check Executor Pool

Thread pool or worker queue system

Performs actual health checks concurrently without overloading services

Health Check API

REST API server

Provides access to current and historical health data for UI and external clients

Health Check UI

Web dashboard

Displays health status of all monitored services in real-time

Alerting Service

Notification system (email, SMS, webhook)

Sends alerts when services become unhealthy or recover

Data Store

Time-series database (e.g., Prometheus, InfluxDB)

Stores health check results and history for analysis and display

Request Flow

1. 1. Scheduler reads service list and check configurations.

2. 2. Scheduler triggers health check tasks at configured intervals.

3. 3. Executor pool performs health checks (ping, HTTP, custom) concurrently.

4. 4. Executor reports results to Data Store.

5. 5. API reads health data from Data Store to serve UI and external queries.

6. 6. Alerting Service subscribes to health status changes and sends notifications.

7. 7. UI displays real-time health status and history to users.

Database Schema

Entities: - Service: id, name, type, check_config (frequency, timeout, check_type) - HealthCheckResult: id, service_id (FK), timestamp, status (up/down/degraded), response_time, details Relationships: - One Service has many HealthCheckResults - HealthCheckResult stores each check's outcome for a service

Scaling Discussion

Bottlenecks

Scheduler overwhelmed by large number of services to check frequently

Executor pool saturates causing delays in health checks

Data Store write/read bottlenecks with high volume of health data

Alerting system overwhelmed by frequent status changes

UI/API latency increases with large data volume

Solutions

Partition services and distribute scheduling across multiple scheduler instances

Scale executor pool horizontally with worker queues and rate limiting per service

Use scalable time-series databases with sharding and retention policies

Implement alert deduplication and rate limiting to reduce noise

Cache recent health data and paginate UI/API responses for performance

Interview Tips

Time: Spend 10 minutes understanding requirements and clarifying scope, 20 minutes designing architecture and data flow, 10 minutes discussing scaling and trade-offs, 5 minutes summarizing.

Explain importance of lightweight, periodic health checks to avoid service overload

Discuss different types of health checks and their trade-offs

Describe how scheduler and executor components work together

Highlight data storage choice for time-series health data

Address alerting strategy to avoid alert fatigue

Discuss scaling challenges and solutions clearly

Show awareness of availability and latency targets

Practice

1. What is the main purpose of a health check in system design?

easy

A. To verify if system components are working properly

B. To increase the system's storage capacity

C. To speed up the system's processing power

D. To change the system's user interface

Solution

1. Step 1: Understand health check purpose

   Health checks are used to confirm if parts of a system are functioning correctly.

2. Step 2: Compare options with purpose

   Only To verify if system components are working properly matches this purpose; others relate to unrelated system improvements.

3. Final Answer:

   To verify if system components are working properly -> Option A

4. Quick Check:

   Health check = Verify system health [OK]

Hint: Health checks confirm system parts work, not improve features [OK]

Common Mistakes:

• Confusing health checks with performance tuning
• Thinking health checks change system features
• Assuming health checks increase capacity

2. Which of the following is a common way to implement a health check endpoint?

easy

A. A complex database query

B. A user login form

C. A file upload service

D. A simple HTTP GET request returning status

Solution

1. Step 1: Identify typical health check method

   Health checks often use simple HTTP GET requests to return system status.

2. Step 2: Eliminate unrelated options

   Database queries, login forms, and file uploads are not standard health check methods.

3. Final Answer:

   A simple HTTP GET request returning status -> Option D

4. Quick Check:

   Health check = Simple HTTP GET [OK]

Hint: Health checks usually use simple HTTP GET endpoints [OK]

Common Mistakes:

• Choosing complex operations for health checks
• Confusing health checks with user features
• Selecting unrelated services like file upload

3. Consider this pseudo-code for a health check endpoint:

function healthCheck() {
  if (database.isConnected() && cache.isAvailable()) {
    return 200;
  } else {
    return 503;
  }
}

What HTTP status code will be returned if the database is connected but the cache is down?

medium

A. 503

B. 500

C. 404

D. 200

Solution

1. Step 1: Analyze condition in code

   The code returns 200 only if both database and cache are healthy.

2. Step 2: Apply given scenario

   Database is connected (true), cache is down (false), so condition fails and returns 503.

3. Final Answer:

   503 -> Option A

4. Quick Check:

   Partial failure returns 503 [OK]

Hint: Both must be healthy for 200; else 503 [OK]

Common Mistakes:

• Assuming 200 if any component is healthy
• Confusing 503 with 404 or 500
• Ignoring the AND condition in code

4. A health check endpoint is returning 200 OK even when the database is down. What is the most likely cause?

medium

A. The HTTP status code 200 means failure

B. The database is actually up

C. The health check does not verify the database connection

D. The cache is down instead

Solution

1. Step 1: Understand health check logic

   If the endpoint returns 200 even when the database is down, it likely does not check the database.

2. Step 2: Evaluate other options

   Database being up contradicts the problem; 200 means success, not failure; cache status irrelevant here.

3. Final Answer:

   The health check does not verify the database connection -> Option C

4. Quick Check:

   Missing DB check causes false 200 [OK]

Hint: If failure shows 200, check what health check verifies [OK]

Common Mistakes:

• Misinterpreting HTTP 200 as failure
• Assuming database is up without proof
• Ignoring health check code logic

5. You design a health check for a distributed system with multiple services. Which approach best ensures accurate overall health status?

hard

A. Check only the main service's status

B. Aggregate health from all critical services before responding

C. Return 200 if any one service is healthy

D. Skip health checks to reduce network load

Solution

1. Step 1: Understand distributed system health

   Overall health depends on all critical services working well, not just one.

2. Step 2: Evaluate options for accuracy

   Aggregating health from all critical services gives a true picture; checking only main or any one service is incomplete; skipping checks loses visibility.

3. Final Answer:

   Aggregate health from all critical services before responding -> Option B

4. Quick Check:

   Aggregate all critical services for true health [OK]

Hint: Combine all critical services' health for accurate status [OK]

Common Mistakes:

• Checking only one service in distributed system
• Assuming any healthy service means overall health
• Skipping health checks to save resources