Health checks in HLD - Scalability & System Analysis

Scalability Analysis - Health checks

Growth Table: Health Checks at Different Scales

Users	Health Check Frequency	Number of Services	Monitoring Load	Alerting Complexity
100 users	Every 30 seconds	1-5 services	Low	Simple
10,000 users	Every 15 seconds	10-50 services	Moderate	Moderate
1,000,000 users	Every 5 seconds	100-500 services	High	Complex
100,000,000 users	Every 1-2 seconds	1000+ services	Very High	Very Complex

First Bottleneck

The first bottleneck is the monitoring system's capacity to process health check requests and responses. As the number of services and frequency of checks increase, the monitoring server CPU, memory, and network bandwidth can become overwhelmed. This leads to delayed or missed health status updates, reducing system reliability.

Scaling Solutions

• Horizontal Scaling: Deploy multiple monitoring servers to distribute health check load.
• Health Check Aggregation: Use local agents or sidecars to aggregate health data before sending to central monitoring.
• Adaptive Frequency: Reduce check frequency for stable services, increase for critical or unstable ones.
• Caching and Throttling: Cache recent health results and throttle redundant checks to reduce load.
• Asynchronous Checks: Use event-driven or push-based health reporting instead of polling.
• Sharding Monitoring Data: Partition monitoring data by service groups or regions to reduce single point load.

Back-of-Envelope Cost Analysis

Assuming each health check request and response is ~1 KB:

• At 1,000 services checked every 5 seconds: 1,000 * (1/5) = 200 checks/sec.
• Network bandwidth: 200 KB/sec (~1.6 Mbps) - manageable on typical servers.
• CPU: Each check requires processing; 200 checks/sec is moderate load.
• Storage: Logs and history can grow quickly; consider retention policies.

At 100,000 services with 1-second checks, load is 100,000 checks/sec, requiring distributed monitoring and efficient aggregation.

Interview Tip

When discussing health checks scalability, start by defining the scale (number of services, check frequency). Identify the monitoring system as the bottleneck. Propose solutions like horizontal scaling, aggregation, and adaptive frequency. Discuss trade-offs between check freshness and system load. Highlight cost and complexity implications.

Self Check Question

Your monitoring system handles 1,000 health check requests per second. Traffic grows 10x. What do you do first and why?

Answer: First, implement horizontal scaling by adding more monitoring servers to distribute the load. This prevents overload and maintains timely health status updates.

Key Result

Health checks scale by increasing monitoring capacity and optimizing check frequency; the monitoring system is the first bottleneck as service count and check frequency grow.

Practice

1. What is the main purpose of a health check in system design?

easy

A. To verify if system components are working properly

B. To increase the system's storage capacity

C. To speed up the system's processing power

D. To change the system's user interface

Solution

1. Step 1: Understand health check purpose

   Health checks are used to confirm if parts of a system are functioning correctly.

2. Step 2: Compare options with purpose

   Only To verify if system components are working properly matches this purpose; others relate to unrelated system improvements.

3. Final Answer:

   To verify if system components are working properly -> Option A

4. Quick Check:

   Health check = Verify system health [OK]

Hint: Health checks confirm system parts work, not improve features [OK]

Common Mistakes:

• Confusing health checks with performance tuning
• Thinking health checks change system features
• Assuming health checks increase capacity

2. Which of the following is a common way to implement a health check endpoint?

easy

A. A complex database query

B. A user login form

C. A file upload service

D. A simple HTTP GET request returning status

Solution

1. Step 1: Identify typical health check method

   Health checks often use simple HTTP GET requests to return system status.

2. Step 2: Eliminate unrelated options

   Database queries, login forms, and file uploads are not standard health check methods.

3. Final Answer:

   A simple HTTP GET request returning status -> Option D

4. Quick Check:

   Health check = Simple HTTP GET [OK]

Hint: Health checks usually use simple HTTP GET endpoints [OK]

Common Mistakes:

• Choosing complex operations for health checks
• Confusing health checks with user features
• Selecting unrelated services like file upload

3. Consider this pseudo-code for a health check endpoint:

function healthCheck() {
  if (database.isConnected() && cache.isAvailable()) {
    return 200;
  } else {
    return 503;
  }
}

What HTTP status code will be returned if the database is connected but the cache is down?

medium

A. 503

B. 500

C. 404

D. 200

Solution

1. Step 1: Analyze condition in code

   The code returns 200 only if both database and cache are healthy.

2. Step 2: Apply given scenario

   Database is connected (true), cache is down (false), so condition fails and returns 503.

3. Final Answer:

   503 -> Option A

4. Quick Check:

   Partial failure returns 503 [OK]

Hint: Both must be healthy for 200; else 503 [OK]

Common Mistakes:

• Assuming 200 if any component is healthy
• Confusing 503 with 404 or 500
• Ignoring the AND condition in code

4. A health check endpoint is returning 200 OK even when the database is down. What is the most likely cause?

medium

A. The HTTP status code 200 means failure

B. The database is actually up

C. The health check does not verify the database connection

D. The cache is down instead

Solution

1. Step 1: Understand health check logic

   If the endpoint returns 200 even when the database is down, it likely does not check the database.

2. Step 2: Evaluate other options

   Database being up contradicts the problem; 200 means success, not failure; cache status irrelevant here.

3. Final Answer:

   The health check does not verify the database connection -> Option C

4. Quick Check:

   Missing DB check causes false 200 [OK]

Hint: If failure shows 200, check what health check verifies [OK]

Common Mistakes:

• Misinterpreting HTTP 200 as failure
• Assuming database is up without proof
• Ignoring health check code logic

5. You design a health check for a distributed system with multiple services. Which approach best ensures accurate overall health status?

hard

A. Check only the main service's status

B. Aggregate health from all critical services before responding

C. Return 200 if any one service is healthy

D. Skip health checks to reduce network load

Solution

1. Step 1: Understand distributed system health

   Overall health depends on all critical services working well, not just one.

2. Step 2: Evaluate options for accuracy

   Aggregating health from all critical services gives a true picture; checking only main or any one service is incomplete; skipping checks loses visibility.

3. Final Answer:

   Aggregate health from all critical services before responding -> Option B

4. Quick Check:

   Aggregate all critical services for true health [OK]

Hint: Combine all critical services' health for accurate status [OK]

Common Mistakes:

• Checking only one service in distributed system
• Assuming any healthy service means overall health
• Skipping health checks to save resources