Health check pattern in Microservices - System Design Exercise

Design: Microservices Health Check System

Design focuses on health check pattern implementation in a microservices environment including health endpoints, aggregation, and monitoring integration. Out of scope are detailed alerting rules and remediation automation.

Functional Requirements

FR1: Each microservice must expose a health check endpoint.

FR2: Health checks should verify service dependencies like databases and external APIs.

FR3: The system should aggregate health status of all microservices.

FR4: Health status must be accessible for monitoring tools and alerting systems.

FR5: Health checks should be lightweight and fast to avoid overhead.

FR6: Support both readiness and liveness probes for container orchestration.

Non-Functional Requirements

NFR1: Handle up to 100 microservices in the system.

NFR2: Health check response time should be under 200ms (p99).

NFR3: System availability target is 99.9% uptime.

NFR4: Health check endpoints must not cause side effects or heavy load.

NFR5: Support secure access to health endpoints to prevent unauthorized use.

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

Key Components

Health check endpoints in each microservice

Health aggregator service or dashboard

Monitoring and alerting tools integration

Service registry or discovery for locating services

Authentication and authorization for health endpoints

Design Patterns

Health check pattern (liveness and readiness probes)

Circuit breaker pattern for dependency checks

Bulkhead pattern to isolate failures

Push vs pull model for health data aggregation

Reference Architecture

                    +---------------------+
                    |  Monitoring System   |
                    +----------+----------+
                               |
                               | Pull aggregated health data
                               v
                    +---------------------+
                    | Health Aggregator    |
                    | (Service or Dashboard)|
                    +----------+----------+
                               |
               +---------------+---------------+
               |                               |
       +-------v-------+               +-------v-------+
       | Microservice 1|               | Microservice N|
       | +-----------+ |               | +-----------+ |
       | | Health    | |               | | Health    | |
       | | Endpoint  | |               | | Endpoint  | |
       | +-----------+ |               | +-----------+ |
       +---------------+               +---------------+

Components

Microservice Health Endpoint

HTTP REST endpoint (e.g., /health, /ready, /live)

Expose health status of the microservice and its dependencies.

Health Aggregator

Custom service or monitoring dashboard

Collect and aggregate health data from all microservices.

Monitoring System

Prometheus, Grafana, or similar

Visualize health status and trigger alerts on failures.

Service Registry

Consul, Eureka, or Kubernetes API

Discover microservice instances to query health endpoints.

Security Layer

API Gateway, OAuth2, mTLS

Protect health endpoints from unauthorized access.

Request Flow

1. 1. Each microservice exposes /health, /ready, and /live endpoints.

2. 2. Health endpoint checks internal status and dependencies (database, external APIs).

3. 3. Health Aggregator queries all microservices' health endpoints periodically.

4. 4. Aggregator compiles overall system health and exposes summary endpoint.

5. 5. Monitoring system pulls aggregated health data for visualization and alerting.

6. 6. Alerts are triggered if any microservice reports unhealthy status.

7. 7. Security layer ensures only authorized systems can access health endpoints.

Database Schema

No persistent database required specifically for health checks. Health status is ephemeral and queried live. If storing history, a simple schema with tables: Services(service_id, name, last_checked), HealthStatus(service_id, timestamp, status, details) can be used.

Scaling Discussion

Bottlenecks

Health Aggregator becomes a bottleneck when querying many services frequently.

Network overhead from frequent health checks impacts service performance.

Security layer adds latency if not optimized.

Monitoring system overwhelmed with too many health metrics.

Solutions

Implement caching and rate limiting in Health Aggregator to reduce load.

Use asynchronous or event-driven health reporting (push model) to reduce polling.

Optimize security with lightweight tokens or mutual TLS to reduce handshake overhead.

Aggregate metrics at microservice level before sending to monitoring to reduce volume.

Interview Tips

Time: Spend 10 minutes understanding requirements and clarifying scope, 20 minutes designing components and data flow, 10 minutes discussing scaling and trade-offs, 5 minutes summarizing.

Explain difference between liveness and readiness probes.

Discuss importance of checking dependencies in health endpoints.

Describe how aggregation helps centralized monitoring.

Highlight security considerations for health endpoints.

Address scaling challenges and mitigation strategies.

Practice

1. What is the main purpose of the health check pattern in microservices?

easy

A. To regularly verify if a service is running and responsive

B. To increase the size of the service database

C. To encrypt communication between services

D. To deploy new versions of the service automatically

Solution

1. Step 1: Understand the health check pattern purpose

   The health check pattern is designed to monitor if a microservice is alive and functioning properly by sending simple requests to it.

2. Step 2: Identify the correct purpose among options

   Only To regularly verify if a service is running and responsive describes this monitoring function, while others describe unrelated tasks like database size, encryption, or deployment.

3. Final Answer:

   To regularly verify if a service is running and responsive -> Option A

4. Quick Check:

   Health check = verify service status [OK]

Hint: Health check means checking if service is alive and working [OK]

Common Mistakes:

• Confusing health check with deployment or encryption
• Thinking health check changes service data
• Assuming health check increases service capacity

2. Which of the following is the correct way to implement a health check endpoint in a microservice?

easy

A. Create an endpoint like /health that returns status 200 if service is healthy

B. Create an endpoint that deletes all data when called

C. Create an endpoint that returns service logs only

D. Create an endpoint that restarts the service automatically

Solution

1. Step 1: Identify typical health check endpoint behavior

   A health check endpoint usually responds with a simple status code like 200 OK to indicate the service is healthy.

2. Step 2: Match this behavior with the options

   Create an endpoint like /health that returns status 200 if service is healthy matches this by returning status 200 on /health. Other options perform unrelated or harmful actions.

3. Final Answer:

   Create an endpoint like /health that returns status 200 if service is healthy -> Option A

4. Quick Check:

   Health endpoint = status 200 OK [OK]

Hint: Health endpoint returns 200 OK if service is fine [OK]

Common Mistakes:

• Confusing health check with data deletion
• Expecting health check to return logs
• Assuming health check restarts service

3. Consider this pseudocode for a health check endpoint:

function healthCheck() {
  if (database.isConnected() && cache.isAvailable()) {
    return { status: 200, message: 'Healthy' };
  } else {
    return { status: 503, message: 'Unhealthy' };
  }
}

What will the endpoint return if the database is connected but the cache is down?

medium

A. An error is thrown

B. { status: 200, message: 'Healthy' }

C. { status: 404, message: 'Not Found' }

D. { status: 503, message: 'Unhealthy' }

Solution

1. Step 1: Analyze the condition in the healthCheck function

   The function returns healthy only if both database.isConnected() and cache.isAvailable() are true.

2. Step 2: Evaluate the given scenario

   Database is connected (true), cache is down (false), so the condition is false and the function returns status 503 with 'Unhealthy'.

3. Final Answer:

   { status: 503, message: 'Unhealthy' } -> Option D

4. Quick Check:

   Both checks true = 200, else 503 [OK]

Hint: Both dependencies must be healthy for 200 OK [OK]

Common Mistakes:

• Assuming partial health returns 200 OK
• Confusing 503 with 404 status
• Expecting an error instead of status response

4. A microservice health check endpoint is implemented as follows:

GET /health
Response: { "status": "ok" }

But monitoring tools report the service as unhealthy. What is the likely problem?

medium

A. The service is actually down and not responding

B. The endpoint does not return an HTTP status code 200

C. The endpoint URL should be /status instead of /health

D. The response body is missing the word 'healthy'

Solution

1. Step 1: Understand monitoring tool expectations

   Most monitoring tools expect the health check endpoint to return HTTP status code 200 to mark service healthy.

2. Step 2: Identify the issue with the current implementation

   The response body contains status 'ok' but if the HTTP status code is not 200, tools may mark it unhealthy.

3. Final Answer:

   The endpoint does not return an HTTP status code 200 -> Option B

4. Quick Check:

   Health check needs HTTP 200 status [OK]

Hint: Health check must return HTTP 200 status code [OK]

Common Mistakes:

• Thinking response body text controls health status
• Assuming endpoint URL name matters
• Ignoring actual service availability

5. You design a microservice system with multiple services. To improve reliability, you want to implement health checks that also verify database and cache connectivity. Which approach best follows the health check pattern?

hard

A. Each service exposes a /health endpoint that always returns 200 regardless of dependency status

B. Only one central service exposes a health check endpoint for all services combined

C. Each service exposes a /health endpoint that returns 200 only if all dependencies (database, cache) are reachable; otherwise returns 503

D. Health checks are done by querying the database directly without service endpoints

Solution

1. Step 1: Understand health check pattern for dependencies

   Health checks should verify the service and its critical dependencies like database and cache to ensure full functionality.

2. Step 2: Evaluate the options for best practice

   Each service exposes a /health endpoint that returns 200 only if all dependencies (database, cache) are reachable; otherwise returns 503 correctly implements a health endpoint that returns 200 only if all dependencies are healthy, otherwise 503. This supports automatic recovery and monitoring.

3. Final Answer:

   Each service exposes a /health endpoint that returns 200 only if all dependencies (database, cache) are reachable; otherwise returns 503 -> Option C

4. Quick Check:

   Health check includes dependencies, returns 200 or 503 [OK]

Hint: Health check must verify all critical dependencies [OK]

Common Mistakes:

• Ignoring dependency status in health check
• Relying on a single central health endpoint
• Checking dependencies outside service endpoints