Bird
Raised Fist0

Health Checks and Automatic Failover in Load Balancers

HLDmediumInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Health Checks and Automatic Failover in Load Balancers
45 min · senior
senior

You are designing the health check and automatic failover mechanism for load balancers serving a 50 million daily active user (DAU) platform. The system must detect unhealthy backend servers quickly and reroute traffic with minimal disruption. The load balancers handle 100K requests per second with a P99 latency target of under 200ms. Health checks must be secure, encrypted, and not overload backend servers. Failover decisions must be consistent across multiple load balancers in a geo-distributed environment.

Constraints
  • Health checks must be encrypted end-to-end to prevent MITM attacks.
  • Health check frequency must balance detection speed and backend load (max 5% CPU overhead).
  • Failover latency must keep P99 request latency under 200ms.
  • System must support geo-distributed load balancers with consistent failover state.
  • Health check results must be eventually consistent within 5 seconds across all load balancers.
📋
Guided Steps
1/5
Step 1
Choosing the Health Check Protocol and Frequency
How would you design the health check mechanism to detect backend failures quickly without overloading servers?
💡 Hint
Consider lightweight protocols and adaptive frequency based on server load.
Key Decision
Use gRPC-based encrypted health check probes with TLS mutual authentication, sending lightweight status requests every 5 seconds, adjustable based on backend CPU usage metrics via Prometheus.
Interviewer Expectation
Mid-level: Names gRPC with TLS for health checks. Senior: Explains mutual TLS prevents MITM, adaptive frequency reduces CPU overhead, and Prometheus metrics integration for dynamic tuning.
⚠️Common Wrong Answer
Using HTTP polling every second without encryption - causes backend CPU spikes and exposes health data to network attackers.
Step 2
Health Status Aggregation and State Sharing
How will load balancers aggregate health check results and share failover state across geo-distributed instances?
💡 Hint
Think about distributed coordination systems that provide consistency guarantees.
Key Decision
Use Apache ZooKeeper ensemble for distributed coordination and state sharing, storing ephemeral nodes representing backend health status with session TTLs to detect failures quickly.
Interviewer Expectation
Mid-level: Mentions ZooKeeper for coordination. Senior: Details ephemeral nodes for automatic failure detection, session timeouts for quick state updates, and how ZooKeeper consistency ensures synchronized failover decisions.
⚠️Common Wrong Answer
Using a relational database for state sharing - leads to high latency and inconsistent failover decisions under network partitions.
Step 3
Failover Decision Logic and Traffic Rerouting
How should the load balancer decide when to failover and reroute traffic to healthy backends?
💡 Hint
Consider threshold-based triggers and circuit breaker patterns to avoid flapping.
Key Decision
Implement circuit breaker logic in Envoy proxy load balancers with thresholds on consecutive failed health checks and integrate with ZooKeeper state to coordinate failover, rerouting traffic only after consensus.
Interviewer Expectation
Mid-level: Describes threshold-based failover. Senior: Explains circuit breaker prevents flapping, ZooKeeper consensus avoids split-brain, and Envoy’s dynamic config reload enables zero-downtime rerouting.
⚠️Common Wrong Answer
Failing over immediately on a single failed health check - causes instability and request failures due to transient errors.
Step 4
Ensuring Security and Encryption in Health Checks
How do you ensure health check data and failover signals are secure and encrypted end-to-end?
💡 Hint
Think about mutual authentication and encryption protocols at transport layer.
Key Decision
Use mutual TLS (mTLS) with certificates issued by an internal PKI for all health check gRPC calls and ZooKeeper client connections, ensuring encrypted and authenticated communication.
Interviewer Expectation
Mid-level: Mentions TLS encryption. Senior: Details mTLS with internal PKI, certificate rotation policies, and how this prevents MITM and unauthorized failover triggers.
⚠️Common Wrong Answer
Using plain TLS without client authentication - vulnerable to impersonation and MITM attacks.
Step 5
API Design for Health Check and Failover System
Sketch 3 core API endpoints: backend health status exposure, load balancer health reporting, and failover notification.
💡 Hint
Consider REST/gRPC endpoints with authentication and payload design for each role.
Key Decision
Backend exposes gRPC HealthCheckService with Check() method returning status; load balancers report aggregated health to ZooKeeper via ephemeral nodes; failover notifications pushed via ZooKeeper watchers triggering Envoy config reloads.
Interviewer Expectation
Mid-level: Names gRPC HealthCheckService and ZooKeeper ephemeral nodes. Senior: Explains how ephemeral nodes reflect live health, watchers enable reactive failover, and secure gRPC APIs prevent spoofing.
⚠️Common Wrong Answer
Designing unencrypted HTTP endpoints without authentication - risks data leaks and unauthorized failover.
🔌
API Design Challenge
Sketch 3 core API endpoints for the health check and failover system: how load balancers report health, how backend servers expose health status, and how failover decisions are communicated.
Hints
  • Consider how the backend server exposes its health status securely.
  • Think about how load balancers aggregate and share health information.
  • Design how failover triggers and notifications propagate to all load balancers.
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Use of encrypted health check protocol with mutual TLS to secure communication
  • Distributed coordination system (ZooKeeper) for consistent failover state sharing
  • Circuit breaker pattern in load balancer to prevent failover flapping
  • Adaptive health check frequency to balance detection speed and backend load
Good to Have - 30%
  • Integration with Prometheus for backend load metrics
  • Use of ephemeral nodes and watchers in ZooKeeper for fast failure detection
  • Dynamic configuration reload in Envoy for zero downtime failover
  • Internal PKI for certificate management and rotation
Bonus - 10%
  • Designing APIs with secure gRPC endpoints and authentication
  • Consideration of geo-distributed consistency and eventual consistency guarantees
Anti-patterns to Avoid
Polling backend health via unencrypted HTTP causes security vulnerabilities and data leaks, exposing sensitive health data to attackers and risking MITM attacks.
Using relational databases for failover state sharing leads to high latency and inconsistent failover decisions under network partitions, causing potential split-brain scenarios.
Failing over immediately on a single failed health check causes instability and request failures due to transient errors, resulting in flapping and degraded user experience.
✅
Model Answer
▼ Reveal

This design uses gRPC with mutual TLS for secure, low-latency health checks that adapt frequency based on backend load metrics from Prometheus. Apache ZooKeeper manages distributed coordination with ephemeral nodes and watchers to maintain consistent failover state across geo-distributed load balancers. Envoy proxies implement circuit breaker patterns to prevent failover flapping and enable zero-downtime traffic rerouting via dynamic config reloads. An internal PKI issues certificates for mTLS, ensuring secure communication and preventing unauthorized failover triggers. The system achieves eventual consistency within 5 seconds for failover state sharing.

Key Decisions
  • Use gRPC with mutual TLS for encrypted health checks to prevent MITM attacks
  • Adjust health check frequency dynamically based on Prometheus CPU metrics to limit backend load to 5%
  • Store backend health status as ephemeral nodes in ZooKeeper with session TTLs for quick failure detection
  • Implement circuit breaker pattern in Envoy to avoid failover flapping and reroute traffic only after ZooKeeper consensus
  • Use ZooKeeper watchers to push failover notifications instantly to all load balancers
  • Employ internal PKI for certificate management ensuring secure mTLS communication
  • Design APIs with secure gRPC endpoints and authentication to prevent spoofing and unauthorized access
  • Ensure eventual consistency of health state within 5 seconds across geo-distributed load balancers
  • gRPC with mutual TLS for encrypted, low-latency health probes minimizing backend load
  • Prometheus integration for adaptive health check frequency based on backend CPU metrics
  • Apache ZooKeeper ephemeral nodes for distributed, consistent health state coordination
  • Envoy proxy with circuit breaker pattern for stable failover and zero-downtime traffic rerouting
  • Mutual TLS with internal PKI for all health check and coordination communications
  • ZooKeeper watchers to propagate failover notifications instantly to all load balancers
  • Backend servers expose standardized gRPC HealthCheckService for uniform health reporting
  • Dynamic Envoy configuration reloads triggered by ZooKeeper state changes for seamless failover

Practice

(1/5)
1. In a system trace, at step 4 the circuit breaker checks the failure count before forwarding a request. What happens if the failure count exceeds the threshold at this step?
easy
A. The circuit breaker opens and immediately returns an error without calling downstream
B. The request is forwarded but logged for monitoring
C. The circuit breaker retries the request twice before opening
D. The request is queued until the downstream service recovers

Solution

  1. Step 1: Trace circuit breaker decision at Step 4

    When failure count exceeds threshold, the circuit breaker transitions to OPEN state and stops forwarding requests.

  2. Step 2: Understand behavior on open circuit

    Requests are immediately failed (B), not forwarded (A), retried (C), or queued (D).

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Open circuit returns error immediately without downstream call [check]
Hint: Open circuit returns error immediately, no downstream call [OK]
Common Mistakes:
2. In a pub-sub system, at step 4 the Delivery Worker reads a message from the topic partition and checks the subscriber's offset. What happens if the subscriber's offset is behind the latest committed offset?
easy
A. The Delivery Worker deletes the message from the topic partition
B. The Delivery Worker skips the message and moves to the next one
C. The Delivery Worker delivers the message to the subscriber and updates the offset
D. The Delivery Worker blocks until the subscriber catches up

Solution

  1. Step 1: Understand subscriber offset semantics

    The subscriber offset tracks the last consumed message. If behind, the Delivery Worker must deliver pending messages to catch up.

  2. Step 2: Delivery Worker behavior

    It delivers the message at the subscriber's offset and then updates the offset to the next message. Skipping or deleting messages (Options A and C) breaks delivery guarantees. Blocking (The Delivery Worker blocks until the subscriber catches up) is unnecessary and stalls the system.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Delivery Worker delivers and advances subscriber offset [check]
Hint: Subscriber offset behind -> deliver message and update offset [check]
Common Mistakes:
3. In a system using Rendezvous Hashing for request routing, at Step 3 the Router computes scores for each node. What happens if a node is temporarily unreachable during this step?
easy
A. The Router waits indefinitely for the unreachable node to respond
B. The Router retries the unreachable node until it responds before proceeding
C. The Router assigns the request to a random node ignoring scores
D. The Router excludes the unreachable node and picks the next highest scoring node

Solution

  1. Step 1: Trace Router behavior at Step 3

    Router computes scores for all nodes; unreachable nodes are excluded to avoid blocking.

  2. Step 2: Understand fallback mechanism

    Router picks the highest scoring reachable node, ensuring availability and progress.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Unreachable nodes excluded -> next highest score chosen [check]
Hint: Exclude unreachable nodes, pick next best [check]
Common Mistakes:
4. For detecting anomalies in high-volume metric streams with low latency, which architecture is better suited and why?
medium
A. Batch processing with hourly aggregation to reduce noise and improve accuracy
B. Polling external ML service every 10 minutes to classify anomalies asynchronously
C. Streaming anomaly detection using windowed statistical models for near real-time alerts
D. Threshold-based alerts combined with manual tuning to approximate anomalies

Solution

  1. Step 1: Analyze latency and volume constraints

    High-volume streams require low-latency detection to be actionable.

  2. Step 2: Match architecture to constraints

    Streaming anomaly detection with windowed models processes data continuously, enabling near real-time alerts, unlike batch or polling which add latency.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Streaming windowed models -> low latency anomaly detection [check]
Hint: High volume + low latency -> streaming anomaly detection [check]
Common Mistakes:
5. For a system targeting 99.9% availability, which trade-off is best when choosing between synchronous replication and asynchronous replication for the primary database?
medium
A. Use asynchronous replication to reduce write latency but accept potential data loss during failover
B. Always use synchronous replication to avoid any data loss
C. Use synchronous replication only for read replicas to improve read availability
D. Use asynchronous replication with no failover mechanism to simplify design

Solution

  1. Step 1: Understand availability vs durability trade-off

    Synchronous replication ensures no data loss but increases write latency and risk of blocking during failures, reducing availability.

  2. Step 2: Evaluate asynchronous replication trade-offs

    Asynchronous replication reduces write latency and improves availability but risks losing recent writes on failover, acceptable for 99.9% availability.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Async replication -> lower latency, possible data loss, fits 99.9% availability [check]
Hint: Async replication trades data loss for availability at 99.9% [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard