Bird
Raised Fist0

Health Checks and Automatic Failover in Load Balancers

HLDmediumInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Health Checks and Automatic Failover in Load Balancers
Designing robust health checks and automatic failover mechanisms to ensure high availability and fault tolerance in load balancers at scale.
mediumLoad BalancersFreq: high
⚙️
Functional Requirements
  • Perform periodic health checks on backend servers to detect failures or degraded performance.
  • Automatically remove unhealthy backend servers from the load balancing pool to prevent routing traffic to them.
  • Reintegrate recovered backend servers after passing health checks to restore capacity.
  • Support multiple health check protocols (HTTP, TCP, ICMP) configurable per backend service.
  • Provide real-time health status monitoring and alerting for backend servers.
  • Ensure minimal impact on request latency during health check and failover operations.
📊
Non-Functional Requirements
  • Support at least 2 billion daily active users (DAU) with backend server pools of up to 10,000 nodes per region.
  • Health check latency must be under 100ms P99 to avoid delaying request routing decisions.
  • System availability SLA of 99.99% to minimize downtime due to backend failures.
  • Automatic failover detection and removal of unhealthy nodes within 5 seconds of failure.
  • Health check system must scale horizontally to handle 925,000 health check probes per second at peak.
❓
Clarifying Questions to Ask
  • What types of backend services and protocols must the health checks support?
  • What is the expected frequency and timeout for health checks per backend node?
  • Should health checks be active (probes) or passive (monitoring traffic failures), or both?
  • How quickly must the system detect and failover from unhealthy nodes?
  • Is there a requirement for stateful session persistence during failover?
  • What monitoring and alerting integrations are expected for health check failures?
Out of Scope
✗ Design of the load balancing algorithm itself (e.g., round-robin, least connections).
✗ Backend server application-level failure recovery or self-healing mechanisms.
✗ Global multi-region failover and disaster recovery strategies.
✗ Client-side retry or fallback logic.
📌
Assumptions
  • 2 billion DAU generating 40 requests per user per day on average.
  • Each health check probe payload is approximately 100 bytes.
  • Each backend pool contains up to 10,000 servers per region.
  • Health checks run every 10 seconds per backend server.
⚡
QPS Calculation
write qps2B users × 40 req/user/day ÷ 86400 sec/day ≈ 925,926 req/sec
peak write qpsAssuming 2× peak factor: 925,926 × 2 = 1,851,852 req/sec
health check qps10,000 servers × (1 probe / 10 sec) = 1,000 probes/sec
read qpsAssuming 5× read amplification for monitoring dashboards and alerts: 1,000 × 5 = 5,000 reads/sec
💾
Storage Estimate
daily storage1,000 probes/sec × 100 bytes × 86400 sec = 8.64 GB/day
retention days30
total storage8.64 GB/day × 30 days = 259.2 GB
with replication259.2 GB × 3 replicas = 777.6 GB total storage
🖥️
Server Estimate
health check workersAssuming each worker handles 10,000 probes/sec, need 1,000 / 10,000 = 0.1 → 1 server for health checks
monitoring dbAssuming Cassandra node handles 100 MB/s write throughput, 8.64 GB/day ≈ 100 KB/s, 1 node sufficient but 3 nodes for replication and HA
🏗️
Architecture Diagram
Architecture Diagram
          +----------------------+
          |   Client Requests     |
          +----------+-----------+
                     |
                     v
          +----------------------+
          |    Load Balancer      |
          +----------+-----------+
                     |
         +-----------+------------+
         |                        |
         v                        v
  +-------------+          +--------------+
  | Backend Srv |          | Backend Srv  |
  +-------------+          +--------------+
         ^                        ^
         |                        |
 +-------+----------------+-------+
 | Health Check Service (Kafka +  |
 | Prometheus + Cassandra)         |
 +-------------------------------+
                     |
          +----------+-----------+
          |  Monitoring & Alerting|
          +----------------------+
🧩
Components
Load Balancer
Routes client requests to healthy backend servers based on health status.
Why thisEnvoy proxy supports dynamic endpoint health checking and integrates with service discovery for real-time failover.
Why notNginx lacks native dynamic health check integration and requires manual config reloads causing downtime.
Health Check Service
Periodically probes backend servers using configurable protocols to determine health status.
Why thisCustom health check workers using gRPC probes allow protocol flexibility and low latency checks.
Why notUsing only passive health checks (e.g., TCP resets) delays failure detection and increases false positives.
Kafka
Asynchronous message queue to decouple health check probe results ingestion from processing and storage.
Why thisKafka handles high throughput (millions of messages/sec) with strong durability and ordering guarantees.
Why notRabbitMQ does not scale linearly at 1M+ messages/sec and has higher latency under load.
Cassandra
Stores health check results with time-series data model for efficient writes and queries.
Why thisCassandra's wide-column model and tunable consistency provide high write throughput and availability.
Why notPostgreSQL requires complex manual sharding at 925K writes/sec; B-tree writes don't scale linearly.
Prometheus
Aggregates health metrics and triggers alerts on unhealthy backend servers.
Why thisPrometheus supports multi-dimensional time series data and flexible alerting rules with low query latency.
Why notNagios lacks scalable metric ingestion and real-time alerting at this scale.
Monitoring & Alerting Dashboard
Visualizes health status and alerts for operators to monitor backend server health.
Why thisGrafana integrates natively with Prometheus for real-time dashboards and alert management.
Why notCustom dashboards require significant engineering effort and lack community support.
🔄
Data Flow
1
Health Check Service → Kafka
Send health probe results asynchronously
Decouples probe execution from processing to handle burst loads

Practice

(1/5)
1. For managing user playlists with frequent updates and offline sync, which storage architecture is best suited given the need for eventual consistency and offline support?
medium
A. Conflict-free replicated data type (CRDT) based distributed store
B. Centralized NoSQL store with eventual consistency but no offline sync
C. Strongly consistent relational database with synchronous replication
D. Local device-only storage with manual sync to server

Solution

  1. Step 1: Identify requirements

    Playlists require offline edits and eventual sync with conflict resolution.

  2. Step 2: Evaluate architectures

    Relational DB (A) blocks offline edits; centralized NoSQL (C) lacks offline sync; local-only (D) risks data loss and complex merges. CRDTs (B) enable conflict-free merges and offline edits.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    CRDTs enable offline edits with automatic conflict resolution [check]
Hint: CRDTs handle offline sync and conflicts automatically [check]
Common Mistakes:
2. During a global network outage caused by BGP misconfiguration, which recovery approach balances speed and safety best for restoring connectivity without causing routing loops?
medium
A. Gradually reintroduce routes with route flap damping to prevent instability
B. Immediately re-advertise all withdrawn routes globally to restore traffic
C. Use static routing entries temporarily to bypass BGP until full recovery
D. Rely on client-side retries and exponential backoff to handle transient failures

Solution

  1. Step 1: Understand recovery trade-offs

    Immediate re-advertisement (A) risks routing loops and instability; static routes (C) don't scale globally; client retries (D) don't fix routing issues.

  2. Step 2: Evaluate gradual reintroduction

    Route flap damping (B) prevents rapid route changes causing instability, balancing speed and safety during recovery.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Route flap damping prevents loops during gradual route restoration [check]
Hint: Gradual route reintroduction with damping avoids loops [check]
Common Mistakes:
3. If the WebSocket server crashes unexpectedly, what is the immediate failure mode experienced by connected clients?
medium
A. Clients automatically reconnect without losing any messages due to built-in buffering.
B. Clients switch to Long Polling automatically to maintain connection without message loss.
C. Clients receive a connection close event and must reconnect; messages sent during downtime are lost unless application-layer retries exist.
D. Clients continue to send messages which are queued by the network until the server recovers.

Solution

  1. Step 1: Understand WebSocket failure behavior

    When the WebSocket server crashes, the TCP connection closes, and clients receive a close event.

  2. Step 2: Trace message delivery impact

    Messages sent during downtime are lost unless the application implements retries. Clients must reconnect manually or via client logic. There is no automatic buffering or fallback to Long Polling.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    WebSocket crash -> connection close event, no automatic buffering [check]
Hint: WebSocket crash -> clients disconnect, no auto-retry [check]
Common Mistakes:
4. At 10x scale, which component in your design explanation is most likely to fail first due to stateful connection bottlenecks, and why?
hard
A. Stateful WebSocket servers because connections are bound to server memory and cannot be easily resharded
B. Stateless API servers because CPU usage spikes with more requests
C. Load balancers because they cannot handle increased DNS queries
D. Databases because they run out of disk space first

Solution

  1. Step 1: Identify stateful components at scale

    WebSocket servers maintain persistent connections bound to server memory, limiting horizontal scaling.

  2. Step 2: Compare with stateless components

    Stateless API servers and load balancers scale horizontally more easily; databases fail later due to storage.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Stateful WebSocket servers bottleneck first due to connection affinity [check]
Hint: Stateful WebSocket servers bottleneck first at scale [check]
Common Mistakes:
5. At 10x scale, which component in a relational database system is most likely to fail first and why?
hard
A. The query optimizer, because it cannot handle complex plans at scale
B. The storage engine, because disk I/O bandwidth saturates first
C. The transaction coordinator, due to increased contention and locking overhead
D. The connection pool, because client connections exceed configured limits

Solution

  1. Step 1: Identify scaling bottlenecks

    At 10x scale, contention on locks and transaction coordination overhead increase dramatically.

  2. Step 2: Why transaction coordinator fails first

    Locking and coordination cause blocking and deadlocks, limiting throughput before storage or connection limits.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Transaction coordinator bottlenecks due to lock contention at scale [check]
Hint: Lock contention chokes transaction coordinator first at scale [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard