Bird
Raised Fist0

Health Checks and Automatic Failover in Load Balancers

HLDmediumInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
▶
Request Trace
Step 1/6
Load Balancer

Sends HTTP health check request

Load Balancer probes backend server health via HTTP

Protocol:HTTP
BeforeLoad Balancer has no recent health status for backend
AfterHealth Check Service receives HTTP probe request
Health Check Service

Processes health check and detects failure

Health Check Service determines backend is unhealthy

Protocol:HTTP Response
BeforeHealth Check Service has last known healthy status
AfterHealth Check Service marks backend as unhealthy
Health Check Service

Publishes failover event to Kafka

Health Check Service emits backend failure event

Protocol:Kafka Producer
BeforeKafka has no recent failover event for backend
AfterKafka stores backend failure event for consumers
Cassandra

Updates backend health state

Cassandra writes new unhealthy status and failover metadata

Protocol:CQL Write
BeforeCassandra has previous healthy backend state
AfterCassandra reflects backend as unhealthy with timestamp
Prometheus

Scrapes metrics from Cassandra and Load Balancer

Prometheus collects health and failover metrics

Protocol:Prometheus Pull
BeforePrometheus has last scrape data with healthy backend
AfterPrometheus metrics updated with backend failure info
Monitoring & Alerting Dashboard

Displays alert for backend failure

Dashboard shows alert based on Prometheus metrics

Protocol:Dashboard API
BeforeDashboard shows no active alerts for backend
AfterDashboard displays backend failure alert to operators
🏗️
Architecture Diagram
Architecture Diagram
┌───────────────────────┐
│     Load Balancer     │
└────────────┬──────────┘
             │ HTTP Health Check
             ▼
┌───────────────────────┐
│ Health Check Service  │
└────────────┬──────────┘
             │ Kafka Event
             ▼
        ┌────────┐
        │ Kafka  │
        └──┬─────┘
           │ Cassandra Write
           ▼
   ┌───────────────┐
   │   Cassandra   │
   └──────┬────────┘
          │ Prometheus Metrics
          ▼
   ┌───────────────────────────────┐
   │        Prometheus             │
   └─────────────┬─────────────────┘
                 │ Metrics Pull
                 ▼
   ┌───────────────────────────────┐
   │ Monitoring & Alerting Dashbd  │
   └───────────────────────────────┘
Design Rationale

This architecture enables automatic failover by integrating health checks with event-driven state updates. The Load Balancer initiates HTTP probes to the backend, which the Health Check Service evaluates. Upon detecting failure, it publishes an event to Kafka, ensuring reliable propagation of the failure state. Cassandra stores the updated backend health status and failover metadata, providing durable state. Prometheus scrapes metrics from Cassandra and the Load Balancer to monitor system health continuously. The Monitoring & Alerting Dashboard visualizes these metrics and alerts operators promptly. This design decouples health detection from failover logic, leverages Kafka for event durability, and uses Prometheus for observability, enabling robust automatic failover in load balancing.

Primary Focus

✓

Practice

(1/5)
1. A web application needs a cache that automatically removes entries after a fixed time regardless of access frequency. Which cache eviction policy best fits this requirement?
easy
A. Least Frequently Used (LFU) eviction, which removes items accessed least often
B. Least Recently Used (LRU) eviction, which removes the oldest accessed items first
C. Time-To-Live (TTL) eviction, which expires entries after a set duration
D. Adaptive Replacement Cache (ARC), which balances recency and frequency

Solution

  1. Step 1: Identify eviction based on time expiration

    TTL eviction removes cache entries strictly after a fixed time, independent of access patterns.

  2. Step 2: Eliminate other policies

    LRU and LFU depend on access recency or frequency, not fixed time. ARC adapts between recency and frequency but does not enforce fixed expiration.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    TTL evicts by fixed time, ignoring usage patterns [check]
Hint: TTL evicts by time, not usage frequency or recency [OK]
Common Mistakes:
2. A distributed key-value store must evenly distribute keys across nodes and handle node failures without massive data reshuffling. Which component primarily enables this?
easy
A. Gossip protocol to propagate membership and failure information
B. Vector clocks to resolve conflicting versions of data
C. Consistent hashing to assign keys to nodes with minimal remapping
D. Quorum-based replication to ensure data durability

Solution

  1. Step 1: Identify the role of consistent hashing

    Consistent hashing distributes keys to nodes such that when nodes join or leave, only a small subset of keys need reassignment, minimizing reshuffling.

  2. Step 2: Eliminate other components

    Vector clocks handle version conflicts but do not distribute keys. Gossip protocol manages cluster membership but not key distribution. Quorum replication ensures durability but does not affect key assignment.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Consistent hashing minimizes key remapping on node changes [check]
Hint: Consistent hashing minimizes key remapping on node changes [OK]
Common Mistakes:
3. In the SLO error budget tracking system, at step 4 the Error Budget Calculator receives aggregated error counts and total request counts. What state does it maintain immediately after processing these inputs?
easy
A. It resets the error budget to full capacity regardless of recent errors
B. It discards error counts if the total request count is below a threshold
C. It queues the error counts for asynchronous processing by the alerting system
D. It updates the current error budget remaining by subtracting recent error rate from total budget

Solution

  1. Step 1: Understand the role of the Error Budget Calculator at step 4

    It updates the remaining error budget by subtracting the recent error rate from the total allowed errors, reflecting current SLO compliance.

  2. Step 2: Exclude other states that contradict error budget tracking logic

    Resetting the budget ignores recent errors; queuing is done later by alerting; discarding counts would lose accuracy.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Error Budget Calculator updates remaining budget after aggregation [check]
Hint: Error budget updates subtract recent errors, not reset or discard [OK]
Common Mistakes:
4. At 10x scale, which component in a consistent hashing database sharding system is most likely to fail first and why?
hard
A. The central shard metadata service, due to increased coordination overhead
B. The individual database shards, because storage capacity limits are reached first
C. The hash ring management service, because updating and propagating ring changes becomes a bottleneck
D. The client-side routing cache, as it cannot handle the increased number of shards efficiently

Solution

  1. Step 1: Identify scaling bottlenecks

    Central metadata service (The central shard metadata service, due to increased coordination overhead) may be distributed or avoided; shards (The individual database shards, because storage capacity limits are reached first) scale horizontally; client caches (The client-side routing cache, as it cannot handle the increased number of shards efficiently) can be sharded or refreshed incrementally.

  2. Step 2: Analyze hash ring management at scale

    Hash ring management (The hash ring management service, because updating and propagating ring changes becomes a bottleneck) requires updating and propagating ring changes to all nodes; at 10x scale, this coordination overhead and update frequency cause it to become the first bottleneck.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Hash ring updates bottleneck first at large scale [check]
Hint: Hash ring updates bottleneck first at large scale [OK]
Common Mistakes:
5. At 10x scale, which component in the distributed logging pipeline is most likely to fail first and why?
hard
A. The Kafka brokers, because partition leader election latency increases with partitions
B. The storage layer, due to inability to handle increased query load
C. The enrichment service, due to CPU bottlenecks on metadata joins
D. The ingestion API servers, because they maintain stateful TCP connections per client

Solution

  1. Step 1: Identify bottlenecks at 10x scale

    Kafka brokers manage partitions and leader elections; more partitions increase coordination overhead.

  2. Step 2: Compare other components

    Storage and enrichment scale horizontally; ingestion APIs are stateless and can be scaled with load balancers.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Kafka leader election latency grows with partitions -> first bottleneck [check]
Hint: Kafka partition leader election slows at scale first [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard