Bird
Raised Fist0

Why Load Balancers Exist and What Problems They Solve

HLDmediumInterview freq: common
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Why Load Balancers Exist and What Problems They Solve
45 min · senior
senior

You are designing the infrastructure for a 50 million daily active user (DAU) startup that provides a latency-sensitive web service. The system must handle sudden traffic spikes, maintain P99 latency under 200ms, and ensure secure encrypted communication. The backend services are horizontally scalable but require efficient traffic distribution and fault tolerance. Offline delivery of requests is not supported; all requests must be routed live.

Constraints
  • Must support 50M DAU with peak QPS of ~600K requests/sec
  • P99 latency under 200ms end-to-end
  • All traffic must be encrypted (TLS termination)
  • No offline delivery; requests must be routed live
  • System must gracefully handle backend server failures without dropping requests
📋
Guided Steps
1/5
Step 1
Identify the core problems load balancers solve at scale
What are the main challenges in handling 50M DAU traffic that justify introducing load balancers?
💡 Hint
Think about traffic distribution, fault tolerance, and latency constraints at this scale.
Key Decision
Use Layer 7 HTTP reverse proxies like Envoy or NGINX to distribute traffic evenly and terminate TLS close to clients, enabling routing decisions based on request metadata.
Interviewer Expectation
Mid-level: Names Envoy or NGINX as load balancer. Senior: Explains TLS termination reduces backend load and Layer 7 routing enables fine-grained traffic control and health checks.
⚠️Common Wrong Answer
Using DNS round-robin alone - leads to uneven traffic distribution and no health checks, causing backend overload and increased latency.
Step 2
Design traffic distribution and fault tolerance mechanisms
How will the load balancer distribute requests and handle backend failures to maintain P99 < 200ms latency?
💡 Hint
Consider health checks, circuit breakers, and retry policies integrated into the load balancer.
Key Decision
Implement active health checks and circuit breakers in Envoy to detect unhealthy backends and reroute traffic dynamically, preventing requests from hitting failed nodes.
Interviewer Expectation
Mid-level: Mentions health checks and retries. Senior: Details active health probes, circuit breaker thresholds, and how these reduce tail latency by avoiding slow or failed backends.
⚠️Common Wrong Answer
Relying on passive failure detection only - causes delayed failure detection and increased P99 latency due to requests hitting unhealthy backends.
Step 3
Ensure secure encrypted communication and TLS termination
Where and how should TLS termination be handled to balance security and backend performance?
💡 Hint
Think about terminating TLS at the edge vs backend and the impact on CPU and latency.
Key Decision
Terminate TLS at the load balancer edge using Envoy’s native TLS termination to offload CPU-intensive crypto operations from backend services and enforce uniform security policies.
Interviewer Expectation
Mid-level: States TLS termination at load balancer. Senior: Explains offloading crypto reduces backend CPU usage and centralizes certificate management, improving scalability and security.
⚠️Common Wrong Answer
Terminating TLS at backend services - increases backend CPU load and complicates certificate management, risking inconsistent security.
Step 4
Design APIs for client interaction and backend registration
What core API endpoints should the load balancer expose for clients and backend services to enable traffic routing and health monitoring?
💡 Hint
Consider endpoints for client request routing, backend health reporting, and service registration/deregistration.
Key Decision
Expose a client-facing endpoint for request proxying, a backend health check endpoint for active probes, and a service registry API (e.g., via Consul or etcd integration) for backend nodes to register/deregister dynamically.
Interviewer Expectation
Mid-level: Lists basic proxy and health check endpoints. Senior: Details integration with service discovery (Consul/etcd), dynamic backend registration, and how these APIs enable zero-downtime deployments and scaling.
⚠️Common Wrong Answer
Hardcoding backend IPs without dynamic registration - leads to manual scaling and downtime during backend changes.
Step 5
Handle sudden traffic spikes and scale load balancers horizontally
How will the load balancing layer scale to handle sudden spikes beyond average traffic without violating latency SLAs?
💡 Hint
Think about horizontal scaling, stateless proxies, and consistent hashing for session affinity.
Key Decision
Deploy multiple stateless Envoy instances behind a DNS-based or anycast IP layer with consistent hashing to maintain session affinity and enable horizontal scaling without sticky state.
Interviewer Expectation
Mid-level: Mentions horizontal scaling of load balancers. Senior: Explains stateless design enables elastic scaling, consistent hashing preserves session affinity without backend state, and anycast reduces latency.
⚠️Common Wrong Answer
Using stateful load balancers with sticky sessions - limits horizontal scaling and causes uneven load distribution under spikes.
🔌
API Design Challenge
Sketch 3 core API endpoints that a load balancer system would expose to clients and backend services.
Hints
  • How does the client initiate a connection and send requests through the load balancer?
  • How does the load balancer perform health checks and route traffic to healthy backend nodes?
  • How does the backend service register or deregister itself with the load balancer?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Explains TLS termination at load balancer edge and why backend termination fails at scale
  • Describes active health checks and circuit breakers to avoid routing to unhealthy backends
  • Names Envoy or NGINX as real load balancer technology with Layer 7 routing
  • Includes dynamic backend registration via service discovery (Consul/etcd)
  • Details horizontal scaling of stateless load balancers with consistent hashing
Good to Have - 30%
  • Mentions anycast or DNS-based load balancer IP distribution
  • Discusses session affinity without sticky state to enable scaling
  • Includes retry policies and timeout configurations in load balancer
  • Describes impact of load balancer design on P99 latency under 200ms
Bonus - 10%
  • Explains how load balancer metrics and telemetry enable proactive scaling
  • Describes zero-downtime backend deployments via dynamic registration APIs
Anti-patterns to Avoid
DNS round-robin without health checks - leads to uneven traffic distribution and backend overload, causing increased latency and potential downtime.
Using stateful load balancers with sticky sessions - limits horizontal scaling and causes uneven load distribution under traffic spikes.
Terminating TLS at backend services - increases backend CPU load and complicates certificate management, risking inconsistent security.
Relying on passive failure detection only - causes delayed failure detection and increased P99 latency due to requests hitting unhealthy backends.
Hardcoding backend IPs without dynamic registration - leads to manual scaling and downtime during backend changes.
✅
Model Answer
▼ Reveal

Load balancers are critical in large-scale systems to distribute traffic evenly, maintain low latency, and ensure fault tolerance. Using Envoy or NGINX as Layer 7 reverse proxies enables TLS termination at the edge, reducing backend CPU load and centralizing security. Active health checks and circuit breakers prevent routing to unhealthy backends, maintaining P99 latency under 200ms. Dynamic backend registration via service discovery tools like Consul or etcd allows zero-downtime scaling. Stateless load balancer instances with consistent hashing and DNS or anycast IP routing enable horizontal scaling and session affinity without sticky state.

Key Decisions
  • Use Envoy for Layer 7 HTTP reverse proxy with native TLS termination to offload backend CPU
  • Implement active health checks and circuit breakers in Envoy to detect and avoid unhealthy backends
  • Integrate with Consul or etcd for dynamic backend service registration and deregistration
  • Design load balancers as stateless instances behind DNS or anycast IP for horizontal scaling
  • Use consistent hashing to maintain session affinity without sticky sessions
  • Expose APIs for client request proxying, backend health reporting, and backend registration
  • Envoy as Layer 7 reverse proxy for TLS termination and intelligent routing
  • Active health checks and circuit breakers in Envoy to maintain low P99 latency
  • TLS termination at load balancer edge to offload backend CPU and centralize security
  • Service discovery integration (Consul or etcd) for dynamic backend registration
  • Stateless load balancer instances enabling horizontal scaling and fault tolerance
  • Consistent hashing for session affinity without sticky state on load balancers
  • DNS-based or anycast IP routing to distribute client traffic across load balancer fleet
  • API endpoints for client request proxying, backend health reporting, and service registration

Practice

(1/5)
1. In the typeahead system request flow, at step 4 the Ranking Service receives candidate suggestions from the Trie Cache. What does the Ranking Service do if the personalization data for the user is missing at this step?
easy
A. It returns an empty suggestion list immediately to avoid stale results
B. It blocks the request until personalization data is fetched synchronously
C. It falls back to global popularity-based ranking without personalization
D. It queries the Trie Cache again with a different prefix to compensate

Solution

  1. Step 1: Trace Ranking Service behavior at step 4

    Ranking Service combines candidate suggestions with personalization data; if missing, it must still return relevant results.

  2. Step 2: Evaluate fallback strategies

    Blocking for personalization data (C) causes latency; empty results (A) degrade UX; querying Trie again (D) is unrelated. Falling back to global popularity ranking ensures responsiveness.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Ranking Service fallback -> global popularity ranking [check]
Hint: Missing personalization -> fallback to global popularity ranking [check]
Common Mistakes:
2. A routing system must quickly update estimated travel times on roads based on live traffic data arriving at high velocity. Which component is best suited to handle this real-time data ingestion and update?
easy
A. Use a distributed stream processing system like Apache Kafka with real-time consumers
B. Use a local in-memory cache updated manually by operators
C. Use a REST API polling service to fetch traffic data every 5 minutes
D. Use a relational database with batch updates every 10 minutes

Solution

  1. Step 1: Identify real-time ingestion needs

    Relational DB with batch updates (A) is too slow for live traffic; polling (C) adds latency and overhead; manual updates (D) are not scalable or real-time.

  2. Step 2: Evaluate streaming system suitability

    Kafka (B) supports high-throughput, low-latency ingestion and real-time processing, ideal for live traffic updates.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Kafka streams handle high-velocity data ingestion with low latency [check]
Hint: Stream processing fits real-time high-velocity data ingestion [check]
Common Mistakes:
3. For a mobile client calling a flaky backend service, which approach best balances responsiveness and failure isolation?
medium
A. Use synchronous calls with retries and exponential backoff
B. Rely on client-side caching to avoid backend calls during failures
C. Implement a circuit breaker with a short timeout and fallback response
D. Use a message queue to buffer requests and process asynchronously

Solution

  1. Step 1: Evaluate synchronous retries (A)

    Retries increase latency and risk cascading failures, bad for mobile responsiveness.

  2. Step 2: Compare asynchronous queue (C) and caching (D)

    Queues add delay and complexity; caching avoids calls but may serve stale data, not ideal for dynamic data.

  3. Step 3: Circuit breaker with short timeout and fallback (B)

    Provides quick failure detection, prevents cascading failures, and returns fallback to maintain responsiveness.

  4. Final Answer:

    Option C -> Option C

  5. Quick Check:

    Circuit breaker with fallback balances isolation and responsiveness [check]
Hint: Circuit breaker with fallback keeps mobile responsive and isolated [OK]
Common Mistakes:
4. If Kafka is removed from the architecture, what exactly breaks in message delivery when Alice sends a message to Bob?
medium
A. Message ordering is preserved but delivery latency increases significantly
B. Alice's send blocks until Bob's server responds; if Bob's server crashes, message is lost with no retry
C. Alice's client immediately shows message as delivered, but Bob never receives it
D. Messages are duplicated because no deduplication occurs without Kafka

Solution

  1. Step 1: Understand Kafka's role in message durability and ordering

    Kafka ensures exactly-once delivery semantics and ordering; without it, deduplication and ordering guarantees are lost.

  2. Step 2: Analyze failure impact

    Without Kafka, messages may be duplicated due to retries and lack of idempotent storage; blocking send or message loss is less direct.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Without Kafka, deduplication and ordering guarantees break -> duplicates occur [check]
Hint: No Kafka -> duplicates and ordering break, not just blocking [check]
Common Mistakes:
5. At 10x scale of concurrent users editing documents globally, which component is most likely to become the bottleneck first, and why?
hard
A. The client UI, because rendering many remote edits overwhelms browser resources
B. The persistent storage layer, because write throughput cannot keep up with operation volume
C. The network layer, because increased latency causes operation reordering and conflicts
D. The OT engine, because it maintains stateful transformation buffers that grow with concurrent edits

Solution

  1. Step 1: Analyze scaling challenges of OT engine

    The OT engine maintains stateful buffers and transforms operations in-memory, which grow with the number of concurrent edits and users.

  2. Step 2: Compare with other components

    Persistent storage can be sharded and scaled; network latency affects but does not bottleneck first; client UI scales per user, not globally.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    OT engine's stateful buffers grow with concurrency, limiting scale first [check]
Hint: OT engine buffers grow with concurrency, limiting scale first [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard