Bird
Raised Fist0

Why Load Balancers Exist and What Problems They Solve

HLDmediumInterview freq: common
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Why Load Balancers Exist and What Problems They Solve
Understanding the critical role of load balancers in scaling, availability, and fault tolerance
mediumLoad BalancersFreq: common
⚙️
Functional Requirements
  • Distribute incoming client requests evenly across multiple backend servers to prevent overload
  • Detect unhealthy backend servers and reroute traffic to healthy ones automatically
  • Support session persistence (sticky sessions) when required by applications
  • Provide SSL/TLS termination to offload cryptographic processing from backend servers
  • Enable dynamic scaling by adding or removing backend servers without downtime
  • Expose health check APIs or endpoints for monitoring backend server status
📊
Non-Functional Requirements
  • Handle peak traffic of at least 1 million requests per second (RPS) with <1ms latency overhead
  • Achieve 99.99% availability to avoid single points of failure
  • Support failover within 100ms to minimize request disruption
  • Maintain consistent request routing with <1% error rate under load
  • Scale horizontally to support 10x traffic growth without redesign
❓
Clarifying Questions to Ask
  • What is the expected peak and average traffic volume (RPS) the load balancer must handle?
  • Are sticky sessions required, or can requests be routed statelessly?
  • Should the load balancer handle SSL/TLS termination or pass through encrypted traffic?
  • What backend server health check mechanisms are preferred (HTTP, TCP, custom)?
  • Is global load balancing across multiple data centers required or just within one region?
  • What are the latency and availability SLAs for the load balancer?
Out of Scope
✗ Designing backend application logic or database scaling
✗ Implementing client-side retry or backoff mechanisms
✗ Detailed SSL certificate management lifecycle
✗ Intrusion detection or advanced security beyond basic TLS termination
📌
Assumptions
  • User base: 2 billion daily active users (DAU)
  • Average requests per user per day: 20
  • Average request size: 500 bytes
  • Backend server capacity: 10,000 RPS per server
⚡
QPS Calculation
write qps2B users × 20 req/user/day ÷ 86400 sec/day ≈ 462,963 req/sec
peak write qpsAssuming 3× peak factor: 462,963 × 3 ≈ 1,388,889 req/sec
read qpsAssuming read requests are 2× write: 462,963 × 2 ≈ 925,926 req/sec
💾
Storage Estimate
daily storage462,963 req/sec × 86400 sec × 500 bytes ≈ 20 TB/day
retention days7
total storage20 TB/day × 7 days = 140 TB
with replication140 TB × 3 replicas = 420 TB total storage
🖥️
Server Estimate
load balancer serversAssuming each LB handles 50,000 RPS, need 1,388,889 ÷ 50,000 ≈ 28 servers
backend serversAt 10,000 RPS per backend, need 1,388,889 ÷ 10,000 ≈ 139 servers
🏗️
Architecture Diagram
Architecture Diagram
          +----------------------+
          |      DNS Resolver     |
          +----------+-----------+
                     |
          +----------v-----------+
          |  Global Load Balancer|
          +----------+-----------+
                     |
          +----------v-----------+
          | Regional Load Balancer|
          +----------+-----------+
                     |
          +----------v-----------+
          |  Health Check Service|
          +----------+-----------+
                     |
          +----------v-----------+
          |  Backend Application |
          +----------------------+
🧩
Components
DNS Resolver
Resolves client domain names to IP addresses of global load balancers
Why thisUsing authoritative DNS with low TTL (e.g., Route 53, Google Cloud DNS) enables global traffic distribution and failover
Why notStatic IPs lack flexibility and cannot route traffic based on health or load
Global Load Balancer (e.g., AWS Global Accelerator, GCP Global LB)
Distributes traffic across regional load balancers based on latency and health
Why thisSupports geo-routing and failover across data centers with low latency using technologies like AWS Global Accelerator or GCP Global Load Balancer
Why notAnycast IP alone lacks fine-grained health checks and dynamic routing capabilities
Regional Load Balancer (e.g., NGINX, Envoy, or AWS ALB)
Balances traffic among backend servers within a region, performs health checks
Why thisSupports Layer 7 routing, SSL termination, and sticky sessions with mature ecosystem tools like NGINX, Envoy, or AWS Application Load Balancer
Why notLayer 4 TCP load balancers (e.g., HAProxy in TCP mode) lack HTTP-level routing features
Health Check Service
Continuously probes backend servers to detect unhealthy instances
Why thisHTTP/TCP health checks using tools like Consul or custom scripts enable fast detection and removal of unhealthy servers
Why notRelying on backend servers to self-report health risks delayed failure detection
Backend Application Servers
Serve application logic and respond to client requests
Why thisStateless or stateful servers behind load balancer enable horizontal scaling using container orchestration platforms like Kubernetes or EC2 instances
Why notDirect client access to backend servers causes uneven load and availability issues
🔄
Data Flow
1
Client → DNS Resolver
Client resolves domain to IP of global load balancer
DNS TTL is low to enable quick failover
2
Client → Global Load Balancer
Client sends request to global LB IP
Global LB routes based on latency and regional health
3
Global Load Balancer → Regional Load Balancer
Forward request to healthy regional LB
Global LB performs health checks on regional LBs
4
Regional Load Balancer → Backend Application Server
Route request to healthy backend server using load balancing algorithm
Supports sticky sessions if required
5
Health Check Service → Backend Application Server
Periodically probe backend servers for health status
Unhealthy servers are removed from routing pool
6
Backend Application Server → Regional Load Balancer
Respond to client request
Response flows back through load balancers transparently
🗄️
LoadBalancerMetrics
PK: region
region STRINGtimestamp TIMESTAMPrequest_count BIGINTerror_rate FLOATlatency_ms FLOAT
Stores metrics per region and time for load balancer monitoring
🗄️
BackendServerHealth
PK: server_id
server_id STRINGcheck_time TIMESTAMPstatus STRINGresponse_time_ms FLOAT
Tracks health check results for backend servers
🔌
GetLoadBalancerStatus
Request
{
  "region": "string"
}
Response
{
  "status": "string",
  "active_servers": "int",
  "error_rate": "float",
  "avg_latency_ms": "float"
}
Returns current status and metrics of load balancer in a region
🔌
UpdateBackendHealth
Request
{
  "server_id": "string",
  "status": "string",
  "response_time_ms": "float"
}
Response
{
  "success": "boolean",
  "message": "string"
}
API for health check service to update backend server health status
🔌
AddBackendServer
Request
{
  "server_id": "string",
  "region": "string",
  "ip_address": "string"
}
Response
{
  "success": "boolean",
  "message": "string"
}
Adds a new backend server to the load balancer pool dynamically

Practice

(1/5)
1. A global CDN operator needs to prevent a misconfiguration from propagating incorrect BGP routes that could cause widespread internet outages. Which component is primarily responsible for detecting and containing such route leaks?
easy
A. Using a centralized DNS firewall to block suspicious IP prefixes
B. Implementing BGP route filters and prefix limits on edge routers
C. Relying on client-side validation of IP addresses before routing
D. Deploying a distributed cache to store valid routing tables

Solution

  1. Step 1: Identify BGP route filtering role

    BGP route filters and prefix limits on edge routers prevent invalid or excessive route announcements, containing route leaks.

  2. Step 2: Eliminate other options

    DNS firewalls (A) do not control BGP routing; client-side validation (C) is ineffective for routing control; distributed caches (D) do not enforce routing policies.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    BGP route filters contain route leaks at the network edge [check]
Hint: Edge routers enforce BGP filters to block bad routes [check]
Common Mistakes:
2. A system requires storing user profiles with flexible schemas and frequent updates to nested attributes. Which MongoDB data modeling pattern best supports this use case?
easy
A. Use an embedded documents pattern to store nested attributes within the user document for atomic updates.
B. Use a normalized pattern with references to separate collections for nested data to avoid duplication.
C. Use a wide-column store pattern to store user attributes as columns for fast columnar queries.
D. Use a graph database pattern to model user profiles and their nested attributes as nodes and edges.

Solution

  1. Step 1: Identify schema flexibility and update frequency

    User profiles have flexible nested attributes updated frequently, favoring atomic updates.

  2. Step 2: Evaluate embedded documents pattern

    Embedding nested attributes allows atomic updates within a single document, reducing complexity and latency.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Embedded documents enable atomic nested updates [check]
Hint: Embed nested data for atomic updates in flexible schemas [OK]
Common Mistakes:
3. For a distributed cache with frequent node joins and leaves, which approach best balances minimal key remapping and low client complexity?
medium
A. Use a centralized directory service to map keys to nodes
B. Use consistent hashing with virtual nodes and client-side slot caching
C. Use simple modulo hashing with client retries on failures
D. Use random key assignment with periodic full cache rebuilds

Solution

  1. Step 1: Evaluate centralized directory approach

    Centralized directory (Use a centralized directory service to map keys to nodes) adds single point of failure and latency, increasing client complexity.

  2. Step 2: Compare consistent hashing with virtual nodes

    Consistent hashing with virtual nodes and client-side caching (Use consistent hashing with virtual nodes and client-side slot caching) minimizes remapping and keeps client complexity manageable.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Consistent hashing + virtual nodes + client caching -> minimal remap, manageable complexity [check]
Hint: Consistent hashing + virtual nodes + client caching balances remap and complexity [OK]
Common Mistakes:
4. Without Kafka in the architecture, what exactly breaks in the message delivery flow when Alice sends a message to Bob?
medium
A. Delivery receipts are sent out of order causing UI inconsistencies
B. Alice's message is encrypted incorrectly causing decryption failures on Bob's side
C. Alice's send blocks until Bob's server responds; if Bob's server crashes, message is lost with no retry
D. Messages are duplicated because of lack of idempotent writes

Solution

  1. Step 1: Understand Kafka's role in decoupling send and delivery

    Kafka buffers messages asynchronously; without it, Chat Server must synchronously contact Bob's server.

  2. Step 2: Trace failure without Kafka

    If Bob's server is slow or crashes, Alice's send blocks or message is lost with no retry, breaking reliability.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    No Kafka -> synchronous send blocks, message loss on Bob's crash [check]
Hint: No Kafka -> synchronous send blocks, no retry, message lost [check]
Common Mistakes:
5. At 10x scale, which component of the Snowflake ID generation system is most likely to become the bottleneck first, and why?
hard
A. The timestamp source, because clock synchronization across machines becomes impossible at scale.
B. The network bandwidth, because ID generation requires frequent coordination between machines.
C. The sequence number, because generating more than its max IDs per millisecond causes blocking delays.
D. The machine ID allocation, because limited machine ID bits cause ID collisions as machines increase.

Solution

  1. Step 1: Analyze timestamp and network constraints

    Timestamp synchronization (A) is challenging but tolerable with NTP; network coordination (D) is minimal as IDs are generated locally.

  2. Step 2: Evaluate sequence number limits

    Sequence number (B) limits per millisecond can cause blocking but can be mitigated by load balancing or sharding.

  3. Step 3: Identify machine ID allocation bottleneck

    Machine ID bits are fixed (e.g., 10 bits), limiting the number of unique machines. At 10x scale, running out of unique machine IDs causes collisions or requires complex reallocation.

  4. Final Answer:

    Option D -> Option D

  5. Quick Check:

    Fixed machine ID bits limit scale -> collisions or reconfiguration needed [check]
Hint: Fixed machine ID bits limit scale before sequence or timestamp issues [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard