Why load balancers distribute traffic in HLD - Design It to Understand It

Design: Load Balancer Traffic Distribution

Focus on the role and design of load balancers in distributing traffic. Out of scope are detailed server implementations and client-side logic.

Functional Requirements

FR1: Distribute incoming client requests evenly across multiple servers

FR2: Ensure high availability and fault tolerance

FR3: Improve system scalability by adding or removing servers without downtime

FR4: Maintain low latency and fast response times

FR5: Handle sudden spikes in traffic gracefully

Non-Functional Requirements

NFR1: Support at least 10,000 concurrent connections

NFR2: API response time p99 under 200ms

NFR3: Availability target of 99.9% uptime

NFR4: Support both HTTP and TCP traffic

NFR5: Minimal single point of failure

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

Key Components

Load balancer (software or hardware)

Backend servers (application servers)

Health check system to monitor server status

DNS or service discovery for load balancer endpoints

Monitoring and logging tools

Design Patterns

Round-robin load balancing

Least connections load balancing

IP hash for session persistence

Active health checks and failover

Horizontal scaling of backend servers

Reference Architecture

Client
  |
  v
Load Balancer (distributes traffic)
  |
  +----> Server 1
  |
  +----> Server 2
  |
  +----> Server 3

Health Checks monitor servers and inform Load Balancer

Components

Load Balancer

Nginx, HAProxy, AWS ELB, or similar

Receives client requests and distributes them evenly to backend servers

Backend Servers

Application servers (e.g., Node.js, Java, Python)

Process client requests and return responses

Health Check System

Built-in load balancer health checks or external monitoring

Continuously checks server health to avoid sending traffic to unhealthy servers

DNS / Service Discovery

Route53, Consul, or similar

Directs clients to the load balancer endpoint

Monitoring and Logging

Prometheus, Grafana, ELK stack

Track traffic distribution, server health, and performance metrics

Request Flow

1. Client sends a request to the load balancer endpoint.

2. Load balancer receives the request and selects a healthy backend server based on the balancing algorithm.

3. Load balancer forwards the request to the selected backend server.

4. Backend server processes the request and sends the response back to the load balancer.

5. Load balancer forwards the response to the client.

6. Health check system continuously monitors backend servers and updates load balancer about server availability.

Database Schema

Not applicable as this design focuses on traffic distribution rather than data storage.

Scaling Discussion

Bottlenecks

Load balancer becoming a single point of failure under high traffic

Backend servers overwhelmed if traffic is not evenly distributed

Health check delays causing traffic to be sent to unhealthy servers

Latency increase if load balancer is overloaded

Solutions

Use multiple load balancers with DNS round-robin or anycast IP for redundancy

Implement auto-scaling for backend servers based on load metrics

Optimize health check frequency and timeout settings for faster failure detection

Use load balancer clustering or distributed load balancing solutions to share traffic load

Interview Tips

Time: Spend 10 minutes understanding requirements and clarifying constraints, 20 minutes designing the load balancer system and explaining components, 10 minutes discussing scaling and failure scenarios, and 5 minutes summarizing.

Explain why distributing traffic prevents server overload and improves availability

Discuss different load balancing algorithms and when to use them

Highlight the importance of health checks to avoid sending traffic to failed servers

Mention how load balancers enable horizontal scaling and fault tolerance

Address potential bottlenecks and realistic solutions to scale the system

Practice

1. Why do load balancers distribute incoming traffic across multiple servers?

easy

A. To prevent any single server from becoming overloaded and improve system reliability

B. To increase the number of servers needed for the system

C. To slow down the response time for users

D. To reduce the total number of user requests

Solution

1. Step 1: Understand server overload

   When too many requests go to one server, it slows down or crashes.

2. Step 2: Role of load balancers

   Load balancers spread requests evenly to keep servers healthy and fast.

3. Final Answer:

   To prevent any single server from becoming overloaded and improve system reliability -> Option A

4. Quick Check:

   Load balancers prevent overload = C [OK]

Hint: Load balancers keep servers from getting too busy [OK]

Common Mistakes:

• Thinking load balancers reduce total requests
• Believing load balancers slow down responses
• Assuming load balancers increase server count unnecessarily

2. Which of the following is a common method used by load balancers to distribute traffic?

easy

A. Bubble sort

B. Binary search

C. Depth-first search

D. Round-robin

Solution

1. Step 1: Identify load balancing methods

   Common methods include round-robin and least connections.

2. Step 2: Match options to known methods

   Round-robin cycles through servers evenly; others are unrelated algorithms.

3. Final Answer:

   Round-robin -> Option D

4. Quick Check:

   Round-robin is a load balancing method = D [OK]

Hint: Round-robin cycles requests evenly [OK]

Common Mistakes:

• Confusing sorting/search algorithms with load balancing
• Choosing unrelated computer science terms
• Ignoring common load balancing strategies

3. Consider a load balancer using the least connections strategy. If Server A has 3 active connections and Server B has 1 active connection, where will the next request be sent?

medium

A. Server B

B. Server A

C. Both servers equally

D. To a new server not listed

Solution

1. Step 1: Understand least connections strategy

   It sends new requests to the server with fewer active connections.

2. Step 2: Compare active connections

   Server A has 3, Server B has 1, so Server B has fewer connections.

3. Final Answer:

   Server B -> Option A

4. Quick Check:

   Least connections picks server with fewer connections = B [OK]

Hint: Send to server with least active connections [OK]

Common Mistakes:

• Choosing server with more connections
• Assuming equal distribution regardless of load
• Thinking new servers are automatically chosen

4. A load balancer is configured to use round-robin but always sends requests to the same server. What is the most likely cause?

medium

A. The servers are all down

B. The load balancer is not tracking the last server used correctly

C. The network is too slow

D. The user requests are too few

Solution

1. Step 1: Understand round-robin behavior

   Round-robin cycles through servers in order to balance load.

2. Step 2: Identify cause of repeated server selection

   If it always picks the same server, it likely lost track of which server was last used.

3. Final Answer:

   The load balancer is not tracking the last server used correctly -> Option B

4. Quick Check:

   Round-robin needs correct tracking = A [OK]

Hint: Round-robin must remember last server used [OK]

Common Mistakes:

• Blaming server downtime without checking load balancer
• Assuming network speed affects server choice
• Thinking low request volume causes imbalance

5. You design a system with a load balancer distributing traffic to 5 servers. Suddenly, one server becomes much slower but still receives equal requests. What change to the load balancing strategy can improve overall system performance?

hard

A. Use round-robin but add more user requests

B. Increase the number of servers without changing strategy

C. Switch from round-robin to least connections to avoid slow server overload

D. Disable the load balancer and send requests to one server

Solution

1. Step 1: Identify problem with round-robin

   Round-robin sends equal requests regardless of server speed, causing slow server overload.

2. Step 2: Choose better strategy

   Least connections sends requests to less busy servers, reducing load on slow server.

3. Final Answer:

   Switch from round-robin to least connections to avoid slow server overload -> Option C

4. Quick Check:

   Least connections adapts to server speed = A [OK]

Hint: Use least connections to avoid slow servers [OK]

Common Mistakes:

• Adding servers without fixing load distribution
• Increasing requests worsens overload
• Disabling load balancer causes single point failure