Why load balancers distribute traffic in HLD - Scalability Evidence

Scalability Analysis - Why load balancers distribute traffic

Growth Table: Traffic and Load Balancer Role

Users	Requests per Second (RPS)	Load Balancer Role	Server Count
100	50	Single server, no load balancer needed	1
10,000	5,000	Single load balancer distributes traffic to few servers	2-4
1,000,000	500,000	Multiple load balancers with health checks and failover	100+
100,000,000	50,000,000	Global load balancers, geo-distribution, CDN integration	Thousands

First Bottleneck: Server Overload Without Load Balancer

At low traffic, one server can handle all requests. As users grow, a single server hits CPU and memory limits first. Without a load balancer, traffic cannot be split, causing slow responses or crashes.

Load balancers prevent this by spreading requests evenly, avoiding overload on any one server.

Scaling Solutions Using Load Balancers

• Horizontal Scaling: Add more servers behind the load balancer to share traffic.
• Health Checks: Load balancers detect unhealthy servers and stop sending traffic to them.
• Session Persistence: Keep user sessions on the same server if needed.
• Global Load Balancing: Use multiple load balancers across regions for geo-distribution.
• Integration with CDN: Offload static content to CDN, reducing load on servers.

Back-of-Envelope Cost Analysis

Assuming 1 server handles ~3000 RPS:

• At 10,000 RPS, need ~4 servers.
• At 500,000 RPS, need ~170 servers.
• Load balancer can handle ~10,000 concurrent connections per instance.
• Network bandwidth: 1 Gbps = 125 MB/s, so servers and load balancers must have sufficient network capacity.

Interview Tip: Structuring Scalability Discussion

Start by explaining the problem of single server limits. Then introduce load balancers as a solution to distribute traffic. Discuss how load balancers improve availability and fault tolerance. Finally, mention scaling strategies and monitoring.

Self Check Question

Your database handles 1000 QPS. Traffic grows 10x. What do you do first?

Answer: Introduce load balancers to distribute traffic across multiple database replicas or application servers to avoid overload and maintain performance.

Key Result

Load balancers are essential to distribute increasing traffic across multiple servers, preventing any single server from becoming a bottleneck and enabling horizontal scaling.

Practice

1. Why do load balancers distribute incoming traffic across multiple servers?

easy

A. To prevent any single server from becoming overloaded and improve system reliability

B. To increase the number of servers needed for the system

C. To slow down the response time for users

D. To reduce the total number of user requests

Solution

1. Step 1: Understand server overload

   When too many requests go to one server, it slows down or crashes.

2. Step 2: Role of load balancers

   Load balancers spread requests evenly to keep servers healthy and fast.

3. Final Answer:

   To prevent any single server from becoming overloaded and improve system reliability -> Option A

4. Quick Check:

   Load balancers prevent overload = C [OK]

Hint: Load balancers keep servers from getting too busy [OK]

Common Mistakes:

• Thinking load balancers reduce total requests
• Believing load balancers slow down responses
• Assuming load balancers increase server count unnecessarily

2. Which of the following is a common method used by load balancers to distribute traffic?

easy

A. Bubble sort

B. Binary search

C. Depth-first search

D. Round-robin

Solution

1. Step 1: Identify load balancing methods

   Common methods include round-robin and least connections.

2. Step 2: Match options to known methods

   Round-robin cycles through servers evenly; others are unrelated algorithms.

3. Final Answer:

   Round-robin -> Option D

4. Quick Check:

   Round-robin is a load balancing method = D [OK]

Hint: Round-robin cycles requests evenly [OK]

Common Mistakes:

• Confusing sorting/search algorithms with load balancing
• Choosing unrelated computer science terms
• Ignoring common load balancing strategies

3. Consider a load balancer using the least connections strategy. If Server A has 3 active connections and Server B has 1 active connection, where will the next request be sent?

medium

A. Server B

B. Server A

C. Both servers equally

D. To a new server not listed

Solution

1. Step 1: Understand least connections strategy

   It sends new requests to the server with fewer active connections.

2. Step 2: Compare active connections

   Server A has 3, Server B has 1, so Server B has fewer connections.

3. Final Answer:

   Server B -> Option A

4. Quick Check:

   Least connections picks server with fewer connections = B [OK]

Hint: Send to server with least active connections [OK]

Common Mistakes:

• Choosing server with more connections
• Assuming equal distribution regardless of load
• Thinking new servers are automatically chosen

4. A load balancer is configured to use round-robin but always sends requests to the same server. What is the most likely cause?

medium

A. The servers are all down

B. The load balancer is not tracking the last server used correctly

C. The network is too slow

D. The user requests are too few

Solution

1. Step 1: Understand round-robin behavior

   Round-robin cycles through servers in order to balance load.

2. Step 2: Identify cause of repeated server selection

   If it always picks the same server, it likely lost track of which server was last used.

3. Final Answer:

   The load balancer is not tracking the last server used correctly -> Option B

4. Quick Check:

   Round-robin needs correct tracking = A [OK]

Hint: Round-robin must remember last server used [OK]

Common Mistakes:

• Blaming server downtime without checking load balancer
• Assuming network speed affects server choice
• Thinking low request volume causes imbalance

5. You design a system with a load balancer distributing traffic to 5 servers. Suddenly, one server becomes much slower but still receives equal requests. What change to the load balancing strategy can improve overall system performance?

hard

A. Use round-robin but add more user requests

B. Increase the number of servers without changing strategy

C. Switch from round-robin to least connections to avoid slow server overload

D. Disable the load balancer and send requests to one server

Solution

1. Step 1: Identify problem with round-robin

   Round-robin sends equal requests regardless of server speed, causing slow server overload.

2. Step 2: Choose better strategy

   Least connections sends requests to less busy servers, reducing load on slow server.

3. Final Answer:

   Switch from round-robin to least connections to avoid slow server overload -> Option C

4. Quick Check:

   Least connections adapts to server speed = A [OK]

Hint: Use least connections to avoid slow servers [OK]

Common Mistakes:

• Adding servers without fixing load distribution
• Increasing requests worsens overload
• Disabling load balancer causes single point failure