Load balancing algorithms (round robin, least connections) in HLD - System Design Exercise

Design: Load Balancer with Round Robin and Least Connections Algorithms

Design focuses on the load balancer component and its algorithms. Backend server internals and client implementations are out of scope.

Functional Requirements

FR1: Distribute incoming client requests evenly across multiple backend servers

FR2: Support at least two load balancing algorithms: round robin and least connections

FR3: Handle up to 10,000 concurrent client connections

FR4: Provide low latency routing with p99 response time under 100ms

FR5: Ensure high availability with 99.9% uptime

FR6: Allow dynamic switching between load balancing algorithms without downtime

Non-Functional Requirements

NFR1: System must handle sudden spikes in traffic gracefully

NFR2: Load balancer should not become a single point of failure

NFR3: Minimal overhead in decision making to keep latency low

NFR4: Backend servers may have different capacities (heterogeneous environment)

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

❓ Question 6

Key Components

Load balancer front-end to receive client requests

Health check module to monitor backend server status

Algorithm module implementing round robin and least connections

Backend server pool management

Metrics and logging for monitoring load distribution

Design Patterns

Round Robin load balancing pattern

Least Connections load balancing pattern

Health checking and failover pattern

Sticky session (if required)

Active-passive or active-active load balancer deployment

Reference Architecture

Client Requests
    |
    v
+-----------------+
|   Load Balancer  |
|  +------------+ |
|  | Algorithm  | |
|  | Module     | |
|  +------------+ |
|  | Health    |  |
|  | Check     |  |
+-----------------+
    |       |       
    v       v       
+-------+ +-------+
|Server1| |Server2|
+-------+ +-------+
    .         .    
    .         .    
+-------+ +-------+
|ServerN| |       |
+-------+ +-------+

Components

Load Balancer Front-end

Nginx or HAProxy or custom TCP/HTTP proxy

Receives client requests and forwards them to backend servers based on selected algorithm

Algorithm Module

Custom logic in load balancer software

Implements round robin and least connections algorithms to select backend server

Health Check Module

Periodic HTTP/TCP probes

Monitors backend server health and removes unhealthy servers from rotation

Backend Server Pool

Set of application servers

Handles actual client requests forwarded by load balancer

Metrics and Logging

Prometheus, Grafana, or built-in logging

Tracks load distribution, server health, and performance

Request Flow

1. Client sends a request to the load balancer.

2. Load balancer receives the request at its front-end.

3. Health check module confirms which backend servers are healthy.

4. Algorithm module selects a backend server based on the chosen algorithm:

5. - Round Robin: picks the next server in a circular order.

6. - Least Connections: picks the server with the fewest active connections.

7. Load balancer forwards the client request to the selected backend server.

8. Backend server processes the request and sends response back through load balancer.

9. Load balancer forwards the response to the client.

Database Schema

Not applicable as this design focuses on runtime load balancing logic without persistent storage.

Scaling Discussion

Bottlenecks

Load balancer becoming a single point of failure under high traffic

Algorithm module latency increasing with large number of backend servers

Health check overhead causing delays or false positives

Uneven load distribution if backend servers have different capacities

Solutions

Deploy multiple load balancer instances with DNS round robin or anycast IP for high availability

Use efficient data structures (e.g., indexed lists, min-heaps) to optimize algorithm selection

Tune health check frequency and timeout to balance accuracy and overhead

Implement weighted round robin or weighted least connections to account for server capacity differences

Interview Tips

Time: Spend 10 minutes clarifying requirements and constraints, 20 minutes designing the architecture and algorithms, 10 minutes discussing scaling and trade-offs, and 5 minutes summarizing.

Explain the difference between round robin and least connections clearly

Discuss how health checks improve reliability

Mention how to handle server failures gracefully

Talk about scaling the load balancer itself to avoid bottlenecks

Highlight how algorithm choice affects load distribution and latency