Routing and load balancing in Microservices - System Design Exercise

Design: Routing and Load Balancing System for Microservices

Design focuses on routing and load balancing components for microservices. Does not cover service implementation details or client-side logic.

Functional Requirements

FR1: Distribute incoming client requests evenly across multiple instances of microservices

FR2: Support dynamic addition and removal of service instances without downtime

FR3: Route requests based on service availability and health status

FR4: Provide low latency routing with minimal overhead

FR5: Support sticky sessions for stateful services when needed

FR6: Handle at least 50,000 concurrent requests with p99 latency under 100ms

FR7: Ensure 99.9% system availability

Non-Functional Requirements

NFR1: Must work in a cloud-native environment with containerized microservices

NFR2: Should support both HTTP and gRPC protocols

NFR3: Load balancer must not become a single point of failure

NFR4: Routing decisions must be consistent and fault tolerant

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

❓ Question 6

Key Components

API Gateway or Ingress Controller

Service Registry and Discovery

Load Balancer (Layer 4 and Layer 7)

Health Check Service

Configuration Management

Metrics and Monitoring

Design Patterns

Round Robin Load Balancing

Least Connections Load Balancing

Consistent Hashing for sticky sessions

Circuit Breaker for fault tolerance

Service Mesh for advanced routing

Reference Architecture

Client
  |
  v
API Gateway / Ingress Controller
  |
  v
Load Balancer (Layer 7)
  |
  v
Service Instances (Microservices)
  |
  v
Service Registry <--> Health Check Service
  
Metrics & Monitoring
  |
  v
Configuration Management

Components

API Gateway / Ingress Controller

Nginx, Envoy, or Kong

Entry point for client requests, performs routing and load balancing at Layer 7

Load Balancer

Envoy, HAProxy, or cloud provider load balancer

Distributes requests evenly across healthy service instances

Service Registry

Consul, Eureka, or etcd

Keeps track of available service instances and their metadata

Health Check Service

Custom or built-in health checks

Monitors service instance health and updates registry

Configuration Management

ConfigMaps, Consul KV store

Manages routing rules, load balancing policies, and service metadata

Metrics and Monitoring

Prometheus, Grafana

Collects metrics on load balancer performance and service health

Request Flow

1. Client sends request to API Gateway

2. API Gateway consults Service Registry to find healthy instances of target microservice

3. Load Balancer applies routing algorithm (e.g., round robin) to select instance

4. Request is forwarded to selected service instance

5. Service instance processes request and responds

6. Health Check Service periodically verifies instance health and updates Service Registry

7. Metrics collected for monitoring and alerting

Database Schema

Entities: - ServiceInstance: id, service_name, ip_address, port, status, last_heartbeat - ServiceRegistry: service_name, list_of_ServiceInstance_ids - HealthCheckLog: instance_id, timestamp, status Relationships: - One ServiceRegistry entry maps to many ServiceInstance entries - HealthCheckLog entries link to ServiceInstance by instance_id

Scaling Discussion

Bottlenecks

API Gateway becoming a single point of failure under high load

Load Balancer overwhelmed by large number of concurrent connections

Service Registry latency causing stale routing information

Health Check delays leading to routing to unhealthy instances

Solutions

Deploy multiple API Gateway instances behind a DNS-based load balancer or cloud load balancer

Use horizontal scaling and connection pooling for Load Balancer components

Implement caching and eventual consistency in Service Registry with fast update propagation

Optimize health check frequency and use lightweight probes to reduce delays

Interview Tips

Time: Spend 10 minutes clarifying requirements and constraints, 20 minutes designing architecture and data flow, 10 minutes discussing scaling and trade-offs, 5 minutes summarizing.

Explain how routing and load balancing improve availability and performance

Discuss trade-offs between different load balancing algorithms

Highlight importance of health checks and service discovery

Address fault tolerance and avoiding single points of failure

Mention monitoring and metrics for operational visibility

Practice

1. What is the main purpose of routing in a microservices architecture?

easy

A. To store data persistently across services

B. To monitor the health of microservices

C. To encrypt communication between services

D. To send requests to the correct microservice based on rules

Solution

1. Step 1: Understand routing role

   Routing directs incoming requests to the right microservice based on predefined rules like URL paths or headers.

2. Step 2: Differentiate routing from other functions

   Storing data, encrypting communication, and monitoring are separate concerns handled by databases, security layers, and monitoring tools respectively.

3. Final Answer:

   To send requests to the correct microservice based on rules -> Option D

4. Quick Check:

   Routing = directing requests [OK]

Hint: Routing directs requests to the right service [OK]

Common Mistakes:

• Confusing routing with data storage
• Mixing routing with security or monitoring
• Thinking routing balances load

2. Which of the following is a correct syntax for defining a load balancer rule that forwards requests to multiple instances evenly?

easy

A. round_robin: [instance1, instance2, instance3]

B. loadbalance = {instance1; instance2; instance3}

C. balance->instances(instance1, instance2, instance3)

D. forward: instance1 | instance2 | instance3

Solution

1. Step 1: Identify common load balancing syntax

   Round robin is a standard load balancing method cycling through instances evenly, often expressed as a list.

2. Step 2: Evaluate options for correct syntax style

   round_robin: [instance1, instance2, instance3] uses a clear list with round_robin keyword, matching common config styles. Others use invalid or uncommon syntax.

3. Final Answer:

   round_robin: [instance1, instance2, instance3] -> Option A

4. Quick Check:

   Round robin uses list syntax [OK]

Hint: Look for standard list syntax with round robin keyword [OK]

Common Mistakes:

• Using semicolons instead of commas
• Incorrect assignment operators
• Using arrows or pipes incorrectly

3. Given the following pseudo-code for a load balancer using weighted routing:

weights = {"serviceA": 3, "serviceB": 1}
requests = 8
for i in range(requests):
    target = weighted_choice(weights)
    print(target)

What is the expected number of requests routed to serviceA?

medium

A. 6

B. 8

C. 4

D. 2

Solution

1. Step 1: Understand weighted routing concept

   Weights define how many times a service should receive requests relative to others. ServiceA has weight 3, serviceB has weight 1, total weight is 4.

2. Step 2: Calculate expected requests for serviceA

   Out of 8 requests, serviceA should get (3/4)*8 = 6 requests on average.

3. Final Answer:

   6 -> Option A

4. Quick Check:

   Weighted share = 6 requests [OK]

Hint: Multiply total requests by service weight fraction [OK]

Common Mistakes:

• Ignoring weights and dividing requests equally
• Confusing total weight with individual weights
• Calculating requests for serviceB instead

4. A load balancer is configured with the following rule:

if (instance.isHealthy()) {
  forwardRequest(instance)
} else {
  skipInstance(instance)
}

However, requests are still being sent to unhealthy instances. What is the most likely cause?

medium

A. Instances are overloaded but still marked healthy

B. Health check logic is not integrated with the load balancer

C. Load balancer is using round robin instead of weighted routing

D. Routing rules are missing URL path matching

Solution

1. Step 1: Analyze health check integration

   The code shows a health check condition, but if the load balancer does not actually use this logic, unhealthy instances may still receive traffic.

2. Step 2: Evaluate other options for relevance

   Round robin vs weighted routing does not affect health checks. Overload does not mark instances unhealthy. URL path matching is unrelated to health status.

3. Final Answer:

   Health check logic is not integrated with the load balancer -> Option B

4. Quick Check:

   Health check integration = key [OK]

Hint: Check if health logic is actually used by load balancer [OK]

Common Mistakes:

• Assuming routing method affects health checks
• Confusing overload with health status
• Ignoring missing integration of health logic

5. You need to design a routing and load balancing system for a microservices app that handles both user requests and background jobs. User requests must be routed based on URL paths, and load balanced evenly. Background jobs should be routed to a separate set of instances with weighted load balancing. Which architecture best fits this requirement?

hard

A. Use DNS-based routing to split traffic, then apply round robin load balancing on all instances

B. Deploy two separate load balancers, one for user requests with weighted balancing, another for jobs with round robin

C. Use a single load balancer with path-based routing directing to two target groups; one uses round robin, the other weighted balancing

D. Route all traffic to a single instance that forwards requests internally based on type

Solution

1. Step 1: Identify routing needs for user requests and jobs

   User requests require path-based routing to separate them from background jobs, which need different load balancing strategies.

2. Step 2: Choose architecture supporting both routing and load balancing rules

   A single load balancer with path-based routing can direct traffic to two target groups. One group uses round robin for user requests, the other weighted for jobs, meeting all requirements efficiently.

3. Final Answer:

   Use a single load balancer with path-based routing directing to two target groups; one uses round robin, the other weighted balancing -> Option C

4. Quick Check:

   Path-based routing + mixed balancing = Use a single load balancer with path-based routing directing to two target groups; one uses round robin, the other weighted balancing [OK]

Hint: Combine path routing with separate load balancing per target group [OK]

Common Mistakes:

• Using weighted balancing for user requests instead of round robin
• Splitting with DNS which lacks path awareness
• Routing all traffic to one instance causing bottlenecks