Routing and load balancing in Microservices - System Design Exercise

Design: Routing and Load Balancing System for Microservices

Design focuses on routing and load balancing components for microservices. Does not cover service implementation details or client-side logic.

Functional Requirements

FR1: Distribute incoming client requests evenly across multiple instances of microservices

FR2: Support dynamic addition and removal of service instances without downtime

FR3: Route requests based on service availability and health status

FR4: Provide low latency routing with minimal overhead

FR5: Support sticky sessions for stateful services when needed

FR6: Handle at least 50,000 concurrent requests with p99 latency under 100ms

FR7: Ensure 99.9% system availability

Non-Functional Requirements

NFR1: Must work in a cloud-native environment with containerized microservices

NFR2: Should support both HTTP and gRPC protocols

NFR3: Load balancer must not become a single point of failure

NFR4: Routing decisions must be consistent and fault tolerant

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

❓ Question 6

Key Components

API Gateway or Ingress Controller

Service Registry and Discovery

Load Balancer (Layer 4 and Layer 7)

Health Check Service

Configuration Management

Metrics and Monitoring

Design Patterns

Round Robin Load Balancing

Least Connections Load Balancing

Consistent Hashing for sticky sessions

Circuit Breaker for fault tolerance

Service Mesh for advanced routing

Reference Architecture

Client
  |
  v
API Gateway / Ingress Controller
  |
  v
Load Balancer (Layer 7)
  |
  v
Service Instances (Microservices)
  |
  v
Service Registry <--> Health Check Service
  
Metrics & Monitoring
  |
  v
Configuration Management

Components

API Gateway / Ingress Controller

Nginx, Envoy, or Kong

Entry point for client requests, performs routing and load balancing at Layer 7

Load Balancer

Envoy, HAProxy, or cloud provider load balancer

Distributes requests evenly across healthy service instances

Service Registry

Consul, Eureka, or etcd

Keeps track of available service instances and their metadata

Health Check Service

Custom or built-in health checks

Monitors service instance health and updates registry

Configuration Management

ConfigMaps, Consul KV store

Manages routing rules, load balancing policies, and service metadata

Metrics and Monitoring

Prometheus, Grafana

Collects metrics on load balancer performance and service health

Request Flow

1. Client sends request to API Gateway

2. API Gateway consults Service Registry to find healthy instances of target microservice

3. Load Balancer applies routing algorithm (e.g., round robin) to select instance

4. Request is forwarded to selected service instance

5. Service instance processes request and responds

6. Health Check Service periodically verifies instance health and updates Service Registry

7. Metrics collected for monitoring and alerting

Database Schema

Entities: - ServiceInstance: id, service_name, ip_address, port, status, last_heartbeat - ServiceRegistry: service_name, list_of_ServiceInstance_ids - HealthCheckLog: instance_id, timestamp, status Relationships: - One ServiceRegistry entry maps to many ServiceInstance entries - HealthCheckLog entries link to ServiceInstance by instance_id

Scaling Discussion

Bottlenecks

API Gateway becoming a single point of failure under high load

Load Balancer overwhelmed by large number of concurrent connections

Service Registry latency causing stale routing information

Health Check delays leading to routing to unhealthy instances

Solutions

Deploy multiple API Gateway instances behind a DNS-based load balancer or cloud load balancer

Use horizontal scaling and connection pooling for Load Balancer components

Implement caching and eventual consistency in Service Registry with fast update propagation

Optimize health check frequency and use lightweight probes to reduce delays

Interview Tips

Time: Spend 10 minutes clarifying requirements and constraints, 20 minutes designing architecture and data flow, 10 minutes discussing scaling and trade-offs, 5 minutes summarizing.

Explain how routing and load balancing improve availability and performance

Discuss trade-offs between different load balancing algorithms

Highlight importance of health checks and service discovery

Address fault tolerance and avoiding single points of failure

Mention monitoring and metrics for operational visibility