Bird
Raised Fist0

Global Server Load Balancing - Routing Traffic Across Data Centers

HLDmediumInterview freq: common
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Global Server Load Balancing - Routing Traffic Across Data Centers
45 min · senior
senior

You are designing a global server load balancing system for a startup with 50 million daily active users (DAU) distributed worldwide. The system must route user traffic to the optimal data center based on latency, availability, and capacity while ensuring encrypted traffic and meeting a P99 latency SLA of under 200ms. The system should gracefully handle data center failures and support dynamic traffic shifts without user disruption.

Constraints
  • All traffic must be encrypted end-to-end using TLS 1.3
  • P99 latency for routing decisions must be under 200ms globally
  • System must support automatic failover and traffic rerouting within 30 seconds of data center failure
  • Traffic routing decisions must consider real-time data center load and network latency
  • Must comply with data sovereignty laws by routing users to region-appropriate data centers
📋
Guided Steps
1/5
Step 1
Choosing the Global Traffic Routing Mechanism
What technology and approach will you use to route user requests globally to the optimal data center?
💡 Hint
Consider DNS-based routing vs. Anycast IP vs. HTTP redirection and their impact on latency and failover.
Key Decision
Use a combination of AWS Route 53 latency-based DNS routing with health checks and GeoDNS policies to direct users to the nearest healthy data center, rejecting Anycast IP due to complex BGP failover delays and HTTP redirection due to added latency and client dependency.
Interviewer Expectation
Mid-level: Names DNS-based routing like Route 53. Senior: Explains why latency-based DNS with health checks is chosen over Anycast IP (BGP convergence delays) and HTTP redirects (extra RTT), and how GeoDNS enforces data sovereignty.
⚠️Common Wrong Answer
Using Anycast IP alone - BGP convergence can take minutes causing prolonged outages and inconsistent routing.
Step 2
Real-Time Data Center Health and Load Monitoring
How will your system collect and process real-time health and load metrics from data centers to inform routing decisions?
💡 Hint
Think about streaming telemetry ingestion and scalable processing systems that can handle high throughput with low latency.
Key Decision
Use Apache Kafka as the ingestion pipeline for streaming health and load metrics from data centers, processed by Apache Flink for real-time aggregation and anomaly detection, rejecting REST polling due to high latency and scalability issues.
Interviewer Expectation
Mid-level: Mentions Kafka for streaming. Senior: Details how Kafka partitions by data center ID for parallelism and Flink for low-latency aggregation and anomaly detection to trigger routing updates.
⚠️Common Wrong Answer
Polling REST APIs from data centers every few seconds - causes high load and delayed detection of failures.
Step 3
Ensuring Encrypted and Secure Traffic Routing
How will you ensure all traffic routed between users and data centers is encrypted end-to-end and meets compliance requirements?
💡 Hint
Consider TLS termination points and certificate management across global data centers.
Key Decision
Terminate TLS at the edge using AWS CloudFront with ACM-managed certificates for global HTTPS termination, then use mutual TLS (mTLS) between edge and data centers to maintain encryption, rejecting TLS termination only at data centers due to increased latency and complexity.
Interviewer Expectation
Mid-level: Mentions TLS termination at edge. Senior: Explains ACM certificate automation, mTLS between edge and data centers for zero trust, and how this reduces latency while maintaining compliance.
⚠️Common Wrong Answer
Terminating TLS only at data centers - increases latency and exposes traffic on internal links.
Step 4
Handling Data Center Failures and Traffic Failover
Describe how your system detects data center failures and reroutes traffic within the SLA of 30 seconds.
💡 Hint
Think about health checks, failure detection thresholds, and DNS TTL tuning.
Key Decision
Implement active health checks via Route 53 with aggressive failure detection (3 consecutive failures within 10 seconds) and low DNS TTL (30 seconds) to enable rapid failover, rejecting high TTL DNS or passive failure detection due to slow reaction times.
Interviewer Expectation
Mid-level: Knows to use health checks and low TTL. Senior: Explains failure detection thresholds, DNS caching behavior, and how to mitigate DNS propagation delays with client-side retries.
⚠️Common Wrong Answer
Using high DNS TTL (5 minutes) - causes slow failover and user requests routed to failed data centers.
Step 5
API Design for Client and Operator Interaction
Design 3 core API endpoints for clients to get routing info, data centers to report health, and operators to update routing policies.
💡 Hint
Consider RESTful design, authentication, and data formats for real-time updates.
Key Decision
Expose a REST API endpoint /route for clients to fetch optimal data center info with JWT authentication; /health-report for data centers to POST health/load metrics into Kafka; and /routing-policy for operators to PATCH routing rules stored in DynamoDB, rejecting polling-based or unauthenticated endpoints.
Interviewer Expectation
Mid-level: Names REST endpoints. Senior: Details authentication, idempotency, data formats (JSON), and integration with Kafka and DynamoDB for scalability and consistency.
⚠️Common Wrong Answer
Using unauthenticated endpoints or polling-based health reporting - leads to security risks and scalability bottlenecks.
🔌
API Design Challenge
Sketch 3 core API endpoints for the Global Server Load Balancer: how clients request routing info, how data centers report health and load, and how operators update routing policies.
Hints
  • How does the client request the best data center endpoint?
  • How do data centers send health and load metrics to the load balancer?
  • How can operators update routing rules or override automatic decisions?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Use latency-based DNS routing with health checks for global traffic distribution
  • Implement streaming ingestion of health metrics via Kafka for scalability
  • Ensure TLS termination at edge with mTLS to data centers for encryption and compliance
  • Design rapid failure detection and DNS failover with low TTL to meet SLA
Good to Have - 30%
  • Use real-time stream processing (Flink) for anomaly detection and load balancing
  • Provide authenticated REST APIs for clients, data centers, and operators
  • Incorporate GeoDNS policies to enforce data sovereignty requirements
Bonus - 10%
  • Explain mitigation of DNS caching issues with client-side retries and low TTL
Anti-patterns to Avoid
✅
Model Answer
▼ Reveal

The Global Server Load Balancing system uses AWS Route 53 latency-based DNS routing combined with GeoDNS policies to route users to the nearest healthy data center while enforcing data sovereignty. Real-time health and load metrics are ingested via Apache Kafka and processed by Apache Flink for anomaly detection and routing updates. TLS termination occurs at the edge using AWS CloudFront with ACM-managed certificates, with mutual TLS securing traffic between edge and data centers. Rapid failover is achieved through aggressive Route 53 health checks and low DNS TTL settings. RESTful APIs secured with JWT enable clients to fetch routing info, data centers to report health metrics, and operators to update routing policies stored in DynamoDB.

Key Decisions
  • Use AWS Route 53 latency-based DNS routing with health checks and GeoDNS to optimize latency and enforce data sovereignty
  • Stream health and load metrics from data centers into Kafka, process with Flink for real-time anomaly detection
  • Terminate TLS at the edge with ACM-managed certificates and use mTLS between edge and data centers for end-to-end encryption
  • Implement aggressive health checks with 3 failures in 10 seconds and low DNS TTL of 30 seconds for rapid failover
  • Design REST APIs with JWT authentication for clients, data centers, and operators, integrating with Kafka and DynamoDB for scalability and consistency
  • AWS Route 53 latency-based DNS routing with GeoDNS for data sovereignty compliance
  • Apache Kafka for ingesting streaming health and load metrics from data centers
  • Apache Flink for real-time processing and anomaly detection on health data
  • AWS CloudFront with ACM-managed TLS certificates for global HTTPS termination
  • Mutual TLS (mTLS) between edge and data centers to maintain end-to-end encryption
  • Aggressive Route 53 health checks with low DNS TTL (30s) for rapid failover
  • REST API endpoints secured with JWT for client routing info, health reporting, and operator policy updates
  • DynamoDB as the backend store for routing policies enabling low-latency updates and consistency

Practice

(1/5)
1. A typeahead system must handle millions of concurrent prefix queries with low latency and high throughput. Which component is best suited to serve these prefix lookups efficiently at scale?
easy
A. A distributed in-memory Trie cache sharded by prefix hash
B. A message queue to buffer prefix queries before processing
C. A full-text search engine like Elasticsearch without caching
D. A relational database with indexed LIKE queries on the prefix column

Solution

  1. Step 1: Identify prefix lookup requirements

    Prefix queries require fast, low-latency lookups optimized for string prefixes, which relational DB LIKE queries are inefficient at scale.

  2. Step 2: Evaluate component suitability

    A distributed in-memory Trie cache supports efficient prefix matching and can be sharded by prefix hash for scalability, unlike message queues or full-text search without caching.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Distributed Trie cache -> efficient prefix lookups at scale [check]
Hint: Trie caches excel at prefix queries, unlike DB LIKE or queues [check]
Common Mistakes:
2. A web service receives highly variable traffic with some clients opening many simultaneous connections, while others open few. Which load balancing algorithm best distributes load evenly across servers?
easy
A. Round Robin, because it cycles through servers regardless of connection count
B. Weighted Round Robin, because it assigns requests based on server weights only
C. IP Hash, because it ensures requests from the same client always go to the same server
D. Least Connections, because it directs new requests to the server with the fewest active connections

Solution

  1. Step 1: Understand traffic variability

    Clients open varying numbers of connections, so servers can become unevenly loaded if requests are assigned blindly.

  2. Step 2: Evaluate load balancing algorithms

    Round Robin ignores current load, IP Hash pins clients to servers (may cause imbalance), Weighted Round Robin uses static weights ignoring dynamic load. Least Connections dynamically balances based on active connections.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Least Connections adapts to real-time load, balancing uneven client connections [check]
Hint: Least Connections adapts to active load, not just request count [check]
Common Mistakes:
3. A messaging system must handle 1 million writes per second and fan-out delivery to millions of subscribers without blocking the sender. Which component best fits this requirement?
easy
A. An in-memory cache like Redis without persistence
B. A relational database with synchronous writes and triggers for fan-out
C. A REST API that synchronously calls each subscriber endpoint
D. A distributed message queue like Kafka with partitioned topics

Solution

  1. Step 1: Identify the need for decoupling high write throughput from delivery

    Relational DB with synchronous writes (A relational database with synchronous writes and triggers for fan-out) blocks on writes and triggers, unsuitable for 1M writes/sec. REST API synchronous calls (A REST API that synchronously calls each subscriber endpoint) block sender and don't scale to millions of subscribers. Redis (An in-memory cache like Redis without persistence) lacks durability and persistence, risking data loss.

  2. Step 2: Recognize Kafka's partitioned topics enable high throughput and fan-out

    Kafka partitions allow parallel writes and ordered delivery per partition, decoupling producers and consumers efficiently.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Kafka partitions -> high throughput + durable fan-out [check]
Hint: Partitioned queues decouple writes and fan-out efficiently [check]
Common Mistakes:
4. At 10x scale, which component in the Q&A feed system is most likely to fail first and why?
hard
A. The search index cluster because it cannot shard queries across more nodes
B. The user profile database because it uses a single master node for writes
C. The feed generation service because it maintains stateful user session caches that cannot be sharded easily
D. The Kafka cluster because partition count cannot increase beyond a fixed limit

Solution

  1. Step 1: Analyze component scalability

    Search clusters (A) scale horizontally with sharding; Kafka (C) can increase partitions; user DB (D) can be sharded or use multi-master.

  2. Step 2: Stateful feed generation bottleneck

    Feed generation maintains per-user session caches and WebSocket connections, which are stateful and tied to specific servers, limiting horizontal scaling and causing early failure.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Stateful feed caches block horizontal scaling, causing first failure [check]
Hint: Stateful caches block scaling before stateless components [check]
Common Mistakes:
5. At 10x scale, which REST API component is most likely to fail first due to stateful session management, and why?
hard
A. Backend database, because it cannot scale writes linearly
B. Load balancer, because it cannot handle increased TCP connections
C. API Gateway, because it maintains stateful authentication sessions in memory per client
D. Cache layer, because it evicts entries too aggressively under load

Solution

  1. Step 1: Identify stateful components in REST API architecture

    API Gateway maintaining in-memory sessions is stateful and tied to client connections.

  2. Step 2: Understand scaling impact of statefulness

    Stateful session storage prevents easy horizontal scaling; new servers can't share session state, causing bottlenecks.

  3. Step 3: Compare with other components

    Load balancers handle stateless TCP connections; databases and caches can be sharded or scaled differently.

  4. Final Answer:

    Option C -> Option C

  5. Quick Check:

    Stateful API Gateway sessions block horizontal scaling first [check]
Hint: Stateful API Gateway sessions block scaling first [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard