Traffic management (routing, splitting) in Microservices - Scalability & System Analysis

Scalability Analysis - Traffic management (routing, splitting)

Traffic Growth and System Changes

Users / Traffic	Routing Complexity	Splitting Use Cases	Infrastructure Needs	Monitoring & Control
100 users	Simple routing rules, mostly static	Rare, manual splitting for testing	Single load balancer, minimal proxies	Basic logs and alerts
10,000 users	Dynamic routing based on service health	Canary releases, A/B testing starts	Multiple load balancers, API gateways	Real-time monitoring dashboards
1,000,000 users	Advanced routing with weighted splits, geo-routing	Automated traffic splitting for experiments	Distributed proxies, service mesh adoption	Automated anomaly detection, tracing
100,000,000 users	Global traffic management, multi-region routing	Complex multi-dimensional splits (device, region, version)	Global DNS, edge proxies, multi-cloud	AI-driven traffic control, self-healing

First Bottleneck

At low to medium scale, the first bottleneck is the routing layer such as API gateways or load balancers. They can become overwhelmed by the number of routing rules and traffic volume, causing increased latency or failures.

As traffic grows, service discovery and configuration management also become bottlenecks, since routing decisions depend on up-to-date service health and versions.

Scaling Solutions

• Horizontal scaling: Add more instances of API gateways and proxies to distribute routing load.
• Service mesh: Offload routing and splitting logic to sidecars for decentralized control.
• Caching routing decisions: Reduce repeated lookups by caching routing rules locally.
• Weighted routing and traffic splitting: Use dynamic weights to gradually shift traffic during deployments.
• Global traffic management: Use DNS-based geo-routing and edge proxies for global scale.
• Automation: Automate routing updates and health checks to avoid stale routes.

Back-of-Envelope Cost Analysis

• At 1M users with 1 request per second each, expect ~1 million requests per second (QPS) at peak.
• Each API gateway instance can handle ~5,000 QPS, so ~200 instances needed for routing layer.
• Service mesh sidecars add CPU and memory overhead per service instance.
• Bandwidth depends on request size; for 1 KB requests, 1M QPS = ~1 GB/s network traffic.
• Storage for routing configs and logs grows with number of rules and traffic volume.

Interview Tip

Start by explaining the routing and splitting needs at different traffic scales. Identify the first bottleneck clearly (usually routing layer). Then discuss specific scaling techniques like horizontal scaling, service mesh, and automation. Use real numbers to justify your approach. Finally, mention monitoring and fallback strategies to maintain reliability.

Self Check

Question: Your routing layer handles 1,000 QPS. Traffic grows 10x to 10,000 QPS. What is your first action and why?

Answer: Add more routing instances (horizontal scaling) and implement load balancing to distribute traffic. This prevents overload and maintains low latency.

Key Result

Traffic management systems first hit bottlenecks at the routing layer as traffic grows; horizontal scaling and service mesh adoption are key to maintaining efficient routing and splitting at scale.

Practice

1. What is the main purpose of traffic routing in microservices architecture?

easy

A. To direct incoming requests to specific services based on rules

B. To store data persistently across services

C. To encrypt communication between services

D. To monitor service health and uptime

Solution

1. Step 1: Understand traffic routing

   Traffic routing means sending requests to the right service based on rules like URL path or user type.

2. Step 2: Identify the main purpose

   Routing helps control where requests go, ensuring they reach the correct microservice.

3. Final Answer:

   To direct incoming requests to specific services based on rules -> Option A

4. Quick Check:

   Routing = directing requests [OK]

Hint: Routing means sending requests to the right place [OK]

Common Mistakes:

• Confusing routing with data storage
• Thinking routing encrypts data
• Mixing routing with monitoring

2. Which of the following is a correct way to define a traffic splitting rule in a service mesh configuration?

easy

A. split:
- weight: 50
service: v1
- weight: 50
service: v2

B. route:
path: /api
service: v1

C. split:
- service: v1
- service: v2
- weight: 100

D. route:
weight: 100
service: v1
path: /home

Solution

1. Step 1: Understand traffic splitting syntax

   Traffic splitting uses weights to divide requests between service versions, e.g., 50% to v1 and 50% to v2.

2. Step 2: Identify correct syntax

   split: - weight: 50 service: v1 - weight: 50 service: v2 correctly assigns weights to services for splitting. Other options mix routing and splitting or have invalid weight placement.

3. Final Answer:

   split: - weight: 50 service: v1 - weight: 50 service: v2 -> Option A

4. Quick Check:

   Splitting uses weights per service [OK]

Hint: Splitting needs weights assigned to each service [OK]

Common Mistakes:

• Confusing routing rules with splitting rules
• Missing weights in splitting definitions
• Placing weights outside service entries

3. Given this traffic splitting configuration, what percentage of requests go to service v2?

split:
  - weight: 70
    service: v1
  - weight: 30
    service: v2

medium

A. 100%

B. 70%

C. 50%

D. 30%

Solution

1. Step 1: Read the weights for each service

   Service v1 has weight 70, and service v2 has weight 30.

2. Step 2: Calculate percentage for v2

   Total weight = 70 + 30 = 100. So, v2 gets 30/100 = 30% of requests.

3. Final Answer:

   30% -> Option D

4. Quick Check:

   Weight 30 means 30% traffic [OK]

Hint: Traffic % = service weight / total weight [OK]

Common Mistakes:

• Adding weights incorrectly
• Assuming equal split without weights
• Confusing service names

4. You have this routing rule:

route:
  path: /user
  service: user-service-v1
  weight: 100

But requests to /user/profile are not reaching user-service-v1. What is the likely problem?

medium

A. Service name is incorrect and causes failure

B. Weight should be split between multiple services

C. The path rule matches only exact /user, not subpaths like /user/profile

D. Routing rules cannot use path matching

Solution

1. Step 1: Analyze the path matching rule

   The rule matches exactly /user, but /user/profile is a subpath and may not match unless wildcard or prefix matching is used.

2. Step 2: Identify why requests fail

   Since /user/profile does not match exactly /user, requests do not route to user-service-v1.

3. Final Answer:

   The path rule matches only exact /user, not subpaths like /user/profile -> Option C

4. Quick Check:

   Exact path matching excludes subpaths [OK]

Hint: Exact path matches exclude subpaths unless wildcard used [OK]

Common Mistakes:

• Assuming weight must be split
• Blaming service name without checking
• Thinking routing ignores paths

5. You want to gradually roll out a new version of a payment service to 10% of users while keeping 90% on the old version. Which traffic management strategy is best suited for this?

hard

A. Use routing based on URL path to send 10% of requests to new service

B. Use traffic splitting with weights 90% to old and 10% to new service

C. Deploy both versions without traffic control and monitor errors

D. Use a load balancer that randomly sends requests without weights

Solution

1. Step 1: Understand gradual rollout needs

   Gradual rollout means controlling what percentage of users see the new version.

2. Step 2: Choose traffic management method

   Traffic splitting with weights allows precise control of request percentages to each version.

3. Step 3: Evaluate other options

   Routing by URL path cannot split traffic by percentage. Random load balancing lacks control. Deploying without control risks all users seeing new version.

4. Final Answer:

   Use traffic splitting with weights 90% to old and 10% to new service -> Option B

5. Quick Check:

   Splitting controls rollout percentages [OK]

Hint: Use weighted splitting for gradual rollout [OK]

Common Mistakes:

• Using URL path routing for percentage split
• Ignoring traffic control during rollout
• Relying on random load balancing