Traffic management (routing, splitting) in Microservices - System Design Guide

Problem Statement

When all user requests are sent to the same version of a microservice, it becomes risky to deploy new features or fixes. A faulty release can cause widespread failures, and it is hard to test changes with real traffic without impacting all users.

Solution

Traffic management directs user requests to different versions or instances of a service based on rules. Routing sends requests to specific service versions, while splitting divides traffic by percentage or conditions. This allows gradual rollout, A/B testing, and quick rollback without downtime.

Architecture

User

→Traffic

↓

→Service v2

This diagram shows user requests flowing into a traffic manager that routes or splits requests between different service versions.

Trade-offs

✓ Pros

→

Enables safe gradual rollout of new features by controlling traffic percentage.

→

Supports A/B testing by routing specific user groups to different service versions.

→

Allows quick rollback by redirecting all traffic to a stable version.

→

Improves system resilience by isolating faulty versions from full traffic.

✗ Cons

→

Adds complexity to deployment and monitoring due to multiple active service versions.

→

Requires careful configuration to avoid uneven load or user experience issues.

→

May increase latency slightly due to routing logic before request reaches service.

Use when deploying new service versions frequently, needing gradual rollout, A/B testing, or canary releases at scale above 1000 requests per second.

Avoid if your system has very low traffic (under 100 req/sec) or if you deploy infrequently and can afford full downtime for updates.

Real World Examples

Netflix

Netflix uses traffic splitting to gradually roll out new streaming service features to a subset of users, minimizing risk of widespread failures.

Uber

Uber routes requests to different microservice versions to test new pricing algorithms on a small user group before full deployment.

Amazon

Amazon uses routing rules to direct traffic to different service versions during deployments, enabling canary releases and quick rollback.

Code Example

The before code sends all requests to one service version, risking full impact of failures. The after code splits requests randomly, sending 80% to the stable version and 20% to the new version, enabling gradual rollout and testing.

Microservices

### Before: No traffic splitting, all requests go to one service version
class ServiceClient:
    def send_request(self, request):
        # Always send to v1
        return self.call_service_v1(request)

    def call_service_v1(self, request):
        # Call service v1 endpoint
        pass


### After: Traffic splitting between v1 and v2
import random

class ServiceClient:
    def send_request(self, request):
        # Split traffic 80% to v1, 20% to v2
        if random.random() < 0.8:
            return self.call_service_v1(request)
        else:
            return self.call_service_v2(request)

    def call_service_v1(self, request):
        # Call service v1 endpoint
        pass

    def call_service_v2(self, request):
        # Call service v2 endpoint
        pass

Alternatives

Blue-Green Deployment

Routes all traffic to either the old or new version, switching instantly without splitting traffic.

Use when: Choose when you want zero overlap between versions and can afford instant cutover.

Feature Flags

Controls feature availability within the same service version without routing traffic differently.

Use when: Choose when you want to toggle features quickly without deploying new service versions.

Summary

Traffic management prevents full impact of faulty deployments by controlling request flow.

Routing and splitting enable gradual rollout, A/B testing, and quick rollback in microservices.

This pattern is essential for safe, scalable deployment in systems with frequent updates.

Practice

1. What is the main purpose of traffic routing in microservices architecture?

easy

A. To direct incoming requests to specific services based on rules

B. To store data persistently across services

C. To encrypt communication between services

D. To monitor service health and uptime

Solution

1. Step 1: Understand traffic routing

   Traffic routing means sending requests to the right service based on rules like URL path or user type.

2. Step 2: Identify the main purpose

   Routing helps control where requests go, ensuring they reach the correct microservice.

3. Final Answer:

   To direct incoming requests to specific services based on rules -> Option A

4. Quick Check:

   Routing = directing requests [OK]

Hint: Routing means sending requests to the right place [OK]

Common Mistakes:

• Confusing routing with data storage
• Thinking routing encrypts data
• Mixing routing with monitoring

2. Which of the following is a correct way to define a traffic splitting rule in a service mesh configuration?

easy

A. split:
- weight: 50
service: v1
- weight: 50
service: v2

B. route:
path: /api
service: v1

C. split:
- service: v1
- service: v2
- weight: 100

D. route:
weight: 100
service: v1
path: /home

Solution

1. Step 1: Understand traffic splitting syntax

   Traffic splitting uses weights to divide requests between service versions, e.g., 50% to v1 and 50% to v2.

2. Step 2: Identify correct syntax

   split: - weight: 50 service: v1 - weight: 50 service: v2 correctly assigns weights to services for splitting. Other options mix routing and splitting or have invalid weight placement.

3. Final Answer:

   split: - weight: 50 service: v1 - weight: 50 service: v2 -> Option A

4. Quick Check:

   Splitting uses weights per service [OK]

Hint: Splitting needs weights assigned to each service [OK]

Common Mistakes:

• Confusing routing rules with splitting rules
• Missing weights in splitting definitions
• Placing weights outside service entries

3. Given this traffic splitting configuration, what percentage of requests go to service v2?

split:
  - weight: 70
    service: v1
  - weight: 30
    service: v2

medium

A. 100%

B. 70%

C. 50%

D. 30%

Solution

1. Step 1: Read the weights for each service

   Service v1 has weight 70, and service v2 has weight 30.

2. Step 2: Calculate percentage for v2

   Total weight = 70 + 30 = 100. So, v2 gets 30/100 = 30% of requests.

3. Final Answer:

   30% -> Option D

4. Quick Check:

   Weight 30 means 30% traffic [OK]

Hint: Traffic % = service weight / total weight [OK]

Common Mistakes:

• Adding weights incorrectly
• Assuming equal split without weights
• Confusing service names

4. You have this routing rule:

route:
  path: /user
  service: user-service-v1
  weight: 100

But requests to /user/profile are not reaching user-service-v1. What is the likely problem?

medium

A. Service name is incorrect and causes failure

B. Weight should be split between multiple services

C. The path rule matches only exact /user, not subpaths like /user/profile

D. Routing rules cannot use path matching

Solution

1. Step 1: Analyze the path matching rule

   The rule matches exactly /user, but /user/profile is a subpath and may not match unless wildcard or prefix matching is used.

2. Step 2: Identify why requests fail

   Since /user/profile does not match exactly /user, requests do not route to user-service-v1.

3. Final Answer:

   The path rule matches only exact /user, not subpaths like /user/profile -> Option C

4. Quick Check:

   Exact path matching excludes subpaths [OK]

Hint: Exact path matches exclude subpaths unless wildcard used [OK]

Common Mistakes:

• Assuming weight must be split
• Blaming service name without checking
• Thinking routing ignores paths

5. You want to gradually roll out a new version of a payment service to 10% of users while keeping 90% on the old version. Which traffic management strategy is best suited for this?

hard

A. Use routing based on URL path to send 10% of requests to new service

B. Use traffic splitting with weights 90% to old and 10% to new service

C. Deploy both versions without traffic control and monitor errors

D. Use a load balancer that randomly sends requests without weights

Solution

1. Step 1: Understand gradual rollout needs

   Gradual rollout means controlling what percentage of users see the new version.

2. Step 2: Choose traffic management method

   Traffic splitting with weights allows precise control of request percentages to each version.

3. Step 3: Evaluate other options

   Routing by URL path cannot split traffic by percentage. Random load balancing lacks control. Deploying without control risks all users seeing new version.

4. Final Answer:

   Use traffic splitting with weights 90% to old and 10% to new service -> Option B

5. Quick Check:

   Splitting controls rollout percentages [OK]

Hint: Use weighted splitting for gradual rollout [OK]

Common Mistakes:

• Using URL path routing for percentage split
• Ignoring traffic control during rollout
• Relying on random load balancing