Traffic management (routing, splitting) in Microservices - System Design Guide

Problem Statement

When all user requests are sent to the same version of a microservice, it becomes risky to deploy new features or fixes. A faulty release can cause widespread failures, and it is hard to test changes with real traffic without impacting all users.

Solution

Traffic management directs user requests to different versions or instances of a service based on rules. Routing sends requests to specific service versions, while splitting divides traffic by percentage or conditions. This allows gradual rollout, A/B testing, and quick rollback without downtime.

Architecture

User

→Traffic

↓

→Service v2

This diagram shows user requests flowing into a traffic manager that routes or splits requests between different service versions.

Trade-offs

✓ Pros

→

Enables safe gradual rollout of new features by controlling traffic percentage.

→

Supports A/B testing by routing specific user groups to different service versions.

→

Allows quick rollback by redirecting all traffic to a stable version.

→

Improves system resilience by isolating faulty versions from full traffic.

✗ Cons

→

Adds complexity to deployment and monitoring due to multiple active service versions.

→

Requires careful configuration to avoid uneven load or user experience issues.

→

May increase latency slightly due to routing logic before request reaches service.

Use when deploying new service versions frequently, needing gradual rollout, A/B testing, or canary releases at scale above 1000 requests per second.

Avoid if your system has very low traffic (under 100 req/sec) or if you deploy infrequently and can afford full downtime for updates.

Real World Examples

Netflix

Netflix uses traffic splitting to gradually roll out new streaming service features to a subset of users, minimizing risk of widespread failures.

Uber

Uber routes requests to different microservice versions to test new pricing algorithms on a small user group before full deployment.

Amazon

Amazon uses routing rules to direct traffic to different service versions during deployments, enabling canary releases and quick rollback.

Code Example

The before code sends all requests to one service version, risking full impact of failures. The after code splits requests randomly, sending 80% to the stable version and 20% to the new version, enabling gradual rollout and testing.

Microservices

### Before: No traffic splitting, all requests go to one service version
class ServiceClient:
    def send_request(self, request):
        # Always send to v1
        return self.call_service_v1(request)

    def call_service_v1(self, request):
        # Call service v1 endpoint
        pass


### After: Traffic splitting between v1 and v2
import random

class ServiceClient:
    def send_request(self, request):
        # Split traffic 80% to v1, 20% to v2
        if random.random() < 0.8:
            return self.call_service_v1(request)
        else:
            return self.call_service_v2(request)

    def call_service_v1(self, request):
        # Call service v1 endpoint
        pass

    def call_service_v2(self, request):
        # Call service v2 endpoint
        pass

Alternatives

Blue-Green Deployment

Routes all traffic to either the old or new version, switching instantly without splitting traffic.

Use when: Choose when you want zero overlap between versions and can afford instant cutover.

Feature Flags

Controls feature availability within the same service version without routing traffic differently.

Use when: Choose when you want to toggle features quickly without deploying new service versions.

Summary

Traffic management prevents full impact of faulty deployments by controlling request flow.

Routing and splitting enable gradual rollout, A/B testing, and quick rollback in microservices.

This pattern is essential for safe, scalable deployment in systems with frequent updates.