Services and networking in Microservices - System Design Guide

Problem Statement

When multiple microservices need to communicate, relying on direct, hardcoded connections causes tight coupling and brittle systems. Without proper networking and service management, requests can fail silently, cause delays, or overload services, leading to poor reliability and scalability.

Solution

Services communicate through well-defined network protocols using service discovery and load balancing. Requests are routed dynamically to healthy instances, and communication can be synchronous or asynchronous. This setup decouples services, improves fault tolerance, and enables scaling by adding or removing service instances transparently.

Architecture

Client App

→API Gateway

↓

Service B

This diagram shows a client sending requests to an API Gateway, which routes requests to multiple microservices (A, B, C). The gateway manages networking and load balancing between services.

Trade-offs

✓ Pros

→

Decouples services allowing independent deployment and scaling.

→

Improves fault tolerance by routing around failed instances.

→

Enables dynamic discovery of services without hardcoded addresses.

→

Supports multiple communication patterns (sync, async).

✗ Cons

→

Adds network latency compared to in-process calls.

→

Increases system complexity with service discovery and routing layers.

→

Requires careful security and authentication management across services.

Use when building distributed microservices architecture with multiple services needing reliable communication and scalability beyond a few instances.

Avoid if your system is a simple monolith or has fewer than 5 services with low traffic, where added networking complexity outweighs benefits.

Real World Examples

Netflix

Uses an API Gateway and service discovery to route millions of streaming requests reliably to thousands of microservice instances.

Uber

Employs service mesh and dynamic routing to connect ride-matching, payments, and notifications services with low latency and high availability.

Amazon

Uses service discovery and load balancing to manage communication between inventory, order, and payment microservices at massive scale.

Code Example

The before code calls a fixed service address, causing tight coupling and no failover. The after code uses a simple load balancer to pick among multiple service instances, enabling better availability and scalability.

Microservices

### Before: Direct service call with hardcoded address
import requests

def get_user_profile(user_id):
    response = requests.get(f'http://service-a:5000/users/{user_id}')
    return response.json()


### After: Using service discovery and load balancing
import requests

SERVICE_A_INSTANCES = ['http://10.0.0.1:5000', 'http://10.0.0.2:5000']

class LoadBalancer:
    def __init__(self, instances):
        self.instances = instances
        self.index = 0

    def get_instance(self):
        instance = self.instances[self.index]
        self.index = (self.index + 1) % len(self.instances)
        return instance

load_balancer = LoadBalancer(SERVICE_A_INSTANCES)

def get_user_profile(user_id):
    base_url = load_balancer.get_instance()
    response = requests.get(f'{base_url}/users/{user_id}')
    return response.json()

Alternatives

Monolithic Architecture

All components run in a single process without network calls between services.

Use when: Choose when your application is small, simple, and does not require independent scaling or deployment.

Serverless Functions

Functions are event-driven and stateless, communicating via events or APIs without persistent service instances.

Use when: Choose when you want to minimize infrastructure management and have highly variable workloads.

Summary

Networking in microservices prevents tight coupling by enabling dynamic service discovery and routing.

It improves reliability by load balancing requests and handling failures gracefully.

Proper networking supports scaling and independent deployment of services.