Mutual TLS between services in Microservices - System Design Guide

Problem Statement

When microservices communicate over the network without verifying each other's identity, attackers can impersonate services or intercept sensitive data. This leads to unauthorized access, data breaches, and loss of trust between services.

Solution

Mutual TLS (mTLS) solves this by requiring both client and server services to present trusted certificates during connection setup. This two-way authentication ensures that each service verifies the other's identity before exchanging data, encrypting communication and preventing impersonation or eavesdropping.

Architecture

┌───────────────┐           ┌───────────────┐
│   Service A   │           │   Service B   │
│  (Client)    │           │  (Server)     │
└──────┬────────┘           └──────┬────────┘
       │ Mutual TLS handshake (certificates exchanged)
       │ ──────────────────────────────→
       │ ←───────────────────────────── 
       │ Encrypted, authenticated communication
       ↓
  Requests/Responses

This diagram shows two microservices performing a mutual TLS handshake where both exchange certificates to authenticate each other before secure communication.

Trade-offs

✓ Pros

→

Ensures both services authenticate each other, preventing impersonation.

→

Encrypts data in transit, protecting against eavesdropping.

→

Improves overall security posture by enforcing strict identity verification.

→

Works well in zero-trust network environments.

✗ Cons

→

Requires managing certificates and a trusted certificate authority, adding operational complexity.

→

Certificate rotation and revocation must be handled carefully to avoid downtime.

→

Adds latency during connection setup due to TLS handshake overhead.

Use mutual TLS when services handle sensitive data, require strong identity verification, or operate in untrusted or zero-trust networks with hundreds or more service instances.

Avoid mutual TLS in small-scale systems with fewer than 10 services where operational overhead outweighs security benefits, or when network is fully trusted and other simpler authentication methods suffice.

Real World Examples

Google

Google uses mutual TLS within its service mesh (Istio) to secure service-to-service communication in Kubernetes clusters, ensuring strong identity verification and encrypted traffic.

Netflix

Netflix employs mutual TLS in its microservices architecture to prevent unauthorized service access and protect sensitive user data during inter-service calls.

Uber

Uber uses mutual TLS to authenticate and encrypt communication between microservices, reducing risk of man-in-the-middle attacks in their distributed system.

Code Example

The before code shows a simple HTTP request without verifying the server or presenting a client certificate. The after code configures the client to present its certificate and verify the server's certificate using the CA, enabling mutual TLS authentication.

Microservices

### Before: Service client without mutual TLS
import requests

response = requests.get('https://service-b.internal/api/data')
print(response.text)

### After: Service client with mutual TLS
import requests

# Paths to client cert and key, and CA cert
client_cert = ('/path/client.crt', '/path/client.key')
ca_cert = '/path/ca.crt'

response = requests.get('https://service-b.internal/api/data', cert=client_cert, verify=ca_cert)
print(response.text)

Alternatives

API Gateway with Token-based Authentication

Centralizes authentication at the gateway using tokens instead of mutual certificate exchange between services.

Use when: Choose when you want simpler certificate management and can trust the gateway to enforce security.

Service Mesh without mTLS

Uses service mesh features like routing and observability but relies on other authentication methods instead of mutual TLS.

Use when: Choose when encryption or mutual authentication is not critical or handled by other layers.

Summary

Mutual TLS prevents impersonation and eavesdropping by requiring both services to authenticate each other with certificates.

It encrypts communication and enforces strong identity verification in microservices architectures.

Managing certificates and handshake overhead are trade-offs to consider before adopting mutual TLS.