Mutual TLS between services in Microservices - System Design Guide

Problem Statement

When microservices communicate over the network without verifying each other's identity, attackers can impersonate services or intercept sensitive data. This leads to unauthorized access, data breaches, and loss of trust between services.

Solution

Mutual TLS (mTLS) solves this by requiring both client and server services to present trusted certificates during connection setup. This two-way authentication ensures that each service verifies the other's identity before exchanging data, encrypting communication and preventing impersonation or eavesdropping.

Architecture

┌───────────────┐           ┌───────────────┐
│   Service A   │           │   Service B   │
│  (Client)    │           │  (Server)     │
└──────┬────────┘           └──────┬────────┘
       │ Mutual TLS handshake (certificates exchanged)
       │ ──────────────────────────────→
       │ ←───────────────────────────── 
       │ Encrypted, authenticated communication
       ↓
  Requests/Responses

This diagram shows two microservices performing a mutual TLS handshake where both exchange certificates to authenticate each other before secure communication.

Trade-offs

✓ Pros

→

Ensures both services authenticate each other, preventing impersonation.

→

Encrypts data in transit, protecting against eavesdropping.

→

Improves overall security posture by enforcing strict identity verification.

→

Works well in zero-trust network environments.

✗ Cons

→

Requires managing certificates and a trusted certificate authority, adding operational complexity.

→

Certificate rotation and revocation must be handled carefully to avoid downtime.

→

Adds latency during connection setup due to TLS handshake overhead.

Use mutual TLS when services handle sensitive data, require strong identity verification, or operate in untrusted or zero-trust networks with hundreds or more service instances.

Avoid mutual TLS in small-scale systems with fewer than 10 services where operational overhead outweighs security benefits, or when network is fully trusted and other simpler authentication methods suffice.

Real World Examples

Google

Google uses mutual TLS within its service mesh (Istio) to secure service-to-service communication in Kubernetes clusters, ensuring strong identity verification and encrypted traffic.

Netflix

Netflix employs mutual TLS in its microservices architecture to prevent unauthorized service access and protect sensitive user data during inter-service calls.

Uber

Uber uses mutual TLS to authenticate and encrypt communication between microservices, reducing risk of man-in-the-middle attacks in their distributed system.

Code Example

The before code shows a simple HTTP request without verifying the server or presenting a client certificate. The after code configures the client to present its certificate and verify the server's certificate using the CA, enabling mutual TLS authentication.

Microservices

### Before: Service client without mutual TLS
import requests

response = requests.get('https://service-b.internal/api/data')
print(response.text)

### After: Service client with mutual TLS
import requests

# Paths to client cert and key, and CA cert
client_cert = ('/path/client.crt', '/path/client.key')
ca_cert = '/path/ca.crt'

response = requests.get('https://service-b.internal/api/data', cert=client_cert, verify=ca_cert)
print(response.text)

Alternatives

API Gateway with Token-based Authentication

Centralizes authentication at the gateway using tokens instead of mutual certificate exchange between services.

Use when: Choose when you want simpler certificate management and can trust the gateway to enforce security.

Service Mesh without mTLS

Uses service mesh features like routing and observability but relies on other authentication methods instead of mutual TLS.

Use when: Choose when encryption or mutual authentication is not critical or handled by other layers.

Summary

Mutual TLS prevents impersonation and eavesdropping by requiring both services to authenticate each other with certificates.

It encrypts communication and enforces strong identity verification in microservices architectures.

Managing certificates and handshake overhead are trade-offs to consider before adopting mutual TLS.

Practice

1. What is the main purpose of using Mutual TLS between microservices?

easy

A. To allow services to communicate without encryption

B. To speed up the communication between services

C. To ensure both services authenticate each other before communication

D. To store service data securely on disk

Solution

1. Step 1: Understand Mutual TLS authentication

   Mutual TLS requires both client and server to present certificates proving their identity.

2. Step 2: Identify the purpose in microservices

   This ensures only trusted services communicate securely, preventing unauthorized access.

3. Final Answer:

   To ensure both services authenticate each other before communication -> Option C

4. Quick Check:

   Mutual TLS = mutual authentication [OK]

Hint: Mutual TLS means both sides prove who they are [OK]

Common Mistakes:

• Thinking it only encrypts data without authentication
• Assuming it speeds up communication
• Confusing it with data storage security

2. Which of the following is the correct step to enable Mutual TLS in a microservice?

easy

A. Disable certificate verification on both services

B. Share the same private key among all services

C. Use plain HTTP instead of HTTPS

D. Configure each service with its own certificate and trust store

Solution

1. Step 1: Identify certificate requirements

   Each service must have its own certificate and trust store to verify others.

2. Step 2: Understand security best practices

   Disabling verification or sharing keys breaks security and is incorrect.

3. Final Answer:

   Configure each service with its own certificate and trust store -> Option D

4. Quick Check:

   Certificates + trust store = Mutual TLS setup [OK]

Hint: Each service needs its own certificate and trust store [OK]

Common Mistakes:

• Disabling certificate verification to simplify setup
• Using HTTP which is unencrypted
• Sharing private keys causing security risks

3. Given two microservices A and B configured with Mutual TLS, what happens if service B presents an expired certificate during handshake?

medium

A. Service A accepts the connection without checks

B. Service A rejects the connection due to invalid certificate

C. Service B automatically renews the certificate

D. The connection proceeds but logs a warning

Solution

1. Step 1: Understand certificate validation in Mutual TLS

   Certificates must be valid and trusted; expired certificates are rejected.

2. Step 2: Identify handshake behavior on invalid certificates

   If service B's certificate is expired, service A will reject the connection to maintain security.

3. Final Answer:

   Service A rejects the connection due to invalid certificate -> Option B

4. Quick Check:

   Expired certificate = connection rejected [OK]

Hint: Expired cert means connection is rejected [OK]

Common Mistakes:

• Assuming expired certs are accepted with warnings
• Thinking certificates auto-renew during handshake
• Believing connection proceeds without checks

4. A microservice fails to establish Mutual TLS with another service. The error logs show "certificate unknown". What is the most likely cause?

medium

A. The service's certificate is not signed by a trusted CA

B. The service is using HTTP instead of HTTPS

C. The private key is missing from the service

D. The service is using a self-signed certificate but trusts it

Solution

1. Step 1: Analyze the error "certificate unknown"

   This error means the certificate presented is not recognized or trusted by the other service.

2. Step 2: Identify cause related to trust

   If the certificate is not signed by a trusted CA, the other service will reject it as unknown.

3. Final Answer:

   The service's certificate is not signed by a trusted CA -> Option A

4. Quick Check:

   Untrusted CA = certificate unknown error [OK]

Hint: Certificate unknown means untrusted CA signature [OK]

Common Mistakes:

• Confusing HTTP usage with certificate errors
• Assuming missing private key causes this error
• Believing self-signed certs are trusted by default

5. You need to design a microservices system with Mutual TLS where services dynamically scale up and down. Which approach best ensures secure and scalable certificate management?

hard

A. Use a centralized certificate authority with automated certificate issuance and rotation

B. Manually generate and distribute certificates to each service instance

C. Disable Mutual TLS during scaling to avoid certificate issues

D. Use the same certificate for all service instances to simplify management

Solution

1. Step 1: Understand challenges of scaling with Mutual TLS

   Dynamic scaling requires automated certificate management to avoid manual errors and delays.

2. Step 2: Evaluate options for secure and scalable management

   A centralized CA with automation allows issuing and rotating certificates securely as instances scale.

3. Step 3: Reject insecure or manual approaches

   Manual distribution is error-prone, disabling TLS reduces security, and sharing certificates risks compromise.

4. Final Answer:

   Use a centralized certificate authority with automated certificate issuance and rotation -> Option A

5. Quick Check:

   Central CA + automation = scalable Mutual TLS [OK]

Hint: Automate certs with central CA for scaling [OK]

Common Mistakes:

• Manually managing certs for each instance
• Disabling Mutual TLS to avoid complexity
• Sharing certificates across instances