Mutual TLS between services in Microservices - System Design Exercise

Design: Mutual TLS Authentication for Microservices

Design focuses on securing communication between microservices using mutual TLS. It excludes client-to-service authentication and certificate authority infrastructure setup beyond basic assumptions.

Functional Requirements

FR1: All microservices must authenticate each other before exchanging data.

FR2: Communication between services must be encrypted to prevent eavesdropping.

FR3: The system should support automatic certificate rotation without downtime.

FR4: Services must reject connections from unauthorized or untrusted services.

FR5: The solution should integrate with existing service discovery mechanisms.

Non-Functional Requirements

NFR1: The system must handle up to 10,000 concurrent service-to-service connections.

NFR2: Latency added by mutual TLS handshake should be under 50ms on average.

NFR3: Availability target is 99.9% uptime for service communication.

NFR4: Certificates must be managed securely and comply with industry best practices.

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

❓ Question 6

Key Components

Certificate Authority (CA) for issuing and signing certificates

Certificate management system or automation tools

Service mesh or sidecar proxies for handling TLS

Service discovery mechanism

Load balancers or API gateways if applicable

Design Patterns

Zero Trust Security Model

Sidecar Proxy Pattern

Certificate Rotation Automation

Service Mesh Integration

Mutual TLS Handshake Process

Reference Architecture

  +----------------+       +----------------+       +----------------+
  |   Service A    | <---> |   Service B    | <---> |   Service C    |
  |  (with mTLS)   |       |  (with mTLS)   |       |  (with mTLS)   |
  +----------------+       +----------------+       +----------------+
          |                        |                        |
          |                        |                        |
          v                        v                        v
  +---------------------------------------------------------------+
  |                   Certificate Authority (CA)                  |
  +---------------------------------------------------------------+
          ^                        ^                        ^
          |                        |                        |
  +----------------+       +----------------+       +----------------+
  | Certificate     |       | Certificate     |       | Certificate     |
  | Management &   |       | Management &   |       | Management &   |
  | Rotation       |       | Rotation       |       | Rotation       |
  +----------------+       +----------------+       +----------------+

Components

Certificate Authority (CA)

PKI system (e.g., HashiCorp Vault, Let's Encrypt internal CA)

Issue and sign certificates for each microservice to enable trust.

Certificate Management & Rotation

Automation tools/scripts or built-in platform features

Automatically renew and rotate certificates without downtime.

Microservices

Any microservice framework (e.g., Spring Boot, Node.js, Go)

Run business logic and establish mutual TLS connections with peers.

Sidecar Proxy / Service Mesh

Istio, Linkerd, Envoy

Handle mutual TLS handshake, encryption, and certificate validation transparently.

Service Discovery

Consul, Kubernetes DNS, or similar

Enable services to find each other and establish secure connections.

Request Flow

1. 1. Each microservice requests a certificate from the Certificate Authority (CA).

2. 2. The CA issues a certificate signed by its private key, trusted by all services.

3. 3. Certificates are installed on services or their sidecar proxies.

4. 4. When Service A wants to communicate with Service B, it initiates a TLS handshake.

5. 5. Both services present their certificates to each other (mutual authentication).

6. 6. Each service verifies the other's certificate against the trusted CA certificate.

7. 7. If verification succeeds, a secure encrypted channel is established.

8. 8. Services exchange data over this encrypted channel.

9. 9. Certificate management system monitors certificate expiry and triggers rotation.

10. 10. Services update certificates seamlessly without dropping connections.

Database Schema

Not applicable as this design focuses on secure communication infrastructure rather than data storage.

Scaling Discussion

Bottlenecks

Certificate Authority becoming a single point of failure under high load.

Latency overhead due to TLS handshake for many concurrent connections.

Managing certificate rotation at scale without service disruption.

Complexity in configuring and maintaining sidecar proxies or service mesh.

Service discovery delays affecting connection establishment.

Solutions

Deploy highly available CA clusters with load balancing and failover.

Use TLS session resumption and connection pooling to reduce handshake overhead.

Automate certificate rotation with zero downtime deployment strategies.

Adopt managed service mesh solutions to reduce operational complexity.

Optimize service discovery caching and health checks for faster lookups.

Interview Tips

Time: Spend 10 minutes understanding requirements and clarifying assumptions, 20 minutes designing architecture and data flow, 10 minutes discussing scaling and trade-offs, 5 minutes summarizing.

Explain the importance of mutual TLS for secure service-to-service communication.

Describe how certificates are issued, validated, and rotated.

Highlight the role of sidecar proxies or service mesh in simplifying mTLS.

Discuss latency and availability trade-offs with TLS handshakes.

Address scaling challenges and how automation helps maintain security at scale.