Service-to-service authentication in Microservices - System Design Exercise

Design: Service-to-Service Authentication System

Design focuses on authentication between microservices only. Authorization, user authentication, and API gateway design are out of scope.

Functional Requirements

FR1: Allow microservices to securely authenticate with each other without user involvement

FR2: Support token-based authentication with short-lived tokens

FR3: Enable services to verify the identity of calling services

FR4: Provide a centralized authentication service for issuing and validating tokens

FR5: Ensure tokens cannot be easily forged or reused after expiration

FR6: Support revocation of tokens in case of compromise

FR7: Allow easy integration with existing microservices architecture

Non-Functional Requirements

NFR1: Must handle up to 10,000 service-to-service authentication requests per second

NFR2: Token validation latency should be under 50ms (p99)

NFR3: System availability must be at least 99.9% uptime

NFR4: Tokens should expire within 5 minutes to reduce risk

NFR5: Authentication service must be horizontally scalable

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

❓ Question 6

Key Components

Authentication Service (Token Issuer)

Token Validation Library or Middleware in each service

Secure Storage for keys or secrets

Service Registry or Discovery

Revocation List or Cache

Logging and Monitoring

Design Patterns

OAuth 2.0 Client Credentials Flow

JWT (JSON Web Tokens) for stateless tokens

Mutual TLS (mTLS) for service identity

Token Introspection Endpoint

Caching token validation results

Reference Architecture

                    +-----------------------+
                    |   Authentication       |
                    |       Service          |
                    |  (Token Issuer &       |
                    |   Validator)           |
                    +-----------+-----------+
                                |
                                | Issue tokens (JWT)
                                |
+----------------+      +-------v--------+      +----------------+
|  Microservice 1 | ---> | Token Validation| <--- |  Microservice 2 |
|  (Client)      |      | Middleware     |      |  (Client)       |
+----------------+      +----------------+      +----------------+

Legend:
- Microservices request tokens from Authentication Service
- Microservices validate tokens locally using middleware
- Authentication Service signs tokens with private key
- Public key distributed to services for verification

Components

Authentication Service

Node.js/Java/Spring Boot (any scalable backend)

Issue signed JWT tokens to services and provide token introspection

Token Validation Middleware

Library in each microservice (e.g., JWT verification library)

Verify token signature, expiration, and claims on each request

Key Management

Vault or KMS (e.g., HashiCorp Vault, AWS KMS)

Securely store and rotate signing keys

Service Registry

Consul/Eureka/Etcd

Allow services to discover Authentication Service endpoints

Revocation Cache

Redis or in-memory cache

Store revoked token IDs for quick lookup

Logging and Monitoring

ELK Stack, Prometheus, Grafana

Track authentication requests, failures, and system health

Request Flow

1. 1. Microservice requests a token from Authentication Service using its credentials.

2. 2. Authentication Service authenticates the service and issues a signed JWT token with a short expiry.

3. 3. Microservice includes the JWT token in requests to other microservices.

4. 4. Receiving microservice uses Token Validation Middleware to verify the token signature and expiry locally.

5. 5. Middleware checks the revocation cache to ensure token is not revoked.

6. 6. If token is valid, request proceeds; otherwise, it is rejected with an authentication error.

7. 7. Authentication Service periodically rotates signing keys and updates public keys distributed to services.

Database Schema

Entities: - ServiceCredentials: service_id (PK), secret, metadata - RevokedTokens: token_id (PK), revoked_at - SigningKeys: key_id (PK), public_key, private_key, created_at, expires_at Relationships: - ServiceCredentials used to authenticate services requesting tokens - RevokedTokens checked during token validation - SigningKeys used by Authentication Service to sign tokens and by services to verify tokens

Scaling Discussion

Bottlenecks

Authentication Service becomes a bottleneck under high token request load

Token validation latency if every request requires remote validation

Key management complexity with frequent key rotations

Revocation cache size and lookup latency

Network latency between services and Authentication Service

Solutions

Scale Authentication Service horizontally behind a load balancer

Use stateless JWT tokens so services can validate tokens locally without remote calls

Implement key rotation with overlapping keys and distribute public keys via service registry or config

Use fast in-memory cache (e.g., Redis) for revocation list with TTL to limit size

Use service mesh or sidecar proxies to handle authentication and reduce network overhead

Interview Tips

Time: Spend 10 minutes clarifying requirements and constraints, 20 minutes designing the architecture and data flow, 10 minutes discussing scaling and trade-offs, and 5 minutes summarizing.

Explain why token-based authentication is suitable for service-to-service communication

Discuss trade-offs between JWT and opaque tokens

Highlight importance of short-lived tokens and revocation

Describe how local token validation reduces latency and load

Mention key management and secure storage as critical for security

Address scaling by horizontal service scaling and caching

Show awareness of real-world challenges like key rotation and token revocation

Practice

1. What is the main purpose of service-to-service authentication in microservices?

easy

A. To ensure that one service can securely verify the identity of another service

B. To speed up communication between services

C. To store data between services

D. To monitor the health of services

Solution

1. Step 1: Understand the role of authentication

   Authentication is about verifying identity to ensure trust between entities.

2. Step 2: Apply to microservices context

   In microservices, service-to-service authentication ensures one service knows it is talking to a trusted service.

3. Final Answer:

   To ensure that one service can securely verify the identity of another service -> Option A

4. Quick Check:

   Authentication means verifying identity = A [OK]

Hint: Authentication means verifying identity between services [OK]

Common Mistakes:

• Confusing authentication with data storage
• Thinking authentication speeds up communication
• Mixing authentication with monitoring

2. Which of the following is a common method used for service-to-service authentication?

easy

A. Using JWT tokens issued by an authentication server

B. Using SQL queries to verify service identity

C. Using CSS styles to secure communication

D. Using HTML forms for authentication

Solution

1. Step 1: Identify valid authentication methods

   JWT tokens are widely used for secure token-based authentication between services.

2. Step 2: Eliminate unrelated options

   SQL queries, CSS, and HTML forms are unrelated to service authentication.

3. Final Answer:

   Using JWT tokens issued by an authentication server -> Option A

4. Quick Check:

   JWT tokens = common authentication method [OK]

Hint: JWT tokens are standard for service authentication [OK]

Common Mistakes:

• Confusing UI technologies with authentication
• Thinking database queries authenticate services
• Mixing frontend and backend concepts

3. Consider this simplified code snippet for service-to-service authentication using JWT:

token = auth_server.issue_token(service_id="serviceA")
if auth_server.verify_token(token):
    print("Access granted")
else:
    print("Access denied")

What will be printed if the token is valid?

medium

A. Access denied

B. Error: token missing

C. Access granted

D. No output

Solution

1. Step 1: Understand token issuance and verification

   The token is issued by the auth server and then verified immediately.

2. Step 2: Check the conditional logic

   If the token is valid, verify_token returns True, so "Access granted" is printed.

3. Final Answer:

   Access granted -> Option C

4. Quick Check:

   Valid token means access granted [OK]

Hint: Valid token means verify_token returns True [OK]

Common Mistakes:

• Assuming token is invalid without checking
• Confusing print outputs
• Ignoring the if-else structure

4. A microservice uses mTLS for service-to-service authentication but fails to connect. Which is the most likely cause?

medium

A. The server service is down

B. The API key is expired

C. The database is unreachable

D. The client service does not have a valid client certificate

Solution

1. Step 1: Understand mTLS requirements

   mTLS requires both client and server to have valid certificates for mutual authentication.

2. Step 2: Identify the cause of failure

   If connection fails due to authentication, missing or invalid client certificate is the likely cause.

3. Final Answer:

   The client service does not have a valid client certificate -> Option D

4. Quick Check:

   mTLS needs valid client cert = B [OK]

Hint: mTLS needs valid client certificate on both sides [OK]

Common Mistakes:

• Blaming server downtime without checking certificates
• Confusing database issues with authentication
• Mixing API keys with mTLS

5. You design a system where multiple microservices authenticate each other using JWT tokens issued by a central auth server. To improve scalability and security, which approach is best?

hard

A. Each service calls the auth server to verify tokens on every request

B. Each service validates tokens locally using the auth server's public key without calling the auth server every time

C. Services share a single API key for all authentication

D. Services trust any token without verification to reduce latency

Solution

1. Step 1: Consider scalability of token verification

   Calling the auth server on every request creates a bottleneck and reduces scalability.

2. Step 2: Use public key verification locally

   JWT tokens can be verified locally using the auth server's public key, improving speed and security.

3. Final Answer:

   Each service validates tokens locally using the auth server's public key without calling the auth server every time -> Option B

4. Quick Check:

   Local JWT verification improves scalability = A [OK]

Hint: Verify JWT locally with public key for scalability [OK]

Common Mistakes:

• Calling auth server on every request causing bottlenecks
• Using shared API keys reduces security
• Skipping token verification breaks security