Secrets management (Vault, AWS Secrets Manager) in Microservices - System Design Guide

Problem Statement

Storing sensitive information like API keys, passwords, or certificates directly in application code or configuration files risks accidental exposure and unauthorized access. Without a centralized and secure way to manage secrets, microservices can leak credentials, leading to security breaches and compliance failures.

Solution

Secrets management systems securely store, control access to, and audit sensitive information. Applications request secrets at runtime from a centralized service that encrypts data at rest and in transit, enforces strict access policies, and rotates secrets automatically to reduce exposure risk.

Architecture

┌───────────────┐       ┌─────────────────────┐       ┌───────────────┐
│  Microservice │──────▶│ Secrets Management   │──────▶│ Secure Storage│
│  (Client)     │       │ Service (Vault, AWS) │       │ (Encrypted DB)│
└───────────────┘       └─────────────────────┘       └───────────────┘
        │                      ▲       │                      ▲
        │                      │       │                      │
        │                      │       │                      │
        └─────────Request──────┘       └─────Access Control───┘

This diagram shows microservices requesting secrets from a centralized secrets management service, which securely stores encrypted secrets and enforces access control.

Trade-offs

✓ Pros

→

Centralizes secret storage, reducing risk of accidental leaks in code or config files.

→

Enforces fine-grained access control and auditing for compliance and security.

→

Supports automatic secret rotation, minimizing exposure time if compromised.

→

Encrypts secrets at rest and in transit, protecting data from unauthorized access.

✗ Cons

→

Introduces a runtime dependency on the secrets management service, which can affect availability.

→

Adds operational complexity and requires secure setup and maintenance.

→

Potential latency overhead when fetching secrets dynamically at runtime.

Use when managing multiple microservices requiring secure, auditable, and scalable secret storage, especially at scale above hundreds of services or when compliance mandates strict controls.

Avoid when building small, simple applications with minimal secrets and low security risk, or when latency sensitivity prohibits runtime secret fetching.

Real World Examples

Netflix

Uses Vault to centrally manage and rotate secrets for thousands of microservices, ensuring secure access and auditability.

Airbnb

Employs AWS Secrets Manager to store database credentials and API keys, enabling automatic rotation and fine-grained access control.

Stripe

Implements Vault to securely distribute encryption keys and API tokens across services with strict access policies.

Code Example

The before code shows a secret hardcoded in the source, risking exposure if code leaks. The after code fetches the secret securely at runtime from Vault, avoiding hardcoding and enabling centralized secret management.

Microservices

### Before: Hardcoded secret in code (bad practice)

class PaymentService:
    def __init__(self):
        self.api_key = "hardcoded-secret-key"

    def process_payment(self):
        print(f"Using API key: {self.api_key}")


### After: Fetch secret from Vault dynamically (good practice)
import hvac

class PaymentService:
    def __init__(self):
        client = hvac.Client(url='https://vault.example.com', token='s.VaultToken')
        secret = client.secrets.kv.v2.read_secret_version(path='payment/api_key')
        self.api_key = secret['data']['data']['api_key']

    def process_payment(self):
        print(f"Using API key: {self.api_key}")

Alternatives

Environment Variables

Stores secrets as environment variables injected into service containers without centralized control or rotation.

Use when: Use for simple deployments with few secrets and no strict compliance requirements.

Configuration Files with Encryption

Secrets are stored encrypted in config files, requiring manual decryption and rotation.

Use when: Choose when centralized secret management is unavailable but encryption is needed.

Summary

Secrets management centralizes and secures sensitive information for microservices.

It enforces access control, auditing, and automatic rotation to reduce risk.

Popular tools like Vault and AWS Secrets Manager enable scalable, secure secret handling.