Secrets management (Vault, AWS Secrets Manager) in Microservices - System Design Guide

Problem Statement

Storing sensitive information like API keys, passwords, or certificates directly in application code or configuration files risks accidental exposure and unauthorized access. Without a centralized and secure way to manage secrets, microservices can leak credentials, leading to security breaches and compliance failures.

Solution

Secrets management systems securely store, control access to, and audit sensitive information. Applications request secrets at runtime from a centralized service that encrypts data at rest and in transit, enforces strict access policies, and rotates secrets automatically to reduce exposure risk.

Architecture

┌───────────────┐       ┌─────────────────────┐       ┌───────────────┐
│  Microservice │──────▶│ Secrets Management   │──────▶│ Secure Storage│
│  (Client)     │       │ Service (Vault, AWS) │       │ (Encrypted DB)│
└───────────────┘       └─────────────────────┘       └───────────────┘
        │                      ▲       │                      ▲
        │                      │       │                      │
        │                      │       │                      │
        └─────────Request──────┘       └─────Access Control───┘

This diagram shows microservices requesting secrets from a centralized secrets management service, which securely stores encrypted secrets and enforces access control.

Trade-offs

✓ Pros

→

Centralizes secret storage, reducing risk of accidental leaks in code or config files.

→

Enforces fine-grained access control and auditing for compliance and security.

→

Supports automatic secret rotation, minimizing exposure time if compromised.

→

Encrypts secrets at rest and in transit, protecting data from unauthorized access.

✗ Cons

→

Introduces a runtime dependency on the secrets management service, which can affect availability.

→

Adds operational complexity and requires secure setup and maintenance.

→

Potential latency overhead when fetching secrets dynamically at runtime.

Use when managing multiple microservices requiring secure, auditable, and scalable secret storage, especially at scale above hundreds of services or when compliance mandates strict controls.

Avoid when building small, simple applications with minimal secrets and low security risk, or when latency sensitivity prohibits runtime secret fetching.

Real World Examples

Netflix

Uses Vault to centrally manage and rotate secrets for thousands of microservices, ensuring secure access and auditability.

Airbnb

Employs AWS Secrets Manager to store database credentials and API keys, enabling automatic rotation and fine-grained access control.

Stripe

Implements Vault to securely distribute encryption keys and API tokens across services with strict access policies.

Code Example

The before code shows a secret hardcoded in the source, risking exposure if code leaks. The after code fetches the secret securely at runtime from Vault, avoiding hardcoding and enabling centralized secret management.

Microservices

### Before: Hardcoded secret in code (bad practice)

class PaymentService:
    def __init__(self):
        self.api_key = "hardcoded-secret-key"

    def process_payment(self):
        print(f"Using API key: {self.api_key}")


### After: Fetch secret from Vault dynamically (good practice)
import hvac

class PaymentService:
    def __init__(self):
        client = hvac.Client(url='https://vault.example.com', token='s.VaultToken')
        secret = client.secrets.kv.v2.read_secret_version(path='payment/api_key')
        self.api_key = secret['data']['data']['api_key']

    def process_payment(self):
        print(f"Using API key: {self.api_key}")

Alternatives

Environment Variables

Stores secrets as environment variables injected into service containers without centralized control or rotation.

Use when: Use for simple deployments with few secrets and no strict compliance requirements.

Configuration Files with Encryption

Secrets are stored encrypted in config files, requiring manual decryption and rotation.

Use when: Choose when centralized secret management is unavailable but encryption is needed.

Summary

Secrets management centralizes and secures sensitive information for microservices.

It enforces access control, auditing, and automatic rotation to reduce risk.

Popular tools like Vault and AWS Secrets Manager enable scalable, secure secret handling.

Practice

1. What is the main purpose of using a secrets management tool like Vault or AWS Secrets Manager in microservices?

easy

A. To monitor the performance of microservices

B. To increase the speed of microservices communication

C. To securely store and manage sensitive information like passwords and API keys

D. To deploy microservices automatically

Solution

1. Step 1: Understand the role of secrets management

   Secrets management tools are designed to keep sensitive data safe and separate from application code.

2. Step 2: Identify the correct purpose

   They securely store and control access to passwords, API keys, and tokens used by microservices.

3. Final Answer:

   To securely store and manage sensitive information like passwords and API keys -> Option C

4. Quick Check:

   Secrets management = Secure storage [OK]

Hint: Secrets tools keep passwords safe, not speed or deployment [OK]

Common Mistakes:

• Confusing secrets management with monitoring or deployment
• Thinking secrets tools improve communication speed
• Assuming secrets are stored inside code

2. Which of the following is the correct way to retrieve a secret value using AWS Secrets Manager CLI?

easy

A. aws secretsmanager get-secret-value --secret-id MySecret

B. aws secretsmanager fetch-secret --id MySecret

C. aws secretmanager get-value --name MySecret

D. aws secrets get-secret --secret MySecret

Solution

1. Step 1: Recall AWS Secrets Manager CLI syntax

   The correct command to get a secret value is 'aws secretsmanager get-secret-value' with the '--secret-id' parameter.

2. Step 2: Match the correct command

   aws secretsmanager get-secret-value --secret-id MySecret matches the exact AWS CLI syntax for retrieving secrets.

3. Final Answer:

   aws secretsmanager get-secret-value --secret-id MySecret -> Option A

4. Quick Check:

   AWS CLI get-secret-value = aws secretsmanager get-secret-value --secret-id MySecret [OK]

Hint: Remember 'get-secret-value' and '--secret-id' for AWS CLI [OK]

Common Mistakes:

• Using incorrect command verbs like 'fetch-secret'
• Mixing parameter names like '--id' instead of '--secret-id'
• Confusing service name as 'secretmanager' instead of 'secretsmanager'

3. Given this Vault CLI command sequence, what will be the output?

vault kv put secret/api-key value=12345
vault kv get -field=value secret/api-key

medium

A. secret/api-key
value=12345

B. value

C. Error: secret not found

D. 12345

Solution

1. Step 1: Understand the Vault put command

   The command 'vault kv put secret/api-key value=12345' stores the key 'value' with '12345' under 'secret/api-key'.

2. Step 2: Understand the Vault get command with '-field=value'

   The command 'vault kv get -field=value secret/api-key' retrieves only the value of the 'value' field, which is '12345'.

3. Final Answer:

   12345 -> Option D

4. Quick Check:

   Vault get -field=value returns the stored secret value [OK]

Hint: Use '-field' to get only the secret value, not full metadata [OK]

Common Mistakes:

• Expecting full secret metadata instead of just the value
• Confusing the output format of Vault CLI commands
• Assuming an error when secret exists

4. You wrote this AWS Secrets Manager policy snippet but your microservice cannot access the secret. What is the error?

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["secretsmanager:GetSecretValue"],
    "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:MySecret"
  }]
}

medium

A. The Action should be 'secretsmanager:RetrieveSecret'

B. The Resource ARN is missing a suffix with random characters

C. The Effect should be 'Deny' instead of 'Allow'

D. The Version date is incorrect

Solution

1. Step 1: Check the Resource ARN format for AWS Secrets Manager

   The ARN for a secret usually ends with a suffix of 6 random characters after the secret name, e.g., 'MySecret-abc123'.

2. Step 2: Identify the missing suffix issue

   The given ARN lacks this suffix, so the policy does not match the actual secret resource.

3. Final Answer:

   The Resource ARN is missing a suffix with random characters -> Option B

4. Quick Check:

   Secrets ARN needs suffix = The Resource ARN is missing a suffix with random characters [OK]

Hint: Secrets ARN always ends with random suffix, include it [OK]

Common Mistakes:

• Using incorrect action names
• Setting Effect to Deny by mistake
• Ignoring ARN suffix requirement

5. You want to rotate a database password stored in Vault automatically every 30 days. Which approach best follows best practices for secrets management?

hard

A. Use Vault's built-in dynamic secrets feature to generate and rotate credentials automatically

B. Manually update the password in Vault and the database every 30 days

C. Store the password in Vault as a static secret and notify the team to rotate it monthly

D. Embed the password in microservice code and update code every 30 days

Solution

1. Step 1: Understand Vault's dynamic secrets feature

   Vault can generate database credentials dynamically and rotate them automatically, improving security and reducing manual work.

2. Step 2: Compare options for best practice

   Using dynamic secrets automates rotation and avoids hardcoding or manual updates, which are error-prone.

3. Final Answer:

   Use Vault's built-in dynamic secrets feature to generate and rotate credentials automatically -> Option A

4. Quick Check:

   Dynamic secrets = automatic rotation [OK]

Hint: Automate rotation with Vault dynamic secrets, avoid manual updates [OK]

Common Mistakes:

• Relying on manual password updates
• Storing static secrets without rotation
• Hardcoding passwords in code