Secrets management (Vault, AWS Secrets Manager) in Microservices - System Design Exercise

Design: Secrets Management System for Microservices

Design covers secret storage, retrieval, rotation, access control, and audit logging. Out of scope are microservice application logic and network infrastructure setup.

Functional Requirements

FR1: Securely store and manage secrets such as API keys, database credentials, and certificates.

FR2: Allow microservices to retrieve secrets dynamically at runtime.

FR3: Support secret rotation without downtime.

FR4: Provide audit logs for secret access and changes.

FR5: Ensure least privilege access control for secrets.

FR6: Integrate with existing microservices architecture.

Non-Functional Requirements

NFR1: Handle up to 10,000 microservice instances accessing secrets concurrently.

NFR2: API response latency for secret retrieval should be under 100ms p99.

NFR3: System availability must be at least 99.9% uptime.

NFR4: Secrets must be encrypted at rest and in transit.

NFR5: Support multi-region deployment for disaster recovery.

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

❓ Question 6

Key Components

Secret storage backend (encrypted database or cloud KMS)

Authentication and authorization service

Secret retrieval API

Secret rotation scheduler

Audit logging system

Cache layer for secrets

Design Patterns

Token-based authentication

Lease and renewal pattern for secrets

Write-ahead logging for audit

Sidecar pattern for secret injection

Cache aside pattern for secret caching

Reference Architecture

                    +---------------------+
                    |  Microservice Apps   |
                    +----------+----------+
                               |
                               | Request secret
                               v
                    +----------+----------+
                    |  Secret Retrieval API|
                    +----------+----------+
                               |
          +--------------------+--------------------+
          |                                         |
+---------v---------+                     +---------v---------+
| Authentication &  |                     |  Cache Layer      |
| Authorization     |                     |  (Redis or Memcached) |
+---------+---------+                     +---------+---------+
          |                                         |
          +--------------------+--------------------+
                               |
                    +----------v----------+
                    | Secret Storage Backend|
                    | (Vault or AWS Secrets |
                    | Manager with KMS)     |
                    +----------+----------+
                               |
                    +----------v----------+
                    | Audit Logging System |
                    +---------------------+

Components

Secret Retrieval API

REST/gRPC service

Handles requests from microservices to fetch secrets securely.

Authentication & Authorization

OAuth2 tokens, IAM roles, or Vault tokens

Verifies identity and permissions of microservices requesting secrets.

Cache Layer

Redis or Memcached

Stores recently accessed secrets to reduce latency and load on storage backend.

Secret Storage Backend

HashiCorp Vault or AWS Secrets Manager with KMS encryption

Securely stores encrypted secrets and manages secret lifecycle.

Audit Logging System

Centralized logging (e.g., ELK stack, CloudWatch Logs)

Records all secret access and modification events for compliance and monitoring.

Secret Rotation Scheduler

Cron jobs or managed rotation features

Automates periodic secret updates without downtime.

Request Flow

1. 1. Microservice sends authenticated request to Secret Retrieval API to get a secret.

2. 2. API verifies the microservice's identity and permissions via Authentication & Authorization component.

3. 3. API checks Cache Layer for the requested secret.

4. 4. If secret is in cache and valid, return it immediately.

5. 5. If not cached, API fetches secret from Secret Storage Backend.

6. 6. Secret Storage Backend decrypts and returns the secret.

7. 7. API stores secret in Cache Layer for future requests.

8. 8. API returns secret to the microservice securely over TLS.

9. 9. Audit Logging System records the access event.

10. 10. Secret Rotation Scheduler periodically updates secrets in the storage backend and invalidates cache.

Database Schema

Entities: - Secret: id (PK), name, encrypted_value, version, created_at, updated_at, rotation_policy - AccessLog: id (PK), secret_id (FK), microservice_id, access_time, action (read/write), success_flag - Microservice: id (PK), name, authentication_credentials, permissions Relationships: - One Microservice can access many Secrets (many-to-many via permissions) - AccessLog links Microservice and Secret with access details

Scaling Discussion

Bottlenecks

Secret Retrieval API becomes overloaded with high concurrent requests.

Cache Layer may have cache misses causing high latency.

Secret Storage Backend throughput limits under heavy load.

Audit Logging System storage and query performance degradation.

Secret Rotation causing temporary unavailability or stale secrets.

Solutions

Scale Secret Retrieval API horizontally behind a load balancer.

Optimize cache hit ratio by tuning TTL and pre-warming cache.

Use highly available and scalable secret storage solutions with multi-region replication.

Archive old logs and use scalable log storage with indexing for audit logs.

Implement zero-downtime secret rotation with versioning and gradual rollout.

Interview Tips

Time: Spend 10 minutes understanding requirements and clarifying constraints, 20 minutes designing architecture and data flow, 10 minutes discussing scaling and trade-offs, 5 minutes summarizing.

Emphasize security best practices: encryption, least privilege, audit logging.

Explain how caching reduces latency and load.

Discuss secret rotation strategies to avoid downtime.

Highlight authentication and authorization importance.

Address scalability and availability with concrete solutions.

Mention real-world tools like Vault and AWS Secrets Manager.

Practice

1. What is the main purpose of using a secrets management tool like Vault or AWS Secrets Manager in microservices?

easy

A. To monitor the performance of microservices

B. To increase the speed of microservices communication

C. To securely store and manage sensitive information like passwords and API keys

D. To deploy microservices automatically

Solution

1. Step 1: Understand the role of secrets management

   Secrets management tools are designed to keep sensitive data safe and separate from application code.

2. Step 2: Identify the correct purpose

   They securely store and control access to passwords, API keys, and tokens used by microservices.

3. Final Answer:

   To securely store and manage sensitive information like passwords and API keys -> Option C

4. Quick Check:

   Secrets management = Secure storage [OK]

Hint: Secrets tools keep passwords safe, not speed or deployment [OK]

Common Mistakes:

• Confusing secrets management with monitoring or deployment
• Thinking secrets tools improve communication speed
• Assuming secrets are stored inside code

2. Which of the following is the correct way to retrieve a secret value using AWS Secrets Manager CLI?

easy

A. aws secretsmanager get-secret-value --secret-id MySecret

B. aws secretsmanager fetch-secret --id MySecret

C. aws secretmanager get-value --name MySecret

D. aws secrets get-secret --secret MySecret

Solution

1. Step 1: Recall AWS Secrets Manager CLI syntax

   The correct command to get a secret value is 'aws secretsmanager get-secret-value' with the '--secret-id' parameter.

2. Step 2: Match the correct command

   aws secretsmanager get-secret-value --secret-id MySecret matches the exact AWS CLI syntax for retrieving secrets.

3. Final Answer:

   aws secretsmanager get-secret-value --secret-id MySecret -> Option A

4. Quick Check:

   AWS CLI get-secret-value = aws secretsmanager get-secret-value --secret-id MySecret [OK]

Hint: Remember 'get-secret-value' and '--secret-id' for AWS CLI [OK]

Common Mistakes:

• Using incorrect command verbs like 'fetch-secret'
• Mixing parameter names like '--id' instead of '--secret-id'
• Confusing service name as 'secretmanager' instead of 'secretsmanager'

3. Given this Vault CLI command sequence, what will be the output?

vault kv put secret/api-key value=12345
vault kv get -field=value secret/api-key

medium

A. secret/api-key
value=12345

B. value

C. Error: secret not found

D. 12345

Solution

1. Step 1: Understand the Vault put command

   The command 'vault kv put secret/api-key value=12345' stores the key 'value' with '12345' under 'secret/api-key'.

2. Step 2: Understand the Vault get command with '-field=value'

   The command 'vault kv get -field=value secret/api-key' retrieves only the value of the 'value' field, which is '12345'.

3. Final Answer:

   12345 -> Option D

4. Quick Check:

   Vault get -field=value returns the stored secret value [OK]

Hint: Use '-field' to get only the secret value, not full metadata [OK]

Common Mistakes:

• Expecting full secret metadata instead of just the value
• Confusing the output format of Vault CLI commands
• Assuming an error when secret exists

4. You wrote this AWS Secrets Manager policy snippet but your microservice cannot access the secret. What is the error?

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["secretsmanager:GetSecretValue"],
    "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:MySecret"
  }]
}

medium

A. The Action should be 'secretsmanager:RetrieveSecret'

B. The Resource ARN is missing a suffix with random characters

C. The Effect should be 'Deny' instead of 'Allow'

D. The Version date is incorrect

Solution

1. Step 1: Check the Resource ARN format for AWS Secrets Manager

   The ARN for a secret usually ends with a suffix of 6 random characters after the secret name, e.g., 'MySecret-abc123'.

2. Step 2: Identify the missing suffix issue

   The given ARN lacks this suffix, so the policy does not match the actual secret resource.

3. Final Answer:

   The Resource ARN is missing a suffix with random characters -> Option B

4. Quick Check:

   Secrets ARN needs suffix = The Resource ARN is missing a suffix with random characters [OK]

Hint: Secrets ARN always ends with random suffix, include it [OK]

Common Mistakes:

• Using incorrect action names
• Setting Effect to Deny by mistake
• Ignoring ARN suffix requirement

5. You want to rotate a database password stored in Vault automatically every 30 days. Which approach best follows best practices for secrets management?

hard

A. Use Vault's built-in dynamic secrets feature to generate and rotate credentials automatically

B. Manually update the password in Vault and the database every 30 days

C. Store the password in Vault as a static secret and notify the team to rotate it monthly

D. Embed the password in microservice code and update code every 30 days

Solution

1. Step 1: Understand Vault's dynamic secrets feature

   Vault can generate database credentials dynamically and rotate them automatically, improving security and reducing manual work.

2. Step 2: Compare options for best practice

   Using dynamic secrets automates rotation and avoids hardcoding or manual updates, which are error-prone.

3. Final Answer:

   Use Vault's built-in dynamic secrets feature to generate and rotate credentials automatically -> Option A

4. Quick Check:

   Dynamic secrets = automatic rotation [OK]

Hint: Automate rotation with Vault dynamic secrets, avoid manual updates [OK]

Common Mistakes:

• Relying on manual password updates
• Storing static secrets without rotation
• Hardcoding passwords in code