Secrets management (Vault, AWS Secrets Manager) in Microservices - Architecture Diagram

System Overview - Secrets management (Vault, AWS Secrets Manager)

This system securely manages sensitive information like passwords, API keys, and certificates for microservices. It ensures secrets are stored safely, accessed only by authorized services, and rotated regularly to reduce risk.

Architecture Diagram

User
  |
  v
Load Balancer
  |
  v
API Gateway
  |
  v
Microservices Cluster
  |
  +-------------------+
  |                   |
  v                   v
Secrets Manager     Cache Layer
(Vault or AWS       (In-memory or
Secrets Manager)    Redis/Memcached)
  |
  v
Secure Storage Backend
(Database or Cloud KMS)

Components

User

client

Initiates requests to microservices

Load Balancer

load_balancer

Distributes incoming requests evenly to API Gateway instances

API Gateway

api_gateway

Routes requests to appropriate microservices and handles authentication

Microservices Cluster

service

Runs business logic and requests secrets when needed

Secrets Manager (Vault or AWS Secrets Manager)

secrets_manager

Stores and manages secrets securely with access control and auditing

Cache Layer

cache

Temporarily stores secrets to reduce latency and load on Secrets Manager

Secure Storage Backend

database

Encrypted storage for secrets managed by the Secrets Manager

Request Flow - 12 Hops

User → Load Balancer

Load Balancer → API Gateway

API Gateway → Microservices Cluster

Microservices Cluster → Cache Layer

Cache Layer → Microservices Cluster

Cache Layer → Secrets Manager

Secrets Manager → Secure Storage Backend

Secrets Manager → Cache Layer

Secrets Manager → Microservices Cluster

Microservices Cluster → API Gateway

API Gateway → Load Balancer

Load Balancer → User

Failure Scenario

Component Fails:Secrets Manager

Impact:Microservices cannot retrieve new secrets; cached secrets still available but may become stale

Mitigation:Use cache to serve secrets temporarily; implement Secrets Manager replication and failover; alert operators for quick recovery

Architecture Quiz - 3 Questions

Which component is responsible for securely storing and managing secrets?

A API Gateway

B Secrets Manager

C Cache Layer

D Load Balancer

Design Principle

This architecture uses a dedicated Secrets Manager to centralize secret storage and access control, combined with a cache layer to reduce latency and improve availability. It separates concerns, secures sensitive data, and ensures microservices can access secrets efficiently and safely.

Practice

1. What is the main purpose of using a secrets management tool like Vault or AWS Secrets Manager in microservices?

easy

A. To monitor the performance of microservices

B. To increase the speed of microservices communication

C. To securely store and manage sensitive information like passwords and API keys

D. To deploy microservices automatically

Solution

1. Step 1: Understand the role of secrets management

   Secrets management tools are designed to keep sensitive data safe and separate from application code.

2. Step 2: Identify the correct purpose

   They securely store and control access to passwords, API keys, and tokens used by microservices.

3. Final Answer:

   To securely store and manage sensitive information like passwords and API keys -> Option C

4. Quick Check:

   Secrets management = Secure storage [OK]

Hint: Secrets tools keep passwords safe, not speed or deployment [OK]

Common Mistakes:

• Confusing secrets management with monitoring or deployment
• Thinking secrets tools improve communication speed
• Assuming secrets are stored inside code

2. Which of the following is the correct way to retrieve a secret value using AWS Secrets Manager CLI?

easy

A. aws secretsmanager get-secret-value --secret-id MySecret

B. aws secretsmanager fetch-secret --id MySecret

C. aws secretmanager get-value --name MySecret

D. aws secrets get-secret --secret MySecret

Solution

1. Step 1: Recall AWS Secrets Manager CLI syntax

   The correct command to get a secret value is 'aws secretsmanager get-secret-value' with the '--secret-id' parameter.

2. Step 2: Match the correct command

   aws secretsmanager get-secret-value --secret-id MySecret matches the exact AWS CLI syntax for retrieving secrets.

3. Final Answer:

   aws secretsmanager get-secret-value --secret-id MySecret -> Option A

4. Quick Check:

   AWS CLI get-secret-value = aws secretsmanager get-secret-value --secret-id MySecret [OK]

Hint: Remember 'get-secret-value' and '--secret-id' for AWS CLI [OK]

Common Mistakes:

• Using incorrect command verbs like 'fetch-secret'
• Mixing parameter names like '--id' instead of '--secret-id'
• Confusing service name as 'secretmanager' instead of 'secretsmanager'

3. Given this Vault CLI command sequence, what will be the output?

vault kv put secret/api-key value=12345
vault kv get -field=value secret/api-key

medium

A. secret/api-key
value=12345

B. value

C. Error: secret not found

D. 12345

Solution

1. Step 1: Understand the Vault put command

   The command 'vault kv put secret/api-key value=12345' stores the key 'value' with '12345' under 'secret/api-key'.

2. Step 2: Understand the Vault get command with '-field=value'

   The command 'vault kv get -field=value secret/api-key' retrieves only the value of the 'value' field, which is '12345'.

3. Final Answer:

   12345 -> Option D

4. Quick Check:

   Vault get -field=value returns the stored secret value [OK]

Hint: Use '-field' to get only the secret value, not full metadata [OK]

Common Mistakes:

• Expecting full secret metadata instead of just the value
• Confusing the output format of Vault CLI commands
• Assuming an error when secret exists

4. You wrote this AWS Secrets Manager policy snippet but your microservice cannot access the secret. What is the error?

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["secretsmanager:GetSecretValue"],
    "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:MySecret"
  }]
}

medium

A. The Action should be 'secretsmanager:RetrieveSecret'

B. The Resource ARN is missing a suffix with random characters

C. The Effect should be 'Deny' instead of 'Allow'

D. The Version date is incorrect

Solution

1. Step 1: Check the Resource ARN format for AWS Secrets Manager

   The ARN for a secret usually ends with a suffix of 6 random characters after the secret name, e.g., 'MySecret-abc123'.

2. Step 2: Identify the missing suffix issue

   The given ARN lacks this suffix, so the policy does not match the actual secret resource.

3. Final Answer:

   The Resource ARN is missing a suffix with random characters -> Option B

4. Quick Check:

   Secrets ARN needs suffix = The Resource ARN is missing a suffix with random characters [OK]

Hint: Secrets ARN always ends with random suffix, include it [OK]

Common Mistakes:

• Using incorrect action names
• Setting Effect to Deny by mistake
• Ignoring ARN suffix requirement

5. You want to rotate a database password stored in Vault automatically every 30 days. Which approach best follows best practices for secrets management?

hard

A. Use Vault's built-in dynamic secrets feature to generate and rotate credentials automatically

B. Manually update the password in Vault and the database every 30 days

C. Store the password in Vault as a static secret and notify the team to rotate it monthly

D. Embed the password in microservice code and update code every 30 days

Solution

1. Step 1: Understand Vault's dynamic secrets feature

   Vault can generate database credentials dynamically and rotate them automatically, improving security and reducing manual work.

2. Step 2: Compare options for best practice

   Using dynamic secrets automates rotation and avoids hardcoding or manual updates, which are error-prone.

3. Final Answer:

   Use Vault's built-in dynamic secrets feature to generate and rotate credentials automatically -> Option A

4. Quick Check:

   Dynamic secrets = automatic rotation [OK]

Hint: Automate rotation with Vault dynamic secrets, avoid manual updates [OK]

Common Mistakes:

• Relying on manual password updates
• Storing static secrets without rotation
• Hardcoding passwords in code