Secrets management (Vault, AWS Secrets Manager) in Microservices - Scalability & System Analysis

Scalability Analysis - Secrets management (Vault, AWS Secrets Manager)

Growth Table: Secrets Management Scaling

Scale	Number of Microservices	Secrets Stored	Request Rate (QPS)	Key Changes
100 users	10-20	100-500	50-200	Single Vault/AWS Secrets Manager instance; low latency; simple access policies
10,000 users	100-200	5,000-10,000	1,000-5,000	Introduce caching at microservice side; enable read replicas; fine-grained access control
1,000,000 users	1,000+	100,000+	50,000-100,000	Use distributed Vault clusters or multi-region AWS Secrets Manager; heavy caching; rate limiting; secrets rotation automation
100,000,000 users	10,000+	1,000,000+	500,000+	Global multi-region deployment; sharding secrets by service or region; advanced monitoring; strict quota enforcement

First Bottleneck

The first bottleneck is the secrets storage backend (Vault or AWS Secrets Manager). At moderate scale, the backend can become overwhelmed by high QPS of secret read requests from many microservices, causing increased latency and throttling.

Scaling Solutions

• Caching: Implement local caching of secrets in microservices with TTL to reduce backend calls.
• Read Replicas: Use Vault clusters or AWS Secrets Manager replicas to distribute read load.
• Horizontal Scaling: Deploy multiple Vault nodes behind a load balancer or use multi-region AWS Secrets Manager.
• Sharding: Partition secrets by service or region to reduce contention.
• Rate Limiting: Enforce request quotas to prevent overload.
• Automation: Automate secret rotation and renewal to avoid stale secrets and reduce manual overhead.

Back-of-Envelope Cost Analysis

• At 10,000 QPS, assuming each secret read is ~1KB, bandwidth = 10,000 KB/s (~10 MB/s).
• Storage: For 100,000 secrets averaging 1KB each, total storage ~100 MB (small, but grows with metadata and versions).
• CPU/Memory: Vault nodes need enough CPU to handle encryption/decryption and network I/O; AWS Secrets Manager is managed but costs scale with requests.
• Network: Ensure network capacity to handle peak QPS without latency spikes.

Interview Tip

Start by identifying the main components: secrets storage, microservices, and access patterns. Discuss bottlenecks focusing on request rates and latency. Propose caching and replication early. Highlight security concerns like access control and rotation. Structure your answer by scale and how each solution addresses specific bottlenecks.

Self Check Question

Your database handles 1000 QPS for secret reads. Traffic grows 10x to 10,000 QPS. What do you do first?

Answer: Implement caching at the microservice level to reduce direct reads from the secrets backend, and add read replicas or scale Vault nodes horizontally to distribute load.

Key Result

Secrets storage backend becomes the first bottleneck as request rates grow; caching and replication are key to scaling.

Practice

1. What is the main purpose of using a secrets management tool like Vault or AWS Secrets Manager in microservices?

easy

A. To monitor the performance of microservices

B. To increase the speed of microservices communication

C. To securely store and manage sensitive information like passwords and API keys

D. To deploy microservices automatically

Solution

1. Step 1: Understand the role of secrets management

   Secrets management tools are designed to keep sensitive data safe and separate from application code.

2. Step 2: Identify the correct purpose

   They securely store and control access to passwords, API keys, and tokens used by microservices.

3. Final Answer:

   To securely store and manage sensitive information like passwords and API keys -> Option C

4. Quick Check:

   Secrets management = Secure storage [OK]

Hint: Secrets tools keep passwords safe, not speed or deployment [OK]

Common Mistakes:

• Confusing secrets management with monitoring or deployment
• Thinking secrets tools improve communication speed
• Assuming secrets are stored inside code

2. Which of the following is the correct way to retrieve a secret value using AWS Secrets Manager CLI?

easy

A. aws secretsmanager get-secret-value --secret-id MySecret

B. aws secretsmanager fetch-secret --id MySecret

C. aws secretmanager get-value --name MySecret

D. aws secrets get-secret --secret MySecret

Solution

1. Step 1: Recall AWS Secrets Manager CLI syntax

   The correct command to get a secret value is 'aws secretsmanager get-secret-value' with the '--secret-id' parameter.

2. Step 2: Match the correct command

   aws secretsmanager get-secret-value --secret-id MySecret matches the exact AWS CLI syntax for retrieving secrets.

3. Final Answer:

   aws secretsmanager get-secret-value --secret-id MySecret -> Option A

4. Quick Check:

   AWS CLI get-secret-value = aws secretsmanager get-secret-value --secret-id MySecret [OK]

Hint: Remember 'get-secret-value' and '--secret-id' for AWS CLI [OK]

Common Mistakes:

• Using incorrect command verbs like 'fetch-secret'
• Mixing parameter names like '--id' instead of '--secret-id'
• Confusing service name as 'secretmanager' instead of 'secretsmanager'

3. Given this Vault CLI command sequence, what will be the output?

vault kv put secret/api-key value=12345
vault kv get -field=value secret/api-key

medium

A. secret/api-key
value=12345

B. value

C. Error: secret not found

D. 12345

Solution

1. Step 1: Understand the Vault put command

   The command 'vault kv put secret/api-key value=12345' stores the key 'value' with '12345' under 'secret/api-key'.

2. Step 2: Understand the Vault get command with '-field=value'

   The command 'vault kv get -field=value secret/api-key' retrieves only the value of the 'value' field, which is '12345'.

3. Final Answer:

   12345 -> Option D

4. Quick Check:

   Vault get -field=value returns the stored secret value [OK]

Hint: Use '-field' to get only the secret value, not full metadata [OK]

Common Mistakes:

• Expecting full secret metadata instead of just the value
• Confusing the output format of Vault CLI commands
• Assuming an error when secret exists

4. You wrote this AWS Secrets Manager policy snippet but your microservice cannot access the secret. What is the error?

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["secretsmanager:GetSecretValue"],
    "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:MySecret"
  }]
}

medium

A. The Action should be 'secretsmanager:RetrieveSecret'

B. The Resource ARN is missing a suffix with random characters

C. The Effect should be 'Deny' instead of 'Allow'

D. The Version date is incorrect

Solution

1. Step 1: Check the Resource ARN format for AWS Secrets Manager

   The ARN for a secret usually ends with a suffix of 6 random characters after the secret name, e.g., 'MySecret-abc123'.

2. Step 2: Identify the missing suffix issue

   The given ARN lacks this suffix, so the policy does not match the actual secret resource.

3. Final Answer:

   The Resource ARN is missing a suffix with random characters -> Option B

4. Quick Check:

   Secrets ARN needs suffix = The Resource ARN is missing a suffix with random characters [OK]

Hint: Secrets ARN always ends with random suffix, include it [OK]

Common Mistakes:

• Using incorrect action names
• Setting Effect to Deny by mistake
• Ignoring ARN suffix requirement

5. You want to rotate a database password stored in Vault automatically every 30 days. Which approach best follows best practices for secrets management?

hard

A. Use Vault's built-in dynamic secrets feature to generate and rotate credentials automatically

B. Manually update the password in Vault and the database every 30 days

C. Store the password in Vault as a static secret and notify the team to rotate it monthly

D. Embed the password in microservice code and update code every 30 days

Solution

1. Step 1: Understand Vault's dynamic secrets feature

   Vault can generate database credentials dynamically and rotate them automatically, improving security and reducing manual work.

2. Step 2: Compare options for best practice

   Using dynamic secrets automates rotation and avoids hardcoding or manual updates, which are error-prone.

3. Final Answer:

   Use Vault's built-in dynamic secrets feature to generate and rotate credentials automatically -> Option A

4. Quick Check:

   Dynamic secrets = automatic rotation [OK]

Hint: Automate rotation with Vault dynamic secrets, avoid manual updates [OK]

Common Mistakes:

• Relying on manual password updates
• Storing static secrets without rotation
• Hardcoding passwords in code