ConfigMaps and Secrets in Microservices - Scalability & System Analysis

Scalability Analysis - ConfigMaps and Secrets

Growth Table: ConfigMaps and Secrets Scaling

Users/Services	ConfigMaps/Secrets Count	Access Frequency	Storage Size	Update Frequency
100 users / 10 services	~10 ConfigMaps, ~5 Secrets	Low to moderate	Small (KBs)	Rare
10,000 users / 100 services	~100 ConfigMaps, ~50 Secrets	Moderate	Medium (MBs)	Occasional
1,000,000 users / 1,000 services	~1,000 ConfigMaps, ~500 Secrets	High	Large (100s MBs)	Frequent
100,000,000 users / 10,000 services	~10,000 ConfigMaps, ~5,000 Secrets	Very High	Very Large (GBs)	Continuous

First Bottleneck

The first bottleneck is the API server and etcd storage that hold ConfigMaps and Secrets. As the number and size of ConfigMaps and Secrets grow, etcd's storage and read/write throughput limits are reached first. This causes delays in configuration updates and retrievals by microservices.

Scaling Solutions

• Horizontal scaling: Add more API server instances behind a load balancer to distribute requests.
• etcd clustering: Use a properly sized etcd cluster with enough nodes and resources to handle increased storage and QPS.
• Caching: Implement local caching of ConfigMaps and Secrets in microservices to reduce frequent API calls.
• Secret management tools: Integrate external secret managers (e.g., HashiCorp Vault) to offload secret storage and rotation.
• Sharding: Partition ConfigMaps and Secrets by namespaces or service groups to reduce load on single etcd cluster.
• Compression and size limits: Keep ConfigMaps and Secrets small and compress data to reduce storage and network load.

Back-of-Envelope Cost Analysis

• Requests per second: Each microservice may request ConfigMaps/Secrets on startup and periodically. For 1,000 services, assuming 1 request per minute, ~17 QPS; for 10,000 services, ~167 QPS.
• Storage: If average ConfigMap/Secret size is 10 KB, 10,000 ConfigMaps total ~100 MB; 100,000 ConfigMaps ~1 GB.
• Bandwidth: Assuming 167 QPS and 10 KB per request, ~1.67 MB/s network bandwidth needed for config data.
• etcd throughput: etcd can handle ~10,000 QPS but performance degrades with large data sizes and many keys.

Interview Tip

Start by explaining the role of ConfigMaps and Secrets in microservices. Then discuss how they are stored and accessed (etcd, API server). Identify the bottleneck as etcd storage and API server throughput. Suggest caching and external secret managers as solutions. Finally, mention partitioning and compression to optimize storage and network usage.

Self Check Question

Your etcd cluster handles 1,000 QPS for ConfigMap and Secret requests. Traffic grows 10x. What do you do first and why?

Answer: First, add more API server instances and scale the etcd cluster horizontally to handle increased QPS. Also, implement caching in microservices to reduce load on etcd.

Key Result

ConfigMaps and Secrets scale bottleneck first at etcd storage and API server throughput; caching and horizontal scaling of etcd and API servers are key solutions.

Practice

1. What is the primary purpose of a ConfigMap in a microservices environment?

easy

A. To store encrypted passwords and tokens securely

B. To store non-sensitive configuration data separately from application code

C. To manage network traffic between microservices

D. To monitor the health of microservices

Solution

1. Step 1: Understand ConfigMap role

   A ConfigMap is designed to hold configuration data that is not sensitive, such as environment variables or config files, separate from the application code.

2. Step 2: Differentiate from Secrets

   Secrets are used for sensitive data like passwords, not ConfigMaps.

3. Final Answer:

   To store non-sensitive configuration data separately from application code -> Option B

4. Quick Check:

   ConfigMap = Non-sensitive config [OK]

Hint: ConfigMaps hold safe config, Secrets hold sensitive info [OK]

Common Mistakes:

• Confusing ConfigMaps with Secrets
• Thinking ConfigMaps encrypt data
• Assuming ConfigMaps manage network or health

2. Which of the following is the correct way to create a Secret in Kubernetes using a literal value?

easy

A. kubectl create secret generic mysecret --from-literal=password=12345

B. kubectl create configmap mysecret --from-literal=password=12345

C. kubectl secret create mysecret --from-file=password.txt

D. kubectl create secret mysecret --password=12345

Solution

1. Step 1: Recall Kubernetes Secret creation syntax

   The correct command uses kubectl create secret generic with --from-literal to create a secret from a literal key-value pair.

2. Step 2: Eliminate incorrect options

   kubectl create configmap mysecret --from-literal=password=12345 uses configmap instead of secret, kubectl secret create mysecret --from-file=password.txt uses wrong command syntax, kubectl create secret mysecret --password=12345 uses invalid flag.

3. Final Answer:

   kubectl create secret generic mysecret --from-literal=password=12345 -> Option A

4. Quick Check:

   Secret creation uses 'kubectl create secret generic' [OK]

Hint: Secret creation uses 'kubectl create secret generic' [OK]

Common Mistakes:

• Using configmap command instead of secret
• Wrong flags like --password
• Confusing file and literal input

3. Given this Kubernetes pod spec snippet, what will be the value of environment variable API_KEY inside the container?

env:
  - name: API_KEY
    valueFrom:
      secretKeyRef:
        name: mysecret
        key: api_key

medium

A. An empty string because 'api_key' is not defined

B. The literal string 'api_key'

C. The value stored under key 'api_key' in the Secret named 'mysecret'

D. The value of a ConfigMap named 'mysecret'

Solution

1. Step 1: Understand env var from secretKeyRef

   The environment variable API_KEY is set from a Secret named mysecret, specifically the key api_key.

2. Step 2: Clarify what value is assigned

   The value assigned is the secret data stored under that key, not the literal string or empty.

3. Final Answer:

   The value stored under key 'api_key' in the Secret named 'mysecret' -> Option C

4. Quick Check:

   secretKeyRef pulls secret value by key [OK]

Hint: secretKeyRef uses secret's key value, not literal text [OK]

Common Mistakes:

• Assuming it uses the key name as string
• Confusing ConfigMap with Secret
• Thinking it defaults to empty if key missing

4. A developer tries to mount a Secret as a volume in a pod but gets an error. Which of these is the most likely cause?

medium

A. The Secret does not exist or has a different name than specified

B. The ConfigMap is missing required keys

C. The pod spec uses configMapKeyRef instead of secretKeyRef

D. The volume mount path is outside the container filesystem

Solution

1. Step 1: Identify common Secret mount errors

   Mounting a Secret as a volume requires the Secret to exist with the exact name referenced.

2. Step 2: Evaluate other options

   ConfigMap keys missing affect ConfigMaps, not Secrets. Using configMapKeyRef instead of secretKeyRef is for env vars, not volumes. Volume mount path errors cause different errors.

3. Final Answer:

   The Secret does not exist or has a different name than specified -> Option A

4. Quick Check:

   Secret volume mount fails if Secret missing [OK]

Hint: Check Secret name exists before mounting volume [OK]

Common Mistakes:

• Mixing ConfigMap and Secret references
• Ignoring Secret existence before mount
• Confusing env var refs with volume mounts

5. You want to securely provide database credentials to multiple microservices without exposing them in code or logs. Which approach best uses ConfigMaps and Secrets together?

hard

A. Embed credentials directly in microservice code and use ConfigMaps for feature flags

B. Store credentials in ConfigMaps and encrypt them manually; use Secrets for logging configs

C. Put all configs and secrets in ConfigMaps for simplicity; avoid Secrets

D. Store all credentials in a Secret and non-sensitive configs in ConfigMaps; mount both as environment variables

Solution

1. Step 1: Separate sensitive and non-sensitive data

   Best practice is to keep sensitive data like credentials in Secrets and non-sensitive configs in ConfigMaps.

2. Step 2: Use environment variables for flexibility

   Mounting both as environment variables allows microservices to access configs securely without code changes.

3. Final Answer:

   Store all credentials in a Secret and non-sensitive configs in ConfigMaps; mount both as environment variables -> Option D

4. Quick Check:

   Secrets for sensitive, ConfigMaps for safe config [OK]

Hint: Secrets for sensitive data, ConfigMaps for safe config [OK]

Common Mistakes:

• Putting secrets in ConfigMaps unencrypted
• Embedding credentials in code
• Using Secrets for non-sensitive data