ConfigMaps and Secrets in Microservices - System Design Exercise

Design: ConfigMaps and Secrets Management in Microservices

Design focuses on managing ConfigMaps and Secrets for microservices in a containerized environment. Does not cover the internal implementation of microservices or the CI/CD pipeline.

Functional Requirements

FR1: Store configuration data separately from application code

FR2: Securely store sensitive information like passwords and API keys

FR3: Allow microservices to access configuration and secrets at runtime

FR4: Support dynamic updates to configuration without redeploying services

FR5: Ensure access control so only authorized services can read secrets

FR6: Provide audit logging for access to secrets

Non-Functional Requirements

NFR1: Must support at least 100 microservices accessing configs and secrets concurrently

NFR2: Configuration read latency should be under 50ms

NFR3: Secrets must be encrypted at rest and in transit

NFR4: System availability target of 99.9% uptime

NFR5: Support rolling updates of configuration without downtime

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

❓ Question 6

Key Components

Configuration store (ConfigMaps)

Secret management system

Access control and authentication service

Encryption mechanisms

Service discovery or sidecar agents

Audit logging system

Design Patterns

Sidecar pattern for injecting configs and secrets

Observer pattern for dynamic config updates

Role-based access control (RBAC)

Encryption at rest and in transit

Immutable infrastructure with config versioning

Reference Architecture

                    +---------------------+
                    |  Configuration Store |
                    |    (ConfigMaps)      |
                    +----------+----------+
                               |
                               v
+------------+          +---------------------+          +----------------+
| Microservice| <------ | Secret Management   | <------> | Access Control  |
|   Pod      |          | System (Secrets)     |          | & Auth Service  |
+------------+          +----------+----------+          +----------------+
                               |
                               v
                    +---------------------+
                    |   Audit Logging     |
                    +---------------------+

Components

Configuration Store (ConfigMaps)

Kubernetes ConfigMaps or similar

Store non-sensitive configuration data accessible by microservices

Secret Management System

Kubernetes Secrets, HashiCorp Vault, or AWS Secrets Manager

Securely store and provide sensitive data like passwords and API keys

Access Control & Authentication Service

Kubernetes RBAC, OAuth2, or custom auth service

Authenticate microservices and authorize access to secrets

Audit Logging System

Centralized logging system like ELK stack or CloudWatch

Record access and changes to secrets for compliance and monitoring

Sidecar Agent

Container sidecar pattern

Inject configuration and secrets into microservices at runtime securely

Request Flow

1. 1. Microservice pod starts and authenticates with Access Control service.

2. 2. Sidecar agent requests configuration data from Configuration Store.

3. 3. Sidecar agent requests secrets from Secret Management System using authenticated identity.

4. 4. Secret Management System verifies access rights via Access Control service.

5. 5. Configuration and secrets are delivered securely to sidecar agent.

6. 6. Sidecar agent injects configuration and secrets into microservice environment variables or files.

7. 7. Audit Logging system records all access and changes to secrets.

8. 8. When configuration updates occur, Configuration Store notifies sidecar agents to refresh configs dynamically.

Database Schema

Entities: - ConfigMap: id, name, data (key-value pairs), version, last_updated - Secret: id, name, encrypted_data, version, last_rotated - ServiceIdentity: id, service_name, credentials - AccessPolicy: id, service_identity_id, secret_id, permissions - AuditLog: id, service_identity_id, secret_id, action, timestamp Relationships: - ServiceIdentity has many AccessPolicies - AccessPolicy links ServiceIdentity to Secrets with permissions - AuditLog records actions by ServiceIdentity on Secrets

Scaling Discussion

Bottlenecks

Secret Management System becomes a single point of failure under high load

Latency increases when many microservices request secrets simultaneously

Audit Logging storage grows rapidly with many access events

Configuration updates cause spikes in network traffic to sidecars

Solutions

Deploy Secret Management System in a highly available cluster with load balancing

Implement caching of secrets at sidecar level with TTL to reduce repeated calls

Use scalable, partitioned logging storage with retention policies for audit logs

Use event-driven config update notifications with throttling to avoid overload

Interview Tips

Time: Spend 10 minutes clarifying requirements and constraints, 20 minutes designing the architecture and data flow, 10 minutes discussing scaling and security, 5 minutes summarizing.

Explain separation of config and secrets for security and flexibility

Describe how sidecar pattern helps inject configs securely

Discuss authentication and authorization for secret access

Highlight encryption and audit logging importance

Address how dynamic updates and scaling challenges are handled

Practice

1. What is the primary purpose of a ConfigMap in a microservices environment?

easy

A. To store encrypted passwords and tokens securely

B. To store non-sensitive configuration data separately from application code

C. To manage network traffic between microservices

D. To monitor the health of microservices

Solution

1. Step 1: Understand ConfigMap role

   A ConfigMap is designed to hold configuration data that is not sensitive, such as environment variables or config files, separate from the application code.

2. Step 2: Differentiate from Secrets

   Secrets are used for sensitive data like passwords, not ConfigMaps.

3. Final Answer:

   To store non-sensitive configuration data separately from application code -> Option B

4. Quick Check:

   ConfigMap = Non-sensitive config [OK]

Hint: ConfigMaps hold safe config, Secrets hold sensitive info [OK]

Common Mistakes:

• Confusing ConfigMaps with Secrets
• Thinking ConfigMaps encrypt data
• Assuming ConfigMaps manage network or health

2. Which of the following is the correct way to create a Secret in Kubernetes using a literal value?

easy

A. kubectl create secret generic mysecret --from-literal=password=12345

B. kubectl create configmap mysecret --from-literal=password=12345

C. kubectl secret create mysecret --from-file=password.txt

D. kubectl create secret mysecret --password=12345

Solution

1. Step 1: Recall Kubernetes Secret creation syntax

   The correct command uses kubectl create secret generic with --from-literal to create a secret from a literal key-value pair.

2. Step 2: Eliminate incorrect options

   kubectl create configmap mysecret --from-literal=password=12345 uses configmap instead of secret, kubectl secret create mysecret --from-file=password.txt uses wrong command syntax, kubectl create secret mysecret --password=12345 uses invalid flag.

3. Final Answer:

   kubectl create secret generic mysecret --from-literal=password=12345 -> Option A

4. Quick Check:

   Secret creation uses 'kubectl create secret generic' [OK]

Hint: Secret creation uses 'kubectl create secret generic' [OK]

Common Mistakes:

• Using configmap command instead of secret
• Wrong flags like --password
• Confusing file and literal input

3. Given this Kubernetes pod spec snippet, what will be the value of environment variable API_KEY inside the container?

env:
  - name: API_KEY
    valueFrom:
      secretKeyRef:
        name: mysecret
        key: api_key

medium

A. An empty string because 'api_key' is not defined

B. The literal string 'api_key'

C. The value stored under key 'api_key' in the Secret named 'mysecret'

D. The value of a ConfigMap named 'mysecret'

Solution

1. Step 1: Understand env var from secretKeyRef

   The environment variable API_KEY is set from a Secret named mysecret, specifically the key api_key.

2. Step 2: Clarify what value is assigned

   The value assigned is the secret data stored under that key, not the literal string or empty.

3. Final Answer:

   The value stored under key 'api_key' in the Secret named 'mysecret' -> Option C

4. Quick Check:

   secretKeyRef pulls secret value by key [OK]

Hint: secretKeyRef uses secret's key value, not literal text [OK]

Common Mistakes:

• Assuming it uses the key name as string
• Confusing ConfigMap with Secret
• Thinking it defaults to empty if key missing

4. A developer tries to mount a Secret as a volume in a pod but gets an error. Which of these is the most likely cause?

medium

A. The Secret does not exist or has a different name than specified

B. The ConfigMap is missing required keys

C. The pod spec uses configMapKeyRef instead of secretKeyRef

D. The volume mount path is outside the container filesystem

Solution

1. Step 1: Identify common Secret mount errors

   Mounting a Secret as a volume requires the Secret to exist with the exact name referenced.

2. Step 2: Evaluate other options

   ConfigMap keys missing affect ConfigMaps, not Secrets. Using configMapKeyRef instead of secretKeyRef is for env vars, not volumes. Volume mount path errors cause different errors.

3. Final Answer:

   The Secret does not exist or has a different name than specified -> Option A

4. Quick Check:

   Secret volume mount fails if Secret missing [OK]

Hint: Check Secret name exists before mounting volume [OK]

Common Mistakes:

• Mixing ConfigMap and Secret references
• Ignoring Secret existence before mount
• Confusing env var refs with volume mounts

5. You want to securely provide database credentials to multiple microservices without exposing them in code or logs. Which approach best uses ConfigMaps and Secrets together?

hard

A. Embed credentials directly in microservice code and use ConfigMaps for feature flags

B. Store credentials in ConfigMaps and encrypt them manually; use Secrets for logging configs

C. Put all configs and secrets in ConfigMaps for simplicity; avoid Secrets

D. Store all credentials in a Secret and non-sensitive configs in ConfigMaps; mount both as environment variables

Solution

1. Step 1: Separate sensitive and non-sensitive data

   Best practice is to keep sensitive data like credentials in Secrets and non-sensitive configs in ConfigMaps.

2. Step 2: Use environment variables for flexibility

   Mounting both as environment variables allows microservices to access configs securely without code changes.

3. Final Answer:

   Store all credentials in a Secret and non-sensitive configs in ConfigMaps; mount both as environment variables -> Option D

4. Quick Check:

   Secrets for sensitive, ConfigMaps for safe config [OK]

Hint: Secrets for sensitive data, ConfigMaps for safe config [OK]

Common Mistakes:

• Putting secrets in ConfigMaps unencrypted
• Embedding credentials in code
• Using Secrets for non-sensitive data