ConfigMaps and Secrets in Microservices - System Design Exercise

Design: ConfigMaps and Secrets Management in Microservices

Design focuses on managing ConfigMaps and Secrets for microservices in a containerized environment. Does not cover the internal implementation of microservices or the CI/CD pipeline.

Functional Requirements

FR1: Store configuration data separately from application code

FR2: Securely store sensitive information like passwords and API keys

FR3: Allow microservices to access configuration and secrets at runtime

FR4: Support dynamic updates to configuration without redeploying services

FR5: Ensure access control so only authorized services can read secrets

FR6: Provide audit logging for access to secrets

Non-Functional Requirements

NFR1: Must support at least 100 microservices accessing configs and secrets concurrently

NFR2: Configuration read latency should be under 50ms

NFR3: Secrets must be encrypted at rest and in transit

NFR4: System availability target of 99.9% uptime

NFR5: Support rolling updates of configuration without downtime

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

❓ Question 6

Key Components

Configuration store (ConfigMaps)

Secret management system

Access control and authentication service

Encryption mechanisms

Service discovery or sidecar agents

Audit logging system

Design Patterns

Sidecar pattern for injecting configs and secrets

Observer pattern for dynamic config updates

Role-based access control (RBAC)

Encryption at rest and in transit

Immutable infrastructure with config versioning

Reference Architecture

                    +---------------------+
                    |  Configuration Store |
                    |    (ConfigMaps)      |
                    +----------+----------+
                               |
                               v
+------------+          +---------------------+          +----------------+
| Microservice| <------ | Secret Management   | <------> | Access Control  |
|   Pod      |          | System (Secrets)     |          | & Auth Service  |
+------------+          +----------+----------+          +----------------+
                               |
                               v
                    +---------------------+
                    |   Audit Logging     |
                    +---------------------+

Components

Configuration Store (ConfigMaps)

Kubernetes ConfigMaps or similar

Store non-sensitive configuration data accessible by microservices

Secret Management System

Kubernetes Secrets, HashiCorp Vault, or AWS Secrets Manager

Securely store and provide sensitive data like passwords and API keys

Access Control & Authentication Service

Kubernetes RBAC, OAuth2, or custom auth service

Authenticate microservices and authorize access to secrets

Audit Logging System

Centralized logging system like ELK stack or CloudWatch

Record access and changes to secrets for compliance and monitoring

Sidecar Agent

Container sidecar pattern

Inject configuration and secrets into microservices at runtime securely

Request Flow

1. 1. Microservice pod starts and authenticates with Access Control service.

2. 2. Sidecar agent requests configuration data from Configuration Store.

3. 3. Sidecar agent requests secrets from Secret Management System using authenticated identity.

4. 4. Secret Management System verifies access rights via Access Control service.

5. 5. Configuration and secrets are delivered securely to sidecar agent.

6. 6. Sidecar agent injects configuration and secrets into microservice environment variables or files.

7. 7. Audit Logging system records all access and changes to secrets.

8. 8. When configuration updates occur, Configuration Store notifies sidecar agents to refresh configs dynamically.

Database Schema

Entities: - ConfigMap: id, name, data (key-value pairs), version, last_updated - Secret: id, name, encrypted_data, version, last_rotated - ServiceIdentity: id, service_name, credentials - AccessPolicy: id, service_identity_id, secret_id, permissions - AuditLog: id, service_identity_id, secret_id, action, timestamp Relationships: - ServiceIdentity has many AccessPolicies - AccessPolicy links ServiceIdentity to Secrets with permissions - AuditLog records actions by ServiceIdentity on Secrets

Scaling Discussion

Bottlenecks

Secret Management System becomes a single point of failure under high load

Latency increases when many microservices request secrets simultaneously

Audit Logging storage grows rapidly with many access events

Configuration updates cause spikes in network traffic to sidecars

Solutions

Deploy Secret Management System in a highly available cluster with load balancing

Implement caching of secrets at sidecar level with TTL to reduce repeated calls

Use scalable, partitioned logging storage with retention policies for audit logs

Use event-driven config update notifications with throttling to avoid overload

Interview Tips

Time: Spend 10 minutes clarifying requirements and constraints, 20 minutes designing the architecture and data flow, 10 minutes discussing scaling and security, 5 minutes summarizing.

Explain separation of config and secrets for security and flexibility

Describe how sidecar pattern helps inject configs securely

Discuss authentication and authorization for secret access

Highlight encryption and audit logging importance

Address how dynamic updates and scaling challenges are handled