Environment configuration in Microservices - System Design Exercise

Design: Microservices Environment Configuration System

In scope: design of the centralized configuration management system for microservices including storage, retrieval, security, and update mechanisms. Out of scope: the internal implementation details of individual microservices.

Functional Requirements

FR1: Provide a centralized way to manage configuration settings for multiple microservices.

FR2: Support different environments such as development, testing, staging, and production.

FR3: Allow secure storage and retrieval of sensitive configuration data like API keys and passwords.

FR4: Enable dynamic configuration updates without restarting services.

FR5: Support versioning and rollback of configuration changes.

FR6: Provide access control to restrict who can view or modify configurations.

FR7: Ensure high availability and low latency for configuration retrieval.

Non-Functional Requirements

NFR1: Must handle configuration for at least 100 microservices.

NFR2: Configuration retrieval latency should be under 50ms at p99.

NFR3: System availability target is 99.9% uptime (less than 8.77 hours downtime per year).

NFR4: Configuration updates should propagate within 30 seconds to all services.

NFR5: Secure storage must comply with industry best practices for secrets management.

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

❓ Question 6

Key Components

Central configuration storage (e.g., distributed key-value store)

Configuration API service for read/write operations

Authentication and authorization service

Cache layer for fast configuration retrieval

Notification or event system for configuration change propagation

Audit logging for configuration changes

Secrets management integration

Design Patterns

Configuration as Code

Feature Flags

Publish-Subscribe for change notifications

Cache Aside pattern for configuration caching

Role-Based Access Control (RBAC)

Circuit Breaker for configuration service resilience

Reference Architecture

                    +-----------------------+
                    |  Configuration Clients |
                    |  (Microservices)       |
                    +-----------+-----------+
                                |
                                | Fetch Config via API
                                v
                    +-----------------------+
                    | Configuration API     |
                    | Service               |
                    +-----------+-----------+
                                |
               +----------------+----------------+
               |                                 |
       +-------v-------+                 +-------v-------+
       | Cache Layer   |                 | Auth & Audit  |
       | (Redis/Etcd)  |                 | Service       |
       +-------+-------+                 +-------+-------+
               |                                 |
               |                                 |
       +-------v-------------------------------v-------+
       |          Central Configuration Store          |
       |       (Distributed KV Store + Secrets)        |
       +-----------------------------------------------+

Change notifications via pub-sub system to clients for dynamic updates.

Components

Configuration API Service

RESTful API with gRPC support

Handles requests from microservices to read and update configuration data securely.

Central Configuration Store

Distributed key-value store (e.g., Consul, Etcd, or ZooKeeper) with secrets management integration

Stores all configuration data including environment-specific and sensitive information.

Cache Layer

In-memory cache (e.g., Redis)

Provides low-latency access to frequently requested configuration data.

Authentication and Authorization Service

OAuth2 / RBAC system

Ensures only authorized users and services can access or modify configurations.

Audit Logging

Centralized logging system (e.g., ELK stack)

Records all configuration changes for compliance and troubleshooting.

Notification System

Message broker (e.g., Kafka, RabbitMQ, or native pub-sub)

Pushes configuration change events to microservices for dynamic updates.

Request Flow

1. 1. Microservice requests configuration from Configuration API Service.

2. 2. API Service checks Cache Layer for requested configuration.

3. 3. If cache miss, API Service fetches configuration from Central Configuration Store.

4. 4. Configuration data is returned to microservice and cached for future requests.

5. 5. When configuration is updated via API Service, changes are stored in Central Store.

6. 6. API Service publishes configuration change event to Notification System.

7. 7. Microservices subscribed to notifications receive update and refresh their local config cache.

8. 8. Authentication and Authorization Service validates all requests to ensure security.

9. 9. Audit Logging records all configuration read and write operations.

Database Schema

Entities: - ConfigurationEntry: id (PK), service_name, environment, key, value, version, is_secret, created_at, updated_at - User: id (PK), username, role - AccessControl: id (PK), user_id (FK), service_name, permission_type (read/write) - AuditLog: id (PK), user_id (FK), action_type (read/write/update), config_entry_id (FK), timestamp Relationships: - User to AccessControl is 1:N - ConfigurationEntry versions tracked by version field - AuditLog references User and ConfigurationEntry for traceability

Scaling Discussion

Bottlenecks

Central Configuration Store can become a read/write bottleneck under high load.

Cache Layer may become stale or overwhelmed with requests if not properly sized.

Notification System may have delays or message loss affecting dynamic updates.

Authentication Service could slow down requests if not scalable.

Audit Logging may generate large volumes of data impacting storage and query performance.

Solutions

Use distributed and highly available key-value stores with replication and sharding.

Implement cache invalidation strategies and scale cache horizontally.

Use reliable message brokers with persistence and retry mechanisms for notifications.

Scale Authentication Service horizontally and use token-based authentication to reduce load.

Archive old audit logs and use efficient indexing for fast queries.

Interview Tips

Time: Spend 10 minutes understanding requirements and clarifying scope, 20 minutes designing the architecture and data flow, 10 minutes discussing scaling and trade-offs, and 5 minutes summarizing.

Emphasize security and access control for sensitive configuration data.

Explain how caching improves latency and reduces load on central store.

Discuss dynamic configuration updates and how notifications keep services in sync.

Highlight versioning and rollback to ensure safe configuration changes.

Address scalability challenges and how distributed components handle load.

Mention audit logging for compliance and troubleshooting.