Environment configuration in Microservices - System Design Exercise

Design: Microservices Environment Configuration System

In scope: design of the centralized configuration management system for microservices including storage, retrieval, security, and update mechanisms. Out of scope: the internal implementation details of individual microservices.

Functional Requirements

FR1: Provide a centralized way to manage configuration settings for multiple microservices.

FR2: Support different environments such as development, testing, staging, and production.

FR3: Allow secure storage and retrieval of sensitive configuration data like API keys and passwords.

FR4: Enable dynamic configuration updates without restarting services.

FR5: Support versioning and rollback of configuration changes.

FR6: Provide access control to restrict who can view or modify configurations.

FR7: Ensure high availability and low latency for configuration retrieval.

Non-Functional Requirements

NFR1: Must handle configuration for at least 100 microservices.

NFR2: Configuration retrieval latency should be under 50ms at p99.

NFR3: System availability target is 99.9% uptime (less than 8.77 hours downtime per year).

NFR4: Configuration updates should propagate within 30 seconds to all services.

NFR5: Secure storage must comply with industry best practices for secrets management.

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

❓ Question 6

Key Components

Central configuration storage (e.g., distributed key-value store)

Configuration API service for read/write operations

Authentication and authorization service

Cache layer for fast configuration retrieval

Notification or event system for configuration change propagation

Audit logging for configuration changes

Secrets management integration

Design Patterns

Configuration as Code

Feature Flags

Publish-Subscribe for change notifications

Cache Aside pattern for configuration caching

Role-Based Access Control (RBAC)

Circuit Breaker for configuration service resilience

Reference Architecture

                    +-----------------------+
                    |  Configuration Clients |
                    |  (Microservices)       |
                    +-----------+-----------+
                                |
                                | Fetch Config via API
                                v
                    +-----------------------+
                    | Configuration API     |
                    | Service               |
                    +-----------+-----------+
                                |
               +----------------+----------------+
               |                                 |
       +-------v-------+                 +-------v-------+
       | Cache Layer   |                 | Auth & Audit  |
       | (Redis/Etcd)  |                 | Service       |
       +-------+-------+                 +-------+-------+
               |                                 |
               |                                 |
       +-------v-------------------------------v-------+
       |          Central Configuration Store          |
       |       (Distributed KV Store + Secrets)        |
       +-----------------------------------------------+

Change notifications via pub-sub system to clients for dynamic updates.

Components

Configuration API Service

RESTful API with gRPC support

Handles requests from microservices to read and update configuration data securely.

Central Configuration Store

Distributed key-value store (e.g., Consul, Etcd, or ZooKeeper) with secrets management integration

Stores all configuration data including environment-specific and sensitive information.

Cache Layer

In-memory cache (e.g., Redis)

Provides low-latency access to frequently requested configuration data.

Authentication and Authorization Service

OAuth2 / RBAC system

Ensures only authorized users and services can access or modify configurations.

Audit Logging

Centralized logging system (e.g., ELK stack)

Records all configuration changes for compliance and troubleshooting.

Notification System

Message broker (e.g., Kafka, RabbitMQ, or native pub-sub)

Pushes configuration change events to microservices for dynamic updates.

Request Flow

1. 1. Microservice requests configuration from Configuration API Service.

2. 2. API Service checks Cache Layer for requested configuration.

3. 3. If cache miss, API Service fetches configuration from Central Configuration Store.

4. 4. Configuration data is returned to microservice and cached for future requests.

5. 5. When configuration is updated via API Service, changes are stored in Central Store.

6. 6. API Service publishes configuration change event to Notification System.

7. 7. Microservices subscribed to notifications receive update and refresh their local config cache.

8. 8. Authentication and Authorization Service validates all requests to ensure security.

9. 9. Audit Logging records all configuration read and write operations.

Database Schema

Entities: - ConfigurationEntry: id (PK), service_name, environment, key, value, version, is_secret, created_at, updated_at - User: id (PK), username, role - AccessControl: id (PK), user_id (FK), service_name, permission_type (read/write) - AuditLog: id (PK), user_id (FK), action_type (read/write/update), config_entry_id (FK), timestamp Relationships: - User to AccessControl is 1:N - ConfigurationEntry versions tracked by version field - AuditLog references User and ConfigurationEntry for traceability

Scaling Discussion

Bottlenecks

Central Configuration Store can become a read/write bottleneck under high load.

Cache Layer may become stale or overwhelmed with requests if not properly sized.

Notification System may have delays or message loss affecting dynamic updates.

Authentication Service could slow down requests if not scalable.

Audit Logging may generate large volumes of data impacting storage and query performance.

Solutions

Use distributed and highly available key-value stores with replication and sharding.

Implement cache invalidation strategies and scale cache horizontally.

Use reliable message brokers with persistence and retry mechanisms for notifications.

Scale Authentication Service horizontally and use token-based authentication to reduce load.

Archive old audit logs and use efficient indexing for fast queries.

Interview Tips

Time: Spend 10 minutes understanding requirements and clarifying scope, 20 minutes designing the architecture and data flow, 10 minutes discussing scaling and trade-offs, and 5 minutes summarizing.

Emphasize security and access control for sensitive configuration data.

Explain how caching improves latency and reduces load on central store.

Discuss dynamic configuration updates and how notifications keep services in sync.

Highlight versioning and rollback to ensure safe configuration changes.

Address scalability challenges and how distributed components handle load.

Mention audit logging for compliance and troubleshooting.

Practice

1. What is the main purpose of environment configuration in microservices?

easy

A. To write all configuration directly inside the code

B. To hardcode database credentials in the source files

C. To separate settings from code for easier management

D. To avoid using any configuration for faster deployment

Solution

1. Step 1: Understand environment configuration role

   Environment configuration means keeping settings like URLs, credentials, and flags outside the code.

2. Step 2: Identify benefits of separating settings

   This separation allows the same code to run in different environments (dev, test, prod) safely and easily.

3. Final Answer:

   To separate settings from code for easier management -> Option C

4. Quick Check:

   Settings separate from code = B [OK]

Hint: Settings outside code means environment configuration [OK]

Common Mistakes:

• Confusing configuration with code logic
• Hardcoding sensitive data inside source files
• Ignoring environment differences

2. Which of the following is the correct way to access an environment variable named DB_HOST in a microservice?

easy

A. getEnv('DB_HOST')

B. config.get('DB_HOST')

C. env.DB_HOST()

D. process.env.DB_HOST

Solution

1. Step 1: Identify common environment variable access syntax

   In many microservice platforms, environment variables are accessed via process.env.VARIABLE_NAME.

2. Step 2: Match the correct syntax for DB_HOST

   The correct way is process.env.DB_HOST, which reads the variable from the environment.

3. Final Answer:

   process.env.DB_HOST -> Option D

4. Quick Check:

   Environment variables use process.env = A [OK]

Hint: Use process.env.VAR to read environment variables [OK]

Common Mistakes:

• Using function calls instead of direct access
• Confusing config libraries with environment variables
• Using incorrect object names like env or getEnv

3. Given this code snippet in a microservice:

const port = process.env.PORT || 3000;
console.log(port);

If the environment variable PORT is set to 8080, what will be printed?

medium

A. 8080

B. undefined

C. null

D. 3000

Solution

1. Step 1: Understand the fallback logic

   The code uses process.env.PORT || 3000, meaning if PORT is set, use it; otherwise, use 3000.

2. Step 2: Apply the given environment variable value

   Since PORT is set to 8080, the variable port will be 8080.

3. Final Answer:

   8080 -> Option A

4. Quick Check:

   PORT set to 8080 means output 8080 [OK]

Hint: If env var exists, use it; else fallback value [OK]

Common Mistakes:

• Assuming fallback value always prints
• Confusing undefined with fallback
• Ignoring environment variable presence

4. A microservice fails to read environment variables after deployment. Which is the most likely cause?

medium

A. Environment variables were not set in the deployment environment

B. The code uses process.env to read variables

C. The microservice has no network connection

D. The source code has syntax errors unrelated to config

Solution

1. Step 1: Identify common reasons for missing environment variables

   If the microservice cannot read environment variables, often they were not set or loaded properly in the deployment environment.

2. Step 2: Eliminate other options

   Using process.env is correct syntax; network issues or unrelated syntax errors won't cause missing env vars.

3. Final Answer:

   Environment variables were not set in the deployment environment -> Option A

4. Quick Check:

   Missing env vars usually mean not set in environment [OK]

Hint: Check if env vars are set in deployment environment [OK]

Common Mistakes:

• Blaming code syntax for missing env vars
• Ignoring deployment environment setup
• Assuming network issues cause env var problems

5. You want to deploy the same microservice code to development, staging, and production environments. Which approach best uses environment configuration to handle different database URLs safely?

hard

A. Hardcode all database URLs in the source code and comment/uncomment as needed

B. Use environment variables to set the database URL for each environment separately

C. Store all database URLs in a single config file checked into source control

D. Use a random database URL generated at runtime

Solution

1. Step 1: Understand the need for environment-specific settings

   Each environment (dev, staging, prod) has different database URLs for safety and isolation.

2. Step 2: Choose a method that separates config from code and supports environment differences

   Using environment variables allows setting different URLs without changing code or risking secrets in source control.

3. Step 3: Evaluate other options

   Hardcoding or single config files risk errors and security issues; random URLs are impractical.

4. Final Answer:

   Use environment variables to set the database URL for each environment separately -> Option B

5. Quick Check:

   Env vars per environment = safe config management [OK]

Hint: Use env vars for environment-specific secrets and URLs [OK]

Common Mistakes:

• Hardcoding secrets in code
• Checking sensitive config into source control
• Using random or unsafe config values