Dynamic configuration updates in Microservices - System Design Exercise

Design: Dynamic Configuration Update System for Microservices

Design covers configuration storage, distribution, update propagation, and security. Does not cover microservice internal logic or deployment pipelines.

Functional Requirements

FR1: Allow microservices to fetch configuration settings at startup

FR2: Support real-time updates to configuration without restarting services

FR3: Ensure consistency of configuration across all instances of a microservice

FR4: Provide rollback capability to previous configuration versions

FR5: Secure access to configuration data

FR6: Support high availability and low latency for configuration fetch requests

Non-Functional Requirements

NFR1: Handle up to 10,000 microservice instances concurrently

NFR2: Configuration update propagation latency under 5 seconds

NFR3: System availability of 99.9% uptime

NFR4: Configuration data size per microservice limited to 1MB

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

❓ Question 6

Key Components

Central configuration store (e.g., distributed key-value store)

Configuration distribution mechanism (push via pub/sub or pull via polling)

Cache layer in microservices for local config access

Versioning and rollback management

Authentication and authorization for config access

Monitoring and alerting for config update failures

Design Patterns

Publish-Subscribe for update notifications

Cache-Aside pattern for local config caching

Leader election for update coordination

Circuit breaker for config fetch failures

Immutable configuration versions

Reference Architecture

                +---------------------+
                |  Configuration Store |
                |  (e.g., etcd, Consul) |
                +----------+----------+
                           |
           +---------------+---------------+
           |                               |
   +-------v-------+               +-------v-------+
   | Config Update |               | Config Update |
   | Publisher     |               | Publisher     |
   +-------+-------+               +-------+-------+
           |                               |
           | Pub/Sub (e.g., Kafka, NATS)  |
           +---------------+---------------+
                           |
                 +---------v---------+
                 | Microservice A     |
                 | (Config Cache +   |
                 |  Config Listener)  |
                 +---------+---------+
                           |
                 +---------v---------+
                 | Microservice B     |
                 | (Config Cache +   |
                 |  Config Listener)  |
                 +-------------------+

Components

Configuration Store

etcd or Consul

Stores all configuration data centrally with versioning and supports watch APIs for change notifications.

Config Update Publisher

Kafka or NATS

Publishes configuration change events to notify microservices about updates.

Microservice Config Listener

Client library in microservices

Listens to update events and refreshes local cached configuration without restart.

Local Config Cache

In-memory cache (e.g., local hashmap)

Provides fast access to configuration data within microservice instances.

Authentication & Authorization

OAuth2 or mTLS

Secures access to configuration data and update channels.

Request Flow

1. 1. Microservice instance starts and fetches initial configuration from Configuration Store.

2. 2. Microservice caches configuration locally for fast access.

3. 3. Configuration admin updates config in Configuration Store with a new version.

4. 4. Configuration Store triggers an event to Config Update Publisher.

5. 5. Config Update Publisher broadcasts update event via Pub/Sub system.

6. 6. Microservice Config Listener receives update event and fetches new config version from Configuration Store.

7. 7. Microservice updates local cache with new configuration without restarting.

8. 8. If needed, rollback can be triggered by updating Configuration Store to previous version.

Database Schema

Entities: - ConfigurationEntry: id (PK), service_name, version, config_data (JSON), created_at - ConfigurationVersionHistory: id (PK), configuration_entry_id (FK), version, config_data, created_at Relationships: - One ConfigurationEntry per microservice - Multiple ConfigurationVersionHistory entries per ConfigurationEntry for versioning and rollback

Scaling Discussion

Bottlenecks

Configuration Store read/write throughput under heavy update load

Pub/Sub system overwhelmed by large number of update events

Microservice instances overwhelmed by frequent config fetches

Network latency causing delayed config propagation

Security overhead impacting performance

Solutions

Use distributed Configuration Store cluster with sharding and replication

Partition Pub/Sub topics by service or region to reduce load

Implement exponential backoff and batching for config fetches in microservices

Deploy edge caches or CDN-like proxies near microservices

Use lightweight authentication tokens and optimize TLS handshakes

Interview Tips

Time: Spend 10 minutes clarifying requirements and constraints, 20 minutes designing architecture and data flow, 10 minutes discussing scaling and trade-offs, 5 minutes summarizing and answering questions.

Emphasize importance of real-time updates without service restarts

Discuss trade-offs between push vs pull update mechanisms

Highlight security considerations for configuration data

Explain versioning and rollback strategies

Address scalability challenges and mitigation techniques

Practice

1. What is the main benefit of using dynamic configuration updates in microservices?

easy

A. Encrypt all service communication by default

B. Change service settings without restarting or downtime

C. Reduce the size of service containers

D. Increase the number of microservices automatically

Solution

1. Step 1: Understand dynamic configuration updates

   Dynamic configuration updates allow changing settings while the service is running.

2. Step 2: Identify the main benefit

   This avoids restarting services, preventing downtime and improving flexibility.

3. Final Answer:

   Change service settings without restarting or downtime -> Option B

4. Quick Check:

   Dynamic config = no downtime updates [OK]

Hint: Dynamic config means no restart needed for changes [OK]

Common Mistakes:

• Confusing dynamic config with scaling services
• Thinking it reduces container size
• Assuming it encrypts communication automatically

2. Which of the following is a common method for microservices to receive dynamic configuration updates?

easy

A. Polling a central configuration server periodically

B. Hardcoding config values in source code

C. Restarting the service every hour

D. Using static environment variables only

Solution

1. Step 1: Review methods for dynamic config updates

   Common methods include polling or push from a central config server.

2. Step 2: Identify the correct method

   Polling a config server lets services fetch updates regularly without restart.

3. Final Answer:

   Polling a central configuration server periodically -> Option A

4. Quick Check:

   Polling config server = dynamic updates [OK]

Hint: Dynamic config needs active update fetching, not static values [OK]

Common Mistakes:

• Thinking hardcoded values can update dynamically
• Assuming restart is needed for updates
• Believing static env vars update automatically

3. Consider this pseudocode for a microservice fetching config updates:

config = load_config()
while True:
    if config_server.has_update():
        config = config_server.get_update()
    sleep(10)
    print(config['feature_flag'])

What will this code do when the feature_flag changes on the config server?

medium

A. Print the old feature_flag value forever

B. Crash because config_server.has_update() is undefined

C. Print nothing because of missing initial config

D. Update and print the new feature_flag value every 10 seconds

Solution

1. Step 1: Analyze the update check loop

   The code checks if the config server has an update, then fetches it.

2. Step 2: Understand the print behavior

   Every 10 seconds it prints the current feature_flag from the updated config.

3. Final Answer:

   Update and print the new feature_flag value every 10 seconds -> Option D

4. Quick Check:

   Polling loop updates config = prints new value [OK]

Hint: Polling loop updates config then prints latest value [OK]

Common Mistakes:

• Assuming config never updates after initial load
• Thinking code crashes due to undefined methods (assumed defined)
• Ignoring the sleep and print inside the loop

4. A microservice uses a push-based config update system but sometimes misses updates. What is a likely cause?

medium

A. The service does not acknowledge receipt of updates

B. The service polls the config server too frequently

C. The service restarts after every update

D. The config server uses static environment variables

Solution

1. Step 1: Understand push-based config update mechanism

   Push systems send updates and expect acknowledgments to confirm delivery.

2. Step 2: Identify why updates are missed

   If the service does not acknowledge, the server may not resend or confirm updates.

3. Final Answer:

   The service does not acknowledge receipt of updates -> Option A

4. Quick Check:

   Missing ack = missed updates in push system [OK]

Hint: Push updates need acknowledgments to avoid missing data [OK]

Common Mistakes:

• Confusing push with polling frequency issues
• Assuming restarts cause missed updates
• Thinking static env vars relate to push update failures

5. You design a microservices system with dynamic config updates using a central config server. To ensure minimal latency and high availability, which approach is best?

hard

A. Embed config in each service and restart services on config change

B. Use a single config server with clients polling every second

C. Deploy multiple config server replicas with push notifications and local caching

D. Use static config files updated manually on each service host

Solution

1. Step 1: Consider latency and availability needs

   Multiple replicas reduce single points of failure and improve response times.

2. Step 2: Evaluate update delivery methods

   Push notifications with local caching reduce latency and avoid constant polling.

3. Final Answer:

   Deploy multiple config server replicas with push notifications and local caching -> Option C

4. Quick Check:

   Replicas + push + cache = low latency & high availability [OK]

Hint: Replicas + push + cache = best for latency and availability [OK]

Common Mistakes:

• Relying on single server causes downtime
• Polling every second wastes resources
• Restarting services causes downtime
• Manual static updates are error-prone and slow