Dynamic configuration updates in Microservices - System Design Guide

Problem Statement

When configuration changes require service restarts, the system experiences downtime or degraded performance. This leads to slow reaction to critical updates like feature toggles or security patches, causing poor user experience and operational risks.

Solution

Dynamic configuration updates allow services to change their settings at runtime without restarting. Services subscribe to a central configuration service or watch configuration sources, applying changes immediately to adapt behavior seamlessly.

Architecture

Configuration

Storage

→Configuration Service

↓

Microservice B

Microservice B

This diagram shows a central configuration storage pushing updates via a configuration service to multiple microservices, which dynamically load new settings without restarting.

Trade-offs

✓ Pros

→

Enables immediate application of critical configuration changes without downtime.

→

Improves operational agility by decoupling configuration from deployment cycles.

→

Reduces risk of errors from manual restarts or stale configurations.

✗ Cons

→

Adds complexity in managing configuration consistency and update propagation.

→

Requires careful design to handle partial failures or rollback of bad configs.

→

May increase resource usage due to continuous monitoring or polling.

Use when your system has frequent configuration changes impacting behavior, especially in microservices with high availability requirements and when downtime for restarts is unacceptable.

Avoid if configuration rarely changes or if your system can tolerate restarts without impacting users, typically in small-scale or batch-processing systems.

Real World Examples

Netflix

Netflix uses dynamic configuration updates to toggle features and adjust streaming parameters in real-time without interrupting user sessions.

Uber

Uber applies dynamic config updates to adjust surge pricing algorithms and routing logic instantly across its microservices.

Amazon

Amazon uses dynamic configuration to manage feature flags and service thresholds dynamically, enabling rapid experimentation and scaling.

Code Example

The before code loads configuration once at startup, so changes require restart. The after code subscribes to a configuration service that pushes updates. The service updates its config dynamically and changes behavior immediately without restart.

Microservices

### Before: Static config loaded once at startup
class Service:
    def __init__(self):
        self.config = load_config()

    def run(self):
        if self.config['feature_enabled']:
            print('Feature ON')
        else:
            print('Feature OFF')


### After: Dynamic config with update callback
import threading
import time

class ConfigService:
    def __init__(self):
        self.config = {'feature_enabled': False}
        self.subscribers = []

    def subscribe(self, callback):
        self.subscribers.append(callback)

    def update_config(self, new_config):
        self.config = new_config
        for cb in self.subscribers:
            cb(self.config)

class Service:
    def __init__(self, config_service):
        self.config = {}
        config_service.subscribe(self.on_config_update)

    def on_config_update(self, new_config):
        self.config = new_config
        print(f'Config updated: {self.config}')

    def run(self):
        if self.config.get('feature_enabled'):
            print('Feature ON')
        else:
            print('Feature OFF')

# Simulate config changes
config_service = ConfigService()
service = Service(config_service)

# Initial run
service.run()

# Update config dynamically
config_service.update_config({'feature_enabled': True})

# Run again with updated config
service.run()

Alternatives

Static configuration with redeployment

Requires service restart or redeployment to apply configuration changes.

Use when: Choose when configuration changes are rare and system downtime is acceptable.

Feature flags with external toggle service

Focuses on toggling features dynamically but may not cover all configuration types.

Use when: Choose when you need fine-grained control over feature enablement rather than full config updates.

Summary

Dynamic configuration updates let services change settings at runtime without restarting.

This improves uptime and agility by applying critical changes immediately.

It requires careful design to handle consistency, failures, and resource use.

Practice

1. What is the main benefit of using dynamic configuration updates in microservices?

easy

A. Encrypt all service communication by default

B. Change service settings without restarting or downtime

C. Reduce the size of service containers

D. Increase the number of microservices automatically

Solution

1. Step 1: Understand dynamic configuration updates

   Dynamic configuration updates allow changing settings while the service is running.

2. Step 2: Identify the main benefit

   This avoids restarting services, preventing downtime and improving flexibility.

3. Final Answer:

   Change service settings without restarting or downtime -> Option B

4. Quick Check:

   Dynamic config = no downtime updates [OK]

Hint: Dynamic config means no restart needed for changes [OK]

Common Mistakes:

• Confusing dynamic config with scaling services
• Thinking it reduces container size
• Assuming it encrypts communication automatically

2. Which of the following is a common method for microservices to receive dynamic configuration updates?

easy

A. Polling a central configuration server periodically

B. Hardcoding config values in source code

C. Restarting the service every hour

D. Using static environment variables only

Solution

1. Step 1: Review methods for dynamic config updates

   Common methods include polling or push from a central config server.

2. Step 2: Identify the correct method

   Polling a config server lets services fetch updates regularly without restart.

3. Final Answer:

   Polling a central configuration server periodically -> Option A

4. Quick Check:

   Polling config server = dynamic updates [OK]

Hint: Dynamic config needs active update fetching, not static values [OK]

Common Mistakes:

• Thinking hardcoded values can update dynamically
• Assuming restart is needed for updates
• Believing static env vars update automatically

3. Consider this pseudocode for a microservice fetching config updates:

config = load_config()
while True:
    if config_server.has_update():
        config = config_server.get_update()
    sleep(10)
    print(config['feature_flag'])

What will this code do when the feature_flag changes on the config server?

medium

A. Print the old feature_flag value forever

B. Crash because config_server.has_update() is undefined

C. Print nothing because of missing initial config

D. Update and print the new feature_flag value every 10 seconds

Solution

1. Step 1: Analyze the update check loop

   The code checks if the config server has an update, then fetches it.

2. Step 2: Understand the print behavior

   Every 10 seconds it prints the current feature_flag from the updated config.

3. Final Answer:

   Update and print the new feature_flag value every 10 seconds -> Option D

4. Quick Check:

   Polling loop updates config = prints new value [OK]

Hint: Polling loop updates config then prints latest value [OK]

Common Mistakes:

• Assuming config never updates after initial load
• Thinking code crashes due to undefined methods (assumed defined)
• Ignoring the sleep and print inside the loop

4. A microservice uses a push-based config update system but sometimes misses updates. What is a likely cause?

medium

A. The service does not acknowledge receipt of updates

B. The service polls the config server too frequently

C. The service restarts after every update

D. The config server uses static environment variables

Solution

1. Step 1: Understand push-based config update mechanism

   Push systems send updates and expect acknowledgments to confirm delivery.

2. Step 2: Identify why updates are missed

   If the service does not acknowledge, the server may not resend or confirm updates.

3. Final Answer:

   The service does not acknowledge receipt of updates -> Option A

4. Quick Check:

   Missing ack = missed updates in push system [OK]

Hint: Push updates need acknowledgments to avoid missing data [OK]

Common Mistakes:

• Confusing push with polling frequency issues
• Assuming restarts cause missed updates
• Thinking static env vars relate to push update failures

5. You design a microservices system with dynamic config updates using a central config server. To ensure minimal latency and high availability, which approach is best?

hard

A. Embed config in each service and restart services on config change

B. Use a single config server with clients polling every second

C. Deploy multiple config server replicas with push notifications and local caching

D. Use static config files updated manually on each service host

Solution

1. Step 1: Consider latency and availability needs

   Multiple replicas reduce single points of failure and improve response times.

2. Step 2: Evaluate update delivery methods

   Push notifications with local caching reduce latency and avoid constant polling.

3. Final Answer:

   Deploy multiple config server replicas with push notifications and local caching -> Option C

4. Quick Check:

   Replicas + push + cache = low latency & high availability [OK]

Hint: Replicas + push + cache = best for latency and availability [OK]

Common Mistakes:

• Relying on single server causes downtime
• Polling every second wastes resources
• Restarting services causes downtime
• Manual static updates are error-prone and slow