Dynamic configuration updates in Microservices - System Design Guide

Problem Statement

When configuration changes require service restarts, the system experiences downtime or degraded performance. This leads to slow reaction to critical updates like feature toggles or security patches, causing poor user experience and operational risks.

Solution

Dynamic configuration updates allow services to change their settings at runtime without restarting. Services subscribe to a central configuration service or watch configuration sources, applying changes immediately to adapt behavior seamlessly.

Architecture

Configuration

Storage

→Configuration Service

↓

Microservice B

Microservice B

This diagram shows a central configuration storage pushing updates via a configuration service to multiple microservices, which dynamically load new settings without restarting.

Trade-offs

✓ Pros

→

Enables immediate application of critical configuration changes without downtime.

→

Improves operational agility by decoupling configuration from deployment cycles.

→

Reduces risk of errors from manual restarts or stale configurations.

✗ Cons

→

Adds complexity in managing configuration consistency and update propagation.

→

Requires careful design to handle partial failures or rollback of bad configs.

→

May increase resource usage due to continuous monitoring or polling.

Use when your system has frequent configuration changes impacting behavior, especially in microservices with high availability requirements and when downtime for restarts is unacceptable.

Avoid if configuration rarely changes or if your system can tolerate restarts without impacting users, typically in small-scale or batch-processing systems.

Real World Examples

Netflix

Netflix uses dynamic configuration updates to toggle features and adjust streaming parameters in real-time without interrupting user sessions.

Uber

Uber applies dynamic config updates to adjust surge pricing algorithms and routing logic instantly across its microservices.

Amazon

Amazon uses dynamic configuration to manage feature flags and service thresholds dynamically, enabling rapid experimentation and scaling.

Code Example

The before code loads configuration once at startup, so changes require restart. The after code subscribes to a configuration service that pushes updates. The service updates its config dynamically and changes behavior immediately without restart.

Microservices

### Before: Static config loaded once at startup
class Service:
    def __init__(self):
        self.config = load_config()

    def run(self):
        if self.config['feature_enabled']:
            print('Feature ON')
        else:
            print('Feature OFF')


### After: Dynamic config with update callback
import threading
import time

class ConfigService:
    def __init__(self):
        self.config = {'feature_enabled': False}
        self.subscribers = []

    def subscribe(self, callback):
        self.subscribers.append(callback)

    def update_config(self, new_config):
        self.config = new_config
        for cb in self.subscribers:
            cb(self.config)

class Service:
    def __init__(self, config_service):
        self.config = {}
        config_service.subscribe(self.on_config_update)

    def on_config_update(self, new_config):
        self.config = new_config
        print(f'Config updated: {self.config}')

    def run(self):
        if self.config.get('feature_enabled'):
            print('Feature ON')
        else:
            print('Feature OFF')

# Simulate config changes
config_service = ConfigService()
service = Service(config_service)

# Initial run
service.run()

# Update config dynamically
config_service.update_config({'feature_enabled': True})

# Run again with updated config
service.run()

Alternatives

Static configuration with redeployment

Requires service restart or redeployment to apply configuration changes.

Use when: Choose when configuration changes are rare and system downtime is acceptable.

Feature flags with external toggle service

Focuses on toggling features dynamically but may not cover all configuration types.

Use when: Choose when you need fine-grained control over feature enablement rather than full config updates.

Summary

Dynamic configuration updates let services change settings at runtime without restarting.

This improves uptime and agility by applying critical changes immediately.

It requires careful design to handle consistency, failures, and resource use.