Dynamic configuration updates in Microservices - System Design Exercise

Design: Dynamic Configuration Update System for Microservices

Design covers configuration storage, distribution, update propagation, and security. Does not cover microservice internal logic or deployment pipelines.

Functional Requirements

FR1: Allow microservices to fetch configuration settings at startup

FR2: Support real-time updates to configuration without restarting services

FR3: Ensure consistency of configuration across all instances of a microservice

FR4: Provide rollback capability to previous configuration versions

FR5: Secure access to configuration data

FR6: Support high availability and low latency for configuration fetch requests

Non-Functional Requirements

NFR1: Handle up to 10,000 microservice instances concurrently

NFR2: Configuration update propagation latency under 5 seconds

NFR3: System availability of 99.9% uptime

NFR4: Configuration data size per microservice limited to 1MB

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

❓ Question 6

Key Components

Central configuration store (e.g., distributed key-value store)

Configuration distribution mechanism (push via pub/sub or pull via polling)

Cache layer in microservices for local config access

Versioning and rollback management

Authentication and authorization for config access

Monitoring and alerting for config update failures

Design Patterns

Publish-Subscribe for update notifications

Cache-Aside pattern for local config caching

Leader election for update coordination

Circuit breaker for config fetch failures

Immutable configuration versions

Reference Architecture

                +---------------------+
                |  Configuration Store |
                |  (e.g., etcd, Consul) |
                +----------+----------+
                           |
           +---------------+---------------+
           |                               |
   +-------v-------+               +-------v-------+
   | Config Update |               | Config Update |
   | Publisher     |               | Publisher     |
   +-------+-------+               +-------+-------+
           |                               |
           | Pub/Sub (e.g., Kafka, NATS)  |
           +---------------+---------------+
                           |
                 +---------v---------+
                 | Microservice A     |
                 | (Config Cache +   |
                 |  Config Listener)  |
                 +---------+---------+
                           |
                 +---------v---------+
                 | Microservice B     |
                 | (Config Cache +   |
                 |  Config Listener)  |
                 +-------------------+

Components

Configuration Store

etcd or Consul

Stores all configuration data centrally with versioning and supports watch APIs for change notifications.

Config Update Publisher

Kafka or NATS

Publishes configuration change events to notify microservices about updates.

Microservice Config Listener

Client library in microservices

Listens to update events and refreshes local cached configuration without restart.

Local Config Cache

In-memory cache (e.g., local hashmap)

Provides fast access to configuration data within microservice instances.

Authentication & Authorization

OAuth2 or mTLS

Secures access to configuration data and update channels.

Request Flow

1. 1. Microservice instance starts and fetches initial configuration from Configuration Store.

2. 2. Microservice caches configuration locally for fast access.

3. 3. Configuration admin updates config in Configuration Store with a new version.

4. 4. Configuration Store triggers an event to Config Update Publisher.

5. 5. Config Update Publisher broadcasts update event via Pub/Sub system.

6. 6. Microservice Config Listener receives update event and fetches new config version from Configuration Store.

7. 7. Microservice updates local cache with new configuration without restarting.

8. 8. If needed, rollback can be triggered by updating Configuration Store to previous version.

Database Schema

Entities: - ConfigurationEntry: id (PK), service_name, version, config_data (JSON), created_at - ConfigurationVersionHistory: id (PK), configuration_entry_id (FK), version, config_data, created_at Relationships: - One ConfigurationEntry per microservice - Multiple ConfigurationVersionHistory entries per ConfigurationEntry for versioning and rollback

Scaling Discussion

Bottlenecks

Configuration Store read/write throughput under heavy update load

Pub/Sub system overwhelmed by large number of update events

Microservice instances overwhelmed by frequent config fetches

Network latency causing delayed config propagation

Security overhead impacting performance

Solutions

Use distributed Configuration Store cluster with sharding and replication

Partition Pub/Sub topics by service or region to reduce load

Implement exponential backoff and batching for config fetches in microservices

Deploy edge caches or CDN-like proxies near microservices

Use lightweight authentication tokens and optimize TLS handshakes

Interview Tips

Time: Spend 10 minutes clarifying requirements and constraints, 20 minutes designing architecture and data flow, 10 minutes discussing scaling and trade-offs, 5 minutes summarizing and answering questions.

Emphasize importance of real-time updates without service restarts

Discuss trade-offs between push vs pull update mechanisms

Highlight security considerations for configuration data

Explain versioning and rollback strategies

Address scalability challenges and mitigation techniques