Feature toggles in Microservices - System Design Exercise

Design: Feature Toggle System for Microservices

Design the feature toggle management system and its integration with microservices. Out of scope: detailed UI design for management console, specific microservice business logic.

Functional Requirements

FR1: Allow enabling or disabling features dynamically without redeploying services

FR2: Support gradual rollout of features to a subset of users

FR3: Provide a centralized management interface for toggles

FR4: Ensure low latency toggle checks for microservices

FR5: Support different toggle types: boolean, percentage rollout, user-based targeting

FR6: Allow toggles to be updated in real-time with minimal impact

FR7: Provide audit logs for toggle changes

FR8: Ensure toggle state consistency across distributed services

Non-Functional Requirements

NFR1: Handle 10,000 toggle checks per second with p99 latency under 20ms

NFR2: 99.9% uptime for toggle service

NFR3: Support up to 1000 concurrent toggle updates per minute

NFR4: Microservices must not be tightly coupled to toggle service availability

NFR5: Toggle data size should be optimized for fast retrieval

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

❓ Question 6

Key Components

Central toggle configuration service with API

Toggle storage database (fast key-value store)

Cache layer for low latency reads

Management UI for toggle control

Audit logging system

SDK or client library for microservices to query toggles

Message queue or pub/sub for toggle update notifications

Design Patterns

Cache-aside pattern for toggle caching

Publish-subscribe for real-time toggle updates

Circuit breaker pattern to handle toggle service unavailability

Feature flag pattern for conditional code execution

Canary release and gradual rollout strategies

Reference Architecture

                +---------------------+
                |  Management Console |
                +----------+----------+
                           |
                           | REST API
                           v
                +---------------------+       +----------------+
                | Feature Toggle      |<----->| Audit Logging  |
                | Configuration       |       +----------------+
                | Service             |
                +----------+----------+
                           |
               +-----------+------------+
               |                        |
       +-------v-------+        +-------v-------+
       | Toggle Store  |        | Message Queue |
       | (Redis/Etcd)  |        | (Pub/Sub)     |
       +-------+-------+        +-------+-------+
               |                        |
               | Cache-aside            | Notifications
               |                        |
       +-------v-------+        +-------v-------+
       | Microservices |        | Microservices |
       | (Toggle SDK)  |        | (Toggle SDK)  |
       +---------------+        +---------------+

Components

Feature Toggle Configuration Service

Node.js/Java Spring Boot REST API

Central service to manage toggle states and provide toggle data to microservices

Toggle Store

Redis or Etcd

Fast key-value store to hold current toggle states for low latency access

Message Queue

Kafka or RabbitMQ

Publish-subscribe system to notify microservices of toggle updates in real-time

Management Console

React or Angular web app

User interface for product owners to create, update, and monitor feature toggles

Audit Logging System

Elasticsearch or relational DB

Store history of toggle changes for compliance and debugging

Toggle SDK

Lightweight client library in microservice language

Provide microservices with APIs to check toggle states efficiently and handle updates

Request Flow

1. 1. Product owner updates toggle state via Management Console.

2. 2. Console sends update request to Feature Toggle Configuration Service.

3. 3. Configuration Service validates and writes toggle state to Toggle Store.

4. 4. Configuration Service publishes toggle update event to Message Queue.

5. 5. Microservices subscribe to Message Queue and receive toggle update notifications.

6. 6. Microservices update local cache of toggles via Toggle SDK.

7. 7. On feature check, microservice queries local cache for toggle state to decide feature behavior.

8. 8. Audit Logging System records toggle changes asynchronously.

Database Schema

Entities: - FeatureToggle: id (PK), name, type (boolean, percentage, user-targeted), description, created_at, updated_at - ToggleState: toggle_id (FK), value (boolean or percentage), conditions (JSON for user targeting), updated_at - AuditLog: id (PK), toggle_id (FK), user_id, old_value, new_value, timestamp Relationships: - FeatureToggle 1:N ToggleState (history or environment-specific states) - FeatureToggle 1:N AuditLog

Scaling Discussion

Bottlenecks

Toggle Store read latency under heavy load

Message Queue throughput for high frequency toggle updates

Configuration Service CPU and memory under many concurrent requests

Microservice SDK cache synchronization delays

Audit Logging storage growth and query performance

Solutions

Use Redis cluster or Etcd cluster for horizontal scaling of toggle store

Partition Message Queue topics and use consumer groups for parallel processing

Deploy multiple instances of Configuration Service behind load balancer

Implement local cache with TTL and fallback to store for cache misses

Archive old audit logs and use efficient indexing for queries

Interview Tips

Time: Spend 10 minutes clarifying requirements and constraints, 20 minutes designing architecture and data flow, 10 minutes discussing scaling and trade-offs, 5 minutes summarizing.

Explain importance of low latency toggle checks in microservices

Discuss trade-offs between consistency and availability for toggle updates

Describe caching strategy and real-time update mechanism

Highlight audit and security considerations

Show awareness of scaling challenges and mitigation strategies

Practice

1. What is the main purpose of using feature toggles in microservices?

easy

A. To enable or disable features without changing the code

B. To increase the number of microservices in the system

C. To replace the API Gateway functionality

D. To store user data securely

Solution

1. Step 1: Understand feature toggles concept

   Feature toggles allow turning features on or off dynamically without code deployment.

2. Step 2: Compare options with feature toggles purpose

   Only To enable or disable features without changing the code correctly describes this purpose; others are unrelated.

3. Final Answer:

   To enable or disable features without changing the code -> Option A

4. Quick Check:

   Feature toggles = Enable/disable features [OK]

Hint: Feature toggles control features without code changes [OK]

Common Mistakes:

• Confusing feature toggles with service scaling
• Thinking toggles replace API Gateway
• Assuming toggles store user data

2. Which of the following is the correct way to check a feature toggle named newUI in a microservice code snippet?

easy

A. if featureToggle.isActive('newUI') { /* use new UI */ }

B. if featureToggle['newUI'] == true then { /* use new UI */ }

C. if featureToggle.newUI = true { /* use new UI */ }

D. if (featureToggle.isEnabled('newUI')) { /* use new UI */ }

Solution

1. Step 1: Identify correct syntax for feature toggle check

   Common pattern is calling a method like isEnabled('featureName') returning boolean.

2. Step 2: Evaluate each option's syntax

   if (featureToggle.isEnabled('newUI')) { /* use new UI */ } uses correct method and syntax. if featureToggle['newUI'] == true then { /* use new UI */ } mixes syntax styles incorrectly. if featureToggle.newUI = true { /* use new UI */ } uses assignment instead of comparison. if featureToggle.isActive('newUI') { /* use new UI */ } uses a wrong method name.

3. Final Answer:

   if (featureToggle.isEnabled('newUI')) { /* use new UI */ } -> Option D

4. Quick Check:

   Correct method call = if (featureToggle.isEnabled('newUI')) { /* use new UI */ } [OK]

Hint: Look for method isEnabled with feature name string [OK]

Common Mistakes:

• Using assignment '=' instead of comparison
• Mixing syntax from different languages
• Using incorrect method names like isActive

3. Consider this pseudocode for a microservice using feature toggles:

if (featureToggle.isEnabled('betaFeature')) {
  return 'Beta feature active';
} else {
  return 'Beta feature inactive';
}

If the toggle betaFeature is OFF, what will be the output?

medium

A. 'Beta feature active'

B. Error: toggle not found

C. 'Beta feature inactive'

D. No output

Solution

1. Step 1: Understand toggle state effect on code flow

   If betaFeature is OFF, isEnabled returns false, so else branch runs.

2. Step 2: Identify output from else branch

   Else branch returns 'Beta feature inactive'.

3. Final Answer:

   'Beta feature inactive' -> Option C

4. Quick Check:

   Toggle OFF = else output [OK]

Hint: Toggle OFF triggers else branch output [OK]

Common Mistakes:

• Assuming toggle OFF triggers if branch
• Expecting error when toggle is off
• Ignoring else branch output

4. A developer wrote this code snippet to check a feature toggle but it always activates the feature regardless of toggle state:

if (featureToggle.isEnabled = true) {
  enableFeature();
} else {
  disableFeature();
}

What is the main error causing this behavior?

medium

A. Feature toggle name is incorrect

B. Using assignment '=' instead of comparison '==' in the if condition

C. Missing parentheses around the condition

D. Calling the wrong method name for toggle check

Solution

1. Step 1: Analyze the if condition syntax

   The condition uses assignment '=' which sets isEnabled to true, always true.

2. Step 2: Identify correct comparison operator

   It should use '==' or a method call to compare, not assignment.

3. Final Answer:

   Using assignment '=' instead of comparison '==' in the if condition -> Option B

4. Quick Check:

   Assignment in if condition causes always true [OK]

Hint: Check if condition uses '==' not '=' [OK]

Common Mistakes:

• Confusing assignment '=' with equality '=='
• Assuming method name is wrong without checking syntax
• Ignoring parentheses importance

5. In a microservices system, you want to safely roll out a new payment feature using feature toggles. Which design approach best supports gradual rollout and quick rollback?

hard

A. Use a centralized feature toggle service with API Gateway to control toggle states dynamically

B. Hardcode toggle values in each microservice and redeploy to change them

C. Use environment variables set at deployment time to control toggles

D. Deploy separate microservices for old and new features without toggles

Solution

1. Step 1: Identify requirements for gradual rollout and rollback

   We need dynamic control over feature toggles without redeploying services.

2. Step 2: Evaluate design options

   Use a centralized feature toggle service with API Gateway to control toggle states dynamically uses centralized toggle service and API Gateway for dynamic control, ideal for gradual rollout and rollback. Options A and B require redeployment or static config. Deploy separate microservices for old and new features without toggles lacks toggle control.

3. Final Answer:

   Use a centralized feature toggle service with API Gateway to control toggle states dynamically -> Option A

4. Quick Check:

   Central toggle service + API Gateway = safe rollout [OK]

Hint: Central toggle service enables dynamic control [OK]

Common Mistakes:

• Using static toggles requiring redeployment
• Ignoring API Gateway role in toggle management
• Deploying separate services instead of toggling