Feature flags in Microservices - System Design Exercise

Design: Feature Flag Management System

Design the feature flag service, API, and integration approach with microservices. Out of scope: detailed UI design and specific microservice implementations.

Functional Requirements

FR1: Allow enabling or disabling features dynamically without redeploying services

FR2: Support targeting feature flags to specific user groups or environments

FR3: Provide a dashboard for product managers to control flags

FR4: Ensure low latency flag evaluation in microservices

FR5: Support gradual rollouts (percentage-based) of features

FR6: Audit changes to feature flags for compliance

FR7: Integrate with multiple microservices in different languages

Non-Functional Requirements

NFR1: Handle 100,000 concurrent users evaluating flags

NFR2: API response latency for flag evaluation under 10ms (p99)

NFR3: 99.9% uptime for the feature flag service

NFR4: Support eventual consistency for flag updates within 1 minute

NFR5: Secure access to flag management dashboard

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

❓ Question 6

Key Components

Feature flag storage database

Flag evaluation service or SDKs in microservices

Management dashboard backend and frontend

API gateway or proxy for flag evaluation requests

Audit logging system

Cache layer for fast flag retrieval

Design Patterns

Cache-aside pattern for flag caching

Publish-subscribe for flag update notifications

Circuit breaker for fallback if flag service is down

Role-based access control for dashboard

Canary releases and gradual rollout patterns

Reference Architecture

  +-------------------+       +-------------------+       +-------------------+
  |                   |       |                   |       |                   |
  |  Management       |       |  Feature Flag     |       |  Microservices    |
  |  Dashboard        | <---> |  Service API      | <---> |  (with SDKs)      |
  |  (Web UI + API)   |       |                   |       |                   |
  +-------------------+       +-------------------+       +-------------------+
           |                            |                            |
           |                            |                            |
           |                            v                            |
           |                   +-------------------+               |
           |                   |  Flag Storage DB   |               |
           |                   +-------------------+               |
           |                            |                            |
           |                            v                            |
           |                   +-------------------+               |
           |                   |  Cache Layer      | <-------------+
           |                   +-------------------+               |
           |                            |                            |
           |                            v                            |
           |                   +-------------------+               |
           |                   |  Audit Logging     |               |
           |                   +-------------------+               |
           +--------------------------------------------------------+

Components

Management Dashboard

React + Node.js backend

Allows product managers to create, update, and target feature flags securely

Feature Flag Service API

RESTful API with Node.js or Go

Handles flag CRUD operations and serves flag data to microservices

Flag Storage Database

PostgreSQL

Stores feature flag definitions, targeting rules, and metadata

Cache Layer

Redis

Caches flag data for low latency retrieval by microservices

Audit Logging System

Elasticsearch + Kibana

Records all changes to feature flags for compliance and troubleshooting

Microservice SDKs

Language-specific SDKs (e.g., Java, Python, Node.js)

Embedded in microservices to evaluate flags locally with cache and fallback

Request Flow

1. 1. Product manager logs into the Management Dashboard and creates or updates a feature flag with targeting rules.

2. 2. Dashboard backend validates and sends the flag data to the Feature Flag Service API.

3. 3. Feature Flag Service stores the flag data in the PostgreSQL database and updates the Redis cache.

4. 4. Audit Logging System records the change event.

5. 5. Microservices periodically fetch updated flags from the Feature Flag Service API or subscribe to update notifications.

6. 6. Microservice SDK caches flags locally and evaluates them on incoming user requests with targeting rules.

7. 7. If the cache is stale or missing, SDK fetches fresh flag data from the cache layer or API.

8. 8. Microservice uses the evaluation result to enable or disable features dynamically.

Database Schema

Entities: - FeatureFlag(id PK, name, description, type [boolean, multivariate], created_at, updated_at) - TargetingRule(id PK, feature_flag_id FK, attribute, operator, value, rollout_percentage) - AuditLog(id PK, feature_flag_id FK, user_id, action, timestamp, old_value, new_value) Relationships: - One FeatureFlag has many TargetingRules - AuditLog references FeatureFlag and user who made changes

Scaling Discussion

Bottlenecks

High read load on Feature Flag Service API causing latency spikes

Cache invalidation delays leading to stale flag evaluations

Audit logging volume growing rapidly with frequent flag changes

SDKs fetching flags too frequently increasing network overhead

Solutions

Implement aggressive caching with TTL and cache-aside pattern to reduce API calls

Use publish-subscribe messaging (e.g., Kafka) to push flag updates to SDKs for near real-time cache invalidation

Archive old audit logs and use scalable storage like Elasticsearch clusters

Add local SDK caching with exponential backoff and fallback to default flag values if service is unreachable

Interview Tips

Time: Spend 10 minutes understanding requirements and clarifying scope, 20 minutes designing components and data flow, 10 minutes discussing scaling and trade-offs, 5 minutes summarizing.

Explain why dynamic feature toggling is important for continuous delivery

Discuss trade-offs between consistency and latency in flag evaluation

Highlight caching strategies to meet low latency requirements

Mention security and auditability for compliance

Describe how gradual rollouts reduce risk

Show awareness of SDK design for multiple languages and fallback mechanisms

Practice

1. What is the main purpose of using feature flags in microservices?

easy

A. To increase database storage capacity

B. To enable or disable features without deploying new code

C. To improve network bandwidth

D. To encrypt communication between services

Solution

1. Step 1: Understand feature flags concept

   Feature flags allow toggling features on or off dynamically without changing the codebase.

2. Step 2: Identify the main benefit in microservices

   This helps in testing, gradual rollout, and quick disabling of features without redeployment.

3. Final Answer:

   To enable or disable features without deploying new code -> Option B

4. Quick Check:

   Feature flags = toggle features dynamically [OK]

Hint: Feature flags toggle features without code changes [OK]

Common Mistakes:

• Confusing feature flags with database scaling
• Thinking feature flags improve network speed
• Assuming feature flags handle encryption

2. Which of the following is the correct way to check a feature flag named new_ui_enabled in a microservice?

easy

A. featureFlags.enable('new_ui_enabled')

B. if (featureFlags.check('new_ui_enabled') == false) { /* use new UI */ }

C. if (featureFlags.isEnabled('new_ui_enabled')) { /* use new UI */ }

D. featureFlags.remove('new_ui_enabled')

Solution

1. Step 1: Identify correct method to check flag status

   Checking if a feature flag is enabled usually uses a method like isEnabled returning true or false.

2. Step 2: Analyze options

   if (featureFlags.isEnabled('new_ui_enabled')) { /* use new UI */ } correctly checks if the flag is enabled before using the feature. if (featureFlags.check('new_ui_enabled') == false) { /* use new UI */ } incorrectly uses check and false condition. featureFlags.enable('new_ui_enabled') and featureFlags.remove('new_ui_enabled') modify flags, not check them.

3. Final Answer:

   if (featureFlags.isEnabled('new_ui_enabled')) { /* use new UI */ } -> Option C

4. Quick Check:

   Check flag with isEnabled() = if (featureFlags.isEnabled('new_ui_enabled')) { /* use new UI */ } [OK]

Hint: Use isEnabled() to check feature flags status [OK]

Common Mistakes:

• Using enable() or remove() to check flags
• Checking flag with wrong method or negation
• Confusing flag check with flag update

3. Consider this pseudocode snippet in a microservice:

if (featureFlags.isEnabled('beta_feature')) {
  return 'Beta feature active';
} else {
  return 'Beta feature inactive';
}

What will be the output if the flag beta_feature is set to false?

medium

A. Beta feature inactive

B. Beta feature active

C. Error: flag not found

D. No output

Solution

1. Step 1: Understand flag value effect on condition

   The condition checks if beta_feature is enabled (true). If false, it goes to else branch.

2. Step 2: Determine output when flag is false

   Since the flag is false, the else block executes returning 'Beta feature inactive'.

3. Final Answer:

   Beta feature inactive -> Option A

4. Quick Check:

   Flag false triggers else = Beta feature inactive [OK]

Hint: False flag runs else block output [OK]

Common Mistakes:

• Assuming false flag runs if block
• Expecting error when flag is false
• Ignoring else branch output

4. A developer wrote this code to disable a feature using a feature flag:

if (featureFlags.isEnabled('dark_mode')) {
  disableDarkMode();
}

Why might this code not work as intended?

medium

A. It disables dark mode when the flag is enabled, which is opposite logic

B. The method disableDarkMode() does not exist

C. Feature flags cannot control UI features

D. The flag name should be 'enable_dark_mode'

Solution

1. Step 1: Analyze the condition logic

   The code disables dark mode if the flag is enabled, which is opposite of expected behavior (usually enabled flag means enable feature).

2. Step 2: Identify correct logic for disabling feature

   To disable dark mode when flag is false, the condition should check if flag is disabled or negate the check.

3. Final Answer:

   It disables dark mode when the flag is enabled, which is opposite logic -> Option A

4. Quick Check:

   Enabled flag should enable, not disable [OK]

Hint: Enabled flag usually means feature ON, not OFF [OK]

Common Mistakes:

• Confusing enable and disable logic
• Assuming flag controls only backend features
• Using wrong flag names without checking

5. You want to roll out a new payment feature gradually using feature flags in a microservices system. Which design approach is best to ensure minimal impact and easy rollback?

hard

A. Disable all other microservices during rollout to avoid conflicts

B. Deploy new code with feature always enabled and monitor logs manually

C. Hardcode feature flag values in each microservice and update code to change flags

D. Use a centralized feature flag service with percentage rollout and fallback to old payment service

Solution

1. Step 1: Understand gradual rollout with feature flags

   Gradual rollout means enabling feature for a small percentage of users first, then increasing over time.

2. Step 2: Identify scalable and safe design

   A centralized flag service allows dynamic control and percentage rollout. Fallback ensures quick rollback if issues arise.

3. Step 3: Evaluate other options

   Deploy new code with feature always enabled and monitor logs manually lacks control and rollback ease. Hardcode feature flag values in each microservice and update code to change flags requires code changes for flag updates, not scalable. Disable all other microservices during rollout to avoid conflicts is disruptive and unnecessary.

4. Final Answer:

   Use a centralized feature flag service with percentage rollout and fallback to old payment service -> Option D

5. Quick Check:

   Centralized flags + gradual rollout = Use a centralized feature flag service with percentage rollout and fallback to old payment service [OK]

Hint: Centralized flags + gradual rollout = safe deployment [OK]

Common Mistakes:

• Hardcoding flags causing frequent deployments
• Disabling services unnecessarily during rollout
• Ignoring rollback strategies