Feature flags in Microservices - Scalability & System Analysis

Scalability Analysis - Feature flags

Growth Table: Feature Flags at Different Scales

Users	Requests per Second	Flag Evaluations per Second	Storage Size	Latency Impact	Complexity
100 users	~10-50 RPS	~10-50	KBs (few flags)	Negligible	Simple flag storage in memory or DB
10,000 users	~1,000-5,000 RPS	~1,000-5,000	MBs (hundreds of flags)	Low latency needed	Use caching, distributed config store
1,000,000 users	~100,000 RPS	~100,000	GBs (thousands of flags, segments)	Must be <10ms per eval	Use CDN, caching, distributed flag evaluation service
100,000,000 users	~10,000,000 RPS	~10,000,000	TBs (complex targeting, analytics)	Highly optimized, near real-time	Global distributed system, sharding, edge caching

First Bottleneck

The first bottleneck is the feature flag evaluation service and its data store. As user count and requests grow, the system must evaluate flags quickly for each request. The database or config store can become overwhelmed by read requests or complex targeting rules, causing latency spikes.

Scaling Solutions

• Caching: Use in-memory caches (e.g., Redis, local caches) to store flag data and evaluation results to reduce DB load.
• Horizontal Scaling: Add more instances of the flag evaluation service behind a load balancer to handle more concurrent requests.
• Read Replicas: Use database read replicas to distribute read traffic for flag configurations.
• Sharding: Partition flag data by user segments or regions to reduce data size per node.
• CDN and Edge Caching: Cache flag data closer to users to reduce latency and network load.
• Asynchronous Updates: Push flag changes asynchronously to services to avoid blocking requests.
• Feature Flag Evaluation SDKs: Use client-side evaluation where possible to reduce server load.

Back-of-Envelope Cost Analysis

• At 1M users with 100K RPS, assuming each flag evaluation is 1KB data read, total bandwidth ~100MB/s.
• Storage for flags and targeting rules grows with complexity; expect GBs at million-user scale.
• CPU usage depends on evaluation complexity; caching reduces CPU by avoiding repeated computations.
• Network bandwidth and latency critical; edge caching reduces cross-region traffic.

Interview Tip

Start by explaining what feature flags are and why they matter. Then discuss how load grows with users and requests. Identify the bottleneck clearly (flag evaluation and data store). Propose concrete scaling solutions like caching, horizontal scaling, and sharding. Mention trade-offs like consistency vs latency. Use real numbers to show understanding.

Self Check Question

Your database handles 1000 QPS for flag data reads. Traffic grows 10x to 10,000 QPS. What do you do first?

Answer: Add caching layer (e.g., Redis or in-memory cache) to reduce direct DB reads and improve response time before scaling the database or services.

Key Result

Feature flag evaluation service and its data store become the first bottleneck as user requests grow; caching and horizontal scaling are key to maintain low latency and high throughput.

Practice

1. What is the main purpose of using feature flags in microservices?

easy

A. To increase database storage capacity

B. To enable or disable features without deploying new code

C. To improve network bandwidth

D. To encrypt communication between services

Solution

1. Step 1: Understand feature flags concept

   Feature flags allow toggling features on or off dynamically without changing the codebase.

2. Step 2: Identify the main benefit in microservices

   This helps in testing, gradual rollout, and quick disabling of features without redeployment.

3. Final Answer:

   To enable or disable features without deploying new code -> Option B

4. Quick Check:

   Feature flags = toggle features dynamically [OK]

Hint: Feature flags toggle features without code changes [OK]

Common Mistakes:

• Confusing feature flags with database scaling
• Thinking feature flags improve network speed
• Assuming feature flags handle encryption

2. Which of the following is the correct way to check a feature flag named new_ui_enabled in a microservice?

easy

A. featureFlags.enable('new_ui_enabled')

B. if (featureFlags.check('new_ui_enabled') == false) { /* use new UI */ }

C. if (featureFlags.isEnabled('new_ui_enabled')) { /* use new UI */ }

D. featureFlags.remove('new_ui_enabled')

Solution

1. Step 1: Identify correct method to check flag status

   Checking if a feature flag is enabled usually uses a method like isEnabled returning true or false.

2. Step 2: Analyze options

   if (featureFlags.isEnabled('new_ui_enabled')) { /* use new UI */ } correctly checks if the flag is enabled before using the feature. if (featureFlags.check('new_ui_enabled') == false) { /* use new UI */ } incorrectly uses check and false condition. featureFlags.enable('new_ui_enabled') and featureFlags.remove('new_ui_enabled') modify flags, not check them.

3. Final Answer:

   if (featureFlags.isEnabled('new_ui_enabled')) { /* use new UI */ } -> Option C

4. Quick Check:

   Check flag with isEnabled() = if (featureFlags.isEnabled('new_ui_enabled')) { /* use new UI */ } [OK]

Hint: Use isEnabled() to check feature flags status [OK]

Common Mistakes:

• Using enable() or remove() to check flags
• Checking flag with wrong method or negation
• Confusing flag check with flag update

3. Consider this pseudocode snippet in a microservice:

if (featureFlags.isEnabled('beta_feature')) {
  return 'Beta feature active';
} else {
  return 'Beta feature inactive';
}

What will be the output if the flag beta_feature is set to false?

medium

A. Beta feature inactive

B. Beta feature active

C. Error: flag not found

D. No output

Solution

1. Step 1: Understand flag value effect on condition

   The condition checks if beta_feature is enabled (true). If false, it goes to else branch.

2. Step 2: Determine output when flag is false

   Since the flag is false, the else block executes returning 'Beta feature inactive'.

3. Final Answer:

   Beta feature inactive -> Option A

4. Quick Check:

   Flag false triggers else = Beta feature inactive [OK]

Hint: False flag runs else block output [OK]

Common Mistakes:

• Assuming false flag runs if block
• Expecting error when flag is false
• Ignoring else branch output

4. A developer wrote this code to disable a feature using a feature flag:

if (featureFlags.isEnabled('dark_mode')) {
  disableDarkMode();
}

Why might this code not work as intended?

medium

A. It disables dark mode when the flag is enabled, which is opposite logic

B. The method disableDarkMode() does not exist

C. Feature flags cannot control UI features

D. The flag name should be 'enable_dark_mode'

Solution

1. Step 1: Analyze the condition logic

   The code disables dark mode if the flag is enabled, which is opposite of expected behavior (usually enabled flag means enable feature).

2. Step 2: Identify correct logic for disabling feature

   To disable dark mode when flag is false, the condition should check if flag is disabled or negate the check.

3. Final Answer:

   It disables dark mode when the flag is enabled, which is opposite logic -> Option A

4. Quick Check:

   Enabled flag should enable, not disable [OK]

Hint: Enabled flag usually means feature ON, not OFF [OK]

Common Mistakes:

• Confusing enable and disable logic
• Assuming flag controls only backend features
• Using wrong flag names without checking

5. You want to roll out a new payment feature gradually using feature flags in a microservices system. Which design approach is best to ensure minimal impact and easy rollback?

hard

A. Disable all other microservices during rollout to avoid conflicts

B. Deploy new code with feature always enabled and monitor logs manually

C. Hardcode feature flag values in each microservice and update code to change flags

D. Use a centralized feature flag service with percentage rollout and fallback to old payment service

Solution

1. Step 1: Understand gradual rollout with feature flags

   Gradual rollout means enabling feature for a small percentage of users first, then increasing over time.

2. Step 2: Identify scalable and safe design

   A centralized flag service allows dynamic control and percentage rollout. Fallback ensures quick rollback if issues arise.

3. Step 3: Evaluate other options

   Deploy new code with feature always enabled and monitor logs manually lacks control and rollback ease. Hardcode feature flag values in each microservice and update code to change flags requires code changes for flag updates, not scalable. Disable all other microservices during rollout to avoid conflicts is disruptive and unnecessary.

4. Final Answer:

   Use a centralized feature flag service with percentage rollout and fallback to old payment service -> Option D

5. Quick Check:

   Centralized flags + gradual rollout = Use a centralized feature flag service with percentage rollout and fallback to old payment service [OK]

Hint: Centralized flags + gradual rollout = safe deployment [OK]

Common Mistakes:

• Hardcoding flags causing frequent deployments
• Disabling services unnecessarily during rollout
• Ignoring rollback strategies