Bird
Raised Fist0

A/B Testing Platform for ML Models - Experiment Management

SYSTEM-DESIGNhardInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
⚖️
Feature Flagging vs Shadow Testing

In an ML model deployment scenario where you need to evaluate model performance with minimal user impact, would you use feature flagging or shadow testing? Justify your choice.

📊
Comparison
DimensionFeature FlaggingShadow Testing
Control over exposureHighMedium
Infrastructure costMediumHigh
Latency impactMediumLow
User feedback integrationHighLow
ScalabilityHighMedium
✅
Verdict
Default to feature flagging for scenarios requiring precise control and rollback capabilities; switch to shadow testing when infrastructure allows and real-world validation is needed without user impact.
💡
Key Insight
Feature flagging excels in controlled rollouts, while shadow testing is ideal for validation without user impact.
⚠️
Interview Trap
❌ Wrong: Always use feature flagging because it allows for controlled rollouts.
This ignores the infrastructure cost and latency impact of managing multiple flags.
✅ Consider the trade-offs between control and infrastructure cost, and choose based on the specific constraints of your deployment scenario.
Feature Flagging
"Allows controlled rollout and easy rollback of features to subsets of users."
Weaknesses
  • Requires careful user segmentation to avoid biased results.
  • Potentially high latency if not implemented efficiently.
  • Complexity in managing multiple flags across environments.
Ideal When
  • When n > 100K users and you need precise control over exposure.
  • When latency < 100ms is acceptable for feature toggling.
  • When you have a dedicated team to manage feature flags.
Facebook uses feature flagging to roll out new features to a small percentage of users before a full release.
Shadow Testing
"Allows testing of new models in production without affecting user experience."
Weaknesses
  • Requires additional infrastructure to handle duplicate requests.
  • Can be costly in terms of compute resources.
  • Does not provide direct user feedback on new features.
Ideal When
  • When you have excess compute capacity to handle shadow requests.
  • When latency < 50ms is critical and cannot be compromised.
  • When n > 1M requests per day and you need real-world data for validation.
Netflix uses shadow testing to validate new recommendation algorithms without impacting user experience.
🐍
Feature Flagging - Example
▼ Show
Python
def feature_flag(user_id, feature_flags):
    if feature_flags.get('new_model', False):
        return 'new_model'
    return 'old_model'
🐍
Shadow Testing - Example
▼ Show
Python
def shadow_test(request, model_a, model_b):
    response_a = model_a.predict(request)
    response_b = model_b.predict(request)
    log_comparison(response_a, response_b)
    return response_a
✅
Use Feature Flagging When
  • •When n > 100K users and precise control over exposure is needed.
  • •When you have a dedicated team to manage feature flags.
  • •When rollback capability is critical.
✅
Use Shadow Testing When
  • •When you have excess compute capacity to handle shadow requests.
  • •When latency < 50ms is critical and cannot be compromised.
  • •When you need real-world data for validation without user impact.
↳
Nuances
  • Feature flagging can introduce latency if not optimized, impacting user experience.
  • Shadow testing requires careful logging and analysis to derive actionable insights.
  • Feature flagging is more suitable for gradual rollouts, while shadow testing is better for validation.

Practice

(1/5)
1. In the context of anomaly detection, which of the following best describes the primary advantage of using an Isolation Forest over an Autoencoder?
easy
A. Isolation Forests require labeled data for training.
B. Isolation Forests are more suitable for detecting anomalies in time-series data.
C. Isolation Forests can handle high-dimensional data better than Autoencoders.
D. Isolation Forests are more interpretable and require less feature engineering.

Solution

  1. Step 1: Understand the strengths of Isolation Forests.

    Isolation Forests are known for their simplicity and interpretability, as they isolate anomalies by randomly selecting a feature and then randomly selecting a split value between the maximum and minimum values of that feature.

  2. Step 2: Eliminate wrong options.

    Isolation Forests require labeled data for training. is incorrect because Isolation Forests do not require labeled data. Isolation Forests can handle high-dimensional data better than Autoencoders. is incorrect because Autoencoders are generally better suited for high-dimensional data. Isolation Forests are more suitable for detecting anomalies in time-series data. is incorrect because Isolation Forests are not inherently designed for time-series data.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Isolation Forests = simplicity and less feature engineering [OK]
Hint: Isolation Forests = simplicity and less feature engineering [OK]
Common Mistakes:
2. For a real-time recommendation system, which metric should the ML platform optimize to ensure timely and relevant suggestions?
easy
A. Latency
B. Mean Absolute Error (MAE)
C. Precision
D. Recall

Solution

  1. Step 1: Understand the system's requirement.

    Real-time systems need to provide timely responses, making latency crucial.

  2. Step 2: Eliminate wrong options.

    MAE measures error, Precision and Recall focus on accuracy, not speed.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Real-time needs low latency. [OK]
Hint: Real-time systems prioritize low latency. [OK]
Common Mistakes:
3. In a news feed system, should you prioritize collaborative filtering or content-based filtering when user data is sparse?
medium
A. Content-based filtering, because it doesn't rely on user data.
B. Collaborative filtering, because it captures user preferences better.
C. Collaborative filtering, because it scales better with more users.
D. Content-based filtering, because it requires less computational power.

Solution

  1. Step 1: Analyze data availability

    When user data is sparse, content-based filtering is more effective as it relies on item attributes rather than user interactions.

  2. Step 2: Eliminate wrong options

    A: Collaborative filtering struggles with sparse data. C: It captures preferences well but needs data. D: Computational power is not the main concern here.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Content-based works with sparse user data. [OK]
Hint: Content-based works with sparse user data. [OK]
Common Mistakes:
4. In an HNSW-based vector search system, what is a likely failure mode if the graph's maximum layer is not properly maintained?
medium
A. Increased precision but decreased recall
B. Increased memory usage without affecting speed
C. Decreased search speed due to inefficient traversal
D. Decreased indexing speed but improved search accuracy

Solution

  1. Step 1: Understand the role of the maximum layer.

    The maximum layer affects traversal efficiency.

  2. Step 2: Eliminate wrong options.

    Precision, recall, and memory usage are not directly affected by the maximum layer.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Max layer affects traversal speed. [OK]
Hint: Max layer affects traversal speed. [OK]
Common Mistakes:
5. As a recommendation system scales to 10x the user base, which component is most likely to fail first in maintaining diversity and freshness?
hard
A. Diversity re-ranking layer
B. User profiling module
C. Content ingestion pipeline
D. Collaborative filtering engine

Solution

  1. Step 1: Analyze scalability challenges

    The diversity re-ranking layer may struggle to maintain performance under increased load.

  2. Step 2: Eliminate wrong options

    User profiling (A) and content ingestion (C) are less directly impacted by user scale. Collaborative filtering (D) may slow down but doesn't directly affect diversity.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Diversity re-ranking struggles with scale [OK]
Hint: Diversity re-ranking struggles with scale [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard