Bird
Raised Fist0

A/B Testing for ML Models - Holdout, Interleaving, Causal Inference

THEORYadvancedInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The A/B Testing Anomaly
Started after the latest feature deployment last Friday

Recommendation system for an e-commerce platform serving 5M users daily

Symptom
Model achieves 0.92 AUC offline but drops to 0.75 AUC on live traffic
📡
Signals
metric
Live AUC on production traffic
Dropped from 0.92 to 0.75 after deployment
feature_importance
Feature importance for 'user_last_active_days'
Dropped from 0.18 to 0.02 post-deployment
distribution
Distribution of 'user_last_active_days' in live data
Shifted significantly towards higher values
log
Deployment logs for feature toggles
New feature 'dynamic_pricing' enabled
metric
Offline validation AUC
Stable at 0.92
🔬
Hypotheses
The new feature 'dynamic_pricing' is causing data distribution shift
Supporting Signals
feature_importancedistributionlog
Verify: Check if 'dynamic_pricing' correlates with changes in 'user_last_active_days'
df['dynamic_pricing'].corr(df['user_last_active_days'])
The model is overfitting to the offline validation set
Supporting Signals
metricfeature_importance
Verify: Compare feature distributions between training and live data
sns.histplot(df['user_last_active_days'], kde=True)
🎯
Root Cause
The 'dynamic_pricing' feature caused a shift in user behavior, affecting model performance.
Why It Happened

The feature was not accounted for in the training data, leading to a mismatch in live data distribution.

🔧
Fix Strategy
ImmediateRollback the 'dynamic_pricing' feature toggle
Short-termRetrain the model with updated data including 'dynamic_pricing'
Long-termImplement a continuous integration pipeline for model retraining with feature toggles
🐍
Python Fix
Python
model.fit(updated_training_data_with_dynamic_pricing)
✗
Common Wrong Diagnoses
▼ Show
Assuming model overfitting due to lack of regularization
Why candidates think this: Common cause for performance drop
Why wrong: Regularization wouldn't cause a sudden drop post-deployment
Believing the model is fine due to stable offline AUC
Why candidates think this: Offline metrics suggest no issue
Why wrong: Offline data doesn't reflect live conditions with new features
📊
Monitoring to Add
  • Real-time AUC monitoring on live traffic
  • Feature distribution drift detection
  • Feature toggle impact analysis
📋
Process Change

Include feature toggle impact assessment in pre-deployment checks

🏗️
Architectural Change

Adopt a shadow testing environment for new features

💡
Key Insight
Feature toggles can cause unexpected distribution shifts, impacting live model performance.

Practice

(1/5)
1. A model performs well offline but poorly online. What is the most likely cause of this discrepancy?
medium
A. The online environment has a different data distribution.
B. The model's hyperparameters are not optimized.
C. The model is overfitting to the training data.
D. The offline evaluation used an incorrect metric.

Solution

  1. Step 1: Analyze the difference between offline and online environments.

    Offline evaluation uses historical data, while online evaluation deals with live data, which may have a different distribution.

  2. Step 2: Eliminate other potential causes.

    The model is overfitting to the training data. is incorrect because overfitting would likely affect both offline and online performance. The model's hyperparameters are not optimized. is incorrect because hyperparameter optimization affects both environments. The offline evaluation used an incorrect metric. is incorrect because using an incorrect metric would not cause a discrepancy between environments.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Different data distribution online. [OK]
Hint: Different data distribution online. [OK]
Common Mistakes:
2. A model trained on a dataset shows unexpected bias against a minority group. What is the most likely root cause?
medium
A. The training data is imbalanced.
B. The learning rate is too high.
C. The model has too many parameters.
D. The model uses a linear activation function.

Solution

  1. Step 1: Identify the symptom.

    Bias against a minority group indicates potential data imbalance.

  2. Step 2: Analyze the options.

    Imbalanced data (B) is a common cause of bias. Other options relate to model complexity or training dynamics, not bias directly.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Imbalanced data often leads to bias. [OK]
Hint: Imbalanced data often leads to bias. [OK]
Common Mistakes:
3. You are designing a transformer model for a language translation task. Which normalization technique would you choose and why?
hard
A. Layer Normalization, because it is more suitable for transformer architectures.
B. Batch Normalization, because it is effective for large datasets.
C. Batch Normalization, because it reduces training time.
D. Layer Normalization, because it improves model interpretability.

Solution

  1. Step 1: Consider the architecture of transformers.

    Transformers process sequences and often use self-attention mechanisms, where Layer Normalization is more effective.

  2. Step 2: Evaluate the benefits of Layer Normalization.

    Layer Normalization normalizes across features, which is beneficial for the attention mechanism in transformers.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Layer Norm fits transformers' self-attention. [OK]
Hint: Layer Norm fits transformers' self-attention. [OK]
Common Mistakes:
4. You need to train a model with a learning rate that decreases over time. Which approach would best suit this requirement?
hard
A. Use a constant learning rate with Adam
B. Switch to RMSProp with a fixed decay rate
C. Use AdaGrad, as it inherently decreases learning rate
D. Implement a learning rate scheduler with SGD

Solution

  1. Step 1: Identify the requirement.

    The requirement is for a learning rate that decreases over time.

  2. Step 2: Match the requirement to the solution.

    A learning rate scheduler with SGD allows for controlled decrease over time.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Scheduler with SGD for decreasing lr. [check]
Hint: Scheduler with SGD for decreasing lr. [check]
Common Mistakes:
5. You are tasked with designing a recommendation system for a high-dimensional dataset. Which approach would best handle the curse of dimensionality while maintaining performance?
hard
A. Use a rule-based system to manually curate recommendations.
B. Use collaborative filtering with matrix factorization.
C. Apply a deep learning model with many hidden layers.
D. Implement a simple content-based filtering system.

Solution

  1. Step 1: Identify the challenge of high dimensions.

    Matrix factorization reduces dimensionality effectively.

  2. Step 2: Eliminate wrong options.

    Implement a simple content-based filtering system. doesn't reduce dimensions. Apply a deep learning model with many hidden layers. may overfit due to high dimensions. Use a rule-based system to manually curate recommendations. is not scalable for large datasets.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Matrix factorization reduces dimensions, aids recommendations. [OK]
Hint: Matrix factorization reduces dimensions, aids recommendations. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard