Bird
Raised Fist0

A/B Testing Platform for ML Models - Experiment Management

SYSTEM-DESIGNhardInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🤖
A/B Testing Platform for ML Models - Experiment Management
Design a robust A/B testing platform for managing ML model experiments.
system-designhardFreq: very_highML Platforms
Prerequisites:A/B Testing for ML ModelsOffline vs Online Evaluation
🎯
Problem Framing
Business Objective

Increase revenue by optimizing the performance of ML models through rigorous A/B testing.

ML Objective

Accurately evaluate model performance differences to identify the best performing model variant.

Constraints
  • Latency SLA: P99 < 100ms for model inference
  • Data privacy regulations (GDPR compliance)
  • Scalability to handle 1000s of concurrent experiments
  • Real-time data processing for immediate feedback
  • Robustness to model and data drift
Clarifying Questions
  • What is the expected scale of experiments (number of users, models)?
  • How frequently do models need to be evaluated?
  • What are the key business metrics to track during experiments?
  • Are there any specific privacy concerns or regulations to consider?
  • What is the current infrastructure for data collection and processing?
Out of Scope
✗ Non-ML related A/B testing
✗ Detailed UI/UX design for experiment dashboards
✗ Integration with non-ML systems
📊
Offline Metrics
PR-AUC: To handle class imbalance in click prediction tasks.
Log Loss: Provides a probabilistic measure of model performance.
📈
Online Metrics
CTR: Measures the effectiveness of model changes on user engagement.
Revenue per session: Directly ties model performance to business revenue.
⚖️
Trade-offs

Offline metrics like PR-AUC may show improvement, but if CTR does not increase, the model may not be beneficial in production.

💼
Business Metric

Revenue per session

🗄️
Data Sources
  • User interaction logs (real-time, streaming)
  • Historical transaction data (batch, daily)
  • User profile data (batch, weekly)
  • Ad impression and click data (real-time, streaming)
🏷️
Data Properties
Labeling Strategy

Implicit feedback from user interactions (clicks, conversions) to label data.

Data Volume

Approximately 10 million rows/day, 1 TB/month

Freshness

Real-time for user interaction data, T-1 day for batch updates

⚠️
Data Challenges
  • Sparse labels due to low click-through rates
  • Position bias in ad placements
  • Survivorship bias in user data
  • Cold start problem for new users and items
🔧
Key Features
categorical
User demographics
Personalizes model predictions based on user characteristics.
numerical
Historical click-through rate
Provides baseline engagement level for users.
categorical
Time of day
Captures temporal patterns in user behavior.
categorical
Ad category
Helps model understand user preferences for different ad types.
categorical
Device type
Accounts for differences in user interaction across devices.
⚡
Feature Engineering Notes
Real-time vs Batch

Real-time computation for user interaction features, batch computation for historical features.

Feature Store

Required

🐍
Feature Snippet
Python
def compute_historical_ctr(user_id, interactions):
    clicks = sum(1 for i in interactions if i['clicked'])
    impressions = len(interactions)
    return clicks / impressions if impressions > 0 else 0
🤖
Model Selection
Baseline

Logistic Regression

Proposed Model

Two-Tower Neural Network

The two-tower model efficiently captures user-item interactions and allows precomputation of item embeddings, reducing online latency.
✗
Alternatives Rejected
XGBoost
Fails to capture complex user-item interactions without significant feature engineering.
Collaborative Filtering
Struggles with cold start problems and requires dense user-item matrices.
🏋️
Training Strategy

Hybrid approach with offline training and online fine-tuning to adapt to real-time data changes.

🚀
Serving Infrastructure
LatencyP99 < 100ms end-to-end
Throughput10,000 QPS
ArchitectureTwo-stage (retrieval + ranking) to efficiently handle large candidate sets.
CachingCache precomputed item embeddings with a TTL of 1 hour, invalidate on significant data updates.
FallbackUse rule-based fallback or cached results if the model is unavailable.
📡
Data Drift Signals
Feature distribution drift in user demographics
Label distribution shift in click-through rates
User demographic changes affecting model predictions
📉
Model Drift Signals
AUC degradation
Prediction score distribution shift
Increased prediction variance
🔔
Alerting & Retraining
Retraining Trigger

AUC drops by more than 5% over a week.

Alerting Strategy

Page on-call engineer if AUC drops by 10% in a day, log all prediction errors, auto-recover by rolling back to the last stable model.

🎯
Interview Angles
  • How to handle data privacy in A/B testing?
  • Strategies for managing large-scale experiments efficiently.
  • Balancing offline and online metrics in model evaluation.
⚠️
Common Mistakes
Ignoring data privacy regulations.
Overfitting to offline metrics without considering online performance.
Neglecting the impact of data drift on model performance.
📌
Key Decisions
  • Choosing the right model architecture for latency and scalability.
  • Defining clear metrics that align with business objectives.
  • Implementing robust monitoring and alerting strategies.

Practice

(1/5)
1. For an ad click-through rate prediction model, which metric is most appropriate to optimize for business value?
easy
A. Recall
B. Precision
C. AUC-ROC
D. Accuracy

Solution

  1. Step 1: Understand business objectives.

    CTR prediction models need to balance true positive rate and false positive rate.

  2. Step 2: Eliminate wrong options.

    Accuracy (A) doesn't account for class imbalance. Precision (B) and Recall (C) focus on one aspect of performance, not the trade-off.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    AUC-ROC balances trade-offs in CTR. [OK]
Hint: AUC-ROC balances trade-offs in CTR. [OK]
Common Mistakes:
2. In a production-scale embedding pipeline for text data, which component is responsible for converting raw text into numerical vectors?
easy
A. Data Loader
B. Feature Extractor
C. Tokenizer
D. Batch Normalizer

Solution

  1. Step 1: Identify the role of each component.

    The Tokenizer is responsible for converting raw text into tokens, which are then converted into numerical vectors.

  2. Step 2: Eliminate wrong options.

    Data Loader handles data input, Feature Extractor processes features but not directly from raw text, and Batch Normalizer normalizes data batches.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Tokenizer converts text to tokens. [OK]
Hint: Tokenizer converts text to tokens. [OK]
Common Mistakes:
3. In an ad auction system, if the ads are not being displayed as expected, which failure mode is most likely?
medium
A. Data leakage from training to test set.
B. Latency in fetching user data.
C. Incorrect bid values being used.
D. Overfitting due to too many features.

Solution

  1. Step 1: Identify the symptom

    If ads are not displayed as expected, latency in fetching user data can cause delays, leading to missed ad opportunities.

  2. Step 2: Eliminate wrong options

    • Incorrect bid values: Affects ranking, not display.
    • Data leakage: Affects model accuracy, not display.
    • Overfitting: Affects model performance, not display timing.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Latency affects ad display timing. [OK]
Hint: Latency affects ad display timing. [OK]
Common Mistakes:
4. In a pairwise learning to rank system, what is a common failure mode if the model is not performing well?
medium
A. The model is overfitting due to too many features
B. The model is not capturing the relative order of item pairs
C. The model is not considering the entire list of items
D. The model is using an inappropriate loss function for regression

Solution

  1. Step 1: Identify the failure mode specific to pairwise ranking.

    Pairwise models can fail if they don't capture the relative order of item pairs.

  2. Step 2: Eliminate wrong options.

    Overfitting and inappropriate loss functions are generic issues, and pairwise doesn't consider the entire list, making C incorrect.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Pairwise = focus on item pairs [OK]
Hint: Pairwise = focus on item pairs [OK]
Common Mistakes:
5. A Two-Tower Model for Retrieval is underperforming. What is a likely failure mode if the user and item embeddings are not aligned?
medium
A. The similarity scores between users and items are inaccurate.
B. The model fails to generalize to unseen items.
C. The model's training time increases significantly.
D. The model overfits to the training data.

Solution

  1. Step 1: Identify the impact of misaligned embeddings

    Misaligned embeddings lead to inaccurate similarity scores, affecting retrieval performance.

  2. Step 2: Eliminate wrong options

    The model overfits to the training data. is unrelated to embedding alignment. The model fails to generalize to unseen items. is more about item diversity. The model's training time increases significantly. is not directly caused by embedding misalignment.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Misaligned embeddings cause inaccurate scores. [OK]
Hint: Misaligned embeddings cause inaccurate scores. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard