Bird
Raised Fist0

A/B Testing Platform for ML Models - Experiment Management

SYSTEM-DESIGNhardInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The A/B Test Anomaly
Started last Friday, ongoing for 3 days.

An A/B testing platform managing experiments for a recommendation system serving 5M users daily.

Symptom
Model achieves 0.92 AUC offline but drops to 0.75 AUC on live traffic. Recommendation diversity collapsed, with users seeing repetitive suggestions.
📡
Signals
metric
Live AUC on production traffic
Dropped from 0.92 to 0.75 after Friday's deploy
distribution
Distribution of recommended items
Top 5 items account for 80% of recommendations, up from 30% pre-deploy
feature_importance
Feature importance for 'user_last_active_days'
Dropped from 0.18 to 0.02 after last Friday's deploy
latency
Real-time feature computation latency
Spiked from 50ms to 200ms P99 after deploy
log
Error logs from feature extraction service
Increased frequency of 'timeout' errors post-deploy
🔬
Hypotheses
Feature extraction service is causing latency spikes, affecting real-time features.
Supporting Signals
latencylog
Verify: Check service logs for timeout errors and latency metrics.
import pandas as pd
log_df = pd.read_csv('service_logs.csv')
timeout_errors = log_df[log_df['error'] == 'timeout']
print(timeout_errors['timestamp'].value_counts())
Feature importance shift indicates a data pipeline issue affecting 'user_last_active_days'.
Supporting Signals
feature_importance
Verify: Inspect data pipeline for changes in feature computation logic.
import pandas as pd
feature_df = pd.read_csv('feature_importance.csv')
print(feature_df[feature_df['feature'] == 'user_last_active_days'])
Recommendation diversity collapse due to model overfitting on recent data.
Supporting Signals
distribution
Verify: Analyze training data distribution and compare with live data.
import matplotlib.pyplot as plt
train_data = pd.read_csv('train_data.csv')
live_data = pd.read_csv('live_data.csv')
plt.hist(train_data['item_id'], bins=50, alpha=0.5, label='Train')
plt.hist(live_data['item_id'], bins=50, alpha=0.5, label='Live')
plt.legend()
plt.show()
🎯
Root Cause
Feature extraction service latency caused real-time features to be outdated, leading to poor model performance.
Why It Happened

A recent deployment introduced inefficient code in the feature extraction service, causing timeouts and delays.

🔧
Fix Strategy
ImmediateRollback to the previous version of the feature extraction service.
Short-termOptimize the feature extraction code to reduce latency and prevent timeouts.
Long-termImplement a caching mechanism for real-time features to handle latency spikes.
🐍
Python Fix
Python
# Example Python fix for optimizing feature extraction
import time
def optimized_feature_extraction(data):
    # Simulate optimized processing
    time.sleep(0.05)  # Reduced from 0.2 seconds
    return data.apply(lambda x: x * 2)

# Apply the optimized function
optimized_data = optimized_feature_extraction(raw_data)
✗
Common Wrong Diagnoses
▼ Show
Model overfitting due to recent data skew.
Why candidates think this: The collapse in recommendation diversity suggests overfitting.
Why wrong: Feature importance shift and latency issues point to real-time feature problems, not data skew.
Data leakage in the training set.
Why candidates think this: The offline AUC is high, suggesting a potential data issue.
Why wrong: The stable validation AUC and latency issues indicate the problem is in real-time feature computation.
📊
Monitoring to Add
  • Real-time feature latency monitoring with alerting on P99 > 100ms
  • Feature importance drift detection for key features
  • Recommendation diversity monitoring to ensure top items don't exceed 50%
📋
Process Change

Implement a pre-deployment latency testing protocol for all feature extraction changes.

🏗️
Architectural Change

Introduce a feature store with versioning to decouple feature computation from real-time serving.

💡
Key Insight
Real-time feature latency can drastically affect online model performance, even when offline metrics appear stable.

Practice

(1/5)
1. In the context of LLM serving infrastructure, which component is primarily responsible for reducing latency by reusing previously computed attention outputs?
easy
A. Batching
B. Speculative Decoding
C. KV Cache
D. Load Balancer

Solution

  1. Step 1: Identify the component's role.

    KV Cache stores key-value pairs from previous computations to avoid redundant calculations.

  2. Step 2: Eliminate wrong options.

    Batching reduces overhead by processing multiple requests together, not by reusing computations. Speculative Decoding predicts future tokens, not reuse. Load Balancer distributes requests, unrelated to computation reuse.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    KV Cache reuses computations to cut latency. [OK]
Hint: KV Cache reuses computations to cut latency. [OK]
Common Mistakes:
2. In a DLRM model for CTR prediction, what is a likely failure mode if the embedding dimension is too small?
medium
A. Overfitting to the training data
B. Underfitting due to insufficient feature representation
C. Increased inference latency
D. Gradient vanishing during training

Solution

  1. Step 1: Understand the role of embedding dimensions.

    Small dimensions may not capture enough feature information.

  2. Step 2: Eliminate wrong options.

    Overfitting (A) is unlikely with small dimensions. Latency (C) and gradient issues (D) are unrelated to embedding size.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Small embeddings risk underfitting. [OK]
Hint: Small embeddings risk underfitting. [OK]
Common Mistakes:
3. In a visual search system, when should you prefer using an Approximate Nearest Neighbor (ANN) index over an exact search method?
medium
A. When low latency is crucial
B. When the dataset is small
C. When exact matches are critical
D. When the search space is one-dimensional

Solution

  1. Step 1: Consider the trade-offs

    ANN is preferred when low latency is crucial because it speeds up search at the cost of some accuracy.

  2. Step 2: Eliminate wrong options

    Exact matches require exact search, small datasets don't benefit much from ANN, and one-dimensional search doesn't need ANN complexity.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    ANN for speed, not exactness. [OK]
Hint: ANN for speed, not exactness. [OK]
Common Mistakes:
4. At 10x data scale in a multimodal embedding pipeline, which component is most likely to become a bottleneck first?
hard
A. Model Inference
B. Feature Storage
C. Data Preprocessing
D. Network Bandwidth

Solution

  1. Step 1: Analyze the impact of scaling on each component.

    Feature Storage is likely to become a bottleneck due to increased data volume.

  2. Step 2: Eliminate wrong options.

    Data Preprocessing and Model Inference can be parallelized, and Network Bandwidth is less affected by data volume.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Storage struggles with data scale. [OK]
Hint: Storage struggles with data scale. [OK]
Common Mistakes:
5. At 10x data scale in a fine-tuning pipeline using SFT, RLHF, and DPO, which component is most likely to become a bottleneck first?
hard
A. Data Preprocessing Optimization (DPO)
B. Supervised Fine-Tuning (SFT)
C. Reinforcement Learning from Human Feedback (RLHF)
D. Model Deployment Infrastructure

Solution

  1. Step 1: Analyze the impact of scaling on each component

    DPO involves handling and optimizing large volumes of data, which can become a bottleneck as data scales.

  2. Step 2: Eliminate wrong options

    B: SFT scales with model size, not data volume. C: RLHF is more about feedback processing, not data volume. D: Deployment infrastructure is affected by model size, not data volume.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    DPO bottlenecks with data scale. [OK]
Hint: DPO bottlenecks with data scale. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard