Bird
Raised Fist0

The Weekend Model Collapse

SYSTEM-DESIGNhardInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Weekend Model Collapse
Started last Saturday, ongoing for 3 days.

ETA prediction system for a delivery service, processing 500k requests per day.

Symptom
Model achieves 0.92 AUC offline but drops to 0.75 AUC on live traffic since last weekend.
📡
Signals
metric
Live AUC on production traffic
Dropped from 0.92 to 0.75 since last Saturday
distribution
Distribution of 'day_of_week' feature in live data
Spike in 'Saturday' and 'Sunday' values compared to training data
feature_importance
Feature importance for 'day_of_week'
Importance dropped from 0.15 to 0.03 after last Friday's deploy
latency
Feature extraction latency for 'traffic_density'
Increased from 50ms to 200ms P99 since last Saturday
log
Error logs from feature extraction service
Increased frequency of 'timeout' errors for 'traffic_density' feature
🔬
Hypotheses
Feature extraction latency causing stale data usage
Supporting Signals
latencylog
Verify: Check timestamps of 'traffic_density' feature in logs
df['traffic_density_timestamp'].apply(lambda x: x < current_time - timedelta(minutes=5)).mean()
Model not generalizing well to weekend data
Supporting Signals
distributionfeature_importance
Verify: Compare model performance on weekend vs weekday data
df[df['day_of_week'].isin(['Saturday', 'Sunday'])]['predictions'].mean()
🎯
Root Cause
Feature extraction latency caused stale 'traffic_density' data, leading to poor predictions on weekends.
Why It Happened

A recent deployment increased the load on the feature extraction service, causing timeouts and stale data.

🔧
Fix Strategy
ImmediateRollback to the previous stable version of the feature extraction service.
Short-termOptimize the feature extraction service to handle increased load without timeouts.
Long-termImplement a load balancing mechanism for the feature extraction service to prevent future latency spikes.
🐍
Python Fix
Python
def optimize_feature_extraction():
    # Increase timeout threshold
    set_timeout_threshold(300)
    # Add caching for frequent requests
    enable_caching_for('traffic_density')
    # Scale service horizontally
    scale_service('feature_extraction', instances=5)
✗
Common Wrong Diagnoses
▼ Show
Model overfitting to training data
Why candidates think this: The drop in live AUC suggests poor generalization.
Why wrong: Validation AUC is stable, indicating the model generalizes well to the validation set.
Data leakage in the validation set
Why candidates think this: The discrepancy between offline and online AUC could imply leakage.
Why wrong: No evidence of leakage; the issue is with live data distribution shift.
📊
Monitoring to Add
  • Real-time monitoring of feature extraction latency with alerts for spikes above 100ms
  • Daily distribution checks for key features like 'day_of_week'
  • A/B testing framework to compare model performance on weekend vs weekday data
📋
Process Change

Introduce a pre-deployment load testing phase for all critical services.

🏗️
Architectural Change

Decouple feature extraction from the main prediction pipeline to allow independent scaling.

💡
Key Insight
Latency-induced stale data can mimic model performance issues, highlighting the need for robust feature pipeline monitoring.

Practice

(1/5)
1. In the context of LLM evaluation systems, which component is responsible for providing a standardized set of tasks to assess model performance?
easy
A. Benchmarks
B. LLM-as-Judge
C. Data preprocessing
D. Human evaluation

Solution

  1. Step 1: Identify the role of each component

    Benchmarks provide standardized tasks for evaluation.

  2. Step 2: Eliminate wrong options

    Human evaluation involves subjective assessment, LLM-as-Judge uses models to evaluate, and data preprocessing is unrelated to task standardization.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Benchmarks standardize tasks for fair comparison. [OK]
Hint: Benchmarks standardize tasks for fair comparison. [OK]
Common Mistakes:
2. Which metric is most appropriate for evaluating the performance of a Two-Tower Model for Retrieval?
easy
A. Mean Squared Error (MSE)
B. Log Loss
C. F1 Score
D. Precision at K

Solution

  1. Step 1: Identify the goal of the retrieval model

    The goal is to rank items effectively for users, making ranking metrics like Precision at K suitable.

  2. Step 2: Eliminate wrong options

    Mean Squared Error (MSE) (MSE) is for regression tasks. F1 Score (F1 Score) is for classification tasks. Log Loss (Log Loss) is also for classification tasks.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Use ranking metrics like Precision at K. [OK]
Hint: Use ranking metrics like Precision at K. [OK]
Common Mistakes:
3. In an LLM evaluation system, what failure might occur if the benchmark tasks are not updated regularly?
medium
A. The system will become too complex to manage.
B. The evaluation results will become less relevant over time.
C. The LLM will overfit to the benchmark tasks.
D. The human evaluators will become biased.

Solution

  1. Step 1: Consider the impact of outdated benchmarks

    Outdated benchmarks can lead to less relevant evaluation results as models evolve.

  2. Step 2: Eliminate wrong options

    Complexity and human bias are unrelated to benchmark updates, and overfitting is more about training data.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Outdated benchmarks lead to irrelevant results. [OK]
Hint: Outdated benchmarks lead to irrelevant results. [OK]
Common Mistakes:
4. When transitioning from traditional collaborative filtering to neural collaborative filtering, which trade-off is most significant?
medium
A. Increased interpretability vs. decreased performance
B. Increased data requirements vs. improved model flexibility
C. Increased computational cost vs. improved scalability
D. Decreased model complexity vs. increased overfitting risk

Solution

  1. Step 1: Understand neural collaborative filtering.

    Neural CF models are more flexible but require more data to train effectively.

  2. Step 2: Eliminate wrong options.

    Options A and D are incorrect as neural models are less interpretable and more complex. Increased computational cost vs. improved scalability is incorrect as neural models are computationally expensive and not necessarily more scalable.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Neural CF = more data, more flexibility [OK]
Hint: Neural CF = more data, more flexibility [OK]
Common Mistakes:
5. As a content-based filtering system scales to 10x more items, which component is most likely to become a bottleneck?
hard
A. Item embedding computation
B. User profile storage
C. Recommendation generation
D. User-item interaction logging

Solution

  1. Step 1: Analyze scaling impact

    Scaling to more items increases the demand for computing and updating item embeddings, making it a potential bottleneck.

  2. Step 2: Eliminate wrong options

    A: User profile storage scales with users, not items. C: Recommendation generation is more affected by user scale. D: Interaction logging is more about user activity than item count.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    More items strain embedding computation. [OK]
Hint: More items strain embedding computation. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard