Bird
Raised Fist0

The Weekend Model Collapse

SYSTEM-DESIGNhardInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Weekend Model Collapse
Started last Friday, ongoing for 5 days

A news feed personalization system serving 10M users daily, optimizing for user engagement and diversity of content.

Symptom
AUC dropped from 0.85 to 0.72 over the past week · Recommendation diversity collapsed, with users seeing repetitive content · Real-time feature latency spiked from 50ms to 200ms · A/B test shows offline metrics are stable but online engagement metrics have declined
📡
Signals
metric
Live AUC on production traffic
Dropped from 0.85 to 0.72
distribution
Content diversity in recommendations
Diversity index dropped from 0.65 to 0.30
latency
Real-time feature computation latency
Increased from 50ms to 200ms
feature_importance
Feature importance for 'user_last_active_days'
Dropped from 0.18 to 0.02 after last Thursday's deploy
log
Error logs from feature computation service
Spike in timeout errors since last Friday
🔬
Hypotheses
Feature computation service is causing latency spikes, affecting real-time features.
Supporting Signals
Real-time feature latency spiked from 50ms to 200msSpike in timeout errors since last Friday
Verify: Check service logs for errors and latency metrics for real-time feature computation.
import pandas as pd; df = pd.read_csv('service_logs.csv'); df[df['error'] == 'timeout'].groupby('date').size().plot()
Model is overfitting due to stale training data, causing AUC drop.
Supporting Signals
Live AUC on production traffic droppedFeature 'user_last_active_days' importance dropped
Verify: Analyze training data freshness and compare feature distributions with live data.
import pandas as pd; train_data = pd.read_csv('train_data.csv'); live_data = pd.read_csv('live_data.csv'); pd.concat([train_data['user_last_active_days'], live_data['user_last_active_days']], axis=1).plot(kind='density')
🎯
Root Cause
The feature computation service was updated, introducing a bug that caused latency spikes and stale feature values.
Why It Happened

A recent deployment changed the feature computation logic, leading to increased latency and incorrect feature values.

🔧
Fix Strategy
ImmediateRollback the feature computation service to the previous stable version.
Short-termPatch the feature computation logic to handle edge cases causing latency spikes.
Long-termImplement a robust testing framework for feature computation to catch latency and correctness issues before deployment.
🐍
Python Fix
Python
def compute_features(data):
    try:
        # Corrected logic
        features = data.apply(lambda x: x * 2 if x > 0 else 0)
    except Exception as e:
        log_error(e)
        features = fallback_features(data)
    return features
✗
Common Wrong Diagnoses
▼ Show
The model is simply overfitting due to a lack of regularization.
Why candidates think this: The drop in AUC suggests overfitting, which is a common issue.
Why wrong: The stable offline metrics and sudden latency spike indicate an external factor affecting real-time features, not model overfitting.
The data pipeline is introducing noise, causing the AUC drop.
Why candidates think this: Noise in data can degrade model performance.
Why wrong: The issue is linked to a specific feature computation service update, not the entire data pipeline.
📊
Monitoring to Add
  • Real-time feature latency monitoring with alerts for spikes above 100ms
  • Feature value distribution checks against historical baselines
  • Automated tests for feature computation logic changes
📋
Process Change

Introduce a mandatory code review process for all feature computation logic changes.

🏗️
Architectural Change

Decouple feature computation from the main service to allow independent scaling and monitoring.

💡
Key Insight
Latency spikes in feature computation can silently degrade model performance by providing stale or incorrect feature values.

Practice

(1/5)
1. For a multimodal embedding pipeline, which metric is most appropriate to evaluate the alignment between text and image embeddings?
easy
A. Precision
B. Mean Squared Error
C. Accuracy
D. Cosine Similarity

Solution

  1. Step 1: Understand the evaluation goal.

    We need to evaluate the alignment between embeddings, which is best measured by similarity metrics.

  2. Step 2: Eliminate wrong options.

    Accuracy and Precision are classification metrics, and Mean Squared Error is for regression, not similarity.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Cosine Similarity measures alignment. [OK]
Hint: Cosine Similarity measures alignment. [OK]
Common Mistakes:
2. In the context of a feed ranking system for social media platforms, which component is primarily responsible for determining the order of posts shown to a user?
easy
A. Ranking algorithm
B. Feature extraction module
C. Data ingestion pipeline
D. User interface

Solution

  1. Step 1: Identify the role of each component

    The ranking algorithm is responsible for determining the order of posts based on relevance and user preferences.

  2. Step 2: Eliminate wrong options

    Data ingestion pipeline collects data, feature extraction module processes features, and user interface displays the results but does not determine order.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Ranking algorithm orders posts. [OK]
Hint: Ranking algorithm orders posts. [OK]
Common Mistakes:
3. For a learning to rank system, which metric is most appropriate to evaluate the quality of the ranking?
easy
A. Normalized Discounted Cumulative Gain (NDCG)
B. Mean Squared Error (MSE)
C. Precision
D. Recall

Solution

  1. Step 1: Identify the metric that evaluates ranking quality.

    NDCG is specifically designed to evaluate the quality of rankings.

  2. Step 2: Eliminate wrong options.

    MSE is for regression, precision and recall are for classification, not ranking.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    NDCG = ranking quality [OK]
Hint: NDCG = ranking quality [OK]
Common Mistakes:
4. For an ML monitoring platform focused on drift detection, which metric is most appropriate to measure the effectiveness of the drift detection module?
easy
A. Accuracy
B. KL Divergence
C. F1 Score
D. Precision

Solution

  1. Step 1: Understand the purpose of drift detection

    Drift detection aims to identify changes in data distribution.

  2. Step 2: Match metrics to purpose

    KL Divergence measures the difference between two probability distributions, making it suitable for drift detection.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Distribution change = KL Divergence [OK]
Hint: Distribution change = KL Divergence [OK]
Common Mistakes:
5. When designing a recommendation system to avoid filter bubbles, which approach is more suitable for ensuring content freshness?
medium
A. Popularity-based recommendations
B. Static content lists
C. User-based collaborative filtering
D. Time-decay models

Solution

  1. Step 1: Evaluate each approach's impact on freshness

    Time-decay models prioritize recent content, enhancing freshness.

  2. Step 2: Eliminate wrong options

    Static content lists (A) do not adapt over time. User-based collaborative filtering (C) and popularity-based recommendations (D) may reinforce existing preferences, not freshness.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Time-decay models prioritize recent content [OK]
Hint: Time-decay models prioritize recent content [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard