Bird
Raised Fist0

The Weekend Model Collapse

SYSTEM-DESIGNhardInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Weekend Model Collapse
Started last Friday, ongoing for 5 days.

A two-tower model for content recommendation serving 5M users daily on a streaming platform.

Symptom
AUC dropped from 0.85 to 0.72 over the past week. · Recommendation diversity collapsed, with many users seeing repeated items. · Real-time feature latency spiked from 50ms to 200ms. · A/B test shows offline AUC at 0.85 but online AUC at 0.72.
📡
Signals
metric
Live AUC on production traffic
Dropped from 0.85 to 0.72
distribution
Distribution of recommended items
Top 10 items account for 80% of recommendations
latency
Real-time feature extraction latency
Increased from 50ms to 200ms
feature_importance
Feature importance for 'user_last_active_days'
Dropped from 0.18 to 0.02 after last Thursday's deploy
log
Error logs from feature extraction service
Spike in timeout errors
🔬
Hypotheses
Feature extraction service is causing latency spikes, affecting real-time features.
Supporting Signals
Real-time feature extraction latency increasedError logs show timeout errors
Verify: Check the feature extraction service logs and latency metrics.
import pandas as pd; df = pd.read_csv('latency_logs.csv'); df['latency'].plot()
Model is overfitting to recent data due to a data pipeline issue.
Supporting Signals
Feature importance for 'user_last_active_days' droppedLive AUC dropped while offline AUC is stable
Verify: Analyze the training data distribution and compare with live data.
import seaborn as sns; sns.histplot(df['user_last_active_days'], kde=True)
🎯
Root Cause
Feature extraction service latency caused by a recent code change, leading to stale features.
Why It Happened

A deployment last Thursday introduced inefficient code in the feature extraction service.

🔧
Fix Strategy
ImmediateRollback the feature extraction service to the previous stable version.
Short-termOptimize the feature extraction code to reduce latency.
Long-termImplement a caching mechanism for frequently accessed features to reduce load.
🐍
Python Fix
Python
def optimized_feature_extraction(data):
    # Implement caching
    cache = {}
    for item in data:
        if item not in cache:
            cache[item] = expensive_computation(item)
    return cache
✗
Common Wrong Diagnoses
▼ Show
The model is overfitting due to insufficient training data.
Why candidates think this: The drop in AUC suggests overfitting.
Why wrong: Offline AUC remains stable, indicating the model is not overfitting on the training data.
The recommendation algorithm is inherently biased.
Why candidates think this: The collapse in diversity suggests bias.
Why wrong: The diversity issue is a symptom of stale features, not an inherent algorithmic bias.
📊
Monitoring to Add
  • Real-time feature extraction latency alerts
  • Feature importance drift detection
  • Recommendation diversity metrics
📋
Process Change

Introduce a mandatory code review process for all feature extraction service changes.

🏗️
Architectural Change

Decouple feature extraction from real-time inference to allow independent scaling.

💡
Key Insight
Feature extraction latency can silently degrade model performance by providing stale data, leading to significant online-offline metric discrepancies.

Practice

(1/5)
1. In the context of anomaly detection, which of the following best describes the primary advantage of using an Isolation Forest over an Autoencoder?
easy
A. Isolation Forests require labeled data for training.
B. Isolation Forests are more suitable for detecting anomalies in time-series data.
C. Isolation Forests can handle high-dimensional data better than Autoencoders.
D. Isolation Forests are more interpretable and require less feature engineering.

Solution

  1. Step 1: Understand the strengths of Isolation Forests.

    Isolation Forests are known for their simplicity and interpretability, as they isolate anomalies by randomly selecting a feature and then randomly selecting a split value between the maximum and minimum values of that feature.

  2. Step 2: Eliminate wrong options.

    Isolation Forests require labeled data for training. is incorrect because Isolation Forests do not require labeled data. Isolation Forests can handle high-dimensional data better than Autoencoders. is incorrect because Autoencoders are generally better suited for high-dimensional data. Isolation Forests are more suitable for detecting anomalies in time-series data. is incorrect because Isolation Forests are not inherently designed for time-series data.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Isolation Forests = simplicity and less feature engineering [OK]
Hint: Isolation Forests = simplicity and less feature engineering [OK]
Common Mistakes:
2. In the context of intent classification, which of the following techniques is most suitable for handling imbalanced datasets?
easy
A. Apply SMOTE (Synthetic Minority Over-sampling Technique)
B. Use a simple logistic regression model
C. Use a linear SVM without any modifications
D. Ignore the imbalance and focus on feature engineering

Solution

  1. Step 1: Identify the challenge of imbalanced datasets.

    Imbalanced datasets can lead to biased models that favor the majority class.

  2. Step 2: Evaluate each option.

    Use a simple logistic regression model: Logistic regression without addressing imbalance will likely be biased. Use a linear SVM without any modifications: Linear SVM also suffers from imbalance without adjustments. Ignore the imbalance and focus on feature engineering: Ignoring imbalance is not a solution. Apply SMOTE (Synthetic Minority Over-sampling Technique): SMOTE is specifically designed to handle imbalance by generating synthetic samples.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    SMOTE for imbalance handling. [OK]
Hint: SMOTE is key for imbalanced datasets. [OK]
Common Mistakes:
3. In a spam detection system, which feature is most likely to be effective in identifying spam emails?
easy
A. Email length
B. Presence of certain keywords
C. Sender's email domain
D. Time of day email was sent

Solution

  1. Step 1: Identify effective features for spam detection

    Keywords like 'free', 'win', or 'prize' are often used in spam emails.

  2. Step 2: Eliminate less effective features

    Email length (A) and time of day (D) are not strong indicators of spam. Sender's domain (C) can be spoofed.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Keywords often signal spam. [OK]
Hint: Keywords often signal spam. [OK]
Common Mistakes:
4. What is a likely failure mode if a spam detection system is not robust to adversarial attacks?
medium
A. Model underfitting
B. Increased false positives
C. Model overfitting
D. Increased false negatives

Solution

  1. Step 1: Understand adversarial attack impacts

    Adversarial attacks often aim to bypass detection, leading to false negatives.

  2. Step 2: Eliminate unrelated failure modes

    False positives (B) are less likely from adversarial attacks. Overfitting (C) and underfitting (D) are unrelated to adversarial robustness.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Adversarial attacks increase false negatives. [OK]
Hint: Adversarial attacks increase false negatives. [OK]
Common Mistakes:
5. As a content-based filtering system scales to 10x more items, which component is most likely to become a bottleneck?
hard
A. Item embedding computation
B. User profile storage
C. Recommendation generation
D. User-item interaction logging

Solution

  1. Step 1: Analyze scaling impact

    Scaling to more items increases the demand for computing and updating item embeddings, making it a potential bottleneck.

  2. Step 2: Eliminate wrong options

    A: User profile storage scales with users, not items. C: Recommendation generation is more affected by user scale. D: Interaction logging is more about user activity than item count.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    More items strain embedding computation. [OK]
Hint: More items strain embedding computation. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard