Bird
Raised Fist0

The Weekend Model Collapse

SYSTEM-DESIGNhardInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Weekend Model Collapse
Started last Friday evening, ongoing for 3 days.

A hate speech detection model serving 10M content moderation requests per day for a major social media platform.

Symptom
Model achieves 0.92 AUC offline but drops to 0.68 AUC on live traffic. Real-time feature latency spiked from 50ms to 200ms.
📡
Signals
metric
Live AUC on production traffic
Dropped from 0.92 to 0.68
latency
Real-time feature extraction latency
Increased from 50ms to 200ms
distribution
Distribution of input text lengths
Shifted from mean 50 words to mean 120 words
feature_importance
Feature 'user_last_active_days' importance
Dropped from 0.18 to 0.02 after last Thursday's deploy
log
Error logs from feature extraction service
Increased rate of 'timeout' errors
🔬
Hypotheses
Feature extraction service is causing latency spikes, affecting model performance.
Supporting Signals
Real-time feature extraction latencyError logs from feature extraction service
Verify: Check service logs for error rates and latency distributions.
import pandas as pd; df = pd.read_csv('service_logs.csv'); print(df['error'].value_counts()); print(df['latency'].describe())
Input data distribution shift is causing model performance degradation.
Supporting Signals
Distribution of input text lengths
Verify: Analyze input data distribution before and after the issue started.
import matplotlib.pyplot as plt; df = pd.read_csv('input_data.csv'); plt.hist(df['text_length'], bins=50); plt.title('Text Length Distribution'); plt.xlabel('Text Length'); plt.ylabel('Frequency'); plt.show()
🎯
Root Cause
Feature extraction service latency increased due to a misconfigured timeout setting.
Why It Happened

A recent deployment inadvertently changed the timeout configuration, causing delays in feature extraction.

🔧
Fix Strategy
ImmediateRollback to the previous stable version of the feature extraction service.
Short-termCorrect the timeout configuration in the feature extraction service.
Long-termImplement a more robust timeout handling mechanism and add redundancy to the feature extraction service.
🐍
Python Fix
Python
def set_timeout(config):
    config['timeout'] = 50  # Set to previous stable value
    return config

config = load_config()
config = set_timeout(config)
save_config(config)
✗
Common Wrong Diagnoses
▼ Show
The model itself is overfitting to the training data.
Why candidates think this: The offline AUC is high, suggesting the model performs well on training data.
Why wrong: The issue is not with the model's training but with the feature extraction latency affecting live predictions.
A data pipeline issue caused incorrect data to be fed into the model.
Why candidates think this: The distribution shift in input text lengths suggests a data issue.
Why wrong: The distribution shift is a symptom of increased latency, not a data pipeline error.
📊
Monitoring to Add
  • Real-time feature extraction latency monitoring with alerts for spikes above 100ms.
  • AUC monitoring on live traffic with alerts for drops below 0.85.
  • Error rate monitoring for feature extraction service with alerts for increases above 5%.
📋
Process Change

Implement a deployment checklist that includes verification of critical configuration settings.

🏗️
Architectural Change

Introduce a circuit breaker pattern for the feature extraction service to handle latency spikes gracefully.

💡
Key Insight
Latency in feature extraction can drastically impact real-time model performance, even if the model itself is well-trained.

Practice

(1/5)
1. For a fraud detection system, which metric is most appropriate to optimize for minimizing false positives?
easy
A. Recall
B. Accuracy
C. Precision
D. F1 Score

Solution

  1. Step 1: Understand the impact of false positives.

    Minimizing false positives means correctly identifying non-fraudulent transactions.

  2. Step 2: Choose the metric that reflects this goal.

    Precision is the metric that focuses on reducing false positives.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Precision reduces false positives. [OK]
Hint: Precision reduces false positives. [OK]
Common Mistakes:
2. In a neural collaborative filtering system, what is a likely failure mode if the model is not performing well on new users?
medium
A. Data leakage from training to validation sets.
B. Overfitting due to too many layers in the neural network.
C. Cold start problem due to lack of user interaction data.
D. Underfitting due to insufficient model complexity.

Solution

  1. Step 1: Identify the issue with new users.

    New users often lack interaction data, leading to the cold start problem.

  2. Step 2: Eliminate wrong options.

    Options A and D relate to model complexity, not new user performance. Data leakage from training to validation sets. is about data leakage, not specific to new users.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    New users = cold start issue [OK]
Hint: New users = cold start issue [OK]
Common Mistakes:
3. In a two-stage retrieval system, when should you prioritize a simpler candidate generation model over a complex one?
medium
A. When the ranking stage is computationally expensive and needs more resources.
B. When precision is more important than recall in the candidate generation stage.
C. When the dataset is small and computation resources are abundant.
D. When the ranking model is not capable of handling large candidate sets.

Solution

  1. Step 1: Consider resource allocation

    If the ranking stage is computationally expensive, a simpler candidate generation model can save resources for the ranking stage.

  2. Step 2: Eliminate wrong options

    When the dataset is small and computation resources are abundant. is incorrect because a small dataset doesn't necessitate a simpler model. When precision is more important than recall in the candidate generation stage. is incorrect because recall is typically prioritized in candidate generation. When the ranking model is not capable of handling large candidate sets. is incorrect because the ranking model should handle the candidate set size.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Save resources for ranking if it's costly. [OK]
Hint: Save resources for ranking if it's costly. [OK]
Common Mistakes:
4. As a content-based filtering system scales to 10x more items, which component is most likely to become a bottleneck?
hard
A. Item embedding computation
B. User profile storage
C. Recommendation generation
D. User-item interaction logging

Solution

  1. Step 1: Analyze scaling impact

    Scaling to more items increases the demand for computing and updating item embeddings, making it a potential bottleneck.

  2. Step 2: Eliminate wrong options

    A: User profile storage scales with users, not items. C: Recommendation generation is more affected by user scale. D: Interaction logging is more about user activity than item count.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    More items strain embedding computation. [OK]
Hint: More items strain embedding computation. [OK]
Common Mistakes:
5. As the volume of emails increases 10x, which component of a spam detection system is most likely to fail first?
hard
A. Feature extraction pipeline
B. Model training process
C. Real-time inference engine
D. Data storage system

Solution

  1. Step 1: Consider scalability of components

    Real-time inference engines often struggle with increased load due to latency constraints.

  2. Step 2: Eliminate components less affected by scale

    Feature extraction (A) and data storage (D) can be scaled horizontally. Model training (B) is not real-time.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Inference engines struggle with scale. [OK]
Hint: Inference engines struggle with scale. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard