Bird
Raised Fist0

The Weekend Model Collapse

SYSTEM-DESIGNhardInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Weekend Model Collapse
Started last Friday evening, ongoing for 3 days

A content moderation system for a large-scale LLM serving 10M requests/day, filtering unsafe content.

Symptom
Model achieves 0.92 AUC offline but 0.68 AUC on live traffic · Increased false positives in content moderation alerts · User complaints about safe content being flagged
📡
Signals
metric
Live AUC dropped significantly compared to offline AUC
Live AUC is 0.68 while offline AUC is 0.92
distribution
Shift in input text length distribution
Average text length increased from 120 to 250 characters
feature_importance
Feature 'text_sentiment_score' importance dropped
Importance dropped from 0.15 to 0.03 post-deploy
latency
Inference latency increased
P99 latency increased from 80ms to 150ms
log
Error logs showing increased rate of 'timeout' errors
Timeout errors increased by 40% since Friday
🔬
Hypotheses
Input data distribution shift causing model performance degradation
Supporting Signals
Shift in input text length distributionLive AUC dropped significantly compared to offline AUC
Verify: Check if the model's performance varies with different input text lengths
df.groupby('text_length')['prediction'].mean().plot()
Feature importance change affecting model predictions
Supporting Signals
Feature 'text_sentiment_score' importance dropped
Verify: Analyze model predictions with and without the 'text_sentiment_score' feature
model.feature_importances_['text_sentiment_score']
🎯
Root Cause
Input data distribution shift due to a new feature rollout that increased text length
Why It Happened

A new feature was deployed that encouraged users to write longer texts, affecting model input distribution

🔧
Fix Strategy
ImmediateRollback to the previous model version
Short-termAdjust the model to handle longer input texts by retraining with updated data
Long-termImplement a dynamic input handling mechanism that adapts to input distribution changes
🐍
Python Fix
Python
model = retrain_model(data_with_longer_texts)
✗
Common Wrong Diagnoses
▼ Show
Server overload causing increased latency and errors
Why candidates think this: Increased latency and timeout errors suggest server issues
Why wrong: Latency and errors are symptoms of longer input texts, not the root cause of AUC drop
Model overfitting due to recent retraining
Why candidates think this: AUC drop could be due to overfitting on recent data
Why wrong: Offline AUC remains high, indicating overfitting is not the issue
📊
Monitoring to Add
  • Real-time monitoring of input text length distribution
  • Alert on significant changes in feature importance
  • Monitor AUC on live traffic continuously
📋
Process Change

Implement a pre-deployment check for input distribution changes

🏗️
Architectural Change

Design the system to automatically adapt to input distribution shifts by retraining or adjusting thresholds

💡
Key Insight
Input distribution shifts can drastically affect model performance, especially when new features alter user behavior.

Practice

(1/5)
1. For an A/B test aimed at increasing user engagement, which metric should the platform optimize for?
easy
A. Conversion Rate
B. Session Duration
C. Click-Through Rate (CTR)
D. Bounce Rate

Solution

  1. Step 1: Define user engagement.

    User engagement is often measured by how long users interact with the platform.

  2. Step 2: Eliminate wrong options.

    CTR and Conversion Rate focus on specific actions, while Bounce Rate measures disengagement.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Engagement = time spent. [OK]
Hint: Engagement = time spent. [OK]
Common Mistakes:
2. For a vision-language model designed to generate image captions, which metric should be prioritized to evaluate the quality of the generated captions?
easy
A. Accuracy
B. BLEU score
C. Precision
D. Recall

Solution

  1. Step 1: Understand the task requirements

    Image captioning requires evaluating the quality of generated text against reference captions.

  2. Step 2: Choose the appropriate metric

    BLEU score is specifically designed to evaluate the quality of text generation tasks like translation and captioning.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    BLEU evaluates text generation quality. [OK]
Hint: BLEU evaluates text generation quality. [OK]
Common Mistakes:
3. In a pairwise learning to rank system, what is a common failure mode if the model is not performing well?
medium
A. The model is overfitting due to too many features
B. The model is not capturing the relative order of item pairs
C. The model is not considering the entire list of items
D. The model is using an inappropriate loss function for regression

Solution

  1. Step 1: Identify the failure mode specific to pairwise ranking.

    Pairwise models can fail if they don't capture the relative order of item pairs.

  2. Step 2: Eliminate wrong options.

    Overfitting and inappropriate loss functions are generic issues, and pairwise doesn't consider the entire list, making C incorrect.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Pairwise = focus on item pairs [OK]
Hint: Pairwise = focus on item pairs [OK]
Common Mistakes:
4. When implementing a 'People You May Know' feature, should you use a simple Jaccard similarity or a more complex graph embedding approach? Assume the graph is sparse and dynamic.
medium
A. Use Jaccard similarity because it's computationally efficient for sparse graphs.
B. Use graph embeddings because they are always better than simple metrics.
C. Use Jaccard similarity because it adapts well to dynamic graphs.
D. Use graph embeddings because they capture more complex relationships.

Solution

  1. Step 1: Analyze graph characteristics.

    The graph is sparse and dynamic, requiring methods that capture complex relationships.

  2. Step 2: Evaluate options.

    Jaccard similarity (A) is efficient but too simplistic for capturing complex relationships. Graph embeddings (B) are suitable as they capture more nuanced relationships. Jaccard similarity (C) does not inherently adapt to dynamic graphs. Graph embeddings (D) are not always better, but in this case, they are due to the complexity of relationships.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Embeddings capture complex relationships. [OK]
Hint: Embeddings capture complex relationships. [OK]
Common Mistakes:
5. As the volume of emails increases 10x, which component of a spam detection system is most likely to fail first?
hard
A. Feature extraction pipeline
B. Model training process
C. Real-time inference engine
D. Data storage system

Solution

  1. Step 1: Consider scalability of components

    Real-time inference engines often struggle with increased load due to latency constraints.

  2. Step 2: Eliminate components less affected by scale

    Feature extraction (A) and data storage (D) can be scaled horizontally. Model training (B) is not real-time.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Inference engines struggle with scale. [OK]
Hint: Inference engines struggle with scale. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard