Bird
Raised Fist0

The Weekend Model Collapse

SYSTEM-DESIGNhardInterview freq: medium
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Weekend Model Collapse
Started last Friday evening, ongoing for 3 days.

A session-based recommendation system for an e-commerce platform serving 5M sessions/day.

Symptom
AUC dropped from 0.85 to 0.72 over the past weekend. · Recommendation diversity collapsed, with many users seeing repeated items. · Real-time feature latency spiked from 50ms to 200ms. · A/B test shows offline metrics are stable but online metrics have degraded.
📡
Signals
metric
Live AUC observed in production monitoring dashboard.
Dropped from 0.85 to 0.72 starting Friday evening.
distribution
Distribution of recommended items per session.
Top 5 items account for 80% of recommendations, up from 30%.
latency
Real-time feature extraction latency from logs.
Increased from 50ms to 200ms at P99.
feature_importance
Feature importance from model explainability tool.
Feature 'session_duration' importance dropped from 0.15 to 0.03.
log
Error logs from feature extraction service.
Increased frequency of timeout errors.
🔬
Hypotheses
Feature extraction service is causing latency spikes, affecting real-time recommendations.
Supporting Signals
latencylog
Verify: Check service logs for timeout errors and correlate with latency spikes.
import pandas as pd; df = pd.read_csv('service_logs.csv'); timeout_errors = df[df['error'] == 'timeout']; timeout_errors['timestamp'].hist(); plt.show()
Model is overfitting to recent data due to lack of diversity in training set.
Supporting Signals
distributionfeature_importance
Verify: Analyze training data distribution and compare with live data.
import matplotlib.pyplot as plt; plt.hist(training_data['item_id'], bins=50); plt.title('Training Data Distribution'); plt.show()
🎯
Root Cause
Feature extraction service latency increased due to a recent code deployment that introduced inefficient data processing.
Why It Happened

A recent deployment included a change that inadvertently increased the complexity of data processing in the feature extraction service.

🔧
Fix Strategy
ImmediateRollback the feature extraction service to the previous stable version.
Short-termOptimize the data processing logic in the feature extraction service to reduce latency.
Long-termImplement a canary deployment strategy for feature extraction service changes.
🐍
Python Fix
Python
def optimize_data_processing(data):
    # Simplified processing logic
    return data.apply(lambda x: x * 0.9)

# Apply optimization
optimized_data = optimize_data_processing(raw_data)
✗
Common Wrong Diagnoses
▼ Show
The model is overfitting due to insufficient training data.
Why candidates think this: The drop in feature importance and recommendation diversity suggests overfitting.
Why wrong: The stable validation AUC indicates the model itself is not overfitting.
Data distribution shift in user behavior.
Why candidates think this: The change in recommendation diversity could be due to user behavior changes.
Why wrong: The timing of the issue aligns with a deployment, not a gradual user behavior shift.
📊
Monitoring to Add
  • Real-time latency monitoring for feature extraction service with alerting.
  • Diversity index monitoring for recommended items.
  • Feature importance drift detection in production.
📋
Process Change

Introduce a mandatory code review process for all changes to the feature extraction service.

🏗️
Architectural Change

Decouple feature extraction from real-time recommendation pipeline to allow independent scaling and updates.

💡
Key Insight
Latency spikes in feature extraction can cascade into recommendation quality issues, highlighting the need for robust monitoring and deployment strategies.

Practice

(1/5)
1. For a real-time recommendation system, which metric should the ML platform optimize to ensure timely and relevant suggestions?
easy
A. Latency
B. Mean Absolute Error (MAE)
C. Precision
D. Recall

Solution

  1. Step 1: Understand the system's requirement.

    Real-time systems need to provide timely responses, making latency crucial.

  2. Step 2: Eliminate wrong options.

    MAE measures error, Precision and Recall focus on accuracy, not speed.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Real-time needs low latency. [OK]
Hint: Real-time systems prioritize low latency. [OK]
Common Mistakes:
2. In a two-stage retrieval system, what is the primary role of the candidate generation stage?
easy
A. To filter out irrelevant items from the entire dataset.
B. To rank the candidates based on relevance scores.
C. To apply complex models for precise ranking.
D. To ensure all items are considered for ranking.

Solution

  1. Step 1: Understand the candidate generation role

    The candidate generation stage is responsible for quickly filtering out irrelevant items to reduce the dataset size for the ranking stage.

  2. Step 2: Eliminate wrong options

    To rank the candidates based on relevance scores. is incorrect because ranking is done in the second stage. To apply complex models for precise ranking. is incorrect because complex models are typically used in the ranking stage. To ensure all items are considered for ranking. is incorrect because not all items are considered for ranking, only the filtered candidates.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Candidate generation filters, not ranks. [OK]
Hint: Candidate generation filters, not ranks. [OK]
Common Mistakes:
3. In an A/B testing platform, what is a likely failure mode if the randomization engine is not functioning correctly?
medium
A. The experiment results are biased.
B. Users are not consistently assigned to the same variant.
C. The platform cannot track user interactions.
D. The experiment dashboard fails to update.

Solution

  1. Step 1: Identify the role of the randomization engine.

    It ensures unbiased assignment of users to variants.

  2. Step 2: Eliminate wrong options.

    Session persistence handles consistent assignment, tracking is separate, and dashboard updates are unrelated.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Randomization ensures unbiased results. [OK]
Hint: Randomization ensures unbiased results. [OK]
Common Mistakes:
4. As an agent system scales to handle 10x more tasks, which component is most likely to become a bottleneck?
hard
A. Memory Architecture
B. Data Input Pipeline
C. Planning Algorithm
D. Tool Use Integration

Solution

  1. Step 1: Consider scaling impact

    Handling more tasks increases data and state storage requirements.

  2. Step 2: Identify potential bottlenecks

    Memory Architecture is likely to struggle with increased storage demands, while other components are less directly affected by task volume.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Scaling = Memory bottleneck risk [OK]
Hint: Scaling often stresses memory first. [OK]
Common Mistakes:
5. As the scale of data increases in a semantic search system using Bi-Encoders, which component is likely to fail first and why?
hard
A. Network bandwidth, due to increased data transfer
B. Document storage, due to increased storage requirements
C. Query processing, due to increased latency
D. Embedding computation, due to increased computational load

Solution

  1. Step 1: Analyze the impact of data scale on components

    As data scales, the computational load for embedding computation increases significantly.

  2. Step 2: Eliminate wrong options

    Query processing and network bandwidth are affected but not as critically as embedding computation. Document storage is a concern but not a failure point.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Scale = embedding load [check]
Hint: Scale = embedding load [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard