Bird
Raised Fist0

The Weekend Model Collapse

THEORYadvancedInterview freq: very_high
Choose your preparation mode6 modes available
🐛
The Weekend Model Collapse
Started last weekend and has been recurring every weekend since.

A recommendation system for an e-commerce platform serving 5M users daily.

Symptom
Model accuracy remains stable, but recall drops to 0 on weekends.
📡
Signals
metric
Recall on weekend data
Recall drops to 0 on weekends, but is 0.75 on weekdays.
distribution
Distribution of 'user_activity_time' feature
Shifted significantly on weekends compared to weekdays.
feature_importance
Feature importance of 'user_activity_time'
Dropped from 0.22 to 0.05 after last weekend's deploy.
log
Deployment logs
New feature 'user_activity_time' added last Friday.
metric
AUC on validation set
Stable at 0.88, no drop observed.
🔬
Hypotheses
Covariate shift due to 'user_activity_time' feature on weekends.
Distribution of 'user_activity_time'Feature importance of 'user_activity_time'
Verify: Check if the distribution of 'user_activity_time' differs significantly between weekdays and weekends.
import pandas as pd
import matplotlib.pyplot as plt
weekend_data = df[df['day_of_week'].isin(['Saturday', 'Sunday'])]
weekday_data = df[~df['day_of_week'].isin(['Saturday', 'Sunday'])]
plt.hist(weekend_data['user_activity_time'], alpha=0.5, label='Weekend')
plt.hist(weekday_data['user_activity_time'], alpha=0.5, label='Weekday')
plt.legend()
plt.show()
Concept drift affecting model's understanding of user behavior on weekends.
Recall on weekend data
Verify: Analyze if the model's predictions differ significantly on weekends.
weekend_preds = model.predict(weekend_data)
weekday_preds = model.predict(weekday_data)
print('Weekend prediction mean:', weekend_preds.mean())
print('Weekday prediction mean:', weekday_preds.mean())
🎯
Root Cause
Covariate shift due to the 'user_activity_time' feature behaving differently on weekends.

The feature 'user_activity_time' was introduced without considering its distributional shift on weekends.

🔧
Fix Strategy
ImmediateRollback to the previous model version without the 'user_activity_time' feature.
Short-termAdjust the model to handle the distributional shift of 'user_activity_time' on weekends.
Long-termImplement a monitoring system to detect and adapt to distributional shifts in real-time.
🐍
Python Fix
Python
from sklearn.model_selection import train_test_split
weekend_data, weekday_data = train_test_split(df, test_size=0.5, stratify=df['day_of_week'])
model.fit(weekday_data.drop(columns=['user_activity_time']), weekday_data['target'])
✗
Common Wrong Diagnoses
▼ Show
The model is overfitting to the training data.
Why candidates think this: The stable AUC on validation data suggests overfitting.
Why wrong: Overfitting would affect both weekdays and weekends, not just weekends.
The model has a bug in the weekend data processing pipeline.
Why candidates think this: The issue only occurs on weekends, suggesting a data pipeline issue.
Why wrong: The data pipeline logs show no errors, and the issue aligns with feature distribution changes.
📊
Monitoring to Add
  • Real-time monitoring of feature distributions across different days of the week.
  • Alerts for significant drops in recall or precision on specific days.
  • Automated checks for feature importance changes post-deployment.
📋
Process Change

Introduce a feature impact analysis step before deploying new features.

🏗️
Architectural Change

Design models to be robust to known distributional shifts, such as temporal changes.

💡
Key Insight
Distributional shifts can cause significant performance drops even when overall accuracy remains stable.

Practice

(1/5)
1.
import torch
x = torch.tensor([-1.0, 0.0, 1.0])
output = torch.nn.functional.relu(x)
print(output)
What is the output of this code?
easy
A. tensor([0.0, 0.0, 1.0])
B. tensor([-1.0, 0.0, 1.0])
C. tensor([0.0, 0.0, 0.0])
D. tensor([-1.0, 0.0, 0.0])

Solution

  1. Step 1: Understand ReLU behavior.

    ReLU outputs zero for negative inputs and the input itself for non-negative inputs.
  2. Step 2: Apply ReLU to each element.

    For -1.0, output is 0.0; for 0.0, output is 0.0; for 1.0, output is 1.0.
  3. Final Answer:

    Option A -> Option A
  4. Quick Check:

    ReLU zeros negatives, keeps positives. [OK]
Hint: ReLU zeros negatives, keeps positives. [OK]
Common Mistakes:
2. In the context of Graph Neural Networks, which component is primarily responsible for aggregating information from neighboring nodes?
easy
A. Dropout layer
B. Activation function
C. Message passing mechanism
D. Batch normalization

Solution

  1. Step 1: Identify the role of each component

    The message passing mechanism is designed to aggregate information from neighboring nodes in a graph.
  2. Step 2: Eliminate wrong options

    Activation functions, dropout layers, and batch normalization do not specifically aggregate information from neighboring nodes.
  3. Final Answer:

    Option C -> Option C
  4. Quick Check:

    Message passing aggregates neighbor info in GNNs. [check]
Hint: Message passing aggregates neighbor info in GNNs. [check]
Common Mistakes:
3. Given the following Python code snippet, what will be the output?
import numpy as np

data = np.array([1, 2, 3, 4, 5])

# Simulate offline data processing
processed_data = data * 2

print(processed_data)
easy
A. [1, 2, 3, 4, 5]
B. [2, 4, 6, 8, 10]
C. [1, 4, 9, 16, 25]
D. Error: Data processing requires real-time input

Solution

  1. Step 1: Understand the code operation

    The code multiplies each element of the array by 2.
  2. Step 2: Calculate the expected output

    Each element in the array [1, 2, 3, 4, 5] is multiplied by 2, resulting in [2, 4, 6, 8, 10].
  3. Final Answer:

    Option B -> Option B
  4. Quick Check:

    Array * 2 doubles each element. [OK]
Hint: Array * 2 doubles each element. [OK]
Common Mistakes:
4. In the context of why machine learning exists, which of the following problems is most suitable for a machine learning approach rather than a rule-based system?
easy
A. Predicting the next word in a sentence
B. Calculating the sum of a list of numbers
C. Sorting a list of integers
D. Checking if a number is prime

Solution

  1. Step 1: Identify the nature of the problem.

    Predicting the next word in a sentence involves understanding context and patterns, which are complex and not easily defined by rules.
  2. Step 2: Eliminate rule-based problems.

    Options A, C, and D are deterministic and can be solved with explicit algorithms.
  3. Final Answer:

    Option A -> Option A
  4. Quick Check:

    ML for patterns, rules for logic. [OK]
Hint: ML excels at pattern recognition, not deterministic logic. [OK]
Common Mistakes:
5. You notice that your XGBoost model is overfitting despite using early stopping. What could be the root cause?
medium
A. The max depth of trees is too high, capturing noise in the data.
B. The learning rate is too low, causing the model to converge slowly.
C. The number of boosting rounds is too low, leading to underfitting.
D. The regularization parameters are set too high, restricting model complexity.

Solution

  1. Step 1: Identify overfitting causes

    Overfitting can occur if the model is too complex, such as having trees with high max depth.
  2. Step 2: Eliminate wrong options

    The learning rate is too low, causing the model to converge slowly. is incorrect because a low learning rate would not cause overfitting. The number of boosting rounds is too low, leading to underfitting. is incorrect because too few rounds would cause underfitting, not overfitting. The regularization parameters are set too high, restricting model complexity. is incorrect because high regularization would prevent overfitting, not cause it.
  3. Final Answer:

    Option A -> Option A
  4. Quick Check:

    High max depth = Overfitting risk [OK]
Hint: High max depth = Overfitting risk [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard