Bird
Raised Fist0

The Anomaly Detection Drift

SYSTEM-DESIGNhardInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Anomaly Detection Drift
Started two weeks ago, ongoing

Anomaly detection system using Isolation Forest for fraud detection in a financial transactions platform processing 5M transactions/day.

Symptom
AUC dropped from 0.92 to 0.75 over the past two weeks · Recommendation diversity collapsed, with similar transactions flagged repeatedly · Real-time feature latency spiked from 50ms to 200ms · A/B test shows offline AUC at 0.91 but online AUC at 0.75
📡
Signals
metric
AUC on live traffic
Dropped from 0.92 to 0.75
distribution
Distribution of transaction amounts
Shifted significantly with more high-value transactions
feature_importance
Feature 'transaction_time_of_day' importance
Dropped from 0.15 to 0.03 after last model update
latency
Real-time feature extraction latency
Increased from 50ms to 200ms
log
Model update logs
New autoencoder model deployed two weeks ago
🔬
Hypotheses
Feature distribution shift causing model degradation
Supporting Signals
Distribution of transaction amountsAUC on live traffic
Verify: Check if the feature distribution in training data matches current live data
df_live['transaction_amount'].hist(), df_train['transaction_amount'].hist()
Feature importance change affecting model performance
Supporting Signals
Feature 'transaction_time_of_day' importance
Verify: Analyze feature importance before and after the model update
plot_feature_importance_change(old_model, new_model)
🎯
Root Cause
Feature distribution shift not accounted for in the new autoencoder model
Why It Happened

The new autoencoder model was trained on outdated data that did not reflect the current transaction distribution.

🔧
Fix Strategy
ImmediateRollback to the previous Isolation Forest model
Short-termRetrain the autoencoder model with updated data reflecting current distribution
Long-termImplement a continuous data pipeline that updates training data with recent transactions
🐍
Python Fix
Python
autoencoder.fit(updated_training_data)
✗
Common Wrong Diagnoses
▼ Show
Model overfitting due to complex architecture
Why candidates think this: Complex models can overfit, especially with autoencoders
Why wrong: The issue was not overfitting but a distribution shift not captured in training
Feature extraction latency causing prediction delays
Why candidates think this: Increased latency could impact real-time predictions
Why wrong: Latency was due to logging, not affecting prediction accuracy
📊
Monitoring to Add
  • Real-time monitoring of feature distribution shifts
  • Alerts for significant drops in online AUC
  • Latency monitoring for feature extraction processes
📋
Process Change

Regularly update training datasets with recent data to reflect current distributions

🏗️
Architectural Change

Adopt a data versioning system to ensure models are trained on the most relevant data

💡
Key Insight
Always ensure training data reflects the current live distribution to prevent model drift.

Practice

(1/5)
1. For a session-based recommendation system, which metric is most appropriate to evaluate the model's ability to predict the next item in a sequence?
easy
A. Precision@K
B. Mean Absolute Error (MAE)
C. Root Mean Square Error (RMSE)
D. F1 Score

Solution

  1. Step 1: Identify metrics for ranking tasks

    Precision@K is used to evaluate the relevance of the top-K recommended items.

  2. Step 2: Eliminate metrics for regression/classification

    MAE and RMSE are for regression; F1 Score is for classification.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Precision@K for ranking relevance. [OK]
Hint: Precision@K for ranking relevance. [OK]
Common Mistakes:
2. A delivery forecasting model is consistently overestimating ETAs during peak traffic hours. What is the most likely failure mode?
medium
A. The model is not accounting for real-time traffic data
B. The model has too many parameters
C. The model is underfitting the data
D. The model is overfitting to the training data

Solution

  1. Step 1: Identify the cause of overestimation during peak hours

    Real-time traffic data is crucial for accurate ETA predictions.

  2. Step 2: Eliminate wrong options

    Underfitting (A) and overfitting (D) are general issues, not specific to peak hours. Too many parameters (C) would cause overfitting, not specific time errors.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Real-time data is key for peak accuracy. [OK]
Hint: Real-time data is key for peak accuracy. [OK]
Common Mistakes:
3. In a pairwise learning to rank system, what is a common failure mode if the model is not performing well?
medium
A. The model is overfitting due to too many features
B. The model is not capturing the relative order of item pairs
C. The model is not considering the entire list of items
D. The model is using an inappropriate loss function for regression

Solution

  1. Step 1: Identify the failure mode specific to pairwise ranking.

    Pairwise models can fail if they don't capture the relative order of item pairs.

  2. Step 2: Eliminate wrong options.

    Overfitting and inappropriate loss functions are generic issues, and pairwise doesn't consider the entire list, making C incorrect.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Pairwise = focus on item pairs [OK]
Hint: Pairwise = focus on item pairs [OK]
Common Mistakes:
4. A session-based recommendation model is underperforming. Which of the following is a likely failure mode specific to sequential modeling?
medium
A. Ignoring the order of interactions.
B. Data leakage from test to train set.
C. Using a too small learning rate.
D. Overfitting due to too many features.

Solution

  1. Step 1: Identify sequential modeling issues

    Ignoring the order of interactions can lead to poor performance in sequential models.

  2. Step 2: Eliminate generic issues

    Overfitting and data leakage are general issues; learning rate affects convergence speed, not sequence handling.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Order matters in sequences. [OK]
Hint: Order matters in sequences. [OK]
Common Mistakes:
5. In a visual search system, when should you prefer using an Approximate Nearest Neighbor (ANN) index over an exact search method?
medium
A. When low latency is crucial
B. When the dataset is small
C. When exact matches are critical
D. When the search space is one-dimensional

Solution

  1. Step 1: Consider the trade-offs

    ANN is preferred when low latency is crucial because it speeds up search at the cost of some accuracy.

  2. Step 2: Eliminate wrong options

    Exact matches require exact search, small datasets don't benefit much from ANN, and one-dimensional search doesn't need ANN complexity.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    ANN for speed, not exactness. [OK]
Hint: ANN for speed, not exactness. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard