Bird
Raised Fist0

ML Monitoring Platform - Drift Detection, Alerting, Retraining

SYSTEM-DESIGNhardInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Silent Drift
The issue started two weeks ago and has been persistent since.

A recommendation system serving 5M users daily, designed to suggest personalized content based on user interaction history.

Symptom
Recommendation diversity collapsed, AUC dropped from 0.85 to 0.72 over the past two weeks, and real-time feature latency spiked.
📡
Signals
metric
AUC on live traffic
Dropped from 0.85 to 0.72 over two weeks
distribution
Feature distribution of 'user_activity_level'
Shifted significantly post-deployment, with mean increasing from 0.5 to 0.8
latency
Real-time feature extraction latency
Spiked from P99 < 50ms to P99 = 200ms
feature_importance
Feature 'user_last_active_days' importance
Dropped from 0.18 to 0.02 after last Thursday's deploy
metric
Recommendation diversity index
Collapsed from 0.75 to 0.45
🔬
Hypotheses
Feature distribution shift causing model performance degradation
Supporting Signals
AUC dropFeature distribution shift
Verify: Check if the distribution shift correlates with the AUC drop timeline.
import matplotlib.pyplot as plt
plt.plot(df['date'], df['user_activity_level'].mean())
plt.title('Feature Distribution Over Time')
plt.show()
Increased latency affecting real-time feature extraction
Supporting Signals
Latency spikeAUC drop
Verify: Analyze latency logs to see if spikes correlate with AUC drops.
import matplotlib.pyplot as plt
latency_logs.groupby('timestamp').max().plot()
plt.title('Latency Over Time')
plt.show()
🎯
Root Cause
A recent deployment introduced a bug in the feature extraction pipeline, causing a shift in the 'user_activity_level' feature distribution.
Why It Happened

An upstream data source change was not accounted for in the feature extraction logic, leading to incorrect feature values.

🔧
Fix Strategy
ImmediateRollback to the last known good deployment version.
Short-termPatch the feature extraction logic to handle the upstream data change correctly.
Long-termImplement a robust feature validation step that checks for distribution shifts before deployment.
🐍
Python Fix
Python
# Example fix for feature extraction logic

def extract_features(data):
    # Corrected logic to handle new data source format
    data['user_activity_level'] = data['activity'] / data['max_activity']
    return data

# Apply fix to the pipeline
corrected_data = extract_features(raw_data)
✗
Common Wrong Diagnoses
▼ Show
Model overfitting due to insufficient data
Why candidates think this: AUC drop suggests model generalization issues.
Why wrong: Validation AUC is stable, indicating the model is not overfitting.
Feature importance change indicates feature irrelevance
Why candidates think this: Drop in feature importance suggests it's no longer useful.
Why wrong: Importance drop is a result of distribution shift, not the cause.
📊
Monitoring to Add
  • Real-time feature distribution monitoring
  • Latency anomaly detection
  • AUC live vs. validation discrepancy alerts
📋
Process Change

Introduce a pre-deployment checklist that includes feature distribution checks against historical data.

🏗️
Architectural Change

Design the system to automatically validate feature distributions against a baseline before allowing deployment.

💡
Key Insight
Feature distribution shifts can silently degrade model performance if not continuously monitored.

Practice

(1/5)
1. In an ML platform, which component is primarily responsible for ensuring that features are consistent across training and serving?
easy
A. Model Training Pipeline
B. Serving Infrastructure
C. Data Ingestion Pipeline
D. Feature Store

Solution

  1. Step 1: Identify the role of each component.

    The Feature Store is designed to ensure feature consistency across training and serving.

  2. Step 2: Eliminate wrong options.

    Model Training Pipeline focuses on training models, Serving Infrastructure handles model deployment, and Data Ingestion Pipeline deals with data collection.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Feature Store ensures consistency. [OK]
Hint: Feature Store ensures feature consistency. [OK]
Common Mistakes:
2. In a Retrieval-Augmented Generation (RAG) system, which component is responsible for fetching relevant documents from a large corpus?
easy
A. The retriever model
B. The generator model
C. The encoder model
D. The decoder model

Solution

  1. Step 1: Identify the role of each component.

    The retriever model is designed to fetch relevant documents from a large corpus based on a query.

  2. Step 2: Eliminate wrong options.

    The generator model creates text based on retrieved documents, the encoder processes input data, and the decoder generates output from encoded data.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Retriever fetches, generator creates. [OK]
Hint: Retriever fetches, generator creates. [OK]
Common Mistakes:
3. In an HNSW-based vector search system, what is a likely failure mode if the graph's maximum layer is not properly maintained?
medium
A. Increased precision but decreased recall
B. Increased memory usage without affecting speed
C. Decreased search speed due to inefficient traversal
D. Decreased indexing speed but improved search accuracy

Solution

  1. Step 1: Understand the role of the maximum layer.

    The maximum layer affects traversal efficiency.

  2. Step 2: Eliminate wrong options.

    Precision, recall, and memory usage are not directly affected by the maximum layer.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Max layer affects traversal speed. [OK]
Hint: Max layer affects traversal speed. [OK]
Common Mistakes:
4. At 10x data scale, which component of a model registry is most likely to fail first?
hard
A. Lineage tracking database
B. Governance compliance checks
C. Model deployment pipeline
D. Model versioning system

Solution

  1. Step 1: Consider data scale impact

    Lineage tracking databases often struggle with large data volumes due to the need to store extensive metadata.

  2. Step 2: Eliminate wrong options

    Model versioning system is incorrect as versioning systems are typically designed for scalability. Model deployment pipeline is wrong because deployment pipelines are less affected by data volume. Governance compliance checks is incorrect as compliance checks are not data volume intensive.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Lineage databases struggle with scale [OK]
Hint: Lineage databases struggle with scale [OK]
Common Mistakes:
5. As a recommendation system using neural collaborative filtering scales to 10x more users, which component is most likely to become a bottleneck?
hard
A. Model training time due to increased data volume.
B. Feature extraction pipeline.
C. Real-time recommendation latency.
D. Data storage for user-item interaction history.

Solution

  1. Step 1: Consider the impact of scaling on neural CF.

    Increased data volume significantly affects model training time.

  2. Step 2: Eliminate wrong options.

    Options A and D are less likely bottlenecks as storage and feature extraction can be optimized. Real-time recommendation latency. is affected but not as critically as training time.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Scaling = training time bottleneck [OK]
Hint: Scaling = training time bottleneck [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard