Bird
Raised Fist0

ML Monitoring Platform - Drift Detection, Alerting, Retraining

SYSTEM-DESIGNhardInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🤖
ML Monitoring Platform - Drift Detection, Alerting, Retraining
Design a robust ML monitoring system for drift detection and automated retraining.
system-designhardFreq: very_highML Platforms
Prerequisites:Monitoring ML Models in ProductionDistributional Shift
🎯
Problem Framing
Business Objective

Ensure ML models remain accurate and reliable over time to maintain business performance.

ML Objective

Detect data and model drift to trigger alerts and retraining processes.

Constraints
  • Latency SLA for drift detection alerts < 5 minutes
  • High availability with 99.9% uptime
  • Scalable to handle 1000s of models
  • Compliance with data privacy regulations
  • Minimal false positives in drift detection
Clarifying Questions
  • What types of models are we monitoring (e.g., classification, regression)?
  • How frequently do we expect data distributions to change?
  • What is the acceptable threshold for model performance degradation?
  • Are there specific regulatory requirements for data handling?
  • What is the current infrastructure for model serving and monitoring?
Out of Scope
✗ Model training algorithms
✗ Feature engineering processes
✗ User interface design
✗ Non-ML related system monitoring
📊
Offline Metrics
KL Divergence to measure feature distribution drift
Jensen-Shannon Divergence for label distribution shift
📈
Online Metrics
Model accuracy drop over time
Prediction latency increase
⚖️
Trade-offs

Offline metrics may indicate drift without immediate online performance impact, requiring careful threshold setting to avoid unnecessary retraining.

💼
Business Metric

Overall system reliability and uptime

🗄️
Data Sources
  • Real-time feature logs (JSON, real-time)
  • Historical model predictions (CSV, daily)
  • User feedback data (SQL, hourly)
🏷️
Data Properties
Labeling Strategy

Implicit feedback from user interactions to infer label shifts.

Data Volume

Approximately 10 million rows per day, 500 GB/month

Freshness

Features must be updated in real-time to detect drift promptly.

⚠️
Data Challenges
  • Sparse labels due to implicit feedback
  • Position bias in user interactions
  • Survivorship bias in historical data
  • Cold start for new models
🔧
Key Features
numerical
Feature distribution mean
Detects shifts in central tendency of features.
numerical
Feature distribution variance
Captures spread changes in feature distributions.
numerical
Prediction confidence score
Monitors model certainty over time.
numerical
User feedback count
Tracks volume of implicit feedback for label shift detection.
numerical
Time since last retraining
Helps determine retraining frequency.
⚡
Feature Engineering Notes
Real-time vs Batch

Real-time computation for feature distribution metrics; batch for historical analysis.

Feature Store

Required

🐍
Feature Snippet
Python
import numpy as np

def compute_feature_stats(features):
    mean = np.mean(features, axis=0)
    variance = np.var(features, axis=0)
    return mean, variance
🤖
Model Selection
Baseline

Simple statistical tests (e.g., t-test) for drift detection

Proposed Model

Autoencoder for unsupervised drift detection

Autoencoders can capture complex feature interactions and detect subtle distribution shifts.
✗
Alternatives Rejected
Isolation Forest: Rejected due to high false positive rate in high-dimensional data.
PCA-based methods: Rejected due to inability to capture non-linear feature interactions.
🏋️
Training Strategy

Hybrid approach with offline training for initial model and online updates for drift thresholds.

🚀
Serving Infrastructure
LatencyP99 < 100ms for drift detection inference
ThroughputTarget 1000 QPS for drift checks
ArchitectureSingle-stage inference for simplicity and speed
CachingCache recent feature distributions with a TTL of 1 hour to reduce redundant computations.
FallbackFallback to rule-based alerts if model inference fails.
📡
Data Drift Signals
Significant change in feature mean
Increase in feature variance
Shift in feature correlation matrix
📉
Model Drift Signals
Drop in model accuracy by 5%
Increase in prediction confidence variance
Shift in prediction score distribution
🔔
Alerting & Retraining
Retraining Trigger

Model accuracy drop below 90% or significant feature drift detected.

Alerting Strategy

Page on-call engineer for critical drifts, log all drift events, auto-retrain for non-critical drifts.

🎯
Interview Angles
  • How to balance false positives in drift detection with retraining costs?
  • What are the trade-offs between real-time and batch drift detection?
  • How to ensure compliance with data privacy regulations in monitoring?
⚠️
Common Mistakes
Ignoring the impact of false positives in drift detection
Overlooking the need for real-time feature updates
Failing to define clear retraining triggers
Neglecting to consider data privacy regulations
📌
Key Decisions
  • Choosing autoencoders for drift detection due to their ability to capture complex patterns
  • Implementing a hybrid training strategy to balance initial setup and ongoing updates
  • Defining specific thresholds for retraining triggers to avoid unnecessary costs

Practice

(1/5)
1. In an A/B testing platform for ML models, which component is responsible for ensuring that users are consistently assigned to the same variant across multiple sessions?
easy
A. Randomization Engine
B. User Segmentation Module
C. Experiment Dashboard
D. Session Persistence Layer

Solution

  1. Step 1: Identify the role of each component.

    The Session Persistence Layer ensures consistent user assignment across sessions.

  2. Step 2: Eliminate wrong options.

    Randomization Engine assigns variants initially, User Segmentation Module groups users, and Experiment Dashboard displays results.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Session consistency needs persistence. [OK]
Hint: Session consistency needs persistence. [OK]
Common Mistakes:
2. For evaluating an LLM's ability to generate coherent text, which metric is most appropriate?
easy
A. Precision
B. Accuracy
C. BLEU score
D. Recall

Solution

  1. Step 1: Identify metrics for text generation

    BLEU score measures text generation quality by comparing n-grams.

  2. Step 2: Eliminate wrong options

    Accuracy, precision, and recall are more suited for classification tasks, not text coherence.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    BLEU for text coherence, not classification. [OK]
Hint: BLEU for text coherence, not classification. [OK]
Common Mistakes:
3. In a model registry, what is the primary purpose of model versioning?
easy
A. To improve model accuracy over time
B. To ensure models are always deployed
C. To track changes and updates to models
D. To automatically optimize hyperparameters

Solution

  1. Step 1: Understand model versioning purpose

    Model versioning is primarily about tracking changes and updates to models.

  2. Step 2: Eliminate wrong options

    To improve model accuracy over time is incorrect because versioning doesn't directly improve accuracy. To ensure models are always deployed is wrong because versioning doesn't ensure deployment. To automatically optimize hyperparameters is incorrect as versioning isn't about hyperparameter optimization.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Versioning = Tracking changes [OK]
Hint: Versioning = Tracking changes [OK]
Common Mistakes:
4. In an ML platform, when should you prefer batch processing over real-time processing for feature computation?
medium
A. When computational resources are limited and latency is not critical.
B. When the system requires immediate user feedback.
C. When features need to be updated every second.
D. When the model needs to adapt instantly to new data.

Solution

  1. Step 1: Analyze the constraints.

    Batch processing is suitable when resources are limited and latency is not a priority.

  2. Step 2: Eliminate wrong options.

    Options A, B, and D require real-time processing due to their need for immediacy.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Batch for non-critical latency. [OK]
Hint: Batch processing suits non-critical latency scenarios. [OK]
Common Mistakes:
5. At 10x user scale, which component of a feed ranking system is most likely to become a bottleneck first?
hard
A. Network bandwidth
B. Data storage
C. Model training
D. Feature extraction

Solution

  1. Step 1: Analyze scaling impact

    Feature extraction often involves complex computations that scale with data volume.

  2. Step 2: Eliminate wrong options

    Data storage can be scaled with distributed systems, model training is periodic, and network bandwidth is less likely to bottleneck first.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Feature extraction scales with data. [OK]
Hint: Feature extraction scales with data. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard