Bird
Raised Fist0

ML Monitoring Platform - Drift Detection, Alerting, Retraining

SYSTEM-DESIGNhardInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
⚖️
Statistical Drift Detection vs Model-based Drift Detection

In a high-frequency trading platform with a 50ms latency SLA, would you use statistical drift detection or model-based drift detection? Justify your choice.

📊
Comparison
DimensionStatistical Drift DetectionModel-based Drift Detection
Detection SpeedHighModerate
Computational CostLowHigh
Detection AccuracyModerateHigh
ScalabilityModerateHigh
InterpretabilityHighModerate
✅
Verdict
Default to statistical drift detection for environments with strict latency requirements and smaller datasets. Switch to model-based detection when you have larger datasets and can afford the computational cost for higher accuracy.
💡
Key Insight
Statistical methods excel in low-latency environments, while model-based methods provide comprehensive detection at a computational cost.
⚠️
Interview Trap
❌ Wrong: Always use model-based drift detection for better accuracy.
Ignoring the computational cost and latency constraints in high-frequency environments.
✅ Consider the trade-off between detection speed and computational cost, especially under strict latency constraints.
Statistical Drift Detection
"Fast detection with minimal computational overhead"
Weaknesses
  • Limited to detecting distributional changes, not performance degradation
  • May trigger false positives in high-dimensional data
  • Requires careful threshold tuning to avoid alert fatigue
Ideal When
  • n < 100K
  • latency < 50ms
  • features < 100
Netflix uses statistical drift detection to monitor changes in user behavior patterns quickly without heavy computational costs.
Model-based Drift Detection
"Detects both distributional changes and performance degradation"
Weaknesses
  • Higher computational cost due to model training
  • Slower detection time compared to statistical methods
  • Requires labeled data for retraining
Ideal When
  • n > 100K
  • labels > 10K
  • sufficient compute resources available
Google uses model-based drift detection to ensure their search algorithms adapt to changing web content and user queries.
🐍
Statistical Drift Detection - Example
▼ Show
Python
from scipy.stats import ks_2samp

def detect_drift(data1, data2):
    stat, p_value = ks_2samp(data1, data2)
    return p_value < 0.05

# Example usage
drift_detected = detect_drift(current_data, baseline_data)
🐍
Model-based Drift Detection - Example
▼ Show
Python
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score

def detect_drift(model, X_train, y_train, X_test, y_test):
    model.fit(X_train, y_train)
    predictions = model.predict(X_test)
    return accuracy_score(y_test, predictions) < 0.8

# Example usage
model = RandomForestClassifier()
drift_detected = detect_drift(model, X_train, y_train, X_test, y_test)
✅
Use Statistical Drift Detection When
  • •when n < 100K
  • •when latency < 50ms
  • •when feature count < 100
✅
Use Model-based Drift Detection When
  • •when n > 100K
  • •when labels > 10K
  • •when you have sufficient compute resources
↳
Nuances
  • Statistical methods may miss performance degradation if the distribution remains unchanged.
  • Model-based methods can be overkill for small datasets due to their computational cost.
  • In high-dimensional spaces, statistical methods may produce false positives due to noise.
  • Model-based methods require retraining, which can introduce latency in detection.

Practice

(1/5)
1. For evaluating an LLM's ability to generate coherent text, which metric is most appropriate?
easy
A. Precision
B. Accuracy
C. BLEU score
D. Recall

Solution

  1. Step 1: Identify metrics for text generation

    BLEU score measures text generation quality by comparing n-grams.

  2. Step 2: Eliminate wrong options

    Accuracy, precision, and recall are more suited for classification tasks, not text coherence.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    BLEU for text coherence, not classification. [OK]
Hint: BLEU for text coherence, not classification. [OK]
Common Mistakes:
2. In a multimodal system that integrates vision and language models, which component is responsible for aligning the semantic meaning between image features and text embeddings?
easy
A. Cross-modal attention layer
B. Text encoder
C. Image encoder
D. Output classifier

Solution

  1. Step 1: Identify the role of each component

    The cross-modal attention layer is designed to align and integrate features from different modalities, such as images and text.

  2. Step 2: Eliminate wrong options

    Image encoder and text encoder process their respective modalities independently. The output classifier is responsible for final predictions, not alignment.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Cross-modal aligns semantics between modalities. [OK]
Hint: Cross-modal aligns semantics between modalities. [OK]
Common Mistakes:
3. In an LLM serving system, what is a likely failure mode if the KV Cache is not properly managed?
medium
A. Inaccurate predictions due to speculative decoding
B. Decreased throughput due to batching inefficiencies
C. Higher memory usage leading to system crashes
D. Increased latency due to redundant computations

Solution

  1. Step 1: Identify the role of KV Cache.

    KV Cache prevents redundant computations, reducing latency.

  2. Step 2: Eliminate wrong options.

    Batching inefficiencies affect throughput, not KV Cache. Memory issues are possible but less direct. Speculative decoding affects accuracy, not KV Cache.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    KV Cache mismanagement leads to redundant computations. [OK]
Hint: KV Cache mismanagement leads to redundant computations. [OK]
Common Mistakes:
4. As the user base of a Guardrails System for LLMs scales 10x, which component is most likely to become a bottleneck first?
hard
A. Model training module
B. User feedback loop
C. Data storage system
D. Content filtering module

Solution

  1. Step 1: Analyze component scalability

    Content filtering must process each request, becoming a bottleneck as requests increase.

  2. Step 2: Eliminate wrong options

    Feedback loops and model training are less frequent, and data storage scales more linearly.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Content filtering scales with user requests. [OK]
Hint: Content filtering scales with user requests. [OK]
Common Mistakes:
5. As the number of images in a visual search system scales up by 10x, which component is most likely to become a bottleneck first?
hard
A. Approximate Nearest Neighbor (ANN) index
B. Network bandwidth
C. Serving layer
D. Image embedding model

Solution

  1. Step 1: Analyze scaling impact

    The ANN index is likely to become a bottleneck as it handles search queries, which increase with more data.

  2. Step 2: Eliminate wrong options

    Embedding model scales with compute, serving layer handles requests, and network bandwidth is less likely to be the first bottleneck.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    ANN index struggles first with scale. [OK]
Hint: ANN index struggles first with scale. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard