Bird
Raised Fist0

Model Hub and Model Registry - Versioning, Lineage, Governance

SYSTEM-DESIGNhardInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Versioning Vortex
Started last Friday, ongoing for 3 days.

A recommendation system model serving 5M users daily, using a model registry for version control and governance.

Symptom
Recommendation diversity collapsed, AUC dropped from 0.85 to 0.72 in live traffic, while offline AUC remained stable at 0.86.
📡
Signals
metric
Live AUC dropped significantly after the latest model deployment.
Live AUC dropped from 0.85 to 0.72.
distribution
Distribution of recommended items changed drastically post-deployment.
Top 10 items now account for 80% of recommendations, up from 30%.
feature_importance
Feature importance for 'user_last_active_days' changed significantly.
Importance dropped from 0.18 to 0.02 after last Friday's deploy.
log
Model registry logs show a new model version was deployed last Friday.
Version 3.2 deployed at 10:00 AM on Friday.
latency
Feature extraction latency increased slightly but consistently.
P99 latency increased from 50ms to 70ms.
🔬
Hypotheses
The new model version has a bug causing poor live performance.
Supporting Signals
Live AUC dropped significantlyModel registry logs show a new model version was deployed
Verify: Check the model versioning logs and compare the deployed model's performance with previous versions.
import pandas as pd
log_df = pd.read_csv('model_registry_logs.csv')
log_df[log_df['version'] == '3.2']
Feature distribution shift is causing the model to perform poorly in live traffic.
Supporting Signals
Distribution of recommended items changed drasticallyFeature importance for 'user_last_active_days' changed significantly
Verify: Analyze the feature distributions before and after deployment to identify shifts.
import matplotlib.pyplot as plt
before_deploy = df[df['timestamp'] < '2023-10-20']
after_deploy = df[df['timestamp'] >= '2023-10-20']
plt.hist(before_deploy['user_last_active_days'], bins=50, alpha=0.5, label='Before')
plt.hist(after_deploy['user_last_active_days'], bins=50, alpha=0.5, label='After')
plt.title('Feature Distribution Before and After Deployment')
plt.legend()
Increased feature extraction latency is affecting real-time predictions.
Supporting Signals
Feature extraction latency increased slightly but consistently
Verify: Check the latency logs and correlate with prediction times.
latency_df = pd.read_csv('latency_logs.csv')
latency_df.plot(x='timestamp', y='latency')
🎯
Root Cause
The new model version was trained on a dataset with incorrect feature scaling, leading to poor live performance.
Why It Happened

A bug in the data preprocessing pipeline caused incorrect scaling of features in the training dataset, which was not caught due to insufficient validation checks.

🔧
Fix Strategy
ImmediateRollback to the previous stable model version.
Short-termCorrect the feature scaling in the data preprocessing pipeline and retrain the model.
Long-termImplement automated checks for feature scaling and distribution shifts in the model training pipeline.
🐍
Python Fix
Python
# Correcting feature scaling
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
# Retrain model
model.fit(X_train_scaled, y_train)
✗
Common Wrong Diagnoses
▼ Show
The model is overfitting to the training data.
Why candidates think this: The stable offline AUC suggests overfitting could be an issue.
Why wrong: Overfitting would typically show a discrepancy between training and validation AUC, not just live AUC.
The feature extraction latency is causing prediction delays.
Why candidates think this: Increased latency could affect real-time predictions.
Why wrong: The latency increase is minor and does not correlate with the severity of the AUC drop.
📊
Monitoring to Add
  • Monitor feature distribution changes pre- and post-deployment
  • Add alerts for significant drops in live AUC
  • Track model version performance over time in live traffic
📋
Process Change

Introduce a mandatory checklist for model deployment that includes feature scaling verification.

🏗️
Architectural Change

Enhance the model registry to include automated validation checks for data preprocessing steps.

💡
Key Insight
Model versioning issues often stem from overlooked data preprocessing errors, highlighting the need for robust validation checks.

Practice

(1/5)
1. A bot detection system using device fingerprinting is failing to detect bots effectively. What is the most likely failure mode?
medium
A. The system is not considering user agent strings.
B. The system is not updating fingerprints frequently enough.
C. The system is overly reliant on IP addresses.
D. The system is using outdated behavioral models.

Solution

  1. Step 1: Identify the core issue with device fingerprinting.

    Device fingerprints need frequent updates to remain effective against evolving bots.

  2. Step 2: Eliminate wrong options.

    Behavioral models and IP reliance are not directly related to fingerprinting; user agent strings are not unique enough.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Frequent updates needed for effective fingerprinting [OK]
Hint: Frequent updates needed for effective fingerprinting [OK]
Common Mistakes:
2. In an ad auction system, when should you prefer a simpler linear model over a complex neural network for ranking ads?
medium
A. When interpretability and quick updates are crucial.
B. When the system needs to handle non-linear interactions.
C. When the dataset is extremely large and diverse.
D. When the computational resources are abundant.

Solution

  1. Step 1: Identify the scenario requirements

    In scenarios where interpretability and quick updates are crucial, a simpler linear model is preferred due to its transparency and ease of updating.

  2. Step 2: Eliminate wrong options

    • Extremely large and diverse dataset: Complex models handle this better.
    • Non-linear interactions: Neural networks are better suited.
    • Abundant computational resources: Allows for complex models.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Choose linear for interpretability and updates. [OK]
Hint: Choose linear for interpretability and updates. [OK]
Common Mistakes:
3. In a real-time fraud detection system, should you prioritize model complexity or latency, and why?
medium
A. Model complexity, because it ensures higher accuracy.
B. Latency, because decisions need to be made instantly.
C. Model complexity, because it allows for more features.
D. Latency, because it reduces computational cost.

Solution

  1. Step 1: Consider the system's real-time requirement.

    Real-time systems need quick decisions to prevent fraud.

  2. Step 2: Evaluate the trade-off between complexity and latency.

    Latency is prioritized to ensure timely fraud detection.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Real-time needs instant decisions. [OK]
Hint: Real-time needs instant decisions. [OK]
Common Mistakes:
4. An entity extraction model is failing to recognize entities in new data. What is the most likely cause?
medium
A. The training data lacks diversity
B. The model's hyperparameters are not optimized
C. The model architecture is too simple
D. The model is overfitting to the training data

Solution

  1. Step 1: Identify the symptoms of poor generalization.

    Failure to recognize entities in new data often indicates poor generalization.

  2. Step 2: Evaluate potential causes.

    The model is overfitting to the training data: Overfitting would show high training accuracy but low test accuracy. The model's hyperparameters are not optimized: Hyperparameter optimization affects performance but not generalization directly. The model architecture is too simple: A simple architecture might miss complex patterns but isn't the primary cause of poor generalization. The training data lacks diversity: Lack of diversity in training data leads to poor generalization to new data.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Diverse data ensures better generalization. [OK]
Hint: Diverse data ensures better generalization. [OK]
Common Mistakes:
5. As the number of concurrent experiments increases in an A/B testing platform, which component is most likely to become a bottleneck first?
hard
A. Data Storage Layer
B. Experiment Dashboard
C. Randomization Engine
D. User Segmentation Module

Solution

  1. Step 1: Consider the impact of scaling experiments.

    More experiments generate more data, stressing storage.

  2. Step 2: Eliminate wrong options.

    Dashboard and segmentation handle display and grouping, while randomization is not data-intensive.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    More experiments = more data. [OK]
Hint: More experiments = more data. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard