Bird
Raised Fist0

Experiment Tracking - MLflow, W&B, Model Registry, Lineage

THEORYintermediateInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Feature Importance Anomaly
Issue started after the last model deployment, which occurred three days ago.

A recommendation system model serving personalized content to 500k users daily.

Symptom
Model accuracy remains stable, but recall has dropped to 0.0 for a key user segment.
📡
Signals
metric
Recall for the 'new_users' segment in production
Recall dropped from 0.65 to 0.0 after the last deployment
feature_importance
Feature importance ranking in the model registry
Feature 'user_engagement_score' dropped from rank #3 (0.22) to rank #20 (0.01)
log
Deployment logs from MLflow
No errors or warnings during deployment
distribution
Distribution of 'user_engagement_score' in training vs. production data
Training data mean: 0.5, Production data mean: 0.05
code
Feature engineering script in the model lineage
Recent change in feature scaling method for 'user_engagement_score'
🔬
Hypotheses
Feature scaling issue caused 'user_engagement_score' to be misrepresented.
Supporting Signals
Feature importance rankingDistribution of 'user_engagement_score'
Verify: Check if the scaling method applied in production matches the one used during training.
import numpy as np
train_scaled = (train_data['user_engagement_score'] - np.mean(train_data['user_engagement_score'])) / np.std(train_data['user_engagement_score'])
prod_scaled = (prod_data['user_engagement_score'] - np.mean(prod_data['user_engagement_score'])) / np.std(prod_data['user_engagement_score'])
print(f'Training scaled mean: {np.mean(train_scaled)}, Production scaled mean: {np.mean(prod_scaled)}')
Data pipeline issue causing incorrect feature values in production.
Supporting Signals
Distribution of 'user_engagement_score'
Verify: Inspect the data pipeline logs for anomalies in feature extraction.
pipeline_logs = open('pipeline_logs.txt').read()
print('Anomalies' in pipeline_logs)
🎯
Root Cause
Feature scaling method was changed in the production pipeline but not updated in the training pipeline.
Why It Happened

A recent update to the feature engineering script altered the scaling method without corresponding updates to the training process.

🔧
Fix Strategy
ImmediateRollback to the previous model version with known good feature scaling.
Short-termUpdate the production feature engineering script to match the training pipeline's scaling method.
Long-termImplement a version control system for feature engineering scripts to ensure consistency across environments.
🐍
Python Fix
Python
def scale_feature(data):
    return (data - np.mean(data)) / np.std(data)

train_data['user_engagement_score'] = scale_feature(train_data['user_engagement_score'])
prod_data['user_engagement_score'] = scale_feature(prod_data['user_engagement_score'])
✗
Common Wrong Diagnoses
▼ Show
Assuming the model itself is faulty due to zero recall.
Why candidates think this: Zero recall suggests a severe model issue.
Why wrong: The model's accuracy is stable, indicating the issue is likely data-related.
Believing deployment logs guarantee a successful deployment.
Why candidates think this: No errors in logs imply everything is functioning correctly.
Why wrong: Logs do not capture data pipeline or feature engineering discrepancies.
📊
Monitoring to Add
  • Monitor feature distribution changes between training and production data.
  • Set up alerts for significant drops in feature importance rankings.
  • Track and log all changes to feature engineering scripts in a centralized system.
📋
Process Change

Introduce a mandatory review process for changes in feature engineering scripts, ensuring alignment between training and production environments.

🏗️
Architectural Change

Adopt a feature store to manage and version control feature transformations consistently across training and production.

💡
Key Insight
Feature engineering consistency between training and production is crucial for reliable model performance.

Practice

(1/5)
1. Given the following Python code using Word2Vec, what is the output shape of the word vector for 'king'? 
from gensim.models import Word2Vec
model = Word2Vec(sentences, vector_size=100, window=5, min_count=1, workers=4)
vector = model.wv['king']
easy
A. (100, 1)
B. (1, 100)
C. (100,)
D. (1,)

Solution

  1. Step 1: Understand Word2Vec output

    The Word2Vec model returns a 1D array of the specified vector size for each word.

  2. Step 2: Determine the shape

    The vector size is set to 100, so the output shape is (100,).

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Word2Vec outputs 1D vector of size [OK]
Hint: Word2Vec outputs 1D vector of size [OK]
Common Mistakes:
2. Given the following Python code using Naive Bayes, what will be the output?
from sklearn.naive_bayes import GaussianNB
import numpy as np
X = np.array([[1, 2], [2, 3], [3, 4]])
y = np.array([1, 0, 1])
model = GaussianNB()
model.fit(X, y)
print(model.predict([[2, 2]]))
easy
A. [1]
B. [0]
C. [0, 1]
D. Error due to shape mismatch

Solution

  1. Step 1: Understand the Code Execution

    The code fits a Gaussian Naive Bayes model and predicts the class for a new sample [2, 2].

  2. Step 2: Analyze the Prediction

    Given the training data, the model predicts class 1 for [2, 2] based on the likelihoods.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    GaussianNB predicts based on likelihoods. [[check]]
Hint: GaussianNB predicts based on likelihoods. [[check]]
Common Mistakes:
3. In the context of Support Vector Machines, what is the primary purpose of the margin?
easy
A. To minimize the number of support vectors
B. To maximize the distance between the decision boundary and the nearest data points
C. To ensure the model is non-linear
D. To reduce the dimensionality of the data

Solution

  1. Step 1: Understand the role of the margin

    The margin in SVM is the distance between the decision boundary and the nearest data points from either class.

  2. Step 2: Eliminate wrong options

    To minimize the number of support vectors is incorrect because the number of support vectors is not minimized. To ensure the model is non-linear is incorrect because the margin itself does not ensure non-linearity. To reduce the dimensionality of the data is incorrect because the margin does not reduce dimensionality.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Margin maximizes boundary distance [OK]
Hint: Margin maximizes boundary distance [OK]
Common Mistakes:
4. A model trained on a dataset shows unexpected bias against a minority group. What is the most likely root cause?
medium
A. The training data is imbalanced.
B. The learning rate is too high.
C. The model has too many parameters.
D. The model uses a linear activation function.

Solution

  1. Step 1: Identify the symptom.

    Bias against a minority group indicates potential data imbalance.

  2. Step 2: Analyze the options.

    Imbalanced data (B) is a common cause of bias. Other options relate to model complexity or training dynamics, not bias directly.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Imbalanced data often leads to bias. [OK]
Hint: Imbalanced data often leads to bias. [OK]
Common Mistakes:
5. You are tasked with deploying a real-time fraud detection system. Which approach would you choose to handle class imbalance effectively without significantly increasing latency?
hard
A. Real-time oversampling of incoming data
B. Batch processing with undersampling
C. Implementing focal loss in the model
D. Using a weighted loss function

Solution

  1. Step 1: Consider latency constraints.

    Real-time systems require low-latency solutions.

  2. Step 2: Evaluate focal loss.

    Focal loss can be integrated into the model without affecting real-time processing speed.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Focal loss suits real-time needs. [OK]
Hint: Focal loss suits real-time needs. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard