Bird
Raised Fist0

The ML Development Lifecycle - From Problem to Production

THEORYbeginnerInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Feature Importance Anomaly
Started after the latest feature deployment last Friday.

A recommendation system model for an e-commerce platform serving 500k users daily.

Symptom
Model accuracy remains stable, but recall has dropped to 0.
📡
Signals
metric
Recall metric from the production environment
Recall dropped from 0.75 to 0.00 after last Friday's deploy
feature_importance
Feature importance ranking from the model
Feature 'user_last_active_days' importance dropped from rank #2 (0.18) to rank #15 (0.02) post-deploy
distribution
Distribution of 'user_last_active_days' feature in the training vs. production data
Production data shows 'user_last_active_days' is mostly zero
log
Logs from the feature engineering pipeline
Error in feature extraction script for 'user_last_active_days'
code
Recent changes in the feature engineering code
New logic added to 'user_last_active_days' calculation
🔬
Hypotheses
Feature 'user_last_active_days' is not being calculated correctly in production.
Supporting Signals
Feature importance dropDistribution anomalyLog error
Verify: Check the feature engineering pipeline for errors in 'user_last_active_days' calculation.
df['user_last_active_days'].describe()
The new feature logic introduced a bug affecting 'user_last_active_days'.
Supporting Signals
Code changeLog error
Verify: Review the recent code changes for 'user_last_active_days'.
git diff HEAD~1 HEAD -- feature_engineering.py
🎯
Root Cause
A bug in the feature engineering script caused 'user_last_active_days' to be zero for all users.
Why It Happened

A recent code change introduced a logic error in the feature calculation.

🔧
Fix Strategy
ImmediateRollback the feature engineering script to the previous version.
Short-termCorrect the logic error in the 'user_last_active_days' calculation.
Long-termImplement automated tests for feature engineering scripts to catch such errors before deployment.
🐍
Python Fix
Python
def calculate_user_last_active_days(user_data):
    # Corrected logic
    return (current_date - user_data['last_active_date']).days

# Apply fix
df['user_last_active_days'] = df.apply(calculate_user_last_active_days, axis=1)
✗
Common Wrong Diagnoses
▼ Show
The model is overfitting to the training data.
Why candidates think this: The stable accuracy might suggest overfitting.
Why wrong: Overfitting would not cause recall to drop to zero suddenly after a deployment.
The data pipeline is missing recent data.
Why candidates think this: A sudden drop in recall could be due to missing data.
Why wrong: The issue is specific to a feature calculation error, not missing data.
📊
Monitoring to Add
  • Monitor feature distribution changes in production vs. training data.
  • Set up alerts for drastic drops in feature importance.
  • Implement logging for feature engineering errors.
📋
Process Change

Introduce code reviews specifically for feature engineering changes.

🏗️
Architectural Change

Adopt a feature store to manage and validate feature calculations centrally.

💡
Key Insight
Feature engineering errors can drastically affect model performance even if accuracy metrics appear stable.

Practice

(1/5)
1. 
from sklearn.tree import DecisionTreeClassifier
import numpy as np

X = np.array([[0, 0], [1, 1], [1, 0], [0, 1]])
y = np.array([0, 1, 1, 0])

clf = DecisionTreeClassifier(criterion='gini')
clf.fit(X, y)

print(clf.predict([[0, 0], [1, 1]]))
What is the output of the above code?
easy
A. [0, 0]
B. [1, 0]
C. [0, 1]
D. [1, 1]

Solution

  1. Step 1: Understand the training data.

    The data is perfectly separable, and the decision tree will learn to classify based on the given labels.

  2. Step 2: Predict using the trained model.

    For input [0, 0], the label is 0, and for [1, 1], the label is 1.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Check training labels for predictions. [OK]
Hint: Check training labels for predictions. [OK]
Common Mistakes:
2. In the context of high-dimensional data, which of the following is a primary reason why the curse of dimensionality affects nearest neighbor algorithms?
easy
A. Distances between points become less meaningful as dimensions increase.
B. High-dimensional data increases the risk of overfitting.
C. High-dimensional data requires more computational resources.
D. High-dimensional data leads to more complex decision boundaries.

Solution

  1. Step 1: Understand the curse of dimensionality.

    In high dimensions, the distance between points becomes less informative.

  2. Step 2: Eliminate wrong options.

    High-dimensional data increases the risk of overfitting. is about overfitting, not distance. High-dimensional data requires more computational resources. is about computation, not distance. High-dimensional data leads to more complex decision boundaries. is about decision boundaries, not distance.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    High dimensions dilute distance meaning. [OK]
Hint: High dimensions dilute distance meaning. [OK]
Common Mistakes:
3. When comparing two ML models using A/B testing, why might you choose causal inference over the holdout method?
medium
A. Holdout method is less accurate.
B. Causal inference is faster to implement.
C. Causal inference provides insights into why one model performs better.
D. Holdout method cannot be used with large datasets.

Solution

  1. Step 1: Understand the methods

    Causal inference helps understand the cause-effect relationship, not just performance.

  2. Step 2: Eliminate wrong options

    Holdout is not inherently less accurate or unsuitable for large datasets. Causal inference is not necessarily faster.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Causal inference explains why, not just what. [OK]
Hint: Causal inference explains why, not just what. [OK]
Common Mistakes:
4. A model trained on offline data is underperforming when deployed in a real-time environment. What is the most likely cause?
medium
A. The model was overfitted to the offline data.
B. The offline data was too diverse.
C. The model lacks the ability to process real-time data.
D. The model's hyperparameters were not optimized.

Solution

  1. Step 1: Identify the mismatch between training and deployment

    Models trained on offline data may not handle real-time data well due to differences in data characteristics.

  2. Step 2: Eliminate unrelated causes

    Options A, C, and D do not specifically address the real-time processing capability issue.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Offline-trained models may struggle with real-time data. [OK]
Hint: Offline-trained models may struggle with real-time data. [OK]
Common Mistakes:
5. In a recommendation system, how would you apply feature engineering to improve the model's ability to capture user-item interactions?
hard
A. Create interaction terms between user and item features.
B. Apply one-hot encoding to user and item IDs.
C. Use collaborative filtering to generate features.
D. Normalize user and item features separately.

Solution

  1. Step 1: Identify the need for capturing interactions.

    Recommendation systems benefit from understanding user-item interactions.

  2. Step 2: Evaluate feature engineering techniques.

    Collaborative filtering (A) is a model, not feature engineering. One-hot encoding (B) doesn't capture interactions. Normalizing (D) doesn't enhance interaction capture. Interaction terms (C) explicitly model user-item interactions, improving the system's ability to recommend.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Interaction terms enhance user-item interaction capture. [OK]
Hint: Interaction terms enhance user-item interaction capture. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard