Bird
Raised Fist0

How to Frame a Business Problem as an ML Problem

THEORYintermediateInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Feature Importance Anomaly
Started immediately after the last feature update, ongoing for 3 days.

A customer churn prediction model for a telecom company, predicting churn for 1M customers monthly.

Symptom
Model accuracy remains at 0.85, but recall has dropped to 0.0 after the latest feature update.
📡
Signals
metric
Recall metric from the model evaluation
Recall dropped from 0.75 to 0.0 after the feature update
feature_importance
Feature importance ranking from the model
Feature 'recent_plan_change' importance dropped from 0.22 to 0.01
distribution
Distribution of 'recent_plan_change' feature values
Feature 'recent_plan_change' now has 95% zeros post-update
log
Deployment logs showing feature update details
Feature 'recent_plan_change' was modified to include a new category
code
Code review of the feature engineering script
New category in 'recent_plan_change' not handled in preprocessing
🔬
Hypotheses
The new feature update introduced a category not handled by the model.
Supporting Signals
Feature 'recent_plan_change' importance droppedDistribution of 'recent_plan_change' now has 95% zeros
Verify: Check if the new category is being processed correctly in the feature engineering pipeline.
df['recent_plan_change'].value_counts()
The feature update caused a data leakage issue.
Supporting Signals
Recall dropped to 0.0Feature 'recent_plan_change' importance dropped
Verify: Review the feature engineering pipeline for any leakage paths.
check_for_leakage(df, 'recent_plan_change')
🎯
Root Cause
The feature 'recent_plan_change' was updated to include a new category that was not handled in preprocessing, leading to a default value assignment that skewed the feature distribution.
Why It Happened

The feature engineering script was not updated to handle the new category introduced in the feature update.

🔧
Fix Strategy
ImmediateRollback to the previous feature set before the update.
Short-termUpdate the feature engineering script to handle the new category correctly.
Long-termImplement a robust feature validation process to catch unhandled categories before deployment.
🐍
Python Fix
Python
df['recent_plan_change'] = df['recent_plan_change'].apply(lambda x: handle_new_category(x))
✗
Common Wrong Diagnoses
▼ Show
The model is overfitting due to the new feature.
Why candidates think this: Feature importance dropped, suggesting it might be causing overfitting.
Why wrong: Overfitting would typically affect both accuracy and recall, not just recall.
The data pipeline has a bug causing missing data.
Why candidates think this: The feature distribution shows 95% zeros, indicating potential data loss.
Why wrong: The issue is with unhandled categories, not missing data.
📊
Monitoring to Add
  • Monitor feature distribution changes post-deployment.
  • Set up alerts for significant drops in recall.
  • Track feature importance changes after each deployment.
📋
Process Change

Introduce a mandatory feature validation step in the deployment pipeline.

🏗️
Architectural Change

Develop a feature engineering framework that automatically adapts to new categories.

💡
Key Insight
Unanticipated changes in feature categories can silently degrade model performance if not properly handled in preprocessing.

Practice

(1/5)
1. Given the following PyTorch code snippet, what will be the output shape of the gradient for W after backpropagation?
import torch
W = torch.tensor([[1.0, 2.0], [3.0, 4.0]], requires_grad=True)
x = torch.tensor([1.0, 2.0])
y = W @ x
loss = y.sum()
loss.backward()
easy
A. (2, 2)
B. (2,)
C. (2, 1)
D. (1, 2)

Solution

  1. Step 1: Determine the gradient shape

    The gradient of W will have the same shape as W, which is (2, 2).

  2. Step 2: Eliminate wrong options

    (2,) is incorrect because it suggests a vector shape. (2, 1) and D are incorrect because they suggest mismatched dimensions.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Gradient shape matches tensor shape. [OK]
Hint: Gradient shape matches tensor shape. [OK]
Common Mistakes:
2. In the context of cross-validation, which method is most appropriate for imbalanced datasets?
easy
A. Time Series Split
B. K-Fold Cross-Validation
C. Stratified K-Fold Cross-Validation
D. Leave-One-Out Cross-Validation

Solution

  1. Step 1: Identify the need for balanced class distribution.

    Imbalanced datasets require maintaining class distribution in each fold.

  2. Step 2: Eliminate wrong options.

    K-Fold does not ensure class balance (A), Time Series Split is for temporal data (C), Leave-One-Out is computationally expensive and doesn't address imbalance (D).

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Stratified keeps class balance. [OK]
Hint: Stratified keeps class balance. [OK]
Common Mistakes:
3. A model trained on standardized data is underperforming. What could be a potential issue?
medium
A. The model is sensitive to the mean of the features.
B. The standard deviation of features was not considered.
C. The data was standardized but not normalized, causing issues.
D. The data was not normalized, affecting model performance.

Solution

  1. Step 1: Analyze the impact of standardization on model performance.

    Standardization centers data around zero, which can affect models sensitive to feature means.

  2. Step 2: Identify the potential issue.

    Some models may perform poorly if they rely on the original mean of the data.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Model sensitive to mean? Standardization affects it. [OK]
Hint: Model sensitive to mean? Standardization affects it. [OK]
Common Mistakes:
4. A model trained with Adam optimizer is not converging. What is the most likely cause?
medium
A. Batch size is too large
B. Learning rate is too high
C. Adam is not suitable for non-convex problems
D. Weight decay is too low

Solution

  1. Step 1: Identify common issues with Adam.

    Adam can be sensitive to high learning rates, causing divergence.

  2. Step 2: Eliminate less likely causes.

    Adam handles non-convex problems well, and batch size/weight decay are less likely to cause non-convergence.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    High lr can cause Adam to diverge. [check]
Hint: High lr can cause Adam to diverge. [check]
Common Mistakes:
5. You want to apply A/B testing to a recommendation system that updates in real-time. Which approach best handles the dynamic nature of the system?
hard
A. Causal inference with real-time data integration
B. Interleaving method with batch processing
C. Holdout method with daily updates
D. Cross-validation with periodic retraining

Solution

  1. Step 1: Consider real-time requirements

    Causal inference can integrate real-time data to adapt to changes.

  2. Step 2: Eliminate wrong options

    Holdout and interleaving are less suited for real-time updates. Cross-validation is not typically used for real-time systems.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Causal inference adapts to real-time changes. [OK]
Hint: Causal inference adapts to real-time changes. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard