Bird
Raised Fist0

Why Machine Learning Exists and What Rules Cannot Solve

THEORYbeginnerInterview freq: medium
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Feature Importance Anomaly
Started after the last feature update, ongoing for 3 days

A recommendation system for an e-commerce platform serving 500K users daily

Symptom
Model accuracy is stable, but recall has dropped to 0
📡
Signals
metric
Recall on live data
Recall dropped from 0.75 to 0.00 after the update
feature_importance
Feature importance ranking
Feature 'user_last_active_days' importance dropped from rank #2 (0.18) to rank #15 (0.02) post-deploy
distribution
Distribution of 'user_last_active_days'
Distribution shifted significantly after the update
log
Deployment logs
New feature 'user_last_active_days' calculation method changed
metric
Validation AUC
Validation AUC remains stable at 0.89
🔬
Hypotheses
The new feature calculation is incorrect, affecting recall
Supporting Signals
Recall dropped to 0Feature importance droppedDistribution shifted
Verify: Check the calculation logic of 'user_last_active_days'
df['user_last_active_days'].describe()
The model is overfitting to the training data
Supporting Signals
Validation AUC stableRecall dropped
Verify: Compare training and validation loss curves
plt.plot(history['val_loss'], label='Validation Loss'); plt.plot(history['loss'], label='Training Loss'); plt.legend()
🎯
Root Cause
Incorrect calculation of 'user_last_active_days' feature
Why It Happened

A change in the feature calculation logic during the last update

🔧
Fix Strategy
ImmediateRollback to the previous feature calculation method
Short-termCorrect the calculation logic for 'user_last_active_days'
Long-termImplement a robust feature validation pipeline to catch such errors
🐍
Python Fix
Python
df['user_last_active_days'] = correct_calculation(df)
✗
Common Wrong Diagnoses
▼ Show
The model is overfitting to the training data
Why candidates think this: Stable validation AUC suggests overfitting
Why wrong: The issue is with feature calculation, not model generalization
Data leakage between training and validation sets
Why candidates think this: Stable validation AUC with poor live performance
Why wrong: The problem is isolated to a feature calculation error
📊
Monitoring to Add
  • Monitor recall on live data continuously
  • Track feature importance changes post-deployment
  • Alert on significant distribution shifts in key features
📋
Process Change

Introduce a feature validation step before deployment

🏗️
Architectural Change

Develop a feature calculation module with unit tests

💡
Key Insight
Feature calculation errors can drastically affect model performance, highlighting the need for robust validation processes.

Practice

(1/5)
1. A company implemented an ML model to predict customer churn but found that the model's predictions were not improving business outcomes. What is the most likely root cause?
medium
A. The model is overfitting the training data
B. The business problem was not correctly translated into an ML problem
C. The model's hyperparameters were not optimized
D. The dataset used was too small

Solution

  1. Step 1: Identify the mismatch

    If the model's predictions are not improving business outcomes, it is likely that the business problem was not correctly translated into an ML problem.

  2. Step 2: Eliminate wrong options

    The model is overfitting the training data is incorrect because overfitting would affect prediction accuracy, not business alignment. The model's hyperparameters were not optimized is incorrect because hyperparameter tuning affects model performance, not problem alignment. The dataset used was too small is incorrect because data size affects model accuracy, not problem framing.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Align ML problem with business goals for impact. [OK]
Hint: Align ML problem with business goals for impact. [OK]
Common Mistakes:
2. You are tasked with designing a fraud detection system. How would you ensure that your model does not suffer from data leakage?
hard
A. Ensure that features derived from future transactions are not used in the training set.
B. Include all available transaction data up to the current date in the training set.
C. Use cross-validation with shuffling to ensure robust performance.
D. Use a single train-test split to simplify the evaluation process.

Solution

  1. Step 1: Identify potential leakage sources in fraud detection

    Using future-derived features can cause leakage.

  2. Step 2: Evaluate each option

    Use cross-validation with shuffling to ensure robust performance. might introduce leakage if shuffling is inappropriate, Include all available transaction data up to the current date in the training set. risks future data leakage, Use a single train-test split to simplify the evaluation process. is simplistic and risky.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Avoid future-derived features in training [OK]
Hint: Avoid future-derived features in training [OK]
Common Mistakes:
3. You are tasked with designing a segmentation system for autonomous vehicles that must handle both object detection and segmentation efficiently. Which approach would you choose and why?
hard
A. Semantic segmentation, because it provides class-level information for all pixels.
B. Panoptic segmentation, because it combines both semantic and instance segmentation, providing comprehensive scene understanding.
C. Instance segmentation, because it identifies individual objects, which is crucial for autonomous vehicles.
D. Object detection, because it is faster and sufficient for identifying objects in real-time.

Solution

  1. Step 1: Determine system requirements

    Autonomous vehicles need both object detection and detailed scene understanding.

  2. Step 2: Evaluate segmentation approaches

    Semantic lacks instance detail, instance lacks semantic context, object detection is not segmentation.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Panoptic = Comprehensive scene understanding [OK]
Hint: Panoptic = Comprehensive scene understanding [OK]
Common Mistakes:
4. In a real-time system where predictions need to be made quickly, which approach to linear regression is more suitable and why?
hard
A. Stochastic Gradient Descent because it updates weights more frequently
B. Least Squares because it provides the exact solution
C. Batch Gradient Descent because it processes the entire dataset at once
D. Mini-batch Gradient Descent because it balances speed and accuracy

Solution

  1. Step 1: Consider real-time constraints

    Stochastic Gradient Descent (SGD) updates weights after each data point, allowing for faster, more frequent updates suitable for real-time systems.

  2. Step 2: Eliminate wrong options

    Batch Gradient Descent (A) is too slow for real-time, Least Squares (C) is not iterative, and Mini-batch (D) is a compromise but not as fast as SGD.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    SGD updates weights after each data point. [OK]
Hint: SGD updates weights after each data point. [OK]
Common Mistakes:
5. You need to design a model for a task involving both spatial and temporal hierarchies. Which approach is most suitable?
hard
A. Use a deep CNN-RNN hybrid to capture both spatial and temporal hierarchies.
B. Use a deep CNN for spatial features and ignore temporal features.
C. Use a shallow RNN for temporal features and ignore spatial features.
D. Use a shallow CNN for spatial features and a separate RNN for temporal features.

Solution

  1. Step 1: Identify task requirements

    The task requires capturing both spatial and temporal hierarchies, which suggests a hybrid approach.

  2. Step 2: Eliminate wrong options

    Use a shallow CNN for spatial features and a separate RNN for temporal features. is incorrect because separate models may not capture interactions. Use a shallow RNN for temporal features and ignore spatial features. is incorrect because it ignores spatial features. Use a deep CNN for spatial features and ignore temporal features. is incorrect because it ignores temporal features.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Hybrid models for spatial-temporal tasks. [OK]
Hint: Hybrid models for spatial-temporal tasks. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard