Bird
Raised Fist0

Regression Metrics - MAE, MSE, RMSE, MAPE, Huber - When to Use Which

THEORYbeginnerInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Misleading Metric Mystery
Started after the last model update two weeks ago.

A house price prediction model serving a real estate platform with 500K daily users.

Symptom
Model predictions have high variance, with some predictions being off by a large margin, despite a low RMSE.
📡
Signals
metric
RMSE on validation set
RMSE = 15000, which is lower than expected given the prediction errors observed.
metric
MAE on validation set
MAE = 12000, indicating larger errors than RMSE suggests.
distribution
Distribution of prediction errors
Long tail with some errors exceeding 50,000.
feature_importance
Feature importance for 'location_quality'
Importance dropped from 0.25 to 0.05 after the update.
log
Model update logs
No significant changes in feature engineering or model architecture.
🔬
Hypotheses
The model is overly sensitive to outliers, skewing RMSE.
Supporting Signals
RMSE on validation setMAE on validation setDistribution of prediction errors
Verify: Check the distribution of errors to see if a few large errors are affecting RMSE.
import matplotlib.pyplot as plt; plt.hist(predictions - actuals, bins=50); plt.show()
Feature 'location_quality' is underrepresented, affecting predictions.
Supporting Signals
Feature importance for 'location_quality'
Verify: Analyze the correlation between 'location_quality' and prediction errors.
df['error'] = predictions - actuals; df.groupby('location_quality')['error'].mean().plot()
🎯
Root Cause
The model's reliance on RMSE masked the impact of outliers, leading to misleading performance metrics.
Why It Happened

The update prioritized RMSE without considering its sensitivity to outliers, which were present in the data.

🔧
Fix Strategy
ImmediateSwitch to using MAE for immediate evaluation to better capture error magnitude.
Short-termRe-evaluate the model using Huber loss to reduce outlier sensitivity.
Long-termImplement a robust evaluation framework that includes multiple metrics for comprehensive assessment.
🐍
Python Fix
Python
from sklearn.metrics import mean_absolute_error; mae = mean_absolute_error(actuals, predictions); print('MAE:', mae)
✗
Common Wrong Diagnoses
▼ Show
The model architecture is flawed.
Why candidates think this: High variance in predictions suggests a model issue.
Why wrong: Logs confirmed no architectural changes; the issue was metric-related.
Data preprocessing error.
Why candidates think this: Feature importance drop suggests preprocessing issues.
Why wrong: Preprocessing was unchanged; the problem was with metric interpretation.
📊
Monitoring to Add
  • Regularly monitor MAE alongside RMSE to capture error magnitude.
  • Track distribution of prediction errors to identify outliers.
  • Implement alerts for significant drops in feature importance.
📋
Process Change

Incorporate a metric review step in the model update process to ensure appropriate metric selection.

🏗️
Architectural Change

Design the evaluation pipeline to automatically compare multiple metrics and highlight discrepancies.

💡
Key Insight
Choosing the right regression metric is crucial, as some metrics can mask significant prediction errors due to their sensitivity to outliers.

Practice

(1/5)
1. In the context of A/B testing for ML models, which method is best suited for ensuring that temporal trends do not bias the results?
easy
A. Holdout method
B. Interleaving method
C. Causal inference
D. Cross-validation

Solution

  1. Step 1: Understand the methods

    Interleaving method allows for simultaneous comparison, reducing temporal bias.

  2. Step 2: Eliminate wrong options

    Holdout and cross-validation do not address temporal trends. Causal inference is more about understanding causality than handling temporal trends.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Interleaving reduces temporal bias. [OK]
Hint: Interleaving reduces temporal bias. [OK]
Common Mistakes:
2. In the context of the bias-variance tradeoff, which of the following is a characteristic of a high-bias model?
easy
A. It tends to underfit the training data.
B. It has high flexibility and can fit the training data well.
C. It has high variance and is sensitive to noise in the training data.
D. It performs well on both training and unseen data.

Solution

  1. Step 1: Identify characteristics of high-bias models

    High-bias models are typically simple and underfit the training data.

  2. Step 2: Eliminate wrong options

    It has high flexibility and can fit the training data well. describes high-variance models. It has high variance and is sensitive to noise in the training data. is incorrect as it describes high-variance models. It performs well on both training and unseen data. is incorrect because high-bias models do not perform well on training data.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    High-bias models underfit. [OK]
Hint: High-bias models underfit. [OK]
Common Mistakes:
3. 
import numpy as np
from sklearn.neighbors import NearestNeighbors

X = np.array([[1, 2], [3, 4], [5, 6]])
knn = NearestNeighbors(n_neighbors=2)
knn.fit(X)
distances, indices = knn.kneighbors([[1, 2]])
print(indices)
What does this code output?
easy
A. [[0, 2]]
B. [[1, 0]]
C. [[0, 1]]
D. [[2, 1]]

Solution

  1. Step 1: Understand kneighbors output

    The code finds the 2 nearest neighbors of [1, 2] in X.

  2. Step 2: Calculate distances

    [1, 2] is closest to itself (index 0) and then to [3, 4] (index 1).

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Self is always nearest. [OK]
Hint: Self is always nearest. [OK]
Common Mistakes:
4. In a scenario where computational resources are limited, which approach is more suitable for sequence-to-sequence tasks and why?
medium
A. Use RNNs because they require less memory than attention mechanisms.
B. Use RNNs because they are faster to train than attention mechanisms.
C. Use attention mechanisms because they allow for parallel processing.
D. Use attention mechanisms because they are more interpretable than RNNs.

Solution

  1. Step 1: Evaluate computational efficiency.

    Attention mechanisms allow for parallel processing, making them more efficient with limited resources.

  2. Step 2: Eliminate wrong options.

    Use RNNs because they require less memory than attention mechanisms. is incorrect as attention mechanisms can be more memory efficient due to parallelism. Use RNNs because they are faster to train than attention mechanisms. is incorrect because RNNs are slower due to sequential processing. Use attention mechanisms because they are more interpretable than RNNs. is incorrect as interpretability is not the main concern here.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Parallel processing saves resources. [OK]
Hint: Parallel processing saves resources. [OK]
Common Mistakes:
5. A K-Means model is not converging on a dataset. What is the most likely cause?
medium
A. Centroids initialized too close to each other
B. Too many clusters specified
C. Data is not normalized
D. Using Euclidean distance metric

Solution

  1. Step 1: Identify convergence issues

    Centroids initialized too close can cause slow convergence or getting stuck in local minima.

  2. Step 2: Eliminate other causes

    Too many clusters can cause overfitting but not convergence issues. Data normalization affects distance calculation but not convergence. Euclidean distance is standard for K-Means.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Close centroids = slow convergence [OK]
Hint: Close centroids = slow convergence [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard