Bird
Raised Fist0

Regression Metrics - MAE, MSE, RMSE, MAPE, Huber - When to Use Which

THEORYbeginnerInterview freq: high
Choose your preparation mode6 modes available
โ–ถ
Execution Trace
Step 1/5
MAE calculation

Calculating MAE for predictions [3, 5, 2] and actuals [2, 5, 4] results in MAE = 1.0

๐ŸŽฏ MAE is useful for understanding average error magnitude without outlier influence.
import numpy as np
y_true = np.array([2, 5, 4])
y_pred = np.array([3, 5, 2])
mae = np.mean(np.abs(y_true - y_pred))
MSE calculation

Calculating MSE for predictions [3, 5, 2] and actuals [2, 5, 4] results in MSE = 2.67

๐ŸŽฏ MSE penalizes larger errors more than smaller ones, making it sensitive to outliers.
mse = np.mean((y_true - y_pred) ** 2)
RMSE calculation

Calculating RMSE for predictions [3, 5, 2] and actuals [2, 5, 4] results in RMSE = 1.63

๐ŸŽฏ RMSE provides error in the same units as the target variable, useful for interpretation.
rmse = np.sqrt(mse)
MAPE calculation

Calculating MAPE for predictions [3, 5, 2] and actuals [2, 5, 4] results in MAPE = 0.25

๐ŸŽฏ MAPE is useful for understanding error in percentage terms but struggles with zero values.
mape = np.mean(np.abs((y_true - y_pred) / y_true))
Huber calculation

Calculating Huber loss with ฮด=1 for predictions [3, 5, 2] and actuals [2, 5, 4] results in Huber = 1.0

๐ŸŽฏ Huber loss is robust to outliers and combines the benefits of MAE and MSE.
delta = 1.0
huber_loss = np.where(np.abs(y_true - y_pred) <= delta,
                     0.5 * (y_true - y_pred) ** 2,
                     delta * (np.abs(y_true - y_pred) - 0.5 * delta))
huber = np.mean(huber_loss)
๐Ÿ“Š
Diagram
Diagram
โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚                      Regression Metrics                      โ”‚
โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚ MAE  โ”€โ”€> |y - ลท|                                             โ”‚
โ”‚ MSE  โ”€โ”€> (y - ลท)ยฒ                                            โ”‚
โ”‚ RMSE โ”€โ”€> โˆš((y - ลท)ยฒ)                                         โ”‚
โ”‚ MAPE โ”€โ”€> |(y - ลท) / y|                                       โ”‚
โ”‚ Huber โ”€> { 0.5 * (y - ลท)ยฒ, if |y - ลท| <= ฮด                   โ”‚
โ”‚          { ฮด * (|y - ลท| - 0.5 * ฮด), otherwise                โ”‚
โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚ MAE: Robust to outliers, interpretable as average error      โ”‚
โ”‚ MSE: Penalizes larger errors, sensitive to outliers          โ”‚
โ”‚ RMSE: Similar to MSE, but in original units                  โ”‚
โ”‚ MAPE: Percentage error, problematic with zero values         โ”‚
โ”‚ Huber: Combines MAE and MSE, robust to outliers              โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
  • MAE: Mean Absolute Error, robust to outliers
  • MSE: Mean Squared Error, sensitive to outliers
  • RMSE: Root Mean Squared Error, in original units
  • MAPE: Mean Absolute Percentage Error, problematic with zeros
  • Huber: Combines MAE and MSE, robust to outliers
๐Ÿ”‘
Key Moments
โ–ฒ
Q: Why is MAE considered robust to outliers?
A: MAE calculates the average of absolute errors, treating all errors equally, thus not disproportionately affected by large errors.
Q: What makes Huber loss a combination of MAE and MSE?
A: Huber loss uses a quadratic loss for small errors (like MSE) and linear loss for large errors (like MAE), providing robustness to outliers.
Q: Why might MAPE be problematic when actual values are zero?
A: MAPE involves division by the actual value, which can lead to undefined or infinite values when the actual value is zero.

This visual trace document explains the use of different regression metrics: MAE, MSE, RMSE, MAPE, and Huber. Each metric has unique characteristics that make it suitable for specific scenarios. MAE is robust to outliers, MSE penalizes larger errors, RMSE provides error in original units, MAPE gives percentage error but struggles with zeros, and Huber combines MAE and MSE benefits. Understanding these metrics helps in selecting the right one based on data characteristics and the specific needs of the analysis.

Primary Focus

โœ“ Understanding when to use different regression metrics

Practice

(1/5)
1.
from sklearn.metrics import precision_score, recall_score, f1_score

y_true = [0, 1, 1, 0, 1, 1, 0]
y_pred = [0, 1, 0, 0, 1, 1, 1]

precision = precision_score(y_true, y_pred)
recall = recall_score(y_true, y_pred)
f1 = f1_score(y_true, y_pred)

print(f"Precision: {precision}, Recall: {recall}, F1: {f1}")
What will be the output of this code?
easy
A. Precision: 0.75, Recall: 0.75, F1: 0.75
B. Precision: 0.67, Recall: 0.67, F1: 0.67
C. Precision: 0.50, Recall: 0.67, F1: 0.57
D. Precision: 0.67, Recall: 0.50, F1: 0.57

Solution

  1. Step 1: Calculate precision.

    Precision = TP / (TP + FP) = 2 / (2 + 1) = 0.67
  2. Step 2: Calculate recall.

    Recall = TP / (TP + FN) = 2 / (2 + 2) = 0.50
  3. Step 3: Calculate F1 score.

    F1 = 2 * (Precision * Recall) / (Precision + Recall) = 0.57
  4. Final Answer:

    Option D -> Option D
  5. Quick Check:

    Precision and recall calculations are key. [OK]
Hint: Precision and recall calculations are key. [OK]
Common Mistakes:
2. In the context of handling class imbalance, which technique is specifically designed to adjust the loss function to focus more on hard-to-classify examples?
easy
A. Oversampling
B. Undersampling
C. Cost-sensitive Learning
D. Focal Loss

Solution

  1. Step 1: Identify the technique that modifies the loss function.

    Focal Loss is designed to focus more on hard-to-classify examples by adjusting the loss function.
  2. Step 2: Eliminate wrong options.

    Oversampling and undersampling adjust the dataset, not the loss function. Cost-sensitive learning modifies the cost, not the focus on hard examples.
  3. Final Answer:

    Option D -> Option D
  4. Quick Check:

    Focal Loss focuses on hard examples. [OK]
Hint: Focal Loss focuses on hard examples. [OK]
Common Mistakes:
3.
import torch
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
for epoch in range(10):
    for data, target in dataloader:
        optimizer.zero_grad()
        output = model(data)
        loss = loss_fn(output, target)
        loss.backward()
        optimizer.step()
print(optimizer.param_groups[0]['lr'])
What will be printed after the loop?
easy
A. 0.0
B. 0.01
C. 0.1
D. Error

Solution

  1. Step 1: Understand the code.

    The learning rate is set to 0.1 and not changed in the loop.
  2. Step 2: Check for learning rate updates.

    SGD without a scheduler keeps the learning rate constant.
  3. Final Answer:

    Option C -> Option C
  4. Quick Check:

    SGD without scheduler keeps lr constant. [check]
Hint: SGD without scheduler keeps lr constant. [check]
Common Mistakes:
4. When comparing two ML models using A/B testing, why might you choose causal inference over the holdout method?
medium
A. Holdout method is less accurate.
B. Causal inference is faster to implement.
C. Causal inference provides insights into why one model performs better.
D. Holdout method cannot be used with large datasets.

Solution

  1. Step 1: Understand the methods

    Causal inference helps understand the cause-effect relationship, not just performance.
  2. Step 2: Eliminate wrong options

    Holdout is not inherently less accurate or unsuitable for large datasets. Causal inference is not necessarily faster.
  3. Final Answer:

    Option C -> Option C
  4. Quick Check:

    Causal inference explains why, not just what. [OK]
Hint: Causal inference explains why, not just what. [OK]
Common Mistakes:
5. You need to reduce the dimensionality of a dataset with 10,000 features for a real-time application. Which technique is more suitable and why?
medium
A. t-SNE, because it captures non-linear relationships effectively.
B. PCA, because it is computationally efficient for large datasets.
C. UMAP, because it is faster than t-SNE and captures global structure.
D. LDA, because it reduces dimensions based on class labels.

Solution

  1. Step 1: Consider computational efficiency

    Real-time applications require fast computations.
  2. Step 2: Evaluate each option

    t-SNE is computationally expensive, UMAP is faster but still slower than PCA, and LDA is not suitable for unsupervised tasks.
  3. Final Answer:

    Option B -> Option B
  4. Quick Check:

    PCA for speed in large datasets [OK]
Hint: PCA for speed in large datasets [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard