Regression Metrics - MAE, MSE, RMSE, MAPE, Huber - When to Use Which

THEORYbeginnerInterview freq: high

Choose your preparation mode6 modes available

Learn Trace Debug Trade-off Challenge

🤖

Choosing the right regression metric for your model's needs

theorybeginnerFreq: highEvaluation

Prerequisites:Linear Regression

💡

Why It Exists

Regression metrics were developed to quantify how well a model's predictions match the actual data. Different metrics highlight different aspects of prediction errors, such as absolute differences, squared differences, or percentage errors, allowing for nuanced evaluation depending on the problem context.

📖

What It Is

Regression metrics are tools used to measure the accuracy of a model's predictions. Think of them as different lenses through which you can view the performance of your model, each highlighting different types of errors.

∑

Math Intuition

MAE measures the average magnitude of errors without considering their direction, making it intuitive and easy to interpret. MSE squares the errors, penalizing larger errors more heavily, which can be useful when large errors are particularly undesirable. RMSE is the square root of MSE, bringing the error back to the original scale of the data. MAPE expresses errors as a percentage, which can be more interpretable in some contexts. Huber loss combines the best of MAE and MSE, being less sensitive to outliers than MSE but more robust than MAE for small errors.

⚙️

Core Mechanism

Each metric calculates the error between predicted and actual values in a unique way, emphasizing different aspects of the error.

📋

Step by Step

Calculate the difference between each predicted and actual value.
Apply the specific transformation for the metric (e.g., absolute value for MAE, square for MSE).
Aggregate these transformed errors (e.g., sum or mean).
For some metrics, apply a final transformation (e.g., square root for RMSE).
Output the aggregated error as the metric value.

🎯

Intuition

Imagine you are a teacher grading exams. MAE is like counting how many points each student missed, MSE is like squaring those missed points to penalize larger mistakes more, RMSE is taking the square root of that penalty to bring it back to the original scale, MAPE is looking at the percentage of points missed relative to the total, and Huber is a mix of MAE and MSE, being gentle on small errors but harsh on large ones.

🐍

Python Example

Python

import numpy as np

y_true = np.array([3, -0.5, 2, 7])
y_pred = np.array([2.5, 0.0, 2, 8])

mae = np.mean(np.abs(y_true - y_pred))
mse = np.mean((y_true - y_pred) ** 2)
rmse = np.sqrt(mse)
mape = np.mean(np.abs((y_true - y_pred) / y_true)) * 100

# Huber loss with delta = 1
huber_loss = np.where(np.abs(y_true - y_pred) <= 1, 0.5 * (y_true - y_pred) ** 2, np.abs(y_true - y_pred) - 0.5)
huber = np.mean(huber_loss)

print(mae, mse, rmse, mape, huber)  # output: 0.5, 0.375, 0.612372, 35.714285714285715, 0.3125

✅

When It Works

Use MAE when you want a simple, interpretable metric that treats all errors equally.
Use MSE when large errors are particularly costly and you want to penalize them more.
Use RMSE when you want a metric that is on the same scale as the data and penalizes large errors.
Use MAPE when you need a percentage error, especially useful when dealing with data of varying scales.
Use Huber when you want a balance between sensitivity to outliers and robustness to small errors.

⚠️

When It Breaks

MAE can be misleading when the data has outliers, as it treats all errors equally.
MSE can overly penalize models in the presence of outliers, leading to skewed evaluations.
RMSE, like MSE, is sensitive to outliers and can give a distorted view of model performance if large errors are present.
MAPE can be problematic when actual values are close to zero, leading to infinite or undefined percentage errors.
Huber loss requires tuning of the delta parameter, which can be non-trivial and affect performance if not set correctly.

⇄

vs R-squared

Use This When

You need a direct measure of prediction error magnitude.

Use Alternative When

You want to understand the proportion of variance explained by the model.

🎯

Interview Angles

Interviewer asks: 'Why might you choose RMSE over MAE in a regression task?' → testing understanding of error penalization and scale.
Interviewer asks: 'How would you handle a regression problem with significant outliers?' → assessing knowledge of robust metrics like Huber loss.
Interviewer asks: 'Explain the limitations of using MAPE in a regression model.' → evaluating awareness of percentage error pitfalls with small actual values.

⚠️

Common Mistakes

Believing MSE and RMSE are interchangeable without considering scale differences, leading to misinterpretation of error magnitude.

Assuming MAPE is always better for interpretability, which can lead to issues with zero or near-zero actual values.

Thinking Huber loss is universally better without understanding the need for delta parameter tuning, which can result in suboptimal model evaluation.

📌

Summary

Choose MAE for simplicity and equal treatment of errors, but be cautious with outliers.
Use MSE or RMSE when large errors are particularly undesirable, but watch for outliers.
Consider Huber loss for a balanced approach to error sensitivity and robustness.

Practice

(1/5)

1. In the context of monitoring ML models in production, which of the following best describes data drift?

easy

A. A sudden drop in model accuracy due to a bug in the code.

B. An increase in the model's training time due to larger datasets.

C. A gradual change in the statistical properties of the input data.

D. A decrease in the model's interpretability over time.

Solution

Step 1: Define data drift
Data drift refers to changes in the statistical properties of input data over time.
Step 2: Eliminate wrong options
A sudden drop in model accuracy due to a bug in the code. describes a bug, not data drift. An increase in the model's training time due to larger datasets. is about computational resources, not data properties. A decrease in the model's interpretability over time. is unrelated to data drift.
Final Answer:
Option C -> Option C
Quick Check:
Data drift = change in data stats [OK]

Hint: Data drift = change in data stats [OK]

Common Mistakes:

import torch
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
for epoch in range(10):
    for data, target in dataloader:
        optimizer.zero_grad()
        output = model(data)
        loss = loss_fn(output, target)
        loss.backward()
        optimizer.step()
print(optimizer.param_groups[0]['lr'])

What will be printed after the loop?

easy

A. 0.0

B. 0.01

C. 0.1

D. Error

Solution

Step 1: Understand the code.
The learning rate is set to 0.1 and not changed in the loop.
Step 2: Check for learning rate updates.
SGD without a scheduler keeps the learning rate constant.
Final Answer:
Option C -> Option C
Quick Check:
SGD without scheduler keeps lr constant. [check]

Hint: SGD without scheduler keeps lr constant. [check]

Common Mistakes:

3. In the context of statistical significance testing in ML, which of the following best describes a p-value?

easy

A. The probability of observing the data given that the null hypothesis is true.

B. The probability that the null hypothesis is true.

C. The probability of the alternative hypothesis being true.

D. The probability of making a Type I error.

Solution

Step 1: Understand p-value definition
A p-value is the probability of observing the data, or something more extreme, given that the null hypothesis is true.
Step 2: Eliminate wrong options
The probability that the null hypothesis is true. is incorrect because p-value is not the probability that the null hypothesis is true. The probability of the alternative hypothesis being true. is incorrect because it does not relate to the alternative hypothesis. The probability of making a Type I error. is incorrect because it confuses p-value with Type I error probability.
Final Answer:
Option A -> Option A
Quick Check:
P-value is about data given null hypothesis [OK]

Hint: P-value is about data given null hypothesis [OK]

Common Mistakes:

4. You have a dataset with a large number of features and limited computational resources. Which feature selection method is most appropriate?

medium

A. Wrapper method

B. Embedded method

C. Recursive feature elimination

D. Filter method

Solution

Step 1: Consider computational efficiency.
Filter methods are computationally inexpensive as they do not involve training models.
Step 2: Eliminate wrong options.
Wrapper methods and recursive feature elimination are computationally intensive. Embedded methods require model training, which can be resource-heavy.
Final Answer:
Option D -> Option D
Quick Check:
Filter = low cost, no model. [OK]

Hint: Filter methods are computationally cheap. [OK]

Common Mistakes:

5. You need to design a model for a task involving both spatial and temporal hierarchies. Which approach is most suitable?

hard

A. Use a deep CNN-RNN hybrid to capture both spatial and temporal hierarchies.

B. Use a deep CNN for spatial features and ignore temporal features.

C. Use a shallow RNN for temporal features and ignore spatial features.

D. Use a shallow CNN for spatial features and a separate RNN for temporal features.

Solution

Step 1: Identify task requirements
The task requires capturing both spatial and temporal hierarchies, which suggests a hybrid approach.
Step 2: Eliminate wrong options
Use a shallow CNN for spatial features and a separate RNN for temporal features. is incorrect because separate models may not capture interactions. Use a shallow RNN for temporal features and ignore spatial features. is incorrect because it ignores spatial features. Use a deep CNN for spatial features and ignore temporal features. is incorrect because it ignores temporal features.
Final Answer:
Option A -> Option A
Quick Check:
Hybrid models for spatial-temporal tasks. [OK]

Hint: Hybrid models for spatial-temporal tasks. [OK]

Common Mistakes:

Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard

Regression Metrics - MAE, MSE, RMSE, MAPE, Huber - When to Use Which

Practice

Solution

Step 1: Define data drift

Step 2: Eliminate wrong options

Final Answer:

Quick Check:

Solution

Step 1: Understand the code.

Step 2: Check for learning rate updates.

Final Answer:

Quick Check:

Solution

Step 1: Understand p-value definition

Step 2: Eliminate wrong options

Final Answer:

Quick Check:

Solution

Step 1: Consider computational efficiency.

Step 2: Eliminate wrong options.

Final Answer:

Quick Check:

Solution

Step 1: Identify task requirements

Step 2: Eliminate wrong options

Final Answer:

Quick Check:

Ready to prove it?