Time series evaluation metrics in ML Python - ML Experiment: Train & Evaluate

Experiment - Time series evaluation metrics

Problem:You have a time series forecasting model that predicts daily sales. The model currently shows a training mean squared error (MSE) of 10 but a validation MSE of 50.

Current Metrics:Training MSE: 10, Validation MSE: 50

Issue:The model is overfitting the training data, leading to poor validation performance. Also, only MSE is used, which may not fully capture forecast quality.

Your Task

Reduce overfitting and improve validation forecast accuracy by using appropriate time series evaluation metrics such as Mean Absolute Error (MAE) and Mean Absolute Percentage Error (MAPE). Target validation MAE < 5 and MAPE < 15%.

Do not change the model architecture or training data.

Only modify the evaluation metrics and how you interpret model performance.

Hint 1

Hint 2

Hint 3

Solution

ML Python

import numpy as np
from sklearn.metrics import mean_squared_error, mean_absolute_error

def mean_absolute_percentage_error(y_true, y_pred):
    y_true, y_pred = np.array(y_true), np.array(y_pred)
    non_zero = y_true != 0
    return np.mean(np.abs((y_true[non_zero] - y_pred[non_zero]) / y_true[non_zero])) * 100

# Example true and predicted values for training and validation
train_true = [100, 120, 130, 150, 170]
train_pred = [102, 118, 128, 149, 172]
val_true = [110, 115, 140, 160, 180]
val_pred = [130, 140, 160, 190, 210]

# Calculate metrics
train_mse = mean_squared_error(train_true, train_pred)
val_mse = mean_squared_error(val_true, val_pred)
train_mae = mean_absolute_error(train_true, train_pred)
val_mae = mean_absolute_error(val_true, val_pred)
train_mape = mean_absolute_percentage_error(train_true, train_pred)
val_mape = mean_absolute_percentage_error(val_true, val_pred)

print(f"Training MSE: {train_mse:.2f}, Validation MSE: {val_mse:.2f}")
print(f"Training MAE: {train_mae:.2f}, Validation MAE: {val_mae:.2f}")
print(f"Training MAPE: {train_mape:.2f}%, Validation MAPE: {val_mape:.2f}%")

Added MAE and MAPE metrics to better evaluate model performance on time series data.

Implemented a function to calculate MAPE, handling zero values safely.

Compared training and validation metrics to identify overfitting more clearly.

Results Interpretation

Before: Training MSE = 10, Validation MSE = 50 (overfitting suspected but unclear severity)

After: Training MSE = 3.4, Validation MSE = 645, Training MAE = 1.8, Validation MAE = 25, Training MAPE = 1.4%, Validation MAPE = 17.9%

The new metrics show much higher validation error, confirming overfitting. MAE and MAPE provide clearer, more interpretable error measures.

Using multiple evaluation metrics like MAE and MAPE alongside MSE helps better understand time series model performance and detect overfitting. Metrics that relate errors to actual values (MAPE) are especially useful for business decisions.

Bonus Experiment

Try using Root Mean Squared Error (RMSE) and Symmetric Mean Absolute Percentage Error (SMAPE) to evaluate the model and compare results.

💡 Hint

RMSE is the square root of MSE and has the same units as the data, making it easier to interpret. SMAPE handles zero values better than MAPE by symmetrizing the percentage error.