ARIMA model basics in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - ARIMA model basics

Which metric matters for ARIMA and WHY

For ARIMA models, we focus on error metrics that show how close the model's predictions are to actual values. Common metrics are Mean Absolute Error (MAE), Mean Squared Error (MSE), and Root Mean Squared Error (RMSE). These metrics measure the size of prediction mistakes in simple terms, helping us understand if the model predicts well over time.

Confusion matrix or equivalent visualization

ARIMA is for continuous values, so confusion matrix does not apply. Instead, we use error tables or plots. For example, a table of actual vs predicted values or a line plot showing both series helps visualize prediction quality.

Time | Actual | Predicted | Error
-----|--------|-----------|-------
t1   | 100    | 98        | 2
t2   | 105    | 107       | -2
t3   | 102    | 101       | 1
    

Tradeoff: Bias vs Variance in ARIMA

ARIMA models balance between being too simple (high bias) and too complex (high variance). A simple model may miss patterns (high error), while a complex model may fit noise (overfit). Choosing the right order (p,d,q) controls this tradeoff. Good error metrics help find this balance.

What "good" vs "bad" metric values look like for ARIMA

Good ARIMA models have low MAE, MSE, and RMSE, meaning predictions are close to actual values. For example, an RMSE of 1.5 on a scale of 100 is good, but 20 is bad. Always compare errors to the scale of your data to judge quality.

Common pitfalls in ARIMA metrics

• Ignoring stationarity: ARIMA assumes data is stationary; if not, errors can be misleading.
• Overfitting: Too many parameters can fit noise, lowering training error but hurting future predictions.
• Using accuracy metrics from classification: ARIMA needs error metrics, not accuracy or precision.
• Not checking residuals: Residuals should look like random noise; patterns mean model issues.

Self-check question

Your ARIMA model has an RMSE of 15 on daily sales data where average sales are around 1000 units. Is this good? Why or why not?

Answer: An RMSE of 15 means the average prediction error is 15 units, which is about 1.5% of average sales. This is quite good because errors are small compared to the scale of data. So, the model predicts well.

Key Result

ARIMA model quality is best judged by error metrics like RMSE, which measure how close predictions are to actual values over time.