Time series evaluation metrics in ML Python - Deep Dive

Overview - Time series evaluation metrics

What is it?

Time series evaluation metrics are ways to measure how well a model predicts data points that change over time. These metrics compare the model's predictions to the actual values to see how close they are. They help us understand if the model is good at capturing patterns like trends or seasonality. Without these metrics, we wouldn't know if our time-based predictions are useful or just random guesses.

Why it matters

Time series data is everywhere, like weather, stock prices, or sales over months. If we can't measure how well our models predict this data, we might make bad decisions, like ordering too much stock or missing a weather warning. These metrics help us trust and improve our models, making real-world systems smarter and safer. Without them, predictions would be guesses without proof.

Where it fits

Before learning time series evaluation metrics, you should understand basic time series concepts like trends, seasonality, and how models make predictions. After this, you can learn how to improve models using these metrics or explore advanced topics like anomaly detection or forecasting with uncertainty.

Mental Model

Core Idea

Time series evaluation metrics measure how close a model's predictions are to actual time-ordered data points, helping us judge prediction quality over time.

Think of it like...

It's like checking how well a weather forecast matches the actual weather each day; the better the match, the more reliable the forecast.

Time series data:  ┌───────────────┐
                  │ Actual values  │
                  └──────┬────────┘
                         │
                  ┌──────▼────────┐
                  │ Model predicts │
                  └──────┬────────┘
                         │
                  ┌──────▼────────┐
                  │ Evaluation    │
                  │ Metrics      │
                  └──────────────┘

Build-Up - 7 Steps

1

Foundation

Understanding time series data basics

Concept: Introduce what time series data is and why it is special compared to other data types.

Time series data is a sequence of data points collected or recorded at regular time intervals, like daily temperatures or monthly sales. Unlike random data, time series data has an order and often shows patterns like trends (up or down over time) and seasonality (repeating cycles). Understanding this helps us know why we need special ways to check predictions.

Result

You can recognize time series data and understand its unique features like order and patterns.

Knowing the special nature of time series data is key to choosing the right evaluation methods that respect time order and patterns.

2

Foundation

Why evaluate predictions in time series

Concept: Explain the purpose of evaluation metrics in measuring prediction accuracy over time.

When a model predicts future values, we want to know how close those predictions are to what actually happens. Evaluation metrics give us numbers that summarize this closeness. Because time series data changes over time, we need metrics that consider the order and size of errors to judge if the model is useful.

Result

You understand that evaluation metrics help compare predicted and actual values to measure model quality.

Realizing that evaluation is about trust and improvement helps focus on metrics that reflect meaningful errors in time.

3

Intermediate

Common error metrics: MAE and MSE

🤔Before reading on: do you think Mean Absolute Error (MAE) or Mean Squared Error (MSE) punishes big mistakes more? Commit to your answer.

Concept: Introduce two basic error metrics: MAE and MSE, explaining their differences and uses.

Mean Absolute Error (MAE) calculates the average of absolute differences between predicted and actual values. It treats all errors equally. Mean Squared Error (MSE) squares the errors before averaging, so bigger errors count more. For example, if a prediction is off by 10, MSE counts it as 100, making the model focus on avoiding big mistakes.

Result

You can calculate and interpret MAE and MSE to understand prediction errors.

Knowing how squaring errors changes focus helps choose metrics based on whether big mistakes or average errors matter more.

4

Intermediate

Scale-free metrics: MAPE and SMAPE

🤔Before reading on: do you think percentage error metrics like MAPE work well when actual values are zero? Commit to your answer.

Concept: Explain metrics that express errors as percentages, making them easier to compare across different scales.

Mean Absolute Percentage Error (MAPE) shows error as a percentage of actual values, making it easy to understand. However, it can be problematic when actual values are zero or near zero, causing huge or undefined percentages. Symmetric MAPE (SMAPE) fixes this by using the average of actual and predicted values in the denominator, reducing extreme values and making it more stable.

Result

You understand how to use percentage-based error metrics and their limitations.

Recognizing scale issues in errors helps pick metrics that fairly compare predictions across different value ranges.

5

Intermediate

Evaluating direction: Directional Accuracy

🤔Before reading on: do you think a model that predicts the exact value but wrong direction scores high on directional accuracy? Commit to your answer.

Concept: Introduce a metric that checks if the model predicts the correct direction of change, not just the size of errors.

Directional Accuracy measures how often the model correctly predicts whether the value goes up or down compared to the previous time point. For example, if sales increased last month and the model predicts an increase, that's a correct direction. This metric is useful when the direction matters more than exact values, like in stock trading.

Result

You can assess if a model captures the trend direction even if exact values differ.

Understanding direction helps evaluate models where knowing up or down is more important than precise numbers.

6

Advanced

Handling seasonality with seasonal metrics

🤔Before reading on: do you think standard error metrics capture seasonal pattern errors well? Commit to your answer.

Concept: Explain how some metrics adjust for repeating seasonal patterns to better evaluate models on seasonal data.

Seasonal metrics like Seasonal Mean Absolute Error (SMAE) compare predictions to actual values considering seasonal cycles. For example, sales in December might always be higher. A model that misses this pattern will have higher seasonal errors. These metrics help detect if the model understands and predicts seasonal changes correctly.

Result

You can evaluate models on their ability to capture seasonal patterns, not just overall error.

Knowing seasonal metrics prevents misleading evaluations when data has repeating cycles.

7

Expert

Advanced metrics: CRPS and probabilistic evaluation

🤔Before reading on: do you think point prediction errors fully capture uncertainty in forecasts? Commit to your answer.

Concept: Introduce metrics that evaluate probabilistic forecasts, measuring how well models predict ranges or probabilities, not just single values.

Continuous Ranked Probability Score (CRPS) measures the quality of probabilistic forecasts, which give a range or distribution of possible future values. Unlike point errors, CRPS rewards models that correctly express uncertainty, which is crucial in real-world decisions where risk matters. For example, predicting a 70% chance of rain is more informative than just predicting rain or no rain.

Result

You understand how to evaluate models that predict uncertainty, not just fixed values.

Appreciating probabilistic metrics expands evaluation beyond accuracy to include confidence and risk.

Under the Hood

Time series evaluation metrics work by comparing each predicted value to the actual value at the same time point, then aggregating these differences into a single number. Metrics like MAE sum absolute differences, while MSE squares them to emphasize larger errors. Percentage metrics normalize errors by actual values to handle scale differences. Directional metrics check if the sign of change matches. Probabilistic metrics compare predicted distributions to actual outcomes, often integrating over possible values.

Why designed this way?

These metrics were designed to capture different aspects of prediction quality: size of errors, scale independence, direction correctness, and uncertainty. Early metrics like MAE and MSE were simple and easy to compute, but lacked nuance for time series patterns. Percentage and directional metrics evolved to address scale and trend issues. Probabilistic metrics arose from the need to handle uncertainty in forecasts, especially in fields like weather and finance.

┌───────────────┐      ┌───────────────┐      ┌───────────────┐
│ Actual values │─────▶│ Compare errors│─────▶│ Aggregate into│
└───────────────┘      └───────────────┘      │ metric value  │
                                               └───────────────┘
       ▲                      ▲                      ▲
       │                      │                      │
  Time ordered           Different error         Different
   data points            calculations          aggregation
                           (abs, squared,       (mean, sum, etc.)

Myth Busters - 4 Common Misconceptions

Quick: Does a low MSE always mean the model predicts trends well? Commit yes or no.

Common Belief: A low Mean Squared Error means the model perfectly captures all patterns including trends and seasonality.

Reality: MSE measures average squared errors but does not guarantee the model captures trends or seasonal patterns well. A model can have low MSE but still miss important time-based behaviors.

Why it matters: Relying only on MSE can lead to trusting models that fail to predict important changes, causing poor decisions in practice.

Quick: Can MAPE be used safely when actual values are zero? Commit yes or no.

Common Belief: Mean Absolute Percentage Error (MAPE) works well for all time series data regardless of actual values.

Reality: MAPE is undefined or unstable when actual values are zero or near zero, causing misleadingly large errors.

Why it matters: Using MAPE blindly can cause wrong conclusions about model quality, especially in data with zeros like demand or counts.

Quick: Does directional accuracy measure how close predicted values are to actual values? Commit yes or no.

Common Belief: Directional Accuracy tells how close the predicted values are to actual values numerically.

Reality: Directional Accuracy only measures if the predicted direction (up or down) matches actual direction, ignoring the size of errors.

Why it matters: Confusing directional accuracy with error size can lead to overestimating model usefulness when exact values matter.

Quick: Is evaluating probabilistic forecasts the same as evaluating point predictions? Commit yes or no.

Common Belief: Evaluating probabilistic forecasts uses the same metrics as point predictions like MAE or MSE.

Reality: Probabilistic forecasts require special metrics like CRPS that consider the full predicted distribution, not just single values.

Why it matters: Using point metrics on probabilistic forecasts ignores uncertainty, leading to poor risk assessment.

Expert Zone

1

Some metrics are sensitive to outliers (like MSE) while others (like MAE) are more robust; choosing depends on error tolerance.

2

Directional metrics can be combined with error metrics to get a fuller picture of model performance in trend-sensitive applications.

3

Probabilistic metrics require careful calibration of forecast distributions; a well-calibrated model balances sharpness and reliability.

When NOT to use

Avoid using MAPE or percentage-based metrics when actual values can be zero or very small; instead, use SMAPE or scale-independent metrics. For models where uncertainty matters, do not rely solely on point error metrics; use probabilistic evaluation. Directional accuracy is not suitable when exact values are critical, such as inventory management.

Production Patterns

In production, teams often monitor multiple metrics simultaneously, like MAE for average error and directional accuracy for trend correctness. Probabilistic forecasts are common in weather and finance, evaluated with CRPS. Seasonal metrics are used in retail forecasting to capture holiday effects. Automated alerts trigger when metrics degrade, signaling model retraining.

Connections

Regression evaluation metrics

Time series metrics build on regression metrics by adding time order and scale considerations.

Understanding regression metrics like MAE and MSE helps grasp time series metrics since they extend these ideas to ordered data.

Risk management in finance

Probabilistic time series metrics relate to risk measures by quantifying uncertainty in forecasts.

Knowing how forecast uncertainty is measured helps in financial risk decisions, linking time series evaluation to risk management.

Quality control in manufacturing

Directional accuracy is similar to detecting trends in quality measurements over time.

Recognizing trend correctness in time series connects to spotting shifts in manufacturing processes, improving defect detection.

Common Pitfalls

#1 Using MAPE on data with zero values causes infinite or huge errors.

Wrong approach:

errors = abs((actual - predicted) / actual) * 100 # fails if actual == 0

Correct approach:

errors = abs(actual - predicted) / ((abs(actual) + abs(predicted)) / 2) * 100 # SMAPE handles zeros

Root cause:

Misunderstanding that dividing by zero or near-zero actual values breaks MAPE calculation.

#2 Ignoring direction of change and only minimizing error size.

Wrong approach:

Use only MAE or MSE without checking if model predicts up/down trends correctly.

Correct approach:

Combine MAE with Directional Accuracy metric to evaluate both error size and trend correctness.

Root cause:

Assuming numeric closeness alone guarantees useful predictions in time series.

#3 Evaluating probabilistic forecasts with point error metrics.

Wrong approach:

Calculate MAE between median forecast and actual value, ignoring forecast spread.

Correct approach:

Use CRPS or similar metrics that compare full forecast distribution to actual outcomes.

Root cause:

Not recognizing that uncertainty information requires different evaluation methods.

Key Takeaways

Time series evaluation metrics measure how well models predict ordered data points over time, considering error size, direction, and scale.

Basic metrics like MAE and MSE quantify average errors but differ in sensitivity to large mistakes.

Percentage-based metrics help compare errors across scales but can fail with zero values, requiring alternatives like SMAPE.

Directional accuracy evaluates if models predict the correct trend direction, important when direction matters more than exact values.

Advanced metrics like CRPS assess probabilistic forecasts, capturing uncertainty beyond point predictions.