Time series with RNN in TensorFlow - Model Metrics & Evaluation

Metrics & Evaluation - Time series with RNN

Which metric matters for Time series with RNN and WHY

For time series prediction using RNNs, common metrics include Mean Squared Error (MSE), Mean Absolute Error (MAE), and Root Mean Squared Error (RMSE). These measure how close the predicted values are to the actual values. Lower values mean better predictions.

When the task is classification on time series data, metrics like accuracy, precision, recall, and F1 score become important.

Choosing the right metric depends on the goal: for continuous value prediction, use error metrics; for classification, use classification metrics.

Confusion matrix example for time series classification

    Actual \ Predicted | Positive | Negative
    -------------------|----------|---------
    Positive           |    50    |   10
    Negative           |    5     |   35

    Total samples = 50 + 10 + 5 + 35 = 100
    

From this matrix:

• True Positives (TP) = 50
• False Positives (FP) = 5
• True Negatives (TN) = 35
• False Negatives (FN) = 10

Precision = TP / (TP + FP) = 50 / (50 + 5) = 0.91

Recall = TP / (TP + FN) = 50 / (50 + 10) = 0.83

F1 Score = 2 * (Precision * Recall) / (Precision + Recall) ≈ 0.87

Precision vs Recall tradeoff with examples

In time series classification, sometimes you want to catch all positive events (high recall), even if some false alarms happen (lower precision). For example, detecting machine failures early.

Other times, you want to be very sure when you predict a positive (high precision), like in medical diagnosis from time series data.

Improving one often lowers the other. You must decide which is more important for your use case.

What "good" vs "bad" metric values look like for time series with RNN

For regression tasks:

• Good: RMSE close to 0 means predictions are very close to actual values.
• Bad: High RMSE means large errors in prediction.

For classification tasks:

• Good: Precision and recall above 0.8 show reliable detection.
• Bad: Precision or recall below 0.5 means many mistakes or missed events.

Common pitfalls in metrics for time series with RNN

• Ignoring temporal order: Metrics should consider time dependencies, not just overall accuracy.
• Data leakage: Using future data in training inflates metrics falsely.
• Overfitting: Very low training error but high test error means model memorizes instead of learns.
• Accuracy paradox: High accuracy can be misleading if classes are imbalanced.

Self-check question

Your RNN model for time series classification has 98% accuracy but only 12% recall on the positive class (e.g., failure events). Is it good for production? Why or why not?

Answer: No, it is not good. The model misses 88% of positive events, which is dangerous if those events are critical. High accuracy is misleading because the dataset is likely imbalanced with many negatives. Improving recall is essential.

Key Result

For time series with RNN, use error metrics for prediction tasks and precision/recall for classification; watch for tradeoffs and data leakage.