nn.RNN layer in PyTorch - Model Metrics & Evaluation

Metrics & Evaluation - nn.RNN layer

Which metric matters for nn.RNN layer and WHY

The nn.RNN layer is used for sequence data, often in tasks like text or time series prediction. The key metrics depend on the task:

• For classification tasks: Accuracy, Precision, Recall, and F1-score matter to understand how well the RNN predicts correct classes over sequences.
• For regression tasks: Mean Squared Error (MSE) or Mean Absolute Error (MAE) show how close predictions are to true values.

Because RNNs handle sequences, it is important to evaluate metrics that reflect performance over the entire sequence, not just single points.

Confusion matrix example for nn.RNN classification

    Actual \ Predicted | Positive | Negative
    -------------------|----------|---------
    Positive           |    50    |   10    
    Negative           |    5     |   35    

    Total samples = 50 + 10 + 5 + 35 = 100
    

From this matrix:

• True Positives (TP) = 50
• False Positives (FP) = 5
• True Negatives (TN) = 35
• False Negatives (FN) = 10

Precision = TP / (TP + FP) = 50 / (50 + 5) = 0.91

Recall = TP / (TP + FN) = 50 / (50 + 10) = 0.83

F1-score = 2 * (Precision * Recall) / (Precision + Recall) ≈ 0.87

Precision vs Recall tradeoff with nn.RNN

Imagine an RNN used to detect spam messages:

• High Precision: Most messages marked as spam really are spam. This avoids blocking good messages.
• High Recall: Most spam messages are caught, but some good messages might be wrongly blocked.

If the RNN is tuned for high precision, it misses some spam (low recall). If tuned for high recall, it may mark good messages as spam (low precision).

Choosing the right balance depends on what is worse: missing spam or blocking good messages.

Good vs Bad metric values for nn.RNN layer

For classification tasks:

• Good: Precision and Recall above 0.8, F1-score above 0.8, accuracy close to or above 85%.
• Bad: Precision or Recall below 0.5, F1-score below 0.6, accuracy near random chance (e.g., 50% for binary).

For regression tasks:

• Good: Low MSE or MAE, showing predictions close to true values.
• Bad: High error values, indicating poor prediction quality.

Common pitfalls when evaluating nn.RNN layer

• Accuracy paradox: High accuracy can be misleading if classes are imbalanced. For example, if 90% of sequences belong to one class, predicting that class always gives 90% accuracy but poor real performance.
• Data leakage: Using future sequence data in training or validation can inflate metrics falsely.
• Overfitting: Very high training accuracy but low validation accuracy means the RNN memorized training sequences but cannot generalize.
• Ignoring sequence length: Metrics should consider performance across entire sequences, not just individual time steps.

Self-check question

Your nn.RNN model has 98% accuracy but only 12% recall on the fraud class. Is it good for production? Why or why not?

Answer: No, it is not good. The low recall means the model misses 88% of fraud cases, which is dangerous. High accuracy is misleading because fraud cases are rare. The model needs better recall to catch fraud effectively.

Key Result

For nn.RNN layers, precision, recall, and F1-score are key metrics to evaluate sequence classification quality, while error metrics matter for regression.