Bidirectional RNNs in PyTorch - Model Metrics & Evaluation

Metrics & Evaluation - Bidirectional RNNs

Which metric matters for Bidirectional RNNs and WHY

Bidirectional RNNs are used to understand sequences from both past and future context. The key metrics to check are accuracy for classification tasks and loss for sequence prediction. For tasks like speech recognition or text tagging, precision, recall, and F1 score are important to measure how well the model predicts each class, especially when classes are imbalanced.

Confusion Matrix Example

Suppose a bidirectional RNN classifies words into two classes: Positive (P) and Negative (N). Here is a confusion matrix:

      | Predicted P | Predicted N |
      |-------------|-------------|
      | True P: 50  | 10          |
      | True N: 5   | 35          |
    

Total samples = 50 + 10 + 5 + 35 = 100

From this matrix:

• Precision = TP / (TP + FP) = 50 / (50 + 5) = 0.91
• Recall = TP / (TP + FN) = 50 / (50 + 10) = 0.83
• F1 Score = 2 * (Precision * Recall) / (Precision + Recall) ≈ 0.87

Precision vs Recall Tradeoff

In bidirectional RNNs, depending on the task, you might want to balance precision and recall differently.

• High Precision: Useful when false positives are costly. For example, in medical diagnosis, wrongly predicting a disease when it is not present can cause unnecessary stress and treatment.
• High Recall: Important when missing a positive case is dangerous. For example, in fraud detection, missing a fraud case (false negative) is worse than flagging a normal case.

Bidirectional RNNs help by using context from both directions, which can improve both precision and recall compared to unidirectional models.

What Good vs Bad Metric Values Look Like

For a bidirectional RNN on a balanced classification task:

• Good: Accuracy above 85%, Precision and Recall above 80%, F1 score above 0.8.
• Bad: Accuracy below 60%, Precision or Recall below 50%, F1 score below 0.5.

Low precision means many false alarms. Low recall means many misses. Both reduce usefulness.

Common Pitfalls in Metrics for Bidirectional RNNs

• Accuracy Paradox: High accuracy can be misleading if classes are imbalanced. For example, if 90% of data is class A, predicting all A gives 90% accuracy but zero recall for class B.
• Data Leakage: If future information leaks into training, metrics look better but model fails in real use.
• Overfitting: Very low training loss but high validation loss means model memorizes training data and won't generalize.
• Ignoring Sequence Length: Metrics averaged over sequences of different lengths can hide poor performance on longer sequences.

Self Check

Your bidirectional RNN model has 98% accuracy but only 12% recall on the positive class (e.g., fraud). Is it good for production?

Answer: No, it is not good. The model misses 88% of positive cases, which is dangerous for fraud detection. High accuracy is misleading because most data is negative. You need to improve recall to catch more fraud cases.

Key Result

Bidirectional RNNs improve sequence understanding, but precision, recall, and F1 score best show their true performance, especially on imbalanced data.