Transformer encoder in PyTorch - Model Metrics & Evaluation

Metrics & Evaluation - Transformer encoder

Which metric matters for Transformer encoder and WHY

The Transformer encoder is often used for tasks like text classification, translation, or feature extraction. The key metrics depend on the task:

• Accuracy for classification tasks: shows how many predictions match the true labels.
• Precision and Recall when classes are imbalanced or some errors cost more.
• F1 score balances precision and recall, useful when both matter.
• Loss (Cross-Entropy) during training: tells how well the model fits the data.

Choosing the right metric helps understand if the Transformer encoder is learning useful patterns or not.

Confusion matrix example

For a binary classification task using a Transformer encoder, suppose we have 100 samples:

      | Predicted Positive | Predicted Negative |
      |--------------------|--------------------|
      | True Positive (TP) = 40 | False Positive (FP) = 5 |
      | False Negative (FN) = 10 | True Negative (TN) = 45 |
    

Calculations:

• Precision = TP / (TP + FP) = 40 / (40 + 5) = 0.89
• Recall = TP / (TP + FN) = 40 / (40 + 10) = 0.80
• Accuracy = (TP + TN) / Total = (40 + 45) / 100 = 0.85
• F1 Score = 2 * (Precision * Recall) / (Precision + Recall) ≈ 0.84

Precision vs Recall tradeoff with examples

Transformer encoders can be tuned to favor precision or recall depending on the task:

• High Precision: Important when false positives are costly. For example, in spam detection, marking a good email as spam is bad. So, the model should be very sure before labeling spam.
• High Recall: Important when missing positive cases is costly. For example, in medical diagnosis, missing a disease case is dangerous. The model should catch as many positives as possible, even if some false alarms happen.

Adjusting thresholds or training strategies on the Transformer encoder affects this balance.

What good vs bad metric values look like for Transformer encoder

Good metrics mean the Transformer encoder understands the data well:

• Good: Accuracy > 85%, Precision and Recall both above 80%, F1 score close to these values.
• Bad: Accuracy near random guess (e.g., 50% for binary), Precision or Recall very low (below 50%), F1 score low showing imbalance.

Good metrics show the model predicts correctly and balances errors well.

Common pitfalls in metrics for Transformer encoder

• Accuracy paradox: High accuracy can be misleading if classes are imbalanced. For example, 90% accuracy if 90% samples belong to one class but model ignores the other.
• Data leakage: If test data leaks into training, metrics look unrealistically good.
• Overfitting indicators: Training loss very low but validation loss high means model memorizes training data but fails on new data.
• Ignoring class imbalance: Not using precision, recall, or F1 when classes are uneven can hide poor performance on minority class.

Self-check question

Your Transformer encoder model has 98% accuracy but only 12% recall on the positive class (e.g., fraud detection). Is it good for production? Why or why not?

Answer: No, it is not good. The high accuracy is misleading because the model misses 88% of positive cases (fraud). In fraud detection, recall is critical to catch as many frauds as possible. Low recall means many frauds go undetected, which is risky.

Key Result

For Transformer encoder tasks, balance precision and recall using F1 score to ensure meaningful performance beyond accuracy.