BERT for text classification in PyTorch - Model Metrics & Evaluation

Metrics & Evaluation - BERT for text classification

Which metric matters for BERT text classification and WHY

For BERT text classification, the key metrics are accuracy, precision, recall, and F1 score. Accuracy shows overall correct predictions. Precision tells us how many predicted positives are truly positive. Recall shows how many actual positives were found. F1 balances precision and recall. These matter because text classes can be imbalanced, so accuracy alone can be misleading.

Confusion matrix example

      | Predicted Positive | Predicted Negative |
      |--------------------|--------------------|
      | True Positive (TP): 80  | False Negative (FN): 20 |
      | False Positive (FP): 10 | True Negative (TN): 90  |

      Total samples = TP + FP + TN + FN = 80 + 10 + 90 + 20 = 200

      Precision = TP / (TP + FP) = 80 / (80 + 10) = 0.89
      Recall = TP / (TP + FN) = 80 / (80 + 20) = 0.80
      F1 Score = 2 * (Precision * Recall) / (Precision + Recall) ≈ 0.84
    

Precision vs Recall tradeoff with examples

Imagine a spam email classifier using BERT:

• High precision: Most emails marked as spam really are spam. Good to avoid losing important emails.
• High recall: Most spam emails are caught. Good to keep inbox clean.

Depending on what matters more, you tune the model. For spam, high precision avoids false spam. For medical text classification, high recall avoids missing important cases.

Good vs Bad metric values for BERT text classification

• Good: Accuracy > 85%, Precision and Recall both > 80%, F1 score balanced and high.
• Bad: Accuracy high but recall very low (missing many positives), or precision very low (many false alarms).
• Balanced metrics indicate the model understands both classes well.

Common pitfalls in metrics for BERT text classification

• Accuracy paradox: High accuracy but poor recall if classes are imbalanced.
• Data leakage: If test data leaks into training, metrics look unrealistically good.
• Overfitting: Very high training accuracy but low test accuracy means model memorizes instead of generalizing.
• Ignoring class imbalance: Not using precision/recall or F1 can hide poor performance on minority classes.

Self-check question

Your BERT model for spam detection has 98% accuracy but only 12% recall on spam class. Is it good for production? Why or why not?

Answer: No, it is not good. The model misses 88% of actual spam emails (low recall), so many spam messages get through. High accuracy is misleading because most emails are not spam. You need to improve recall to catch more spam.

Key Result

For BERT text classification, balanced precision, recall, and F1 score are key to ensure the model correctly identifies all classes, especially when data is imbalanced.