RoBERTa and DistilBERT in NLP - Model Metrics & Evaluation

Metrics & Evaluation - RoBERTa and DistilBERT

Which metric matters for RoBERTa and DistilBERT and WHY

RoBERTa and DistilBERT are models used for understanding language. We often use accuracy to see how many answers they get right. But because language tasks can be tricky, precision and recall help us understand if the model is good at finding the right answers without too many mistakes or misses. For example, in sentiment analysis, precision tells us how many positive labels were truly positive, and recall tells us how many positive cases the model found out of all positives.

Confusion matrix example for RoBERTa/DistilBERT classification

      | Predicted Positive | Predicted Negative |
      |--------------------|--------------------|
      | True Positive (TP) = 85  | False Negative (FN) = 15 |
      | False Positive (FP) = 10 | True Negative (TN) = 90  |

      Total samples = 85 + 15 + 10 + 90 = 200

      Precision = TP / (TP + FP) = 85 / (85 + 10) = 0.8947
      Recall = TP / (TP + FN) = 85 / (85 + 15) = 0.85
      Accuracy = (TP + TN) / Total = (85 + 90) / 200 = 0.875
      F1 Score = 2 * (Precision * Recall) / (Precision + Recall) ≈ 0.871
    

Precision vs Recall tradeoff with examples

Imagine RoBERTa is used to detect spam emails. If it marks too many good emails as spam (low precision), users get annoyed. So, high precision is important here.

Now, if DistilBERT is used to find all harmful content in social media posts, missing any harmful post is bad (low recall). So, high recall is important.

Choosing between precision and recall depends on what is worse: false alarms or missed cases.

What "good" vs "bad" metric values look like for RoBERTa and DistilBERT

Good: Precision and recall above 85% means the model finds most correct answers and makes few mistakes. Accuracy above 85% shows overall strong performance.

Bad: Precision or recall below 50% means the model misses many correct answers or makes many wrong predictions. Accuracy near 50% means the model is guessing randomly.

Common pitfalls in evaluating RoBERTa and DistilBERT

• Accuracy paradox: High accuracy can be misleading if classes are unbalanced. For example, if 90% of data is negative, a model always predicting negative gets 90% accuracy but is useless.
• Data leakage: If test data leaks into training, metrics look better but model fails in real use.
• Overfitting: Model performs very well on training data but poorly on new data. Watch for big gaps between training and validation metrics.

Self-check question

Your RoBERTa model has 98% accuracy but only 12% recall on detecting fraud cases. Is it good for production? Why or why not?

Answer: No, it is not good. The model misses 88% of fraud cases (low recall), which is dangerous. High accuracy is misleading because fraud cases are rare. You need to improve recall to catch more fraud.

Key Result

Precision and recall are key to evaluate RoBERTa and DistilBERT, ensuring the model finds correct answers without many misses or false alarms.