Spam detection pipeline in NLP - Model Metrics & Evaluation

Metrics & Evaluation - Spam detection pipeline

Which metric matters for Spam Detection and WHY

In spam detection, precision and recall are the most important metrics.

Precision tells us how many emails marked as spam really are spam. High precision means fewer good emails wrongly blocked.

Recall tells us how many actual spam emails we catch. High recall means fewer spam emails slip through.

We want a balance, but often prioritize high precision to avoid blocking important emails by mistake.

Confusion Matrix Example

      | Predicted Spam | Predicted Not Spam |
      |----------------|--------------------|
      | True Positives (TP) = 80  | False Negatives (FN) = 20 |
      | False Positives (FP) = 10 | True Negatives (TN) = 890 |
    

Total emails = 80 + 20 + 10 + 890 = 1000

Precision = TP / (TP + FP) = 80 / (80 + 10) = 0.89

Recall = TP / (TP + FN) = 80 / (80 + 20) = 0.80

Precision vs Recall Tradeoff with Examples

If we set the spam filter very strict, we catch almost all spam (high recall), but may block many good emails (low precision).

If we set it very loose, we block fewer good emails (high precision), but more spam gets through (low recall).

Example:

• High precision, low recall: Only mark very obvious spam. Few false alarms, but some spam slips through.
• High recall, low precision: Mark many emails as spam. Catch almost all spam, but many good emails get blocked.

Spam filters usually aim for high precision to avoid annoying users by blocking good emails.

Good vs Bad Metric Values for Spam Detection

Good:

• Precision above 0.85 (most flagged emails are really spam)
• Recall above 0.75 (catch most spam emails)
• Accuracy high but not the main focus

Bad:

• Precision below 0.5 (many good emails wrongly blocked)
• Recall below 0.5 (many spam emails missed)
• High accuracy but low precision or recall (accuracy paradox)

Common Metric Pitfalls in Spam Detection

• Accuracy paradox: Because most emails are not spam, a model that always predicts "not spam" can have high accuracy but is useless.
• Data leakage: If spam keywords appear in test data from training, metrics look better but model won't generalize.
• Overfitting: Model performs well on training but poorly on new emails, causing misleading metrics.
• Ignoring class imbalance: Spam is usually a small part of emails, so metrics like accuracy can be misleading.

Self Check

Your spam detection model has 98% accuracy but only 12% recall on spam emails. Is it good for production?

Answer: No, it is not good. The model misses 88% of spam emails (low recall), so many spam messages will reach users. High accuracy is misleading because most emails are not spam. Improving recall is critical.

Key Result

Precision and recall are key; high precision avoids blocking good emails, high recall catches most spam.