Bag of Words and TF-IDF in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - Bag of Words and TF-IDF

Which metric matters for Bag of Words and TF-IDF and WHY

When using Bag of Words or TF-IDF to turn text into numbers for models, the key metrics depend on the task. For classification, accuracy, precision, and recall matter because they show how well the model understands the text features. For ranking or search tasks, precision and recall help measure if the important words are captured well. These metrics tell us if the chosen text representation helps the model make good decisions.

Confusion Matrix Example

Imagine a spam email classifier using TF-IDF features. Here is a confusion matrix after testing 100 emails:

      |                 | Predicted Spam | Predicted Not Spam |
      |-----------------|----------------|--------------------|
      | Actual Spam     | True Positives (TP) = 40 | False Negatives (FN) = 5 |
      | Actual Not Spam | False Positives (FP) = 10 | True Negatives (TN) = 45 |
    

Total emails = 40 + 10 + 5 + 45 = 100

From this, we calculate:

• Precision = TP / (TP + FP) = 40 / (40 + 10) = 0.8
• Recall = TP / (TP + FN) = 40 / (40 + 5) = 0.89
• Accuracy = (TP + TN) / Total = (40 + 45) / 100 = 0.85

Precision vs Recall Tradeoff with Examples

Using Bag of Words or TF-IDF, sometimes the model finds many spam emails but also marks some good emails as spam (high recall, lower precision). Other times, it only marks very sure spam emails (high precision, lower recall).

Example 1: If you want to avoid missing spam (catch all spam), prioritize high recall. You accept some good emails wrongly marked as spam.

Example 2: If you want to avoid marking good emails as spam, prioritize high precision. You accept missing some spam emails.

Choosing Bag of Words or TF-IDF affects this tradeoff because TF-IDF downweights common words, helping precision by focusing on important words.

What Good vs Bad Metric Values Look Like

For text classification using Bag of Words or TF-IDF:

• Good: Precision and recall above 0.8 means the model finds most relevant texts and avoids many mistakes.
• Bad: Precision or recall below 0.5 means the model either misses many relevant texts or marks many irrelevant texts.
• Accuracy alone can be misleading if classes are imbalanced (e.g., spam is rare).

Common Metrics Pitfalls

• Accuracy Paradox: High accuracy can happen if most texts belong to one class, but the model fails on the minority class.
• Data Leakage: If the text features leak future information, metrics look better but model fails in real use.
• Overfitting: Very high training metrics but low test metrics means the model memorized training texts, not learned general patterns.
• Ignoring Class Imbalance: Not using precision and recall can hide poor performance on rare classes.

Self Check

Your spam detection model using TF-IDF has 98% accuracy but only 12% recall on spam emails. Is it good for production?

Answer: No. The model misses 88% of spam emails (low recall), so it fails to catch most spam even if overall accuracy looks high. This is bad for user experience.

Key Result

Precision and recall are key metrics for Bag of Words and TF-IDF to ensure models find relevant text without many mistakes.