Bag of Words (CountVectorizer) in NLP - Model Metrics & Evaluation

Metrics & Evaluation - Bag of Words (CountVectorizer)

Which metric matters for Bag of Words (CountVectorizer) and WHY

When using Bag of Words with CountVectorizer, the main goal is to convert text into numbers for models. The quality of this conversion affects how well the model learns. Metrics like accuracy, precision, and recall matter because they show how well the model understands the text features created by CountVectorizer.

For example, if you use Bag of Words for spam detection, precision tells you how many emails marked as spam really are spam, and recall tells you how many spam emails you caught. These metrics help check if the word counts are helping the model make good decisions.

Confusion Matrix Example

      Actual \ Predicted | Spam | Not Spam
      -------------------|-------|---------
      Spam               |  80   |   20
      Not Spam           |  10   |   90
    

Here, TP=80 (spam correctly found), FP=10 (not spam wrongly marked spam), FN=20 (spam missed), TN=90 (not spam correctly found).

Precision = 80 / (80 + 10) = 0.89
Recall = 80 / (80 + 20) = 0.80

Precision vs Recall Tradeoff

With Bag of Words, sometimes the model finds many spam emails but also marks some good emails as spam (high recall, low precision). Or it marks only very sure spam emails but misses some (high precision, low recall).

For spam filters, high precision is important to avoid losing good emails. For medical text classification, high recall is important to catch all cases.

Good vs Bad Metric Values

Good: Precision and recall both above 0.8 means the Bag of Words features help the model find relevant text patterns well.

Bad: Precision or recall below 0.5 means the word counts are not helping the model distinguish classes well. This could be due to poor vocabulary choice or noisy text.

Common Pitfalls

• Ignoring stop words: Common words like "the" or "and" can add noise if not removed.
• High dimensionality: Bag of Words can create many features, causing overfitting if the dataset is small.
• Data leakage: Using test data to build vocabulary can inflate metrics falsely.
• Accuracy paradox: High accuracy can be misleading if classes are imbalanced.

Self Check

Your model using Bag of Words has 98% accuracy but only 12% recall on spam emails. Is it good?

Answer: No. The model misses most spam emails (low recall). Even with high accuracy, it fails to catch the important class. You need to improve recall by adjusting features or model.

Key Result

Precision and recall are key to evaluate Bag of Words effectiveness; high accuracy alone can be misleading.