TF-IDF (TfidfVectorizer) in NLP - Model Metrics & Evaluation

Metrics & Evaluation - TF-IDF (TfidfVectorizer)

Which metric matters for TF-IDF and WHY

TF-IDF is a way to turn text into numbers by showing how important words are in documents. It is not a model itself but a tool to prepare data for models like classifiers. So, the metrics that matter come from the model using TF-IDF features.

For example, if you use TF-IDF with a spam detector, you want to check precision and recall of the spam classifier. Precision tells you how many emails marked as spam really are spam. Recall tells you how many spam emails you caught.

TF-IDF helps the model by giving good features, but the final evaluation depends on the model's predictions and the task.

Confusion matrix example

Imagine a spam classifier using TF-IDF features. Here is a confusion matrix:

      | Predicted Spam | Predicted Not Spam |
      |----------------|--------------------|
      | True Spam: 80  | False Negative: 20 |
      | False Positive: 10 | True Not Spam: 90  |
    

From this:

• True Positives (TP) = 80 (correct spam detected)
• False Positives (FP) = 10 (good emails marked spam)
• False Negatives (FN) = 20 (spam missed)
• True Negatives (TN) = 90 (correct good emails)

Precision vs Recall tradeoff with examples

In spam detection using TF-IDF features:

• High Precision: Few good emails are wrongly marked as spam. Good for user experience.
• High Recall: Most spam emails are caught. Good for security.

Sometimes improving recall means lowering precision and vice versa. You must choose based on what matters more.

For example, if you want to avoid missing spam, focus on recall. If you want to avoid annoying users, focus on precision.

What good vs bad metric values look like

Good metrics for a spam classifier using TF-IDF might be:

• Precision: 0.9 or higher (90% of emails marked spam are really spam)
• Recall: 0.8 or higher (80% of spam emails caught)
• F1 Score: balances precision and recall, should be high (around 0.85+)

Bad metrics might be:

• Precision below 0.5 (many good emails wrongly marked spam)
• Recall below 0.5 (many spam emails missed)
• Accuracy can be misleading if data is unbalanced (e.g., 95% accuracy but model never detects spam)

Common pitfalls with metrics when using TF-IDF

• Accuracy paradox: If spam is rare, a model that always says "not spam" can have high accuracy but is useless.
• Data leakage: If test data leaks into training, metrics look too good but model fails in real life.
• Overfitting: Model fits training data perfectly but performs poorly on new data. Check metrics on separate test data.
• Ignoring class imbalance: If spam is rare, use precision, recall, and F1 instead of accuracy.

Self-check question

Your spam model using TF-IDF features has 98% accuracy but only 12% recall on spam emails. Is it good for production? Why or why not?

Answer: No, it is not good. The model misses 88% of spam emails (low recall), so many spam messages get through. High accuracy is misleading because most emails are not spam, so the model guesses "not spam" too often. You need to improve recall to catch more spam.

Key Result

TF-IDF is a feature tool; model metrics like precision and recall on tasks using TF-IDF features show true performance.