Text feature basics (CountVectorizer, TF-IDF) in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - Text feature basics (CountVectorizer, TF-IDF)

Which metric matters for this concept and WHY

When working with text features like CountVectorizer and TF-IDF, the key metrics to evaluate are accuracy, precision, and recall of the model using these features. This is because these features transform text into numbers, and the quality of this transformation affects how well the model predicts.

For example, if you use these features in a spam detection model, precision tells you how many emails marked as spam really are spam, and recall tells you how many spam emails you caught. Both matter depending on your goal.

Confusion matrix or equivalent visualization (ASCII)

      Actual \ Predicted | Spam (Positive) | Not Spam (Negative)
      -------------------------------------------------------
      Spam (Positive)    |       TP = 80    |       FN = 20
      Not Spam (Negative)|       FP = 10    |       TN = 90
    

This matrix shows how many emails were correctly or incorrectly classified using text features.

Precision vs Recall tradeoff with concrete examples

Precision is important when you want to avoid false alarms. For example, in spam detection, high precision means fewer good emails are wrongly marked as spam.

Recall is important when you want to catch as many positives as possible. For example, in detecting harmful content, high recall means fewer harmful messages are missed.

CountVectorizer and TF-IDF affect this tradeoff by how well they represent important words. TF-IDF often helps by reducing the weight of common words, improving precision without losing recall.

What "good" vs "bad" metric values look like for this use case

Good metrics:

• Precision and recall both above 0.8 (80%) for balanced performance.
• F1 score (balance of precision and recall) above 0.8.
• Consistent results on training and test data, showing features generalize well.

Bad metrics:

• High precision but very low recall (e.g., precision 0.9, recall 0.3) means many positives missed.
• High recall but very low precision (e.g., recall 0.9, precision 0.2) means many false alarms.
• Very different metrics on training vs test data, indicating overfitting or poor feature representation.

Metrics pitfalls (accuracy paradox, data leakage, overfitting indicators)

• Accuracy paradox: In imbalanced text data (e.g., 95% non-spam), accuracy can be high even if the model ignores spam. Precision and recall give better insight.
• Data leakage: If test data words appear in training in a way that leaks labels, metrics look better but model fails in real use.
• Overfitting: Very high training metrics but low test metrics suggest the text features capture noise, not true patterns.
• Ignoring stop words: CountVectorizer without removing common words can inflate feature space and hurt model quality.

Self-check: Your model has 98% accuracy but 12% recall on spam. Is it good?

No, this model is not good for spam detection. The 98% accuracy is misleading because spam is rare. The 12% recall means it only finds 12 out of 100 spam emails, missing most spam. This would let many spam emails through, which is bad for users.

Key Result

Precision and recall are key to evaluate text features; high accuracy alone can be misleading in imbalanced text tasks.

Practice

1. What does CountVectorizer do in text processing?

easy

A. Calculates the importance of words based on frequency and rarity

B. Counts how many times each word appears in the text

C. Removes stop words from the text

D. Converts text into lowercase only

Solution

1. Step 1: Understand CountVectorizer's role

   CountVectorizer transforms text into a matrix of token counts, counting word occurrences.

2. Step 2: Differentiate from TF-IDF

   Unlike TF-IDF, it does not weigh words by importance, only counts frequency.

3. Final Answer:

   Counts how many times each word appears in the text -> Option B

4. Quick Check:

   CountVectorizer = word counts [OK]

Hint: CountVectorizer counts words, TF-IDF scores importance [OK]

Common Mistakes:

• Confusing CountVectorizer with TF-IDF
• Thinking it removes stop words by default
• Assuming it normalizes text only

2. Which of the following is the correct way to import and create a CountVectorizer in Python?

easy

A. from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer()

B. import CountVectorizer from sklearn.text
vectorizer = CountVectorizer()

C. from sklearn.text import CountVectorizer
vectorizer = CountVectorizer()

D. import CountVectorizer
vectorizer = CountVectorizer()

Solution

1. Step 1: Recall correct sklearn import path

   CountVectorizer is in sklearn.feature_extraction.text module.

2. Step 2: Check syntax correctness

   from sklearn.feature_extraction.text import CountVectorizer vectorizer = CountVectorizer() uses correct import and instantiation syntax.

3. Final Answer:

   from sklearn.feature_extraction.text import CountVectorizer vectorizer = CountVectorizer() -> Option A

4. Quick Check:

   Correct import path and syntax [OK]

Hint: CountVectorizer is in sklearn.feature_extraction.text [OK]

Common Mistakes:

• Using wrong module path for import
• Incorrect import syntax (like import ... from ...)
• Forgetting to instantiate the class

3. What will be the output shape of the matrix after applying CountVectorizer on these two sentences?

sentences = ["I love cats", "Cats love me"]
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(sentences)
print(X.shape)

medium

A. (2, 4)

B. (2, 3)

C. (3, 2)

D. (4, 2)

Solution

1. Step 1: Count unique words in sentences

   Words are: 'i', 'love', 'cats', 'me' -> 4 unique words.

2. Step 2: Understand shape of output matrix

   There are 2 sentences (rows) and 4 unique words (columns), so shape is (2, 4).

3. Final Answer:

   (2, 4) -> Option A

4. Quick Check:

   Rows = sentences, columns = unique words [OK]

Hint: Shape = (number of texts, unique words) [OK]

Common Mistakes:

• Mixing rows and columns in shape
• Counting duplicate words multiple times
• Ignoring case sensitivity (CountVectorizer lowercases by default)

4. Identify the error in this TF-IDF code snippet:

from sklearn.feature_extraction.text import TfidfVectorizer
texts = ["apple banana apple", "banana fruit"]
tfidf = TfidfVectorizer()
X = tfidf.fit_transform(texts)
print(tfidf.get_feature_names())

medium

A. fit_transform() should be called on texts as a string, not list

B. TfidfVectorizer() requires stop_words parameter

C. get_feature_names() is deprecated, should use get_feature_names_out()

D. Import statement is incorrect

Solution

1. Step 1: Check method usage for feature names

   In recent sklearn versions, get_feature_names() is deprecated.

2. Step 2: Use updated method

   Use get_feature_names_out() instead to get feature names without error.

3. Final Answer:

   get_feature_names() is deprecated, should use get_feature_names_out() -> Option C

4. Quick Check:

   Use get_feature_names_out() for TF-IDF features [OK]

Hint: Use get_feature_names_out() with TF-IDF [OK]

Common Mistakes:

• Using deprecated get_feature_names() method
• Passing wrong data type to fit_transform
• Incorrect import paths

5. You want to transform text data so that common words like 'the' and 'is' have less impact, but rare important words have higher scores. Which method should you use?

hard

A. One-hot encoding of words

B. CountVectorizer without stop words

C. Raw word counts from CountVectorizer

D. TF-IDF Vectorizer

Solution

1. Step 1: Understand the goal of reducing common word impact

   Common words appear frequently but carry less meaning, so their impact should be lowered.

2. Step 2: Identify method that weighs words by importance

   TF-IDF scores words higher if they are rare and important, reducing common word impact.

3. Final Answer:

   TF-IDF Vectorizer -> Option D

4. Quick Check:

   TF-IDF = importance weighting [OK]

Hint: Use TF-IDF to weigh rare words higher [OK]

Common Mistakes:

• Using raw counts which treat all words equally
• Assuming stop words removal alone solves importance
• Confusing one-hot encoding with frequency weighting