N-grams in NLP - Model Metrics & Evaluation

Metrics & Evaluation - N-grams

Which metric matters for N-grams and WHY

N-grams help us understand sequences of words or characters in text. When we use N-grams in models like text classifiers or language models, common metrics to check how well the model works include accuracy, precision, recall, and F1 score. These metrics tell us how well the model predicts the right categories or next words based on N-gram patterns.

For example, if we use N-grams to detect spam emails, precision is important to avoid marking good emails as spam. If we use N-grams to find important phrases, recall helps us catch most relevant phrases.

Confusion matrix for N-gram based classification

      | Predicted Spam | Predicted Not Spam |
      |----------------|--------------------|
      | True Positive (TP) = 80  | False Negative (FN) = 20 |
      | False Positive (FP) = 10 | True Negative (TN) = 90  |

      Total samples = 80 + 20 + 10 + 90 = 200

      Precision = TP / (TP + FP) = 80 / (80 + 10) = 0.89
      Recall = TP / (TP + FN) = 80 / (80 + 20) = 0.80
      F1 Score = 2 * (Precision * Recall) / (Precision + Recall) = 2 * (0.89 * 0.80) / (0.89 + 0.80) ≈ 0.84
    

Precision vs Recall tradeoff with N-grams

When using N-grams for tasks like spam detection, there is a tradeoff between precision and recall:

• High Precision: The model marks emails as spam only when very sure. This means fewer good emails are wrongly marked as spam. But it might miss some spam emails (lower recall).
• High Recall: The model catches most spam emails, but might mark some good emails as spam (lower precision).

Choosing which to prioritize depends on what is worse: missing spam or wrongly blocking good emails.

What "good" vs "bad" metric values look like for N-gram models

For N-gram based text classification:

• Good: Precision and recall above 0.8 means the model finds most relevant items and makes few mistakes.
• Bad: Precision or recall below 0.5 means the model misses many relevant items or makes many wrong predictions.
• Accuracy: Can be misleading if classes are imbalanced (e.g., spam is rare). High accuracy might hide poor spam detection.

Common pitfalls in N-gram model evaluation

• Accuracy paradox: High accuracy but poor precision/recall if one class dominates.
• Data leakage: Using future text or test data in training N-grams can inflate metrics falsely.
• Overfitting: Very high N (like 5-grams) may memorize training text, causing poor generalization and misleading metrics.
• Ignoring class imbalance: Not using precision/recall or F1 can hide poor performance on rare classes.

Self-check question

Your N-gram based spam detector has 98% accuracy but only 12% recall on spam emails. Is it good for production? Why or why not?

Answer: No, it is not good. The model misses 88% of spam emails (low recall), so many spam messages get through. High accuracy is misleading because most emails are not spam, so the model just predicts "not spam" most of the time. You need to improve recall to catch more spam.

Key Result

Precision and recall are key metrics for N-gram models; high accuracy alone can be misleading due to class imbalance.

Practice

1. What is an n-gram in natural language processing?

easy

A. A random selection of n words from a text

B. A single word repeated n times

C. A sentence with n words

D. A group of n consecutive words in a text

Solution

1. Step 1: Understand the definition of n-gram

   An n-gram is defined as a sequence of n consecutive words appearing together in text.

2. Step 2: Compare options with definition

   Only A group of n consecutive words in a text correctly describes an n-gram as consecutive words, not random or repeated words.

3. Final Answer:

   A group of n consecutive words in a text -> Option D

4. Quick Check:

   n-gram = consecutive words [OK]

Hint: Remember: n-gram means consecutive words, not random ones [OK]

Common Mistakes:

• Thinking n-gram means repeated words
• Confusing n-gram with sentence length
• Assuming words are randomly picked

2. Which of the following is the correct way to set up a CountVectorizer to extract bigrams in Python?

easy

A. CountVectorizer(ngram_range=(1,1))

B. CountVectorizer(ngram_range=(2,2))

C. CountVectorizer(ngram_range=(0,2))

D. CountVectorizer(ngram_range=(1,3))

Solution

1. Step 1: Understand ngram_range parameter

   ngram_range=(2,2) extracts only bigrams (groups of exactly 2 words).

2. Step 2: Evaluate each option

   CountVectorizer(ngram_range=(1,1)) extracts unigrams only; C is invalid because 0 is not a valid n; D extracts unigrams to trigrams.

3. Final Answer:

   CountVectorizer(ngram_range=(2,2)) -> Option B

4. Quick Check:

   bigrams = ngram_range (2,2) [OK]

Hint: Set ngram_range=(2,2) for only bigrams [OK]

Common Mistakes:

• Using (1,1) which extracts unigrams
• Using (0,2) which is invalid
• Using (1,3) which extracts multiple n-grams

3. What will be the output tokens when extracting trigrams from the sentence 'I love machine learning' using CountVectorizer(ngram_range=(3,3))?

medium

A. ['I love machine', 'love machine learning']

B. ['I love', 'love machine', 'machine learning']

C. ['I', 'love', 'machine', 'learning']

D. ['I love machine learning']

Solution

1. Step 1: Understand trigram extraction

   Trigrams are groups of 3 consecutive words. The sentence has 4 words, so possible trigrams are words 1-3 and 2-4.

2. Step 2: List trigrams from the sentence

   First trigram: 'I love machine', second trigram: 'love machine learning'.

3. Final Answer:

   ['I love machine', 'love machine learning'] -> Option A

4. Quick Check:

   Trigrams = groups of 3 words [OK]

Hint: Count groups of 3 consecutive words for trigrams [OK]

Common Mistakes:

• Listing bigrams instead of trigrams
• Listing single words instead of groups
• Combining all words as one token

4. Identify the error in this code snippet for extracting bigrams:

from sklearn.feature_extraction.text import CountVectorizer
text = ['hello world']
vectorizer = CountVectorizer(ngram_range=(1,2))
vectorizer.fit_transform(text)
print(vectorizer.get_feature_names())

medium

A. The text should be a string, not a list

B. The ngram_range should be (2,2) to extract only bigrams

C. The method get_feature_names() is deprecated and should be get_feature_names_out()

D. CountVectorizer cannot extract bigrams

Solution

1. Step 1: Check method usage

   In recent sklearn versions, get_feature_names() is deprecated; get_feature_names_out() is the correct method.

2. Step 2: Validate other parts

   ngram_range=(1,2) is valid for unigrams and bigrams; text as list is correct; CountVectorizer supports bigrams.

3. Final Answer:

   get_feature_names() is deprecated and should be get_feature_names_out() -> Option C

4. Quick Check:

   Use get_feature_names_out() for features [OK]

Hint: Use get_feature_names_out() instead of deprecated get_feature_names() [OK]

Common Mistakes:

• Thinking ngram_range=(1,2) is wrong for bigrams
• Assuming text must be a string, not list
• Believing CountVectorizer can't extract bigrams

5. You want to build a text prediction model that uses both unigrams and bigrams but excludes any n-grams containing stop words like 'the' or 'and'. Which approach is best?

hard

A. Use CountVectorizer with ngram_range=(1,2) and stop_words='english'

B. Use CountVectorizer with ngram_range=(2,2) and no stop words removal

C. Use CountVectorizer with ngram_range=(1,1) and manually remove stop words after extraction

D. Use CountVectorizer with ngram_range=(1,3) and stop_words=None

Solution

1. Step 1: Understand requirements

   We need unigrams and bigrams, and want to exclude stop words in any n-gram.

2. Step 2: Evaluate options

   Use CountVectorizer with ngram_range=(1,2) and stop_words='english' uses ngram_range=(1,2) for unigrams and bigrams and removes stop words automatically. Others either miss unigrams, include stop words, or include trigrams.

3. Final Answer:

   Use CountVectorizer with ngram_range=(1,2) and stop_words='english' -> Option A

4. Quick Check:

   Unigrams + bigrams + stop word removal = Use CountVectorizer with ngram_range=(1,2) and stop_words='english' [OK]

Hint: Set ngram_range and stop_words='english' to filter stop words [OK]

Common Mistakes:

• Not removing stop words from bigrams
• Using wrong ngram_range missing unigrams
• Including trigrams when not needed