N-grams in NLP - Model Metrics & Evaluation

Metrics & Evaluation - N-grams

Which metric matters for N-grams and WHY

N-grams help us understand sequences of words or characters in text. When we use N-grams in models like text classifiers or language models, common metrics to check how well the model works include accuracy, precision, recall, and F1 score. These metrics tell us how well the model predicts the right categories or next words based on N-gram patterns.

For example, if we use N-grams to detect spam emails, precision is important to avoid marking good emails as spam. If we use N-grams to find important phrases, recall helps us catch most relevant phrases.

Confusion matrix for N-gram based classification

      | Predicted Spam | Predicted Not Spam |
      |----------------|--------------------|
      | True Positive (TP) = 80  | False Negative (FN) = 20 |
      | False Positive (FP) = 10 | True Negative (TN) = 90  |

      Total samples = 80 + 20 + 10 + 90 = 200

      Precision = TP / (TP + FP) = 80 / (80 + 10) = 0.89
      Recall = TP / (TP + FN) = 80 / (80 + 20) = 0.80
      F1 Score = 2 * (Precision * Recall) / (Precision + Recall) = 2 * (0.89 * 0.80) / (0.89 + 0.80) ≈ 0.84
    

Precision vs Recall tradeoff with N-grams

When using N-grams for tasks like spam detection, there is a tradeoff between precision and recall:

• High Precision: The model marks emails as spam only when very sure. This means fewer good emails are wrongly marked as spam. But it might miss some spam emails (lower recall).
• High Recall: The model catches most spam emails, but might mark some good emails as spam (lower precision).

Choosing which to prioritize depends on what is worse: missing spam or wrongly blocking good emails.

What "good" vs "bad" metric values look like for N-gram models

For N-gram based text classification:

• Good: Precision and recall above 0.8 means the model finds most relevant items and makes few mistakes.
• Bad: Precision or recall below 0.5 means the model misses many relevant items or makes many wrong predictions.
• Accuracy: Can be misleading if classes are imbalanced (e.g., spam is rare). High accuracy might hide poor spam detection.

Common pitfalls in N-gram model evaluation

• Accuracy paradox: High accuracy but poor precision/recall if one class dominates.
• Data leakage: Using future text or test data in training N-grams can inflate metrics falsely.
• Overfitting: Very high N (like 5-grams) may memorize training text, causing poor generalization and misleading metrics.
• Ignoring class imbalance: Not using precision/recall or F1 can hide poor performance on rare classes.

Self-check question

Your N-gram based spam detector has 98% accuracy but only 12% recall on spam emails. Is it good for production? Why or why not?

Answer: No, it is not good. The model misses 88% of spam emails (low recall), so many spam messages get through. High accuracy is misleading because most emails are not spam, so the model just predicts "not spam" most of the time. You need to improve recall to catch more spam.

Key Result

Precision and recall are key metrics for N-gram models; high accuracy alone can be misleading due to class imbalance.