Stemming (Porter, Snowball) in NLP - Model Metrics & Evaluation

Metrics & Evaluation - Stemming (Porter, Snowball)

Which metric matters for Stemming and WHY

In stemming, the main goal is to reduce words to their root form. The key metric is accuracy of normalization, which means how well the stemmer groups related words together without losing meaning.

We also look at precision and recall in the context of information retrieval or text classification tasks that use stemming. Precision measures how many of the stemmed words are correctly grouped, while recall measures how many related words are successfully captured by the stemmer.

Good stemming improves downstream tasks by reducing word variations, so metrics like F1 score on those tasks also matter.

Confusion matrix or equivalent visualization

For stemming, a confusion matrix can show how often words are correctly or incorrectly stemmed:

          | Correctly Stemmed | Incorrectly Stemmed
    ------|-------------------|-------------------
    Related Words   |       TP          |        FN         
    Unrelated Words |       FP          |        TN         
    

Here:

• TP: Words that should be grouped and are stemmed together.
• FN: Words that should be grouped but are not stemmed together.
• FP: Words that should not be grouped but are stemmed together.
• TN: Words that should not be grouped and are not stemmed together.

Precision vs Recall tradeoff with examples

If a stemmer is too aggressive (like Porter sometimes is), it may group unrelated words together, increasing false positives and lowering precision.

If a stemmer is too conservative (like Snowball can be), it may miss grouping related words, increasing false negatives and lowering recall.

Example: Grouping "running" and "runner" correctly is good recall. But grouping "run" and "rung" incorrectly lowers precision.

Choosing the right stemmer depends on whether you want to avoid mixing unrelated words (high precision) or capture all related forms (high recall).

What "good" vs "bad" metric values look like for stemming

Good stemming:

• High precision (e.g., > 0.85): Most grouped words are truly related.
• High recall (e.g., > 0.85): Most related words are grouped.
• Balanced F1 score (e.g., > 0.85) showing good overall performance.

Bad stemming:

• Low precision (e.g., < 0.6): Many unrelated words grouped together.
• Low recall (e.g., < 0.6): Many related words missed.
• Unbalanced metrics showing over- or under-stemming.

Common pitfalls in stemming metrics

• Accuracy paradox: High accuracy can be misleading if most words are unique and not stemmed.
• Data leakage: Evaluating on the same text used to tune the stemmer can inflate metrics.
• Overstemming: Aggressive stemming merges unrelated words, hurting precision.
• Understemming: Conservative stemming misses related words, hurting recall.
• Ignoring downstream impact: Metrics should consider how stemming affects tasks like search or classification.

Self-check question

Your stemmer has 98% accuracy but only 12% recall on grouping related words. Is it good for production? Why or why not?

Answer: No, it is not good. The very low recall means it misses most related words, so it fails to group them properly. High accuracy here is misleading because most words are unique and not grouped. This stemmer will not help tasks that rely on grouping word forms.

Key Result

Effective stemming balances precision and recall to group related words without mixing unrelated ones.