Stemming (Porter, Snowball) in NLP - Model Metrics & Evaluation

Metrics & Evaluation - Stemming (Porter, Snowball)

Which metric matters for Stemming and WHY

In stemming, the main goal is to reduce words to their root form. The key metric is accuracy of normalization, which means how well the stemmer groups related words together without losing meaning.

We also look at precision and recall in the context of information retrieval or text classification tasks that use stemming. Precision measures how many of the stemmed words are correctly grouped, while recall measures how many related words are successfully captured by the stemmer.

Good stemming improves downstream tasks by reducing word variations, so metrics like F1 score on those tasks also matter.

Confusion matrix or equivalent visualization

For stemming, a confusion matrix can show how often words are correctly or incorrectly stemmed:

          | Correctly Stemmed | Incorrectly Stemmed
    ------|-------------------|-------------------
    Related Words   |       TP          |        FN         
    Unrelated Words |       FP          |        TN         
    

Here:

• TP: Words that should be grouped and are stemmed together.
• FN: Words that should be grouped but are not stemmed together.
• FP: Words that should not be grouped but are stemmed together.
• TN: Words that should not be grouped and are not stemmed together.

Precision vs Recall tradeoff with examples

If a stemmer is too aggressive (like Porter sometimes is), it may group unrelated words together, increasing false positives and lowering precision.

If a stemmer is too conservative (like Snowball can be), it may miss grouping related words, increasing false negatives and lowering recall.

Example: Grouping "running" and "runner" correctly is good recall. But grouping "run" and "rung" incorrectly lowers precision.

Choosing the right stemmer depends on whether you want to avoid mixing unrelated words (high precision) or capture all related forms (high recall).

What "good" vs "bad" metric values look like for stemming

Good stemming:

• High precision (e.g., > 0.85): Most grouped words are truly related.
• High recall (e.g., > 0.85): Most related words are grouped.
• Balanced F1 score (e.g., > 0.85) showing good overall performance.

Bad stemming:

• Low precision (e.g., < 0.6): Many unrelated words grouped together.
• Low recall (e.g., < 0.6): Many related words missed.
• Unbalanced metrics showing over- or under-stemming.

Common pitfalls in stemming metrics

• Accuracy paradox: High accuracy can be misleading if most words are unique and not stemmed.
• Data leakage: Evaluating on the same text used to tune the stemmer can inflate metrics.
• Overstemming: Aggressive stemming merges unrelated words, hurting precision.
• Understemming: Conservative stemming misses related words, hurting recall.
• Ignoring downstream impact: Metrics should consider how stemming affects tasks like search or classification.

Self-check question

Your stemmer has 98% accuracy but only 12% recall on grouping related words. Is it good for production? Why or why not?

Answer: No, it is not good. The very low recall means it misses most related words, so it fails to group them properly. High accuracy here is misleading because most words are unique and not grouped. This stemmer will not help tasks that rely on grouping word forms.

Key Result

Effective stemming balances precision and recall to group related words without mixing unrelated ones.

Practice

1. What is the main purpose of stemming in Natural Language Processing?

easy

A. To reduce words to their base or root form

B. To translate text into another language

C. To count the number of words in a sentence

D. To generate synonyms for words

Solution

1. Step 1: Understand stemming concept

   Stemming simplifies words by cutting off suffixes to get the root form.

2. Step 2: Compare options with stemming goal

   Only To reduce words to their base or root form describes reducing words to their base form, which is the goal of stemming.

3. Final Answer:

   To reduce words to their base or root form -> Option A

4. Quick Check:

   Stemming = base form reduction [OK]

Hint: Stemming cuts word endings to find the root [OK]

Common Mistakes:

• Confusing stemming with translation
• Thinking stemming counts words
• Mixing stemming with synonym generation

2. Which of the following is the correct way to import the Porter Stemmer from NLTK in Python?

easy

A. from nltk.stem import PorterStemmer

B. import nltk.PorterStemmer

C. from nltk import PorterStemmer

D. import PorterStemmer from nltk.stem

Solution

1. Step 1: Recall correct import syntax in Python

   Python imports use 'from module import class' format for specific classes.

2. Step 2: Match with NLTK Porter Stemmer import

   The correct import is 'from nltk.stem import PorterStemmer' as it imports the class from the stem module.

3. Final Answer:

   from nltk.stem import PorterStemmer -> Option A

4. Quick Check:

   Correct import uses 'from nltk.stem import PorterStemmer' [OK]

Hint: Use 'from module import class' for specific imports [OK]

Common Mistakes:

• Using dot notation incorrectly in import
• Trying to import class directly from nltk
• Wrong order of import keywords

3. What is the output of the following Python code using Porter Stemmer?

from nltk.stem import PorterStemmer
ps = PorterStemmer()
words = ['running', 'runs', 'runner']
stemmed = [ps.stem(word) for word in words]
print(stemmed)

medium

A. ['run', 'run', 'run']

B. ['running', 'runs', 'runner']

C. ['run', 'run', 'runner']

D. ['runn', 'run', 'runn']

Solution

1. Step 1: Apply Porter Stemmer to each word

   Porter Stemmer reduces 'running' and 'runs' to 'run', but 'runner' remains 'runner' because it is treated differently.

2. Step 2: List the stemmed results

   The list becomes ['run', 'run', 'runner'] after stemming.

3. Final Answer:

   ['run', 'run', 'runner'] -> Option C

4. Quick Check:

   Porter stems 'running' and 'runs' to 'run' [OK]

Hint: Porter stems common verb forms to root, but some nouns stay [OK]

Common Mistakes:

• Assuming all words stem to the same root
• Confusing stemmed output with original words
• Expecting 'runner' to stem to 'run'

4. Identify the error in this Snowball Stemmer usage code snippet:

from nltk.stem import SnowballStemmer
stemmer = SnowballStemmer('english')
words = ['happiness', 'happier', 'happy']
stemmed_words = [stemmer.stem(word) for word in words]
print(stemmed_words)

medium

A. The stem method should be called as stemmer.stem_word(word)

B. No error; code runs correctly and prints stemmed words

C. SnowballStemmer requires language name in uppercase

D. SnowballStemmer must be imported from nltk.stem.snowball

Solution

1. Step 1: Check SnowballStemmer import and usage

   Importing from nltk.stem and initializing with 'english' is correct and case-insensitive.

2. Step 2: Verify method call and output

   The stem method is correctly called as stemmer.stem(word), and the code prints stemmed words without error.

3. Final Answer:

   No error; code runs correctly and prints stemmed words -> Option B

4. Quick Check:

   SnowballStemmer usage is correct as shown [OK]

Hint: SnowballStemmer language is lowercase string, stem() method used [OK]

Common Mistakes:

• Using uppercase language name incorrectly
• Calling non-existent stem_word method
• Wrong import path for SnowballStemmer

5. You want to preprocess text data by stemming words but keep the original word if it is shorter than 4 characters. Which Python code snippet using Porter Stemmer correctly implements this?

hard

A. stemmed = [ps.stem(word) for word in words if len(word) >= 4]

B. stemmed = [ps.stem(word) if len(word) < 4 else word for word in words]

C. stemmed = [word for word in words if len(word) < 4 else ps.stem(word)]

D. stemmed = [word if len(word) < 4 else ps.stem(word) for word in words]

Solution

1. Step 1: Understand the condition for stemming

   Words shorter than 4 characters should remain unchanged; others should be stemmed.

2. Step 2: Check list comprehension syntax

   stemmed = [word if len(word) < 4 else ps.stem(word) for word in words] uses correct if-else inside list comprehension: 'word if len(word) < 4 else ps.stem(word)'.

3. Final Answer:

   stemmed = [word if len(word) < 4 else ps.stem(word) for word in words] -> Option D

4. Quick Check:

   Keep short words, stem others with if-else [OK]

Hint: Use 'word if condition else stem(word)' in list comprehension [OK]

Common Mistakes:

• Swapping if-else order in comprehension
• Using if without else causing missing elements
• Incorrect syntax mixing if-else and for