Text preprocessing (tokenization, stemming, lemmatization) in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - Text preprocessing (tokenization, stemming, lemmatization)

Which metric matters for this concept and WHY

For text preprocessing steps like tokenization, stemming, and lemmatization, the key metrics focus on accuracy of text normalization. This means how well the processed text matches the intended meaning or structure for the model. Metrics include tokenization accuracy (correctly splitting text), and normalization accuracy (correctly reducing words to base forms). These matter because poor preprocessing leads to noisy or misleading input, hurting model performance.

Confusion matrix or equivalent visualization

While confusion matrices are common for classification, here we use a token-level accuracy table to show preprocessing quality.

Tokenization Results:
+----------------+----------------+----------------+
| Original Token | Correct Split? | Count          |
+----------------+----------------+----------------+
| "running"     | Yes            | 100            |
| "runing"      | No             | 5              |
+----------------+----------------+----------------+

Stemming/Lemmatization Results:
+----------------+----------------+----------------+
| Original Word  | Correct Base?  | Count          |
+----------------+----------------+----------------+
| "running"     | "run" (Yes)   | 100            |
| "ran"         | "run" (Yes)   | 50             |
| "runs"        | "run" (Yes)   | 80             |
| "runned"      | Incorrect      | 3              |
+----------------+----------------+----------------+
    

Precision vs Recall tradeoff with concrete examples

In text preprocessing, the tradeoff is between over-normalization and under-normalization.

• Over-normalization (High Precision, Low Recall): Only very sure words are stemmed or lemmatized, so fewer errors but many words remain unprocessed. This keeps precision high (correct base forms) but recall low (many words not normalized).
• Under-normalization (Low Precision, High Recall): Aggressive stemming changes many words, catching most variants but also making mistakes (wrong base forms). This increases recall but lowers precision.

Example: For a search engine, high recall is better to find all relevant documents, even if some words are wrongly normalized. For a grammar checker, high precision is better to avoid false corrections.

What "good" vs "bad" metric values look like for this use case

Good preprocessing:

• Tokenization accuracy > 98% (almost all words split correctly)
• Stemming/Lemmatization accuracy > 90% (most words reduced correctly)
• Low error rate in base form assignment

Bad preprocessing:

• Tokenization accuracy < 90% (many words incorrectly split or joined)
• Stemming/Lemmatization accuracy < 70% (many words wrongly changed)
• High noise in processed text causing model confusion

Metrics pitfalls

• Ignoring context: Stemming can cut words wrongly without context, hurting meaning.
• Overfitting preprocessing: Tailoring stemming rules too much to training data can fail on new text.
• Data leakage: Using test data to tune preprocessing can inflate accuracy falsely.
• Accuracy paradox: High tokenization accuracy may hide poor lemmatization quality.
• Ignoring downstream impact: Good preprocessing metrics alone don't guarantee better model results.

Self-check question

Your text preprocessing pipeline has 99% tokenization accuracy but only 60% lemmatization accuracy. Is this good enough for a sentiment analysis model? Why or why not?

Answer: Not necessarily. While tokenization is excellent, low lemmatization accuracy means many words are not properly normalized. This can confuse the model by treating similar words as different, hurting sentiment detection. Improving lemmatization or using better normalization is important.

Key Result

High tokenization and lemmatization accuracy are key to clean text input, directly impacting model quality.

Practice

1. What is the main purpose of tokenization in text preprocessing?

easy

A. To split text into smaller units like words or tokens

B. To convert words to their base dictionary form

C. To remove stop words from the text

D. To reduce words to their root form by cutting suffixes

Solution

1. Step 1: Understand tokenization

   Tokenization means breaking text into smaller pieces called tokens, usually words.

2. Step 2: Compare with other preprocessing steps

   Lemmatization and stemming change words, but tokenization just splits text.

3. Final Answer:

   To split text into smaller units like words or tokens -> Option A

4. Quick Check:

   Tokenization = splitting text [OK]

Hint: Tokenization means splitting text into words or tokens [OK]

Common Mistakes:

• Confusing tokenization with stemming or lemmatization
• Thinking tokenization changes word forms
• Assuming tokenization removes words

2. Which of the following Python code snippets correctly performs stemming using NLTK's PorterStemmer?

easy

A. from nltk.tokenize import word_tokenize
word_tokenize('running')

B. import re
re.sub(r'\W+', '', 'running')

C. from nltk.stem import WordNetLemmatizer
wl = WordNetLemmatizer()
wl.lemmatize('running')

D. from nltk.stem import PorterStemmer
ps = PorterStemmer()
ps.stem('running')

Solution

1. Step 1: Identify stemming code

   PorterStemmer from nltk.stem is used for stemming; calling stem('running') returns stemmed word.

2. Step 2: Check other options

   from nltk.tokenize import word_tokenize word_tokenize('running') is tokenization, C is lemmatization, D is regex substitution, not stemming.

3. Final Answer:

   from nltk.stem import PorterStemmer ps = PorterStemmer() ps.stem('running') -> Option D

4. Quick Check:

   PorterStemmer.stem() = stemming [OK]

Hint: PorterStemmer.stem() is for stemming, not tokenization or lemmatization [OK]

Common Mistakes:

• Confusing stemming with lemmatization functions
• Using tokenization code for stemming
• Using regex instead of stemmer

3. What will be the output of the following Python code using NLTK's WordNetLemmatizer?

from nltk.stem import WordNetLemmatizer
wl = WordNetLemmatizer()
print(wl.lemmatize('better', pos='a'))

medium

A. good

B. bet

C. better

D. betters

Solution

1. Step 1: Understand lemmatization with POS

   Using pos='a' means adjective. Lemmatizer converts 'better' (comparative adjective) to base form 'good'.

2. Step 2: Check output possibilities

   Stemming would cut to 'bet', but lemmatization returns dictionary base word 'good'.

3. Final Answer:

   good -> Option A

4. Quick Check:

   Lemmatize('better', pos='a') = 'good' [OK]

Hint: Lemmatizer with pos='a' returns base adjective form [OK]

Common Mistakes:

• Expecting stemming output instead of lemmatization
• Ignoring pos parameter in lemmatization
• Assuming output is unchanged word

4. Identify the error in this code snippet for stemming a list of words using NLTK's PorterStemmer:

from nltk.stem import PorterStemmer
ps = PorterStemmer()
words = ['running', 'jumps', 'easily']
stemmed = [ps.stem(words)]
print(stemmed)

medium

A. The list comprehension syntax is incorrect

B. PorterStemmer is not imported correctly

C. ps.stem() expects a single word, but a list was passed

D. The print statement is missing parentheses

Solution

1. Step 1: Check ps.stem() input

   ps.stem() expects a single string word, but words (a list) was passed directly.

2. Step 2: Identify correct usage

   We should stem each word individually, e.g., using list comprehension: [ps.stem(w) for w in words].

3. Final Answer:

   ps.stem() expects a single word, but a list was passed -> Option C

4. Quick Check:

   Stem() needs single word input [OK]

Hint: Stem words one by one, not as a whole list [OK]

Common Mistakes:

• Passing list directly to stem() instead of iterating
• Confusing stemming with tokenization
• Incorrect list comprehension syntax

5. You want to preprocess text data by first splitting sentences into words, then reducing words to their dictionary base form, and finally removing suffixes to get root forms. Which sequence of steps is correct?

hard

A. Tokenization -> Lemmatization -> Stemming

B. Tokenization -> Stemming -> Lemmatization

C. Lemmatization -> Stemming -> Tokenization

D. Stemming -> Tokenization -> Lemmatization

Solution

1. Step 1: Understand preprocessing order

   First, split text into tokens (words) using tokenization.

2. Step 2: Apply stemming then lemmatization

   Stemming reduces words to root forms by removing suffixes; lemmatization converts words to dictionary base form.

3. Step 3: Verify option sequence

   Tokenization -> Stemming -> Lemmatization matches the correct order: Tokenization -> Stemming -> Lemmatization.

4. Final Answer:

   Tokenization -> Stemming -> Lemmatization -> Option B

5. Quick Check:

   Tokenize first, then stem, then lemmatize [OK]

Hint: Tokenize first, then stem, then lemmatize for best results [OK]

Common Mistakes:

• Applying stemming before tokenization
• Skipping tokenization step
• Confusing order of lemmatization and stemming