Lemmatization in NLP - Model Metrics & Evaluation

Metrics & Evaluation - Lemmatization

Which metric matters for Lemmatization and WHY

Lemmatization is about turning words into their base form, like "running" to "run". To check if a lemmatizer works well, we use accuracy. Accuracy here means how many words are correctly changed to their base form out of all words tested. This is important because a wrong base form can change the meaning and hurt later tasks like search or translation.

Confusion matrix for Lemmatization

          | Predicted Correct | Predicted Incorrect
    ------|-------------------|-------------------
    Actual Correct   |        TP = 85       |       FN = 15      
    Actual Incorrect |        FP = 10       |       TN = 90      

    Total words tested = 200

    TP: Words correctly lemmatized
    FN: Words that should be lemmatized but were not
    FP: Words incorrectly changed
    TN: Words correctly left unchanged
    

Precision vs Recall tradeoff in Lemmatization

Precision tells us: Of all words the model changed, how many were correct? High precision means fewer wrong changes.

Recall tells us: Of all words that needed changing, how many did the model catch? High recall means fewer missed changes.

For example, if a search engine uses lemmatization, high precision is important so it does not change words wrongly and confuse results. But if a language learning app uses it, high recall is important to catch all word forms and teach the base word.

Good vs Bad metric values for Lemmatization

• Good: Accuracy above 90%, Precision and Recall both above 85%. This means most words are correctly lemmatized and few mistakes happen.
• Bad: Accuracy below 70%, Precision or Recall below 60%. This means many words are wrongly changed or missed, hurting downstream tasks.

Common pitfalls in Lemmatization metrics

• Accuracy paradox: If most words don't need changing, a model that never changes words can have high accuracy but be useless.
• Data leakage: Testing on words seen during training inflates accuracy falsely.
• Overfitting: Model works well on training words but fails on new words, showing high training accuracy but low real accuracy.

Self-check question

Your lemmatization model has 98% accuracy but only 12% recall on words that need changing. Is it good for production? Why or why not?

Answer: No, it is not good. The model misses most words that need lemmatization (low recall), so it fails to convert many words correctly. High accuracy here is misleading because most words don't need changing, so the model just leaves them alone.

Key Result

Accuracy, precision, and recall together show how well a lemmatizer correctly changes words without missing or wrongly changing them.

Practice

1. What is the main purpose of lemmatization in natural language processing?

easy

A. To find the base or dictionary form of a word

B. To count the frequency of words in a text

C. To translate text from one language to another

D. To remove stop words from a sentence

Solution

1. Step 1: Understand the goal of lemmatization

   Lemmatization simplifies words by converting them to their base or dictionary form, like 'running' to 'run'.

2. Step 2: Compare with other options

   Counting words, translating, or removing stop words are different NLP tasks unrelated to lemmatization.

3. Final Answer:

   To find the base or dictionary form of a word -> Option A

4. Quick Check:

   Lemmatization = base form extraction [OK]

Hint: Lemmatization = find root word form [OK]

Common Mistakes:

• Confusing lemmatization with stemming
• Thinking it counts words
• Mixing it with translation tasks

2. Which of the following is the correct way to use the WordNetLemmatizer from NLTK to lemmatize the word 'better' as an adjective?

easy

A. lemmatizer.lemmatize('better', pos='a')

B. lemmatizer.lemmatize('better', pos='v')

C. lemmatizer.lemmatize('better')

D. lemmatizer.lemmatize('better', pos='n')

Solution

1. Step 1: Identify correct POS tag for adjective

   In NLTK, 'a' is the POS tag for adjective, so to lemmatize 'better' as adjective, use pos='a'.

2. Step 2: Check other POS tags

   'v' is verb, 'n' is noun, and no POS defaults to noun, which is incorrect here.

3. Final Answer:

   lemmatizer.lemmatize('better', pos='a') -> Option A

4. Quick Check:

   POS 'a' = adjective lemmatization [OK]

Hint: Use pos='a' for adjectives in lemmatizer [OK]

Common Mistakes:

• Omitting POS tag defaults to noun
• Using wrong POS like 'v' for adjective
• Confusing POS tags with part of speech names

3. What will be the output of the following Python code using NLTK's WordNetLemmatizer?

from nltk.stem import WordNetLemmatizer
lemmatizer = WordNetLemmatizer()
print(lemmatizer.lemmatize('wolves'))

medium

A. 'wolves'

B. Error: missing POS argument

C. 'wolve'

D. 'wolf'

Solution

1. Step 1: Understand default POS in lemmatize()

   By default, lemmatize() assumes POS='n' (noun). 'wolves' is plural noun.

2. Step 2: Lemmatize plural noun

   The lemmatizer converts plural nouns to singular, so 'wolves' becomes 'wolf'.

3. Final Answer:

   'wolf' -> Option D

4. Quick Check:

   Plural noun 'wolves' -> singular 'wolf' [OK]

Hint: Default POS='n' converts plurals to singular [OK]

Common Mistakes:

• Expecting output to be unchanged plural
• Thinking POS argument is mandatory
• Confusing lemmatization with stemming

4. Consider this code snippet:

from nltk.stem import WordNetLemmatizer
lemmatizer = WordNetLemmatizer()
word = 'running'
print(lemmatizer.lemmatize(word))

Why does the output remain 'running' instead of 'run'?

medium

A. Because the lemmatizer cannot process verbs

B. Because the default POS is noun, and 'running' as noun stays unchanged

C. Because the word is misspelled

D. Because lemmatization always returns the original word

Solution

1. Step 1: Check default POS in lemmatize()

   Without specifying POS, lemmatize() treats words as nouns by default.

2. Step 2: Analyze 'running' as noun

   As a noun, 'running' is valid and unchanged, so output remains 'running'.

3. Final Answer:

   Because the default POS is noun, and 'running' as noun stays unchanged -> Option B

4. Quick Check:

   Default POS noun keeps 'running' unchanged [OK]

Hint: Specify POS='v' to lemmatize verbs correctly [OK]

Common Mistakes:

• Assuming lemmatizer always changes words
• Not specifying POS for verbs
• Thinking 'running' is misspelled

5. You want to lemmatize the sentence 'The striped bats are hanging on their feet.' correctly using NLTK. Which approach will give the best lemmatization results?

hard

A. Lemmatize each word without POS tags

B. Remove stop words before lemmatization

C. Lemmatize each word with POS tags obtained from POS tagging

D. Use stemming instead of lemmatization

Solution

1. Step 1: Understand importance of POS tags in lemmatization

   Lemmatization accuracy improves when each word's part of speech is known and used.

2. Step 2: Compare approaches

   Lemmatizing without POS tags may give wrong base forms; stemming changes words roughly; removing stop words doesn't improve lemmatization.

3. Final Answer:

   Lemmatize each word with POS tags obtained from POS tagging -> Option C

4. Quick Check:

   POS tagging + lemmatization = best accuracy [OK]

Hint: Use POS tags for accurate lemmatization [OK]

Common Mistakes:

• Skipping POS tagging before lemmatization
• Confusing stemming with lemmatization
• Thinking stop word removal affects lemmatization