Lemmatization in NLP - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of Lemmatization with POS Tag

What is the output of the following Python code using NLTK's WordNetLemmatizer when lemmatizing the word 'better' as an adjective?

NLP

from nltk.stem import WordNetLemmatizer
lemmatizer = WordNetLemmatizer()
result = lemmatizer.lemmatize('better', pos='a')
print(result)

A "better"

B "bet"

C "good"

D "bettered"

💡 Hint

Consider how lemmatization uses part of speech to find the base form.

When lemmatizing 'better' as an adjective (pos='a'), the lemmatizer returns its base form 'good'. Without specifying pos, it would return 'better'.

🧠 Conceptual

intermediate

Purpose of Lemmatization in Text Processing

Which of the following best describes the main purpose of lemmatization in natural language processing?

A To translate text from one language to another

B To remove all punctuation and special characters from text

C To split text into individual words or tokens

D To reduce words to their base or dictionary form considering context

💡 Hint

Think about how lemmatization differs from simple word cutting.

Lemmatization reduces words to their base or dictionary form (lemma) by considering the word's meaning and part of speech, unlike stemming which may just cut off endings.

❓ Metrics

advanced

Evaluating Lemmatization Impact on Text Classification

You apply lemmatization to your training and test datasets before training a text classifier. Which metric is best to compare model performance before and after lemmatization?

A Accuracy on the test set

B Number of unique tokens in the vocabulary

C Training loss only

D Time taken to train the model

💡 Hint

Focus on how well the model predicts unseen data.

Accuracy on the test set measures how well the model generalizes to new data, showing if lemmatization improved understanding.

🔧 Debug

advanced

Identifying the Error in Lemmatization Code

What error will this code raise when run? from nltk.stem import WordNetLemmatizer lemmatizer = WordNetLemmatizer() word = 'running' result = lemmatizer.lemmatize(word) print(result)

NLP

from nltk.stem import WordNetLemmatizer
lemmatizer = WordNetLemmatizer()
word = 'running'
result = lemmatizer.lemmatize(word)
print(result)

A Prints 'run' without error

B Prints 'running' without error

C Raises TypeError due to missing pos argument

D Raises NameError because WordNetLemmatizer is not imported

💡 Hint

Check the default behavior of lemmatize without pos argument.

Without specifying pos, lemmatize treats the word as a noun, so 'running' stays 'running'. No error occurs.

❓ Model Choice

expert

Choosing the Best Lemmatization Approach for Ambiguous Words

You want to lemmatize words in a sentence where words can have multiple parts of speech (e.g., 'saw' as noun or verb). Which approach will give the most accurate lemmatization?

A First perform POS tagging, then lemmatize each word with its POS tag

B Use a stemmer instead of a lemmatizer

C Lemmatize all words assuming they are nouns

D Lemmatize words without specifying POS tags

💡 Hint

Think about how context helps decide the correct base form.

POS tagging before lemmatization helps identify the correct part of speech, so the lemmatizer can find the right base form for ambiguous words.

Practice

1. What is the main purpose of lemmatization in natural language processing?

easy

A. To find the base or dictionary form of a word

B. To count the frequency of words in a text

C. To translate text from one language to another

D. To remove stop words from a sentence

Solution

1. Step 1: Understand the goal of lemmatization

   Lemmatization simplifies words by converting them to their base or dictionary form, like 'running' to 'run'.

2. Step 2: Compare with other options

   Counting words, translating, or removing stop words are different NLP tasks unrelated to lemmatization.

3. Final Answer:

   To find the base or dictionary form of a word -> Option A

4. Quick Check:

   Lemmatization = base form extraction [OK]

Hint: Lemmatization = find root word form [OK]

Common Mistakes:

• Confusing lemmatization with stemming
• Thinking it counts words
• Mixing it with translation tasks

2. Which of the following is the correct way to use the WordNetLemmatizer from NLTK to lemmatize the word 'better' as an adjective?

easy

A. lemmatizer.lemmatize('better', pos='a')

B. lemmatizer.lemmatize('better', pos='v')

C. lemmatizer.lemmatize('better')

D. lemmatizer.lemmatize('better', pos='n')

Solution

1. Step 1: Identify correct POS tag for adjective

   In NLTK, 'a' is the POS tag for adjective, so to lemmatize 'better' as adjective, use pos='a'.

2. Step 2: Check other POS tags

   'v' is verb, 'n' is noun, and no POS defaults to noun, which is incorrect here.

3. Final Answer:

   lemmatizer.lemmatize('better', pos='a') -> Option A

4. Quick Check:

   POS 'a' = adjective lemmatization [OK]

Hint: Use pos='a' for adjectives in lemmatizer [OK]

Common Mistakes:

• Omitting POS tag defaults to noun
• Using wrong POS like 'v' for adjective
• Confusing POS tags with part of speech names

3. What will be the output of the following Python code using NLTK's WordNetLemmatizer?

from nltk.stem import WordNetLemmatizer
lemmatizer = WordNetLemmatizer()
print(lemmatizer.lemmatize('wolves'))

medium

A. 'wolves'

B. Error: missing POS argument

C. 'wolve'

D. 'wolf'

Solution

1. Step 1: Understand default POS in lemmatize()

   By default, lemmatize() assumes POS='n' (noun). 'wolves' is plural noun.

2. Step 2: Lemmatize plural noun

   The lemmatizer converts plural nouns to singular, so 'wolves' becomes 'wolf'.

3. Final Answer:

   'wolf' -> Option D

4. Quick Check:

   Plural noun 'wolves' -> singular 'wolf' [OK]

Hint: Default POS='n' converts plurals to singular [OK]

Common Mistakes:

• Expecting output to be unchanged plural
• Thinking POS argument is mandatory
• Confusing lemmatization with stemming

4. Consider this code snippet:

from nltk.stem import WordNetLemmatizer
lemmatizer = WordNetLemmatizer()
word = 'running'
print(lemmatizer.lemmatize(word))

Why does the output remain 'running' instead of 'run'?

medium

A. Because the lemmatizer cannot process verbs

B. Because the default POS is noun, and 'running' as noun stays unchanged

C. Because the word is misspelled

D. Because lemmatization always returns the original word

Solution

1. Step 1: Check default POS in lemmatize()

   Without specifying POS, lemmatize() treats words as nouns by default.

2. Step 2: Analyze 'running' as noun

   As a noun, 'running' is valid and unchanged, so output remains 'running'.

3. Final Answer:

   Because the default POS is noun, and 'running' as noun stays unchanged -> Option B

4. Quick Check:

   Default POS noun keeps 'running' unchanged [OK]

Hint: Specify POS='v' to lemmatize verbs correctly [OK]

Common Mistakes:

• Assuming lemmatizer always changes words
• Not specifying POS for verbs
• Thinking 'running' is misspelled

5. You want to lemmatize the sentence 'The striped bats are hanging on their feet.' correctly using NLTK. Which approach will give the best lemmatization results?

hard

A. Lemmatize each word without POS tags

B. Remove stop words before lemmatization

C. Lemmatize each word with POS tags obtained from POS tagging

D. Use stemming instead of lemmatization

Solution

1. Step 1: Understand importance of POS tags in lemmatization

   Lemmatization accuracy improves when each word's part of speech is known and used.

2. Step 2: Compare approaches

   Lemmatizing without POS tags may give wrong base forms; stemming changes words roughly; removing stop words doesn't improve lemmatization.

3. Final Answer:

   Lemmatize each word with POS tags obtained from POS tagging -> Option C

4. Quick Check:

   POS tagging + lemmatization = best accuracy [OK]

Hint: Use POS tags for accurate lemmatization [OK]

Common Mistakes:

• Skipping POS tagging before lemmatization
• Confusing stemming with lemmatization
• Thinking stop word removal affects lemmatization