Text preprocessing (tokenization, stemming, lemmatization) in ML Python - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of Tokenization Code

What is the output of the following Python code using NLTK's word_tokenize function?

ML Python

from nltk.tokenize import word_tokenize
text = "Hello there! How's it going?"
tokens = word_tokenize(text)
print(tokens)

A ['Hello', 'there', '!', 'How', "'s", 'it', 'going', '?']

B ['Hello', 'there', '!', 'How', ''s', 'it', 'going', '?']

C ['Hello', 'there', '!', 'How's', 'it', 'going', '?']

D ['Hello', 'there', '!', 'How', 's', 'it', 'going', '?']

💡 Hint

Remember how word_tokenize handles contractions and punctuation.

NLTK's word_tokenize splits contractions like "How's" into ['How', "'s"], keeping the apostrophe as part of the token.

🧠 Conceptual

intermediate

Difference Between Stemming and Lemmatization

Which statement correctly describes the difference between stemming and lemmatization?

A Stemming removes suffixes to get root forms, while lemmatization returns dictionary base forms considering context.

B Lemmatization removes suffixes to get root forms, while stemming returns dictionary base forms considering context.

C Both stemming and lemmatization always return the same root word without context.

D Stemming and lemmatization are identical processes with different names.

💡 Hint

Think about whether the process uses a dictionary or just cuts word endings.

Stemming is a crude process that chops off word endings, often producing non-words. Lemmatization uses vocabulary and morphological analysis to return the base or dictionary form of a word.

❓ Metrics

advanced

Evaluating Tokenization Quality

You have two tokenization methods applied to the sentence: "Cats running faster than dogs." Method A produces 6 tokens, Method B produces 5 tokens. Which metric best helps decide which tokenization is better?

A Use the length of the longest token as the metric.

B Use precision and recall comparing tokens to a gold standard.

C Use model training accuracy directly without token evaluation.

D Token count alone is enough to decide quality.

💡 Hint

Think about how to compare token lists to a correct reference.

Precision and recall compare the tokens produced by a method to a gold standard set of tokens, measuring how many tokens are correctly identified and how many are missed or extra.

🔧 Debug

advanced

Error in Lemmatization Code

What error does this code raise? from nltk.stem import WordNetLemmatizer lemmatizer = WordNetLemmatizer() word = 'running' print(lemmatizer.lemmatize(word))

A TypeError because lemmatize requires a list, not a string.

B NameError because WordNetLemmatizer is not imported.

C SyntaxError due to missing parentheses in print statement.

D It prints 'running' because default pos is 'noun', so no change.

💡 Hint

Check the default part of speech used by lemmatizer.

The lemmatizer defaults to noun pos, so 'running' as a noun stays 'running'. To get 'run', pos='v' (verb) must be specified.

❓ Model Choice

expert

Choosing Preprocessing for Sentiment Analysis

You want to build a sentiment analysis model on social media text with many slang and misspellings. Which preprocessing step combination is best?

A Use only tokenization without stemming or lemmatization to preserve slang.

B Use stemming to reduce words to roots, ignoring slang and misspellings.

C Use lemmatization with POS tagging to normalize words, plus custom slang dictionary.

D Remove all punctuation and lowercase text only, no tokenization.

💡 Hint

Consider how to handle slang and misspellings while normalizing words.

Lemmatization with POS tagging normalizes words properly, and a custom slang dictionary helps handle informal language common in social media, improving model input quality.

Practice

1. What is the main purpose of tokenization in text preprocessing?

easy

A. To split text into smaller units like words or tokens

B. To convert words to their base dictionary form

C. To remove stop words from the text

D. To reduce words to their root form by cutting suffixes

Solution

1. Step 1: Understand tokenization

   Tokenization means breaking text into smaller pieces called tokens, usually words.

2. Step 2: Compare with other preprocessing steps

   Lemmatization and stemming change words, but tokenization just splits text.

3. Final Answer:

   To split text into smaller units like words or tokens -> Option A

4. Quick Check:

   Tokenization = splitting text [OK]

Hint: Tokenization means splitting text into words or tokens [OK]

Common Mistakes:

• Confusing tokenization with stemming or lemmatization
• Thinking tokenization changes word forms
• Assuming tokenization removes words

2. Which of the following Python code snippets correctly performs stemming using NLTK's PorterStemmer?

easy

A. from nltk.tokenize import word_tokenize
word_tokenize('running')

B. import re
re.sub(r'\W+', '', 'running')

C. from nltk.stem import WordNetLemmatizer
wl = WordNetLemmatizer()
wl.lemmatize('running')

D. from nltk.stem import PorterStemmer
ps = PorterStemmer()
ps.stem('running')

Solution

1. Step 1: Identify stemming code

   PorterStemmer from nltk.stem is used for stemming; calling stem('running') returns stemmed word.

2. Step 2: Check other options

   from nltk.tokenize import word_tokenize word_tokenize('running') is tokenization, C is lemmatization, D is regex substitution, not stemming.

3. Final Answer:

   from nltk.stem import PorterStemmer ps = PorterStemmer() ps.stem('running') -> Option D

4. Quick Check:

   PorterStemmer.stem() = stemming [OK]

Hint: PorterStemmer.stem() is for stemming, not tokenization or lemmatization [OK]

Common Mistakes:

• Confusing stemming with lemmatization functions
• Using tokenization code for stemming
• Using regex instead of stemmer

3. What will be the output of the following Python code using NLTK's WordNetLemmatizer?

from nltk.stem import WordNetLemmatizer
wl = WordNetLemmatizer()
print(wl.lemmatize('better', pos='a'))

medium

A. good

B. bet

C. better

D. betters

Solution

1. Step 1: Understand lemmatization with POS

   Using pos='a' means adjective. Lemmatizer converts 'better' (comparative adjective) to base form 'good'.

2. Step 2: Check output possibilities

   Stemming would cut to 'bet', but lemmatization returns dictionary base word 'good'.

3. Final Answer:

   good -> Option A

4. Quick Check:

   Lemmatize('better', pos='a') = 'good' [OK]

Hint: Lemmatizer with pos='a' returns base adjective form [OK]

Common Mistakes:

• Expecting stemming output instead of lemmatization
• Ignoring pos parameter in lemmatization
• Assuming output is unchanged word

4. Identify the error in this code snippet for stemming a list of words using NLTK's PorterStemmer:

from nltk.stem import PorterStemmer
ps = PorterStemmer()
words = ['running', 'jumps', 'easily']
stemmed = [ps.stem(words)]
print(stemmed)

medium

A. The list comprehension syntax is incorrect

B. PorterStemmer is not imported correctly

C. ps.stem() expects a single word, but a list was passed

D. The print statement is missing parentheses

Solution

1. Step 1: Check ps.stem() input

   ps.stem() expects a single string word, but words (a list) was passed directly.

2. Step 2: Identify correct usage

   We should stem each word individually, e.g., using list comprehension: [ps.stem(w) for w in words].

3. Final Answer:

   ps.stem() expects a single word, but a list was passed -> Option C

4. Quick Check:

   Stem() needs single word input [OK]

Hint: Stem words one by one, not as a whole list [OK]

Common Mistakes:

• Passing list directly to stem() instead of iterating
• Confusing stemming with tokenization
• Incorrect list comprehension syntax

5. You want to preprocess text data by first splitting sentences into words, then reducing words to their dictionary base form, and finally removing suffixes to get root forms. Which sequence of steps is correct?

hard

A. Tokenization -> Lemmatization -> Stemming

B. Tokenization -> Stemming -> Lemmatization

C. Lemmatization -> Stemming -> Tokenization

D. Stemming -> Tokenization -> Lemmatization

Solution

1. Step 1: Understand preprocessing order

   First, split text into tokens (words) using tokenization.

2. Step 2: Apply stemming then lemmatization

   Stemming reduces words to root forms by removing suffixes; lemmatization converts words to dictionary base form.

3. Step 3: Verify option sequence

   Tokenization -> Stemming -> Lemmatization matches the correct order: Tokenization -> Stemming -> Lemmatization.

4. Final Answer:

   Tokenization -> Stemming -> Lemmatization -> Option B

5. Quick Check:

   Tokenize first, then stem, then lemmatize [OK]

Hint: Tokenize first, then stem, then lemmatize for best results [OK]

Common Mistakes:

• Applying stemming before tokenization
• Skipping tokenization step
• Confusing order of lemmatization and stemming