Text preprocessing (tokenization, stemming, lemmatization) in ML Python - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of Tokenization Code

What is the output of the following Python code using NLTK's word_tokenize function?

ML Python

from nltk.tokenize import word_tokenize
text = "Hello there! How's it going?"
tokens = word_tokenize(text)
print(tokens)

A ['Hello', 'there', '!', 'How', "'s", 'it', 'going', '?']

B ['Hello', 'there', '!', 'How', ''s', 'it', 'going', '?']

C ['Hello', 'there', '!', 'How's', 'it', 'going', '?']

D ['Hello', 'there', '!', 'How', 's', 'it', 'going', '?']

💡 Hint

Remember how word_tokenize handles contractions and punctuation.

NLTK's word_tokenize splits contractions like "How's" into ['How', "'s"], keeping the apostrophe as part of the token.

🧠 Conceptual

intermediate

Difference Between Stemming and Lemmatization

Which statement correctly describes the difference between stemming and lemmatization?

A Stemming removes suffixes to get root forms, while lemmatization returns dictionary base forms considering context.

B Lemmatization removes suffixes to get root forms, while stemming returns dictionary base forms considering context.

C Both stemming and lemmatization always return the same root word without context.

D Stemming and lemmatization are identical processes with different names.

💡 Hint

Think about whether the process uses a dictionary or just cuts word endings.

Stemming is a crude process that chops off word endings, often producing non-words. Lemmatization uses vocabulary and morphological analysis to return the base or dictionary form of a word.

❓ Metrics

advanced

Evaluating Tokenization Quality

You have two tokenization methods applied to the sentence: "Cats running faster than dogs." Method A produces 6 tokens, Method B produces 5 tokens. Which metric best helps decide which tokenization is better?

A Use the length of the longest token as the metric.

B Use precision and recall comparing tokens to a gold standard.

C Use model training accuracy directly without token evaluation.

D Token count alone is enough to decide quality.

💡 Hint

Think about how to compare token lists to a correct reference.

Precision and recall compare the tokens produced by a method to a gold standard set of tokens, measuring how many tokens are correctly identified and how many are missed or extra.

🔧 Debug

advanced

Error in Lemmatization Code

What error does this code raise? from nltk.stem import WordNetLemmatizer lemmatizer = WordNetLemmatizer() word = 'running' print(lemmatizer.lemmatize(word))

A TypeError because lemmatize requires a list, not a string.

B NameError because WordNetLemmatizer is not imported.

C SyntaxError due to missing parentheses in print statement.

D It prints 'running' because default pos is 'noun', so no change.

💡 Hint

Check the default part of speech used by lemmatizer.

The lemmatizer defaults to noun pos, so 'running' as a noun stays 'running'. To get 'run', pos='v' (verb) must be specified.

❓ Model Choice

expert

Choosing Preprocessing for Sentiment Analysis

You want to build a sentiment analysis model on social media text with many slang and misspellings. Which preprocessing step combination is best?

A Use only tokenization without stemming or lemmatization to preserve slang.

B Use stemming to reduce words to roots, ignoring slang and misspellings.

C Use lemmatization with POS tagging to normalize words, plus custom slang dictionary.

D Remove all punctuation and lowercase text only, no tokenization.

💡 Hint

Consider how to handle slang and misspellings while normalizing words.

Lemmatization with POS tagging normalizes words properly, and a custom slang dictionary helps handle informal language common in social media, improving model input quality.