Text preprocessing (tokenization, stemming, lemmatization) in ML Python - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is tokenization in text preprocessing?

Tokenization is the process of breaking down text into smaller pieces called tokens, usually words or sentences, to make it easier for a computer to understand and analyze.

beginner

Explain stemming in simple terms.

Stemming cuts words down to their root form by removing endings. For example, 'running' becomes 'run'. It helps group similar words together but may not always produce real words.

intermediate

What does lemmatization do differently from stemming?

Lemmatization reduces words to their base or dictionary form called lemma, considering the word's meaning and context. For example, 'better' becomes 'good'. It produces real words unlike stemming.

beginner

Why is text preprocessing important before training a machine learning model?

Text preprocessing cleans and simplifies text data, making it easier for models to learn patterns. It reduces noise, handles variations of words, and improves model accuracy.

intermediate

Give an example of tokenization, stemming, and lemmatization for the sentence: 'The cats are running happily'.

Tokenization: ['The', 'cats', 'are', 'running', 'happily']
Stemming: ['The', 'cat', 'are', 'run', 'happili']
Lemmatization: ['The', 'cat', 'be', 'run', 'happily']

What is the main goal of tokenization?

A Calculate word frequency

B Split text into smaller pieces called tokens

C Convert words to their base form

D Remove stop words from text

Tokenization breaks text into tokens like words or sentences to prepare for analysis.

Which technique removes word endings without considering meaning?

A Stemming

B Lemmatization

C Tokenization

D Parsing

Stemming cuts word endings to get root forms but may not produce real words.

Which method uses word meaning and context to find the base form?

A Normalization

B Stemming

C Tokenization

D Lemmatization

Lemmatization considers meaning and context to return dictionary forms of words.

Why do we preprocess text before machine learning?

A To make text longer

B To translate text to another language

C To clean and simplify text for better learning

D To encrypt the text

Preprocessing cleans and simplifies text so models can learn patterns more effectively.

Which of these is NOT a typical step in text preprocessing?

A Image resizing

B Stemming

C Lemmatization

D Tokenization

Image resizing is unrelated to text preprocessing.

Describe the differences between tokenization, stemming, and lemmatization in text preprocessing.

Think about how each step changes the text and why.

Explain why text preprocessing is important before feeding text data into a machine learning model.

Consider what happens if raw text is used directly.

Practice

1. What is the main purpose of tokenization in text preprocessing?

easy

A. To split text into smaller units like words or tokens

B. To convert words to their base dictionary form

C. To remove stop words from the text

D. To reduce words to their root form by cutting suffixes

Solution

1. Step 1: Understand tokenization

   Tokenization means breaking text into smaller pieces called tokens, usually words.

2. Step 2: Compare with other preprocessing steps

   Lemmatization and stemming change words, but tokenization just splits text.

3. Final Answer:

   To split text into smaller units like words or tokens -> Option A

4. Quick Check:

   Tokenization = splitting text [OK]

Hint: Tokenization means splitting text into words or tokens [OK]

Common Mistakes:

• Confusing tokenization with stemming or lemmatization
• Thinking tokenization changes word forms
• Assuming tokenization removes words

2. Which of the following Python code snippets correctly performs stemming using NLTK's PorterStemmer?

easy

A. from nltk.tokenize import word_tokenize
word_tokenize('running')

B. import re
re.sub(r'\W+', '', 'running')

C. from nltk.stem import WordNetLemmatizer
wl = WordNetLemmatizer()
wl.lemmatize('running')

D. from nltk.stem import PorterStemmer
ps = PorterStemmer()
ps.stem('running')

Solution

1. Step 1: Identify stemming code

   PorterStemmer from nltk.stem is used for stemming; calling stem('running') returns stemmed word.

2. Step 2: Check other options

   from nltk.tokenize import word_tokenize word_tokenize('running') is tokenization, C is lemmatization, D is regex substitution, not stemming.

3. Final Answer:

   from nltk.stem import PorterStemmer ps = PorterStemmer() ps.stem('running') -> Option D

4. Quick Check:

   PorterStemmer.stem() = stemming [OK]

Hint: PorterStemmer.stem() is for stemming, not tokenization or lemmatization [OK]

Common Mistakes:

• Confusing stemming with lemmatization functions
• Using tokenization code for stemming
• Using regex instead of stemmer

3. What will be the output of the following Python code using NLTK's WordNetLemmatizer?

from nltk.stem import WordNetLemmatizer
wl = WordNetLemmatizer()
print(wl.lemmatize('better', pos='a'))

medium

A. good

B. bet

C. better

D. betters

Solution

1. Step 1: Understand lemmatization with POS

   Using pos='a' means adjective. Lemmatizer converts 'better' (comparative adjective) to base form 'good'.

2. Step 2: Check output possibilities

   Stemming would cut to 'bet', but lemmatization returns dictionary base word 'good'.

3. Final Answer:

   good -> Option A

4. Quick Check:

   Lemmatize('better', pos='a') = 'good' [OK]

Hint: Lemmatizer with pos='a' returns base adjective form [OK]

Common Mistakes:

• Expecting stemming output instead of lemmatization
• Ignoring pos parameter in lemmatization
• Assuming output is unchanged word

4. Identify the error in this code snippet for stemming a list of words using NLTK's PorterStemmer:

from nltk.stem import PorterStemmer
ps = PorterStemmer()
words = ['running', 'jumps', 'easily']
stemmed = [ps.stem(words)]
print(stemmed)

medium

A. The list comprehension syntax is incorrect

B. PorterStemmer is not imported correctly

C. ps.stem() expects a single word, but a list was passed

D. The print statement is missing parentheses

Solution

1. Step 1: Check ps.stem() input

   ps.stem() expects a single string word, but words (a list) was passed directly.

2. Step 2: Identify correct usage

   We should stem each word individually, e.g., using list comprehension: [ps.stem(w) for w in words].

3. Final Answer:

   ps.stem() expects a single word, but a list was passed -> Option C

4. Quick Check:

   Stem() needs single word input [OK]

Hint: Stem words one by one, not as a whole list [OK]

Common Mistakes:

• Passing list directly to stem() instead of iterating
• Confusing stemming with tokenization
• Incorrect list comprehension syntax

5. You want to preprocess text data by first splitting sentences into words, then reducing words to their dictionary base form, and finally removing suffixes to get root forms. Which sequence of steps is correct?

hard

A. Tokenization -> Lemmatization -> Stemming

B. Tokenization -> Stemming -> Lemmatization

C. Lemmatization -> Stemming -> Tokenization

D. Stemming -> Tokenization -> Lemmatization

Solution

1. Step 1: Understand preprocessing order

   First, split text into tokens (words) using tokenization.

2. Step 2: Apply stemming then lemmatization

   Stemming reduces words to root forms by removing suffixes; lemmatization converts words to dictionary base form.

3. Step 3: Verify option sequence

   Tokenization -> Stemming -> Lemmatization matches the correct order: Tokenization -> Stemming -> Lemmatization.

4. Final Answer:

   Tokenization -> Stemming -> Lemmatization -> Option B

5. Quick Check:

   Tokenize first, then stem, then lemmatize [OK]

Hint: Tokenize first, then stem, then lemmatize for best results [OK]

Common Mistakes:

• Applying stemming before tokenization
• Skipping tokenization step
• Confusing order of lemmatization and stemming