Text preprocessing (tokenization, stemming, lemmatization) in ML Python - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is tokenization in text preprocessing?

Tokenization is the process of breaking down text into smaller pieces called tokens, usually words or sentences, to make it easier for a computer to understand and analyze.

beginner

Explain stemming in simple terms.

Stemming cuts words down to their root form by removing endings. For example, 'running' becomes 'run'. It helps group similar words together but may not always produce real words.

intermediate

What does lemmatization do differently from stemming?

Lemmatization reduces words to their base or dictionary form called lemma, considering the word's meaning and context. For example, 'better' becomes 'good'. It produces real words unlike stemming.

beginner

Why is text preprocessing important before training a machine learning model?

Text preprocessing cleans and simplifies text data, making it easier for models to learn patterns. It reduces noise, handles variations of words, and improves model accuracy.

intermediate

Give an example of tokenization, stemming, and lemmatization for the sentence: 'The cats are running happily'.

Tokenization: ['The', 'cats', 'are', 'running', 'happily']<br>Stemming: ['The', 'cat', 'are', 'run', 'happili']<br>Lemmatization: ['The', 'cat', 'be', 'run', 'happily']

What is the main goal of tokenization?

A Calculate word frequency

B Split text into smaller pieces called tokens

C Convert words to their base form

D Remove stop words from text

Tokenization breaks text into tokens like words or sentences to prepare for analysis.

Which technique removes word endings without considering meaning?

A Stemming

B Lemmatization

C Tokenization

D Parsing

Stemming cuts word endings to get root forms but may not produce real words.

Which method uses word meaning and context to find the base form?

A Normalization

B Stemming

C Tokenization

D Lemmatization

Lemmatization considers meaning and context to return dictionary forms of words.

Why do we preprocess text before machine learning?

A To make text longer

B To translate text to another language

C To clean and simplify text for better learning

D To encrypt the text

Preprocessing cleans and simplifies text so models can learn patterns more effectively.

Which of these is NOT a typical step in text preprocessing?

A Image resizing

B Stemming

C Lemmatization

D Tokenization

Image resizing is unrelated to text preprocessing.

Describe the differences between tokenization, stemming, and lemmatization in text preprocessing.

Think about how each step changes the text and why.

Explain why text preprocessing is important before feeding text data into a machine learning model.

Consider what happens if raw text is used directly.