Text cleaning pipeline in Data Analysis Python - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the main purpose of a text cleaning pipeline?

A text cleaning pipeline prepares raw text data by removing unwanted parts and fixing errors so that the text is easier to analyze or use in models.

beginner

Name three common steps in a text cleaning pipeline.

Common steps include: 1) Lowercasing all text, 2) Removing punctuation and special characters, 3) Removing stopwords (common words like 'the', 'and').

beginner

Why do we remove stopwords in text cleaning?

Stopwords are very common words that usually do not add useful meaning for analysis, so removing them helps focus on important words.

intermediate

What Python library is commonly used for text cleaning tasks?

The 're' library is used for regular expressions to find and replace patterns, and 'nltk' or 'spaCy' are popular for more advanced text processing.

beginner

How does tokenization fit into a text cleaning pipeline?

Tokenization splits text into smaller pieces like words or sentences, making it easier to analyze or clean each part separately.

Which step is NOT typically part of a text cleaning pipeline?

A Adding random characters

B Removing punctuation

C Converting text to lowercase

D Removing stopwords

Adding random characters would make the text messier, not cleaner.

What does tokenization do in text cleaning?

A Splits text into smaller parts like words

B Changes text to uppercase

C Removes numbers from text

D Combines words into sentences

Tokenization breaks text into smaller pieces such as words or sentences.

Why is lowercasing text important in cleaning?

A To make text colorful

B To reduce differences between words like 'Apple' and 'apple'

C To remove punctuation

D To add spaces between words

Lowercasing helps treat words with different cases as the same word.

Which Python library helps with pattern matching in text cleaning?

A numpy

B matplotlib

C pandas

D re

The 're' library is used for regular expressions to find and replace text patterns.

What are stopwords?

A Numbers in text

B Rare words in text

C Common words like 'the' and 'is' that add little meaning

D Punctuation marks

Stopwords are common words that usually do not add useful meaning.

Describe the main steps you would include in a text cleaning pipeline and why each is important.

Think about how each step makes text easier to analyze.

Explain how you would use Python to clean a text dataset before analysis.

Imagine you have a messy text file and want to prepare it for counting word frequencies.

Practice

1. What is the main purpose of a text cleaning pipeline in data analysis?

easy

A. To translate text into another language

B. To make text uniform and easier to analyze

C. To increase the length of the text

D. To add more punctuation to the text

Solution

1. Step 1: Understand text cleaning goal

   Text cleaning aims to prepare text data by making it consistent and easier to work with.

2. Step 2: Identify correct purpose

   Adding punctuation or increasing length does not help analysis; uniformity does.

3. Final Answer:

   To make text uniform and easier to analyze -> Option B

4. Quick Check:

   Text cleaning = uniform text [OK]

Hint: Text cleaning makes text consistent for analysis [OK]

Common Mistakes:

• Thinking cleaning adds punctuation
• Confusing cleaning with translation
• Believing cleaning increases text length

2. Which of the following Python code snippets correctly converts a string text to lowercase?

easy

A. text.lower()

B. lower(text)

C. text.toLowerCase()

D. text.lowercase()

Solution

1. Step 1: Recall Python string method for lowercase

   Python strings have a method called lower() to convert text to lowercase.

2. Step 2: Check each option

   text.lower() uses correct syntax. Options B, C, D are invalid in Python.

3. Final Answer:

   text.lower() -> Option A

4. Quick Check:

   Python lowercase method = lower() [OK]

Hint: Use text.lower() to lowercase strings in Python [OK]

Common Mistakes:

• Using JavaScript method toLowerCase() in Python
• Trying a non-existent function lower() outside string
• Using incorrect method name like lowercase()

3. Given the following Python code, what is the output of cleaned_text?

import string
text = " Hello, World!  "
cleaned_text = text.lower().strip().translate(str.maketrans('', '', string.punctuation))
print(cleaned_text)

medium

A. " hello world "

B. "hello, world!"

C. "hello world"

D. "HELLO WORLD"

Solution

1. Step 1: Apply lower() and strip()

   The text becomes "hello, world!" after lowering and removing spaces at ends.

2. Step 2: Remove punctuation with translate()

   All punctuation marks like comma and exclamation are removed, resulting in "hello world".

3. Final Answer:

   "hello world" -> Option C

4. Quick Check:

   Lowercase + strip + remove punctuation = "hello world" [OK]

Hint: Chain methods: lower, strip, then remove punctuation [OK]

Common Mistakes:

• Forgetting to strip spaces
• Not removing punctuation correctly
• Confusing case conversion order

4. Identify the error in this text cleaning function and choose the fix:

def clean_text(text):
    text = text.lower()
    text = text.strip
    return text

print(clean_text("  DATA Science  "))

medium

A. Change text.strip to text.strip()

B. Change text.lower() to lower(text)

C. Add text = text.replace() to remove spaces

D. Remove the return statement

Solution

1. Step 1: Check method usage

   strip is a method and must be called with parentheses ().

2. Step 2: Identify fix

   Changing text.strip to text.strip() correctly removes spaces.

3. Final Answer:

   Change text.strip to text.strip() -> Option A

4. Quick Check:

   Method calls need parentheses [OK]

Hint: Always call string methods with parentheses [OK]

Common Mistakes:

• Forgetting parentheses on methods
• Using wrong syntax for lower()
• Removing return statement by mistake

5. You have a list of sentences with inconsistent spacing and punctuation:

texts = ["Data Science!  ", "  Machine Learning.", "AI, is fun!"]

Which code snippet correctly cleans all texts by making them lowercase, removing punctuation, and trimming spaces?

hard

A. cleaned = [t.strip().lower() for t in texts]

B. cleaned = [t.lower().replace(' ', '') for t in texts]

C. cleaned = [t.translate(str.maketrans('', '', string.punctuation)) for t in texts]

D. import string
cleaned = [t.lower().strip().translate(str.maketrans('', '', string.punctuation)) for t in texts]

Solution

1. Step 1: Understand cleaning steps needed

   We need lowercase, remove punctuation, and trim spaces for each text.

2. Step 2: Check each option

   import string cleaned = [t.lower().strip().translate(str.maketrans('', '', string.punctuation)) for t in texts] applies all three steps correctly using list comprehension. Others miss one or more steps.

3. Final Answer:

   import string cleaned = [t.lower().strip().translate(str.maketrans('', '', string.punctuation)) for t in texts] -> Option D

4. Quick Check:

   All steps in list comprehension = import string cleaned = [t.lower().strip().translate(str.maketrans('', '', string.punctuation)) for t in texts] [OK]

Hint: Use list comprehension chaining all cleaning steps [OK]

Common Mistakes:

• Missing punctuation removal
• Not trimming spaces
• Removing spaces incorrectly with replace