Text cleaning pipeline in Data Analysis Python - Time & Space Complexity

Time Complexity: Text cleaning pipeline

O(n * m)

Understanding Time Complexity

We want to understand how the time needed to clean text grows as the amount of text grows.

How does the cleaning process scale when we have more words or sentences?

Scenario Under Consideration

Analyze the time complexity of the following code snippet.

import re

def clean_text(texts):
    cleaned = []
    for text in texts:
        text = text.lower()
        text = re.sub(r'[^a-z ]', '', text)
        words = text.split()
        cleaned.append(words)
    return cleaned

This code cleans a list of text strings by making them lowercase, removing non-letter characters, and splitting into words.

Identify Repeating Operations

Identify the loops, recursion, array traversals that repeat.

• Primary operation: Looping over each text string and processing each character inside the regex substitution and splitting.
• How many times: Once for each text string, and inside each string, operations run proportional to its length.

How Execution Grows With Input

As the number of texts and their length grow, the cleaning time grows roughly in proportion to total characters.

Input Size (n texts)	Approx. Operations (proportional to total characters)
10 (short texts)	Low, quick cleaning
100 (medium texts)	About 10 times more work
1000 (long texts)	About 100 times more work

Pattern observation: The time grows roughly linearly with the total amount of text to clean.

Final Time Complexity

Time Complexity: O(n * m)

This means the cleaning time grows linearly with the number of texts (n) and the average length of each text (m).

Common Mistake

[X] Wrong: "The cleaning time depends only on the number of texts, not their length."

[OK] Correct: Each text's length affects how many characters are processed, so longer texts take more time even if the count is the same.

Interview Connect

Understanding how text cleaning scales helps you explain your approach clearly and shows you think about efficiency in real data tasks.

Self-Check

"What if we added a nested loop to check each word against a list of stopwords? How would the time complexity change?"

Practice

1. What is the main purpose of a text cleaning pipeline in data analysis?

easy

A. To translate text into another language

B. To make text uniform and easier to analyze

C. To increase the length of the text

D. To add more punctuation to the text

Solution

1. Step 1: Understand text cleaning goal

   Text cleaning aims to prepare text data by making it consistent and easier to work with.

2. Step 2: Identify correct purpose

   Adding punctuation or increasing length does not help analysis; uniformity does.

3. Final Answer:

   To make text uniform and easier to analyze -> Option B

4. Quick Check:

   Text cleaning = uniform text [OK]

Hint: Text cleaning makes text consistent for analysis [OK]

Common Mistakes:

• Thinking cleaning adds punctuation
• Confusing cleaning with translation
• Believing cleaning increases text length

2. Which of the following Python code snippets correctly converts a string text to lowercase?

easy

A. text.lower()

B. lower(text)

C. text.toLowerCase()

D. text.lowercase()

Solution

1. Step 1: Recall Python string method for lowercase

   Python strings have a method called lower() to convert text to lowercase.

2. Step 2: Check each option

   text.lower() uses correct syntax. Options B, C, D are invalid in Python.

3. Final Answer:

   text.lower() -> Option A

4. Quick Check:

   Python lowercase method = lower() [OK]

Hint: Use text.lower() to lowercase strings in Python [OK]

Common Mistakes:

• Using JavaScript method toLowerCase() in Python
• Trying a non-existent function lower() outside string
• Using incorrect method name like lowercase()

3. Given the following Python code, what is the output of cleaned_text?

import string
text = " Hello, World!  "
cleaned_text = text.lower().strip().translate(str.maketrans('', '', string.punctuation))
print(cleaned_text)

medium

A. " hello world "

B. "hello, world!"

C. "hello world"

D. "HELLO WORLD"

Solution

1. Step 1: Apply lower() and strip()

   The text becomes "hello, world!" after lowering and removing spaces at ends.

2. Step 2: Remove punctuation with translate()

   All punctuation marks like comma and exclamation are removed, resulting in "hello world".

3. Final Answer:

   "hello world" -> Option C

4. Quick Check:

   Lowercase + strip + remove punctuation = "hello world" [OK]

Hint: Chain methods: lower, strip, then remove punctuation [OK]

Common Mistakes:

• Forgetting to strip spaces
• Not removing punctuation correctly
• Confusing case conversion order

4. Identify the error in this text cleaning function and choose the fix:

def clean_text(text):
    text = text.lower()
    text = text.strip
    return text

print(clean_text("  DATA Science  "))

medium

A. Change text.strip to text.strip()

B. Change text.lower() to lower(text)

C. Add text = text.replace() to remove spaces

D. Remove the return statement

Solution

1. Step 1: Check method usage

   strip is a method and must be called with parentheses ().

2. Step 2: Identify fix

   Changing text.strip to text.strip() correctly removes spaces.

3. Final Answer:

   Change text.strip to text.strip() -> Option A

4. Quick Check:

   Method calls need parentheses [OK]

Hint: Always call string methods with parentheses [OK]

Common Mistakes:

• Forgetting parentheses on methods
• Using wrong syntax for lower()
• Removing return statement by mistake

5. You have a list of sentences with inconsistent spacing and punctuation:

texts = ["Data Science!  ", "  Machine Learning.", "AI, is fun!"]

Which code snippet correctly cleans all texts by making them lowercase, removing punctuation, and trimming spaces?

hard

A. cleaned = [t.strip().lower() for t in texts]

B. cleaned = [t.lower().replace(' ', '') for t in texts]

C. cleaned = [t.translate(str.maketrans('', '', string.punctuation)) for t in texts]

D. import string
cleaned = [t.lower().strip().translate(str.maketrans('', '', string.punctuation)) for t in texts]

Solution

1. Step 1: Understand cleaning steps needed

   We need lowercase, remove punctuation, and trim spaces for each text.

2. Step 2: Check each option

   import string cleaned = [t.lower().strip().translate(str.maketrans('', '', string.punctuation)) for t in texts] applies all three steps correctly using list comprehension. Others miss one or more steps.

3. Final Answer:

   import string cleaned = [t.lower().strip().translate(str.maketrans('', '', string.punctuation)) for t in texts] -> Option D

4. Quick Check:

   All steps in list comprehension = import string cleaned = [t.lower().strip().translate(str.maketrans('', '', string.punctuation)) for t in texts] [OK]

Hint: Use list comprehension chaining all cleaning steps [OK]

Common Mistakes:

• Missing punctuation removal
• Not trimming spaces
• Removing spaces incorrectly with replace