Lowercasing and normalization in NLP - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of this normalization code?

Consider the following Python code that normalizes a text by lowercasing and removing punctuation. What is the printed output?

NLP

import string
text = "Hello, World! Let's normalize this text."
normalized = ''.join(ch for ch in text.lower() if ch not in string.punctuation)
print(normalized)

A Hello World Lets Normalize This Text

B hello, world! let's normalize this text.

C hello world lets normalize this text

D hello world let's normalize this text

💡 Hint

Think about what lowercasing and removing punctuation does to the original text.

The code converts all letters to lowercase and removes all punctuation characters, including commas, apostrophes, and periods.

🧠 Conceptual

intermediate

Why is lowercasing important in text normalization?

Which of the following best explains why lowercasing is a common step in text normalization for machine learning?

A It increases the length of the text to improve model accuracy.

B It removes all punctuation from the text.

C It translates text into a different language.

D It reduces the number of unique words by treating 'Apple' and 'apple' as the same word.

💡 Hint

Think about how case differences affect word counts.

Lowercasing helps reduce vocabulary size by treating words that differ only in case as the same, which simplifies the model's learning.

❓ Metrics

advanced

How does normalization affect model accuracy?

You train two text classification models: Model A uses raw text, Model B uses normalized text (lowercased, punctuation removed). Which outcome is most likely?

A Model A achieves higher accuracy because raw text has more information.

B Model B achieves higher accuracy because normalization reduces noise and vocabulary size.

C Both models have the same accuracy because normalization does not affect text data.

D Model B performs worse because removing punctuation removes important meaning.

💡 Hint

Consider how noise and vocabulary size affect learning.

Normalization reduces noise and vocabulary size, helping the model generalize better and often improving accuracy.

🔧 Debug

advanced

Identify the error in this normalization code

What error does this code raise when run? import string text = "Normalize THIS!" normalized = ''.join(ch for ch in text.lower() if ch != string.punctuation) print(normalized)

A No error, prints 'normalize this!'

B TypeError: 'in <string>' requires string as left operand, not 'str'

C TypeError: '!=' not supported between instances of 'str' and 'str'

D The code prints 'normalize this' without punctuation

💡 Hint

Look carefully at how punctuation is checked in the condition.

No runtime error is raised. The condition 'ch != string.punctuation' always evaluates to True because a single character never equals the entire 'string.punctuation' string, so no punctuation is removed. Use 'ch not in string.punctuation' instead.

❓ Model Choice

expert

Choosing the best normalization for noisy text data

You have a dataset of social media posts with many uppercase letters, emojis, and punctuation. Which normalization approach is best before training a sentiment analysis model?

A Lowercase all text, remove punctuation, and remove emojis

B Keep original casing, keep punctuation, remove emojis

C Lowercase all text, keep punctuation and emojis

D Remove punctuation and emojis, keep original casing

💡 Hint

Consider what noise elements can confuse the model and what information is useful.

Lowercasing reduces vocabulary size, removing punctuation and emojis reduces noise that may not help sentiment detection, improving model focus.

Practice

1. What is the main purpose of lowercasing text in Natural Language Processing?

easy

A. To translate text into another language

B. To make all letters small so words like 'Apple' and 'apple' are treated the same

C. To remove all punctuation marks from the text

D. To split sentences into words

Solution

1. Step 1: Understand what lowercasing does

   Lowercasing changes all letters in text to small letters.

2. Step 2: Understand why lowercasing is used

   This helps treat words like 'Apple' and 'apple' as the same word, improving consistency.

3. Final Answer:

   To make all letters small so words like 'Apple' and 'apple' are treated the same -> Option B

4. Quick Check:

   Lowercasing = uniform word form [OK]

Hint: Lowercase to treat same words equally [OK]

Common Mistakes:

• Confusing lowercasing with removing punctuation
• Thinking lowercasing translates text
• Believing lowercasing splits sentences

2. Which of the following Python code snippets correctly converts a string text to lowercase?

easy

A. text.lowercase()

B. lower(text)

C. text.toLowerCase()

D. text.lower()

Solution

1. Step 1: Recall Python string method for lowercasing

   Python strings have a method called lower() to convert text to lowercase.

2. Step 2: Check each option

   text.lower() uses text.lower(), which is correct. lower(text) is not a Python function. text.toLowerCase() is JavaScript style. text.lowercase() is not a valid method.

3. Final Answer:

   text.lower() -> Option D

4. Quick Check:

   Python lowercase method = lower() [OK]

Hint: Python lowercase method is .lower() [OK]

Common Mistakes:

• Using JavaScript syntax in Python
• Calling non-existent methods like lowercase()
• Trying to use a function named lower() instead of method

3. What will be the output of this Python code?

text = 'Café'
normalized = text.lower()
print(normalized)

medium

A. 'café'

B. 'cafe'

C. 'CAFÉ'

D. 'Cafe'

Solution

1. Step 1: Apply lower() method on the string 'Café'

   The lower() method converts all uppercase letters to lowercase but does not remove accents.

2. Step 2: Understand effect on accented characters

   The accented 'é' remains unchanged because lower() does not normalize accents.

3. Final Answer:

   'café' -> Option A

4. Quick Check:

   lower() keeps accents, just lowers letters [OK]

Hint: lower() changes case but keeps accents [OK]

Common Mistakes:

• Assuming accents are removed by lower()
• Expecting uppercase output
• Confusing normalization with lowercasing

4. The following code aims to lowercase and normalize text but has an error:

import unicodedata
text = 'Café'
normalized = unicodedata.normalize('NFKD', text).lower()
print(normalized)

What is the error and how to fix it?

medium

A. normalize returns a string with accents separated; fix by removing accents after normalization

B. Calling lower() before normalize; fix by swapping the calls

C. lower() returns a string; normalize expects bytes, fix by encoding first

D. No error; code works correctly

Solution

1. Step 1: Understand what normalize('NFKD') does

   It decomposes accented characters into base character plus accent marks.

2. Step 2: Check the code behavior

   After normalization, accents are separate characters, so lower() works but accents remain. To remove accents, you must filter out combining marks after normalization.

3. Final Answer:

   normalize returns a string with accents separated; fix by removing accents after normalization -> Option A

4. Quick Check:

   Normalization decomposes accents; remove them explicitly [OK]

Hint: Normalize then remove accents explicitly [OK]

Common Mistakes:

• Thinking lower() removes accents
• Swapping normalize and lower() calls incorrectly
• Assuming no extra step needed to remove accents

5. You want to preprocess text data by lowercasing and removing accents for a machine learning model. Which Python code snippet correctly does this?

hard

A. import unicodedata
text = 'Café'
text = unicodedata.normalize('NFKD', text)
print(text)

B. text = 'Café'
text = text.lower()
print(text)

C. import unicodedata
text = 'Café'
text = text.lower()
text = ''.join(c for c in unicodedata.normalize('NFKD', text) if not unicodedata.combining(c))
print(text)

D. text = 'Café'
text = text.upper()
print(text)

Solution

1. Step 1: Lowercase the text

   Use text.lower() to convert all letters to lowercase.

2. Step 2: Normalize and remove accents

   Use unicodedata.normalize('NFKD', text) to decompose accents, then remove combining characters to strip accents.

3. Step 3: Combine steps correctly

   import unicodedata text = 'Café' text = text.lower() text = ''.join(c for c in unicodedata.normalize('NFKD', text) if not unicodedata.combining(c)) print(text) does both steps properly: lowercasing first, then normalization and accent removal.

4. Final Answer:

   import unicodedata text = 'Café' text = text.lower() text = ''.join(c for c in unicodedata.normalize('NFKD', text) if not unicodedata.combining(c)) print(text) -> Option C

5. Quick Check:

   Lowercase + normalize + remove accents = clean text [OK]

Hint: Lowercase first, then normalize and remove accents [OK]

Common Mistakes:

• Skipping accent removal after normalization
• Using upper() instead of lower()
• Normalizing without removing combining characters