Text preprocessing pipelines in NLP - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of this text tokenization code?

Consider the following Python code that tokenizes a sentence into words using a simple split method. What is the output list?

NLP

sentence = "Hello, world! Welcome to AI." 
tokens = sentence.split()
print(tokens)

A ['Hello,', 'world!', 'Welcome', 'to', 'AI.']

B ['Hello', 'world', 'Welcome', 'to', 'AI']

C ['Hello', ',', 'world', '!', 'Welcome', 'to', 'AI', '.']

D ['Hello world Welcome to AI']

💡 Hint

The split() method splits by spaces and does not remove punctuation.

The split() method splits the string by spaces, so punctuation remains attached to words.

🧠 Conceptual

intermediate

Which step is NOT typically part of a text preprocessing pipeline?

In a typical text preprocessing pipeline for NLP tasks, which of the following steps is usually NOT included?

A Applying stemming or lemmatization

B Removing stopwords

C Training a neural network model

D Lowercasing all text

💡 Hint

Preprocessing prepares data before modeling.

Training a model is a separate step after preprocessing; preprocessing focuses on cleaning and preparing text.

❓ Metrics

advanced

What is the vocabulary size after preprocessing?

Given the following list of sentences, after converting all text to lowercase, removing punctuation, and tokenizing, what is the size of the vocabulary (unique words)? Sentences: 1. "AI is fun!" 2. "Fun with AI and machine learning." 3. "Learning AI is exciting."

NLP

import string
sentences = ["AI is fun!", "Fun with AI and machine learning.", "Learning AI is exciting."]
vocab = set()
for sent in sentences:
    sent = sent.lower()
    sent = sent.translate(str.maketrans('', '', string.punctuation))
    tokens = sent.split()
    vocab.update(tokens)
print(len(vocab))

A 10

B 8

C 9

D 7

💡 Hint

Count unique words after cleaning and splitting.

After cleaning and tokenizing, the unique words are: 'ai', 'is', 'fun', 'with', 'and', 'machine', 'learning', 'exciting' totaling 8.

🔧 Debug

advanced

Why does this text normalization code raise an error?

Examine the code below that attempts to normalize text by removing digits and converting to lowercase. Why does it raise an error?

NLP

import re
text = "AI version 2.0 is here!"
normalized = re.sub('[0-9]+', '', text).lower()
print(normalized)

A Raises AttributeError because lower() is called on None

B Raises TypeError because re.sub returns bytes, not string

C Raises SyntaxError due to incorrect regex pattern

D No error; the code runs and outputs 'ai version . is here!'

💡 Hint

Check the return type of re.sub and method chaining.

re.sub returns a string, so calling lower() on it is valid. The code runs without error.

❓ Model Choice

expert

Which model is best suited for text classification after preprocessing?

After completing text preprocessing (tokenization, stopword removal, and vectorization), which model below is generally best for classifying short text documents?

A Recurrent Neural Network (RNN) or LSTM tailored for sequences

B Convolutional Neural Network (CNN) designed for images

C Linear Regression model

D K-Means clustering algorithm

💡 Hint

Consider models that handle sequences and context well.

RNNs and LSTMs are designed to handle sequential data like text, making them suitable for text classification.