First NLP pipeline - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of this tokenization code?

Given the following Python code using NLTK, what is the output of the tokens variable?

NLP

import nltk
nltk.download('punkt')
from nltk.tokenize import word_tokenize
text = "Hello world! This is a test."
tokens = word_tokenize(text)
print(tokens)

A ['Hello world!', 'This is a test.']

B ['Hello', 'world', '!', 'This', 'is', 'a', 'test', '.']

C ['Hello', 'world!', 'This', 'is', 'a', 'test.']

D ['Hello', 'world', 'This', 'is', 'a', 'test']

💡 Hint

Think about how word_tokenize splits punctuation as separate tokens.

The word_tokenize function splits the text into words and punctuation marks separately, so punctuation like '!' and '.' become their own tokens.

❓ Model Choice

intermediate

Which model is best for sentiment analysis in an NLP pipeline?

You want to build a simple NLP pipeline to classify movie reviews as positive or negative. Which model is most suitable?

A A convolutional neural network for image classification

B A K-Means clustering model

C A linear regression model

D A pretrained BERT model fine-tuned on sentiment data

💡 Hint

Sentiment analysis is a text classification task; choose a model designed for text understanding.

BERT is a powerful language model that can be fine-tuned for sentiment classification tasks, unlike clustering or regression models or image CNNs.

❓ Hyperparameter

advanced

Which hyperparameter affects the number of words considered in a Bag-of-Words model?

In a Bag-of-Words NLP pipeline using CountVectorizer, which hyperparameter controls the maximum number of words (features) to keep?

A stop_words

B min_df

C max_features

D ngram_range

💡 Hint

This parameter limits the vocabulary size by frequency.

max_features sets the maximum number of features (words) to keep based on frequency, controlling vocabulary size.

❓ Metrics

advanced

Which metric is best to evaluate an imbalanced text classification model?

You trained an NLP model to detect spam emails, but spam emails are only 5% of your data. Which metric is best to evaluate your model?

A Precision and Recall

B Accuracy

C Mean Squared Error

D R-squared

💡 Hint

Accuracy can be misleading when classes are imbalanced.

Precision and recall give better insight on performance for imbalanced classes by measuring false positives and false negatives.

🔧 Debug

expert

Why does this NLP pipeline code raise a KeyError?

Consider this code snippet for text preprocessing:

from sklearn.feature_extraction.text import CountVectorizer
texts = ['I love AI', 'AI is fun']
vectorizer = CountVectorizer(stop_words='english')
X = vectorizer.fit_transform(texts)
print(vectorizer.vocabulary_['AI'])

Why does it raise a KeyError for 'AI'?

NLP

from sklearn.feature_extraction.text import CountVectorizer
texts = ['I love AI', 'AI is fun']
vectorizer = CountVectorizer(stop_words='english')
X = vectorizer.fit_transform(texts)
print(vectorizer.vocabulary_['AI'])

A 'AI' is not in the vocabulary because CountVectorizer lowercases all tokens by default

B 'AI' is removed because it is considered a stop word in English

C The vocabulary_ attribute is not a dictionary, causing the error

D The fit_transform method was not called before accessing vocabulary_

💡 Hint

Check how CountVectorizer processes text tokens before building vocabulary.

CountVectorizer lowercases all tokens by default, so 'AI' becomes 'ai'. Accessing 'AI' (uppercase) causes KeyError.