Document processing pipeline in NLP - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Key step order in a document processing pipeline

Which of the following sequences correctly represents the typical order of steps in a document processing pipeline?

A Lemmatization → Tokenization → Feature Extraction → Stopword Removal

B Tokenization → Stopword Removal → Lemmatization → Feature Extraction

C Stopword Removal → Feature Extraction → Tokenization → Lemmatization

D Feature Extraction → Tokenization → Lemmatization → Stopword Removal

💡 Hint

Think about how raw text is first broken down before cleaning and then converted to numbers.

The typical pipeline starts by splitting text into tokens (words), then removing common words (stopwords), followed by reducing words to their base form (lemmatization), and finally converting text into numerical features.

❓ Predict Output

intermediate

Output of tokenizing and removing stopwords

What is the output of this Python code snippet?

NLP

import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize

text = "The quick brown fox jumps over the lazy dog"
stop_words = set(stopwords.words('english'))
tokens = word_tokenize(text)
filtered = [w for w in tokens if w.lower() not in stop_words]
print(filtered)

A ['The', 'quick', 'brown', 'fox', 'jumps', 'lazy', 'dog']

B ['The', 'quick', 'brown', 'fox', 'jumps', 'over', 'the', 'lazy', 'dog']

C ['quick', 'brown', 'fox', 'jumps', 'over', 'lazy', 'dog']

D ['quick', 'brown', 'fox', 'jumps', 'lazy', 'dog']

💡 Hint

Stopwords like 'the' and 'over' are removed.

The code tokenizes the sentence, then removes common English stopwords like 'the' and 'over'. The remaining words are returned in a list.

❓ Hyperparameter

advanced

Choosing n-gram range for feature extraction

In a document processing pipeline using TF-IDF vectorization, which n-gram range setting is best to capture both single words and pairs of words?

A ngram_range=(1,1)

B ngram_range=(2,2)

C ngram_range=(1,2)

D ngram_range=(2,3)

💡 Hint

You want to include both single words and two-word phrases.

Setting ngram_range=(1,2) includes unigrams (single words) and bigrams (pairs of words), which helps capture more context.

❓ Metrics

advanced

Evaluating document classification with imbalanced classes

Which metric is most appropriate to evaluate a document classification model when classes are imbalanced?

A F1 Score

B Precision

C Accuracy

D Mean Squared Error

💡 Hint

Consider a metric that balances precision and recall.

F1 Score balances precision and recall, making it suitable for imbalanced datasets where accuracy can be misleading.

🔧 Debug

expert

Identifying error in document vectorization code

What error does this code raise when run, and why?

NLP

from sklearn.feature_extraction.text import TfidfVectorizer

docs = ["Data science is fun", "Machine learning is powerful"]
vectorizer = TfidfVectorizer(stop_words='english')
X = vectorizer.fit_transform(docs)
print(X.toarray())
print(vectorizer.get_feature_names_out())

A AttributeError: 'TfidfVectorizer' object has no attribute 'get_feature_names' because the method was renamed

B TypeError: stop_words parameter must be a list, not a string

C ValueError: Input documents must be non-empty strings

D No error, prints the TF-IDF matrix and feature names

💡 Hint

Check the latest method name for getting feature names in sklearn.

In recent sklearn versions, get_feature_names() was replaced by get_feature_names_out(). Using the old method causes AttributeError.