Document processing pipeline in NLP - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Key step order in a document processing pipeline

Which of the following sequences correctly represents the typical order of steps in a document processing pipeline?

A Lemmatization → Tokenization → Feature Extraction → Stopword Removal

B Tokenization → Stopword Removal → Lemmatization → Feature Extraction

C Stopword Removal → Feature Extraction → Tokenization → Lemmatization

D Feature Extraction → Tokenization → Lemmatization → Stopword Removal

💡 Hint

Think about how raw text is first broken down before cleaning and then converted to numbers.

The typical pipeline starts by splitting text into tokens (words), then removing common words (stopwords), followed by reducing words to their base form (lemmatization), and finally converting text into numerical features.

❓ Predict Output

intermediate

Output of tokenizing and removing stopwords

What is the output of this Python code snippet?

NLP

import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize

text = "The quick brown fox jumps over the lazy dog"
stop_words = set(stopwords.words('english'))
tokens = word_tokenize(text)
filtered = [w for w in tokens if w.lower() not in stop_words]
print(filtered)

A ['The', 'quick', 'brown', 'fox', 'jumps', 'lazy', 'dog']

B ['The', 'quick', 'brown', 'fox', 'jumps', 'over', 'the', 'lazy', 'dog']

C ['quick', 'brown', 'fox', 'jumps', 'over', 'lazy', 'dog']

D ['quick', 'brown', 'fox', 'jumps', 'lazy', 'dog']

💡 Hint

Stopwords like 'the' and 'over' are removed.

The code tokenizes the sentence, then removes common English stopwords like 'the' and 'over'. The remaining words are returned in a list.

❓ Hyperparameter

advanced

Choosing n-gram range for feature extraction

In a document processing pipeline using TF-IDF vectorization, which n-gram range setting is best to capture both single words and pairs of words?

A ngram_range=(1,1)

B ngram_range=(2,2)

C ngram_range=(1,2)

D ngram_range=(2,3)

💡 Hint

You want to include both single words and two-word phrases.

Setting ngram_range=(1,2) includes unigrams (single words) and bigrams (pairs of words), which helps capture more context.

❓ Metrics

advanced

Evaluating document classification with imbalanced classes

Which metric is most appropriate to evaluate a document classification model when classes are imbalanced?

A F1 Score

B Precision

C Accuracy

D Mean Squared Error

💡 Hint

Consider a metric that balances precision and recall.

F1 Score balances precision and recall, making it suitable for imbalanced datasets where accuracy can be misleading.

🔧 Debug

expert

Identifying error in document vectorization code

What error does this code raise when run, and why?

NLP

from sklearn.feature_extraction.text import TfidfVectorizer

docs = ["Data science is fun", "Machine learning is powerful"]
vectorizer = TfidfVectorizer(stop_words='english')
X = vectorizer.fit_transform(docs)
print(X.toarray())
print(vectorizer.get_feature_names_out())

A AttributeError: 'TfidfVectorizer' object has no attribute 'get_feature_names' because the method was renamed

B TypeError: stop_words parameter must be a list, not a string

C ValueError: Input documents must be non-empty strings

D No error, prints the TF-IDF matrix and feature names

💡 Hint

Check the latest method name for getting feature names in sklearn.

In recent sklearn versions, get_feature_names() was replaced by get_feature_names_out(). Using the old method causes AttributeError.

Practice

1. What is the main purpose of a document processing pipeline in NLP?

easy

A. To break down text tasks into smaller, manageable steps

B. To store documents in a database

C. To translate documents into multiple languages

D. To generate random text from documents

Solution

1. Step 1: Understand the pipeline concept

   A document processing pipeline divides a big task into smaller steps to handle text better.

2. Step 2: Identify the main goal

   The goal is to make complex text easier to process by breaking it down.

3. Final Answer:

   To break down text tasks into smaller, manageable steps -> Option A

4. Quick Check:

   Pipeline purpose = break down tasks [OK]

Hint: Think of a pipeline as a step-by-step recipe for text [OK]

Common Mistakes:

• Confusing pipeline with storage or translation
• Thinking pipeline generates text
• Ignoring the step-by-step nature

2. Which of the following is the correct order of steps in a simple document processing pipeline?

easy

A. Stopword Removal -> Lemmatization -> Tokenization

B. Lemmatization -> Tokenization -> Stopword Removal

C. Tokenization -> Stopword Removal -> Lemmatization

D. Tokenization -> Lemmatization -> Stopword Removal

Solution

1. Step 1: Recall common pipeline steps

   Tokenization splits text into words, stopword removal deletes common words, lemmatization reduces words to base form.

2. Step 2: Determine logical order

   First split text (tokenize), then remove stopwords, then lemmatize remaining words.

3. Final Answer:

   Tokenization -> Stopword Removal -> Lemmatization -> Option C

4. Quick Check:

   Order = tokenize, remove stopwords, lemmatize [OK]

Hint: Split text first, then clean, then normalize words [OK]

Common Mistakes:

• Removing stopwords before tokenizing
• Lemmatizing before tokenizing
• Mixing step order randomly

3. Given this Python snippet in a document pipeline:

text = "Cats are running fast"
tokens = text.lower().split()
filtered = [w for w in tokens if w not in ['are', 'is', 'the']]
print(filtered)

What is the output?

medium

A. ['cats', 'running', 'fast']

B. ['Cats', 'are', 'running', 'fast']

C. ['cats', 'are', 'running', 'fast']

D. ['running', 'fast']

Solution

1. Step 1: Lowercase and split text

   "Cats are running fast" becomes ['cats', 'are', 'running', 'fast'] after lower() and split().

2. Step 2: Remove stopwords

   Words 'are', 'is', 'the' are removed, so 'are' is removed from the list.

3. Final Answer:

   ['cats', 'running', 'fast'] -> Option A

4. Quick Check:

   Stopwords removed = ['cats', 'running', 'fast'] [OK]

Hint: Lowercase then remove stopwords from tokens [OK]

Common Mistakes:

• Not lowercasing before filtering
• Including stopwords in output
• Confusing original and filtered lists

4. This code is part of a document pipeline:

def clean_text(text):
    tokens = text.split()
    tokens = [t.lower() for t in tokens]
    tokens = [t for t in tokens if t not in stopwords]
    tokens = lemmatize(tokens)
    return tokens

stopwords = ['and', 'the', 'is']

print(clean_text("The cats and dogs are playing"))

What is the error here?

medium

A. text.split() should be text.lower().split()

B. lemmatize function is not defined

C. stopwords list is empty

D. tokens list is not returned

Solution

1. Step 1: Check function definitions

   The code calls lemmatize(tokens) but no lemmatize function is defined or imported.

2. Step 2: Verify other parts

   stopwords list is defined, tokens are returned, and text is split correctly.

3. Final Answer:

   lemmatize function is not defined -> Option B

4. Quick Check:

   Missing lemmatize function causes error [OK]

Hint: Check if all functions used are defined or imported [OK]

Common Mistakes:

• Assuming lemmatize is built-in
• Ignoring missing function errors
• Thinking stopwords list is empty

5. You want to build a document processing pipeline that extracts keywords from large documents. Which sequence of steps is best?

hard

A. POS Tagging -> Keyword Extraction -> Tokenization -> Stopword Removal

B. Keyword Extraction -> Tokenization -> Stopword Removal -> POS Tagging

C. Stopword Removal -> Tokenization -> Keyword Extraction -> POS Tagging

D. Tokenization -> Stopword Removal -> POS Tagging -> Keyword Extraction

Solution

1. Step 1: Understand keyword extraction needs

   Extracting keywords requires clean tokens and knowing word types (POS tags) to pick important words.

2. Step 2: Arrange logical steps

   First tokenize text, remove stopwords to clean, then tag parts of speech, finally extract keywords based on tags.

3. Final Answer:

   Tokenization -> Stopword Removal -> POS Tagging -> Keyword Extraction -> Option D

4. Quick Check:

   Pipeline order = tokenize, clean, tag, extract [OK]

Hint: Clean tokens before tagging and extracting keywords [OK]

Common Mistakes:

• Extracting keywords before tokenizing
• Tagging before cleaning tokens
• Wrong step order breaks pipeline