Document processing pipeline in NLP - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to load a document as a string.

NLP

with open('document.txt', 'r') as file:
    text = file.[1]()

Drag options to blanks, or click blank then click option'

Areadlines

Breadline

Cread

Dopen

Attempts:

3 left

💡 Hint

Common Mistakes

Using readline() reads only one line, not the full document.

Using readlines() returns a list of lines, not a single string.

The read() method reads the entire file content as a single string, which is needed to process the whole document.

2fill in blank

medium

Complete the code to split the document text into sentences.

NLP

import nltk
nltk.download('punkt')
sentences = nltk.tokenize.[1](text)

Drag options to blanks, or click blank then click option'

Aword_tokenize

Bsplit

Ctokenize

Dsent_tokenize

Attempts:

3 left

💡 Hint

Common Mistakes

Using word_tokenize splits text into words, not sentences.

Using split is a basic string method and won't handle sentence boundaries properly.

The sent_tokenize function splits text into sentences, which is essential for sentence-level processing.

3fill in blank

hard

Fix the error in the code to remove stopwords from the token list.

NLP

from nltk.corpus import stopwords
stop_words = set(stopwords.words('english'))
tokens = ['this', 'is', 'a', 'test']
filtered = [word for word in tokens if word [1] stop_words]

Drag options to blanks, or click blank then click option'

Ain

Bnot in

C==

D!=

Attempts:

3 left

💡 Hint

Common Mistakes

Using in keeps only stopwords, which is the opposite of what we want.

We want to keep words that are not in the stopwords list, so we use not in.

4fill in blank

hard

Fill both blanks to create a dictionary of word counts from tokens.

NLP

word_counts = [1]()
for word in tokens:
    word_counts[word] = word_counts.get(word, [2]) + 1

Drag options to blanks, or click blank then click option'

Adict

B0

C1

Dlist

Attempts:

3 left

💡 Hint

Common Mistakes

Using list() instead of dict() causes errors.

Using 1 as default count causes counts to start at 2.

We start with an empty dictionary dict(). The get method returns 0 if the word is not yet counted, then we add 1.

5fill in blank

hard

Fill all three blanks to create a TF-IDF vectorizer and transform documents.

NLP

from sklearn.feature_extraction.text import [1]
vectorizer = [2](stop_words='english')
X = vectorizer.[3](documents)

Drag options to blanks, or click blank then click option'

ATfidfVectorizer

Cfit_transform

Dfit

Attempts:

3 left

💡 Hint

Common Mistakes

Using fit alone returns the model, not the transformed data.

Using wrong class names causes import errors.

TfidfVectorizer converts text to TF-IDF features. We create an instance and then call fit_transform to learn and transform the documents.

Practice

1. What is the main purpose of a document processing pipeline in NLP?

easy

A. To break down text tasks into smaller, manageable steps

B. To store documents in a database

C. To translate documents into multiple languages

D. To generate random text from documents

Solution

1. Step 1: Understand the pipeline concept

   A document processing pipeline divides a big task into smaller steps to handle text better.

2. Step 2: Identify the main goal

   The goal is to make complex text easier to process by breaking it down.

3. Final Answer:

   To break down text tasks into smaller, manageable steps -> Option A

4. Quick Check:

   Pipeline purpose = break down tasks [OK]

Hint: Think of a pipeline as a step-by-step recipe for text [OK]

Common Mistakes:

• Confusing pipeline with storage or translation
• Thinking pipeline generates text
• Ignoring the step-by-step nature

2. Which of the following is the correct order of steps in a simple document processing pipeline?

easy

A. Stopword Removal -> Lemmatization -> Tokenization

B. Lemmatization -> Tokenization -> Stopword Removal

C. Tokenization -> Stopword Removal -> Lemmatization

D. Tokenization -> Lemmatization -> Stopword Removal

Solution

1. Step 1: Recall common pipeline steps

   Tokenization splits text into words, stopword removal deletes common words, lemmatization reduces words to base form.

2. Step 2: Determine logical order

   First split text (tokenize), then remove stopwords, then lemmatize remaining words.

3. Final Answer:

   Tokenization -> Stopword Removal -> Lemmatization -> Option C

4. Quick Check:

   Order = tokenize, remove stopwords, lemmatize [OK]

Hint: Split text first, then clean, then normalize words [OK]

Common Mistakes:

• Removing stopwords before tokenizing
• Lemmatizing before tokenizing
• Mixing step order randomly

3. Given this Python snippet in a document pipeline:

text = "Cats are running fast"
tokens = text.lower().split()
filtered = [w for w in tokens if w not in ['are', 'is', 'the']]
print(filtered)

What is the output?

medium

A. ['cats', 'running', 'fast']

B. ['Cats', 'are', 'running', 'fast']

C. ['cats', 'are', 'running', 'fast']

D. ['running', 'fast']

Solution

1. Step 1: Lowercase and split text

   "Cats are running fast" becomes ['cats', 'are', 'running', 'fast'] after lower() and split().

2. Step 2: Remove stopwords

   Words 'are', 'is', 'the' are removed, so 'are' is removed from the list.

3. Final Answer:

   ['cats', 'running', 'fast'] -> Option A

4. Quick Check:

   Stopwords removed = ['cats', 'running', 'fast'] [OK]

Hint: Lowercase then remove stopwords from tokens [OK]

Common Mistakes:

• Not lowercasing before filtering
• Including stopwords in output
• Confusing original and filtered lists

4. This code is part of a document pipeline:

def clean_text(text):
    tokens = text.split()
    tokens = [t.lower() for t in tokens]
    tokens = [t for t in tokens if t not in stopwords]
    tokens = lemmatize(tokens)
    return tokens

stopwords = ['and', 'the', 'is']

print(clean_text("The cats and dogs are playing"))

What is the error here?

medium

A. text.split() should be text.lower().split()

B. lemmatize function is not defined

C. stopwords list is empty

D. tokens list is not returned

Solution

1. Step 1: Check function definitions

   The code calls lemmatize(tokens) but no lemmatize function is defined or imported.

2. Step 2: Verify other parts

   stopwords list is defined, tokens are returned, and text is split correctly.

3. Final Answer:

   lemmatize function is not defined -> Option B

4. Quick Check:

   Missing lemmatize function causes error [OK]

Hint: Check if all functions used are defined or imported [OK]

Common Mistakes:

• Assuming lemmatize is built-in
• Ignoring missing function errors
• Thinking stopwords list is empty

5. You want to build a document processing pipeline that extracts keywords from large documents. Which sequence of steps is best?

hard

A. POS Tagging -> Keyword Extraction -> Tokenization -> Stopword Removal

B. Keyword Extraction -> Tokenization -> Stopword Removal -> POS Tagging

C. Stopword Removal -> Tokenization -> Keyword Extraction -> POS Tagging

D. Tokenization -> Stopword Removal -> POS Tagging -> Keyword Extraction

Solution

1. Step 1: Understand keyword extraction needs

   Extracting keywords requires clean tokens and knowing word types (POS tags) to pick important words.

2. Step 2: Arrange logical steps

   First tokenize text, remove stopwords to clean, then tag parts of speech, finally extract keywords based on tags.

3. Final Answer:

   Tokenization -> Stopword Removal -> POS Tagging -> Keyword Extraction -> Option D

4. Quick Check:

   Pipeline order = tokenize, clean, tag, extract [OK]

Hint: Clean tokens before tagging and extracting keywords [OK]

Common Mistakes:

• Extracting keywords before tokenizing
• Tagging before cleaning tokens
• Wrong step order breaks pipeline