First NLP pipeline

Introduction

We use an NLP pipeline to turn text into useful information step-by-step. It helps computers understand human language.

You want to find the main topics in customer reviews.

You need to check if emails are spam or not.

You want to translate sentences from one language to another.

You want to find names of people or places in news articles.

You want to summarize long documents into short points.

Syntax

NLP

from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB

pipeline = Pipeline([
    ('vectorizer', CountVectorizer()),
    ('classifier', MultinomialNB())
])

The pipeline is a list of steps, each with a name and a tool.

Text data flows through each step in order.

Examples

This pipeline removes common English words before classifying.

NLP

pipeline = Pipeline([
    ('vectorizer', CountVectorizer(stop_words='english')),
    ('classifier', MultinomialNB())
])

This pipeline uses single words and pairs of words to understand text better.

NLP

pipeline = Pipeline([
    ('vectorizer', CountVectorizer(ngram_range=(1,2))),
    ('classifier', MultinomialNB())
])

Sample Model

This program creates a simple NLP pipeline that turns text into numbers and then classifies if the text is positive or negative. It trains on some examples and tests on others, then shows predictions and accuracy.

NLP

from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Sample text data and labels
texts = [
    'I love this movie',
    'This film was terrible',
    'Amazing acting and story',
    'I did not like the film',
    'Best movie ever',
    'Worst movie I have seen'
]
labels = [1, 0, 1, 0, 1, 0]  # 1=positive, 0=negative

# Split data into train and test sets
X_train, X_test, y_train, y_test = train_test_split(texts, labels, test_size=0.33, random_state=42)

# Create the NLP pipeline
pipeline = Pipeline([
    ('vectorizer', CountVectorizer()),
    ('classifier', MultinomialNB())
])

# Train the model
pipeline.fit(X_train, y_train)

# Predict on test data
predictions = pipeline.predict(X_test)

# Calculate accuracy
accuracy = accuracy_score(y_test, predictions)

print(f'Predictions: {predictions}')
print(f'Accuracy: {accuracy:.2f}')

Important Notes

Always split your data into training and testing to check if your model works well.

CountVectorizer turns words into numbers that the model can understand.

MultinomialNB is a simple and fast classifier good for text data.

Summary

An NLP pipeline processes text step-by-step to make predictions.

Use vectorizers to convert text into numbers.

Train and test your pipeline to see how well it works.

Practice

1. What is the main purpose of an NLP pipeline in machine learning?

easy

A. To translate text into different languages automatically

B. To store large amounts of text data

C. To process text step-by-step for making predictions

D. To create images from text

Solution

1. Step 1: Understand the role of an NLP pipeline

   An NLP pipeline breaks down text processing into steps like cleaning, vectorizing, and modeling.

2. Step 2: Identify the goal of these steps

   The goal is to prepare text data so a model can make predictions, such as classifying or understanding text.

3. Final Answer:

   To process text step-by-step for making predictions -> Option C

4. Quick Check:

   NLP pipeline = step-by-step text processing for predictions [OK]

Hint: Remember: pipeline means step-by-step processing [OK]

Common Mistakes:

• Thinking pipeline stores data only
• Confusing pipeline with translation tools
• Assuming pipeline creates images

2. Which of the following is the correct way to import a text vectorizer from scikit-learn for an NLP pipeline?

easy

A. import CountVectorizer from sklearn.text

B. from sklearn.feature_extraction.text import CountVectorizer

C. from sklearn.vectorizer import TextCount

D. import text_vectorizer from sklearn.feature

Solution

1. Step 1: Recall the correct module for text vectorizers

   Scikit-learn provides CountVectorizer in the feature_extraction.text module.

2. Step 2: Check the import syntax

   The correct syntax is: from sklearn.feature_extraction.text import CountVectorizer.

3. Final Answer:

   from sklearn.feature_extraction.text import CountVectorizer -> Option B

4. Quick Check:

   Correct import = from sklearn.feature_extraction.text import CountVectorizer [OK]

Hint: Remember: CountVectorizer is in feature_extraction.text [OK]

Common Mistakes:

• Using wrong module names
• Incorrect import syntax
• Confusing class names

3. Given the following code snippet, what will be the output of print(X.toarray())?

from sklearn.feature_extraction.text import CountVectorizer
texts = ['cat and dog', 'dog and mouse']
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(texts)
print(X.toarray())

medium

A. [[1 1 1 0]
[1 0 1 1]]

B. [[1 0 1 1]
[1 1 0 1]]

C. [[1 1 0 1]
[1 0 1 1]]

D. [[0 1 1 1]
[1 1 1 0]]

Solution

1. Step 1: Identify the vocabulary from the texts

   The texts are 'cat and dog' and 'dog and mouse'. The unique words are: 'and', 'cat', 'dog', 'mouse'. CountVectorizer sorts them alphabetically: ['and', 'cat', 'dog', 'mouse'].

2. Step 2: Map each text to counts of these words

   First text: 'cat and dog' -> counts: and=1, cat=1, dog=1, mouse=0 -> [1 1 1 0]. Second text: 'dog and mouse' -> counts: and=1, cat=0, dog=1, mouse=1 -> [1 0 1 1].

3. Final Answer:

   [[1 1 1 0] [1 0 1 1]] -> Option A

4. Quick Check:

   Vocabulary order and counts match [[1 1 1 0] [1 0 1 1]] [OK]

Hint: Remember: CountVectorizer sorts words alphabetically [OK]

Common Mistakes:

• Mixing word order in output
• Confusing counts of words
• Assuming different vocabulary order

4. You wrote this code but get an error: AttributeError: 'CountVectorizer' object has no attribute 'transform_text'. What is the likely fix?

from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer()
vectorizer.transform_text(['hello world'])

medium

A. Replace transform_text with transform

B. Import CountVectorizer from a different module

C. Call fit before transform_text

D. Use fit_transform_text instead

Solution

1. Step 1: Identify the incorrect method name

   The error says 'CountVectorizer' has no method 'transform_text'. The correct method is 'transform'.

2. Step 2: Correct the method call

   Replace transform_text with transform to fix the error.

3. Final Answer:

   Replace transform_text with transform -> Option A

4. Quick Check:

   Correct method name is transform [OK]

Hint: Check method names carefully in docs [OK]

Common Mistakes:

• Using non-existent method names
• Not reading error messages
• Trying to call fit_transform_text which doesn't exist

5. You want to build a simple NLP pipeline that converts text to numbers and then trains a logistic regression model to classify text. Which sequence of steps is correct?

hard

A. Predict on new text -> Vectorize text -> Train logistic regression

B. Train logistic regression -> Vectorize text -> Predict on new text

C. Vectorize text -> Predict on new text -> Train logistic regression

D. Vectorize text -> Train logistic regression -> Predict on new text

Solution

1. Step 1: Understand the pipeline order

   First, text must be converted to numbers using vectorization before training a model.

2. Step 2: Follow logical flow

   After vectorizing, train the logistic regression model, then use it to predict on new vectorized text.

3. Final Answer:

   Vectorize text -> Train logistic regression -> Predict on new text -> Option D

4. Quick Check:

   Correct pipeline order = Vectorize text -> Train logistic regression -> Predict on new text [OK]

Hint: Always vectorize before training or predicting [OK]

Common Mistakes:

• Trying to train before vectorizing
• Predicting before training
• Skipping vectorization step