Sentiment analysis with scikit-learn in ML Python

Introduction

Sentiment analysis helps us understand if text shows positive or negative feelings. Using scikit-learn makes it easy to build a simple model for this.

To find out if customer reviews are happy or unhappy.

To check if social media posts are positive or negative about a topic.

To analyze feedback from surveys quickly.

To sort emails or messages by mood.

To monitor brand reputation by scanning online comments.

Syntax

ML Python

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import accuracy_score

# Prepare text data and labels
texts = ["I love this!", "This is bad.", "Amazing product", "Not good", "I am happy"]
labels = [1, 0, 1, 0, 1]  # 1=positive, 0=negative

# Convert text to numbers
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(texts)

# Split data
X_train, X_test, y_train, y_test = train_test_split(X, labels, test_size=0.2, random_state=42)

# Train model
model = MultinomialNB()
model.fit(X_train, y_train)

# Predict
predictions = model.predict(X_test)

# Check accuracy
accuracy = accuracy_score(y_test, predictions)

CountVectorizer changes words into numbers the model can understand.

MultinomialNB is a simple and effective model for text classification.

Examples

This removes common English words like 'the' or 'and' to focus on important words.

ML Python

vectorizer = CountVectorizer(stop_words='english')
X = vectorizer.fit_transform(texts)

Alpha controls smoothing to handle words not seen in training.

ML Python

model = MultinomialNB(alpha=0.5)
model.fit(X_train, y_train)

Predict sentiment for new text by transforming it first.

ML Python

predictions = model.predict(vectorizer.transform(["I hate this"]))

Sample Model

This program trains a simple model to tell if text is positive or negative. It shows accuracy and compares predictions to actual labels.

ML Python

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import accuracy_score

# Sample text data and labels
texts = [
    "I love this product",
    "This is the worst thing ever",
    "Absolutely fantastic experience",
    "I hate it",
    "Not good at all",
    "I am very happy",
    "Terrible service",
    "Best purchase I've made",
    "Awful, will not buy again",
    "Really pleased with this"
]
labels = [1, 0, 1, 0, 0, 1, 0, 1, 0, 1]  # 1=positive, 0=negative

# Convert text to numeric features
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(texts)

# Split into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, labels, test_size=0.3, random_state=1)

# Train the Naive Bayes model
model = MultinomialNB()
model.fit(X_train, y_train)

# Predict on test data
predictions = model.predict(X_test)

# Calculate accuracy
accuracy = accuracy_score(y_test, predictions)

print(f"Accuracy: {accuracy:.2f}")
print("Test predictions:", predictions)
print("Actual labels:  ", y_test)

Important Notes

More data usually means better results.

Try different models or add more text cleaning for improvement.

Accuracy shows how often the model guesses right.

Summary

Sentiment analysis finds if text is positive or negative.

Scikit-learn helps turn words into numbers and trains a simple model.

Check model accuracy to see how well it works.

Practice

1. What is the main goal of sentiment analysis in machine learning?

easy

A. To generate new text based on input

B. To count the number of words in a sentence

C. To translate text from one language to another

D. To find out if text expresses positive or negative feelings

Solution

1. Step 1: Understand sentiment analysis purpose

   Sentiment analysis is about detecting emotions or opinions in text, usually positive or negative feelings.

2. Step 2: Compare options with definition

   Only To find out if text expresses positive or negative feelings matches this goal. Others describe different tasks like counting words or translation.

3. Final Answer:

   To find out if text expresses positive or negative feelings -> Option D

4. Quick Check:

   Sentiment analysis = Detect positive/negative feelings [OK]

Hint: Sentiment analysis = positive or negative text feelings [OK]

Common Mistakes:

• Confusing sentiment analysis with translation
• Thinking it counts words instead of feelings
• Mixing it up with text generation

2. Which scikit-learn class is commonly used to convert text into numbers for sentiment analysis?

easy

A. LinearRegression

B. CountVectorizer

C. KMeans

D. DecisionTreeClassifier

Solution

1. Step 1: Identify text to number conversion tool

   CountVectorizer turns text into a matrix of word counts, a common step before training models.

2. Step 2: Check other options

   LinearRegression predicts numbers, KMeans clusters data, DecisionTreeClassifier classifies but does not convert text directly.

3. Final Answer:

   CountVectorizer -> Option B

4. Quick Check:

   Text to numbers = CountVectorizer [OK]

Hint: CountVectorizer converts text to numbers [OK]

Common Mistakes:

• Choosing regression or clustering classes for text conversion
• Confusing model classes with vectorizers
• Using classifiers directly on raw text

3. What will be the output of this code snippet?

from sklearn.feature_extraction.text import CountVectorizer
texts = ['I love this movie', 'I hate this movie']
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(texts)
print(X.toarray())

medium

A. [[0 1 1 1 1]
[1 1 0 1 1]]

B. [[1 1 1 1 0]
[1 1 1 0 1]]

C. [[1 1 0 1 1]
[1 0 1 1 1]]

D. [[1 0 1 1 1]
[1 1 1 0 1]]

Solution

1. Step 1: Identify vocabulary order

   CountVectorizer sorts words alphabetically: ['hate', 'i', 'love', 'movie', 'this']

2. Step 2: Map texts to counts

   'I love this movie' counts: hate=0, i=1, love=1, movie=1, this=1; 'I hate this movie' counts: hate=1, i=1, love=0, movie=1, this=1

3. Final Answer:

   [[0 1 1 1 1] [1 1 0 1 1]] -> Option A

4. Quick Check:

   Alphabetical words count matrix matches [[0 1 1 1 1] [1 1 0 1 1]] [OK]

Hint: CountVectorizer sorts words alphabetically [OK]

Common Mistakes:

• Assuming original word order instead of alphabetical
• Mixing up counts for 'love' and 'hate'
• Forgetting to convert sparse matrix to array

4. Identify the error in this sentiment analysis code snippet:

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.linear_model import LogisticRegression
texts = ['good', 'bad']
labels = [1, 0]
vectorizer = CountVectorizer()
X = vectorizer.transform(texts)
model = LogisticRegression()
model.fit(X, labels)

medium

A. Labels should be strings, not integers

B. LogisticRegression cannot be used for sentiment analysis

C. Using transform before fit_transform on vectorizer

D. Texts list should be empty

Solution

1. Step 1: Check vectorizer usage

   CountVectorizer must be fit with fit_transform before transform to learn vocabulary.

2. Step 2: Identify error in code

   Code calls transform without fitting first, causing error because vocabulary is unknown.

3. Final Answer:

   Using transform before fit_transform on vectorizer -> Option C

4. Quick Check:

   Vectorizer must fit before transform [OK]

Hint: Always fit vectorizer before transform [OK]

Common Mistakes:

• Calling transform without fitting vectorizer
• Thinking labels must be strings
• Believing LogisticRegression is unsuitable here

5. You trained a sentiment analysis model with scikit-learn and got 90% accuracy on training data but only 60% on new reviews. What should you do to improve real-world performance?

hard

A. Use more diverse training data and tune model parameters

B. Reduce training data size to avoid overfitting

C. Remove stop words from test data only

D. Switch to a regression model instead of classification

Solution

1. Step 1: Understand overfitting signs

   High training accuracy but low test accuracy means model learned training data too well but fails on new data.

2. Step 2: Choose improvement strategy

   Using more diverse data and tuning parameters helps model generalize better. Reducing data or removing stop words only on test is wrong. Regression is not suitable for classification tasks.

3. Final Answer:

   Use more diverse training data and tune model parameters -> Option A

4. Quick Check:

   Overfitting fix = More data + tuning [OK]

Hint: Fix overfitting with more data and tuning [OK]

Common Mistakes:

• Reducing training data size worsens model
• Altering test data only is incorrect
• Confusing classification with regression