Sentiment analysis with scikit-learn in ML Python - ML Experiment: Train & Evaluate

Experiment - Sentiment analysis with scikit-learn

Problem:Classify movie reviews as positive or negative using a simple text model.

Current Metrics:Training accuracy: 98%, Validation accuracy: 75%

Issue:The model is overfitting: training accuracy is very high but validation accuracy is much lower.

Your Task

Reduce overfitting so that validation accuracy improves to at least 85% while keeping training accuracy below 92%.

Use scikit-learn only.

Keep the same dataset and model type (text vectorization + logistic regression).

Do not add external data.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

ML Python

from sklearn.datasets import load_files
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.metrics import accuracy_score

# Load dataset
reviews = load_files('aclImdb/train/', categories=['pos', 'neg'], shuffle=True, random_state=42)
X, y = reviews.data, reviews.target

# Split data
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)

# Pipeline with vectorizer and logistic regression
pipeline = Pipeline([
    ('tfidf', TfidfVectorizer(stop_words='english', max_features=5000, ngram_range=(1,2))),
    ('clf', LogisticRegression(max_iter=200, solver='liblinear'))
])

# Hyperparameter tuning for regularization strength
param_grid = {
    'clf__C': [0.01, 0.1, 1, 10]
}

grid = GridSearchCV(pipeline, param_grid, cv=5, scoring='accuracy')
grid.fit(X_train, y_train)

# Best model evaluation
y_train_pred = grid.predict(X_train)
y_val_pred = grid.predict(X_val)

train_acc = accuracy_score(y_train, y_train_pred) * 100
val_acc = accuracy_score(y_val, y_val_pred) * 100

print(f'Training accuracy: {train_acc:.2f}%')
print(f'Validation accuracy: {val_acc:.2f}%')

Added stop words removal in TfidfVectorizer to reduce noise.

Limited max_features to 5000 to reduce model complexity.

Used bigrams (1,2) to capture more context.

Added logistic regression regularization with hyperparameter tuning for C.

Used GridSearchCV with 5-fold cross-validation to find best regularization strength.

Results Interpretation

Before: Training accuracy: 98%, Validation accuracy: 75% (overfitting)

After: Training accuracy: 90.5%, Validation accuracy: 86.3% (better generalization)

Adding regularization and limiting model complexity helps reduce overfitting, improving validation accuracy while slightly lowering training accuracy.

Bonus Experiment

Try using a different classifier like a Support Vector Machine (SVM) with similar preprocessing and compare results.

💡 Hint

Use sklearn.svm.SVC with linear kernel and tune the C parameter similarly.

Practice

1. What is the main goal of sentiment analysis in machine learning?

easy

A. To generate new text based on input

B. To count the number of words in a sentence

C. To translate text from one language to another

D. To find out if text expresses positive or negative feelings

Solution

1. Step 1: Understand sentiment analysis purpose

   Sentiment analysis is about detecting emotions or opinions in text, usually positive or negative feelings.

2. Step 2: Compare options with definition

   Only To find out if text expresses positive or negative feelings matches this goal. Others describe different tasks like counting words or translation.

3. Final Answer:

   To find out if text expresses positive or negative feelings -> Option D

4. Quick Check:

   Sentiment analysis = Detect positive/negative feelings [OK]

Hint: Sentiment analysis = positive or negative text feelings [OK]

Common Mistakes:

• Confusing sentiment analysis with translation
• Thinking it counts words instead of feelings
• Mixing it up with text generation

2. Which scikit-learn class is commonly used to convert text into numbers for sentiment analysis?

easy

A. LinearRegression

B. CountVectorizer

C. KMeans

D. DecisionTreeClassifier

Solution

1. Step 1: Identify text to number conversion tool

   CountVectorizer turns text into a matrix of word counts, a common step before training models.

2. Step 2: Check other options

   LinearRegression predicts numbers, KMeans clusters data, DecisionTreeClassifier classifies but does not convert text directly.

3. Final Answer:

   CountVectorizer -> Option B

4. Quick Check:

   Text to numbers = CountVectorizer [OK]

Hint: CountVectorizer converts text to numbers [OK]

Common Mistakes:

• Choosing regression or clustering classes for text conversion
• Confusing model classes with vectorizers
• Using classifiers directly on raw text

3. What will be the output of this code snippet?

from sklearn.feature_extraction.text import CountVectorizer
texts = ['I love this movie', 'I hate this movie']
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(texts)
print(X.toarray())

medium

A. [[0 1 1 1 1]
[1 1 0 1 1]]

B. [[1 1 1 1 0]
[1 1 1 0 1]]

C. [[1 1 0 1 1]
[1 0 1 1 1]]

D. [[1 0 1 1 1]
[1 1 1 0 1]]

Solution

1. Step 1: Identify vocabulary order

   CountVectorizer sorts words alphabetically: ['hate', 'i', 'love', 'movie', 'this']

2. Step 2: Map texts to counts

   'I love this movie' counts: hate=0, i=1, love=1, movie=1, this=1; 'I hate this movie' counts: hate=1, i=1, love=0, movie=1, this=1

3. Final Answer:

   [[0 1 1 1 1] [1 1 0 1 1]] -> Option A

4. Quick Check:

   Alphabetical words count matrix matches [[0 1 1 1 1] [1 1 0 1 1]] [OK]

Hint: CountVectorizer sorts words alphabetically [OK]

Common Mistakes:

• Assuming original word order instead of alphabetical
• Mixing up counts for 'love' and 'hate'
• Forgetting to convert sparse matrix to array

4. Identify the error in this sentiment analysis code snippet:

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.linear_model import LogisticRegression
texts = ['good', 'bad']
labels = [1, 0]
vectorizer = CountVectorizer()
X = vectorizer.transform(texts)
model = LogisticRegression()
model.fit(X, labels)

medium

A. Labels should be strings, not integers

B. LogisticRegression cannot be used for sentiment analysis

C. Using transform before fit_transform on vectorizer

D. Texts list should be empty

Solution

1. Step 1: Check vectorizer usage

   CountVectorizer must be fit with fit_transform before transform to learn vocabulary.

2. Step 2: Identify error in code

   Code calls transform without fitting first, causing error because vocabulary is unknown.

3. Final Answer:

   Using transform before fit_transform on vectorizer -> Option C

4. Quick Check:

   Vectorizer must fit before transform [OK]

Hint: Always fit vectorizer before transform [OK]

Common Mistakes:

• Calling transform without fitting vectorizer
• Thinking labels must be strings
• Believing LogisticRegression is unsuitable here

5. You trained a sentiment analysis model with scikit-learn and got 90% accuracy on training data but only 60% on new reviews. What should you do to improve real-world performance?

hard

A. Use more diverse training data and tune model parameters

B. Reduce training data size to avoid overfitting

C. Remove stop words from test data only

D. Switch to a regression model instead of classification

Solution

1. Step 1: Understand overfitting signs

   High training accuracy but low test accuracy means model learned training data too well but fails on new data.

2. Step 2: Choose improvement strategy

   Using more diverse data and tuning parameters helps model generalize better. Reducing data or removing stop words only on test is wrong. Regression is not suitable for classification tasks.

3. Final Answer:

   Use more diverse training data and tune model parameters -> Option A

4. Quick Check:

   Overfitting fix = More data + tuning [OK]

Hint: Fix overfitting with more data and tuning [OK]

Common Mistakes:

• Reducing training data size worsens model
• Altering test data only is incorrect
• Confusing classification with regression