Sentiment analysis with scikit-learn in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - Sentiment analysis with scikit-learn

Which metric matters for Sentiment analysis with scikit-learn and WHY

In sentiment analysis, we want to know how well the model can correctly identify positive and negative feelings in text. The key metrics are Precision, Recall, and F1-score.

Precision tells us how many of the texts the model labeled as positive (or negative) are actually correct. This is important if we want to avoid false alarms.

Recall tells us how many of the actual positive (or negative) texts the model found. This matters if missing a sentiment is costly.

F1-score balances precision and recall, giving a single number to understand overall performance.

Accuracy is also used but can be misleading if classes are imbalanced (e.g., many more positive than negative reviews).

Confusion matrix example

Suppose we have 100 reviews: 50 positive and 50 negative. The model predicts as follows:

      | Predicted Positive | Predicted Negative |
      |--------------------|--------------------|
      | True Positive (TP) = 40 | False Negative (FN) = 10 |
      | False Positive (FP) = 5 | True Negative (TN) = 45 |
    

Calculations:

• Precision = TP / (TP + FP) = 40 / (40 + 5) = 0.89
• Recall = TP / (TP + FN) = 40 / (40 + 10) = 0.80
• F1-score = 2 * (0.89 * 0.80) / (0.89 + 0.80) ≈ 0.84
• Accuracy = (TP + TN) / Total = (40 + 45) / 100 = 0.85

Precision vs Recall tradeoff with examples

If we want to be very sure that positive reviews are truly positive, we focus on high precision. This means fewer false positives but might miss some positive reviews (lower recall).

If we want to catch as many positive reviews as possible, we focus on high recall. This means fewer false negatives but might include some false positives (lower precision).

For example, a company analyzing customer feedback might prefer high recall to not miss any unhappy customers. But if they want to send special offers only to truly happy customers, they might prefer high precision.

What good vs bad metric values look like

Good metrics:

• Precision and recall above 0.80 show the model is reliable in identifying sentiments.
• F1-score above 0.80 means a good balance.
• Accuracy above 0.85 is good if classes are balanced.

Bad metrics:

• Precision or recall below 0.50 means many errors in predictions.
• F1-score below 0.60 shows poor overall performance.
• High accuracy but low recall or precision indicates the model might be guessing the majority class.

Common pitfalls in metrics for sentiment analysis

• Accuracy paradox: High accuracy can be misleading if one sentiment class dominates the data.
• Data leakage: If test data leaks into training, metrics look better but model fails in real use.
• Overfitting: Very high training accuracy but low test accuracy means the model memorizes training data, not generalizing well.
• Ignoring class imbalance: Not using precision and recall can hide poor performance on minority classes.

Self-check question

Your sentiment analysis model has 98% accuracy but only 12% recall on negative reviews. Is it good for production? Why or why not?

Answer: No, it is not good. The model misses 88% of negative reviews (low recall), which means many negative sentiments are not detected. High accuracy is likely due to many positive reviews dominating the data. This can cause poor user experience if negative feedback is ignored.

Key Result

Precision, recall, and F1-score are key to evaluate sentiment analysis models, especially when classes are imbalanced.

Practice

1. What is the main goal of sentiment analysis in machine learning?

easy

A. To generate new text based on input

B. To count the number of words in a sentence

C. To translate text from one language to another

D. To find out if text expresses positive or negative feelings

Solution

1. Step 1: Understand sentiment analysis purpose

   Sentiment analysis is about detecting emotions or opinions in text, usually positive or negative feelings.

2. Step 2: Compare options with definition

   Only To find out if text expresses positive or negative feelings matches this goal. Others describe different tasks like counting words or translation.

3. Final Answer:

   To find out if text expresses positive or negative feelings -> Option D

4. Quick Check:

   Sentiment analysis = Detect positive/negative feelings [OK]

Hint: Sentiment analysis = positive or negative text feelings [OK]

Common Mistakes:

• Confusing sentiment analysis with translation
• Thinking it counts words instead of feelings
• Mixing it up with text generation

2. Which scikit-learn class is commonly used to convert text into numbers for sentiment analysis?

easy

A. LinearRegression

B. CountVectorizer

C. KMeans

D. DecisionTreeClassifier

Solution

1. Step 1: Identify text to number conversion tool

   CountVectorizer turns text into a matrix of word counts, a common step before training models.

2. Step 2: Check other options

   LinearRegression predicts numbers, KMeans clusters data, DecisionTreeClassifier classifies but does not convert text directly.

3. Final Answer:

   CountVectorizer -> Option B

4. Quick Check:

   Text to numbers = CountVectorizer [OK]

Hint: CountVectorizer converts text to numbers [OK]

Common Mistakes:

• Choosing regression or clustering classes for text conversion
• Confusing model classes with vectorizers
• Using classifiers directly on raw text

3. What will be the output of this code snippet?

from sklearn.feature_extraction.text import CountVectorizer
texts = ['I love this movie', 'I hate this movie']
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(texts)
print(X.toarray())

medium

A. [[0 1 1 1 1]
[1 1 0 1 1]]

B. [[1 1 1 1 0]
[1 1 1 0 1]]

C. [[1 1 0 1 1]
[1 0 1 1 1]]

D. [[1 0 1 1 1]
[1 1 1 0 1]]

Solution

1. Step 1: Identify vocabulary order

   CountVectorizer sorts words alphabetically: ['hate', 'i', 'love', 'movie', 'this']

2. Step 2: Map texts to counts

   'I love this movie' counts: hate=0, i=1, love=1, movie=1, this=1; 'I hate this movie' counts: hate=1, i=1, love=0, movie=1, this=1

3. Final Answer:

   [[0 1 1 1 1] [1 1 0 1 1]] -> Option A

4. Quick Check:

   Alphabetical words count matrix matches [[0 1 1 1 1] [1 1 0 1 1]] [OK]

Hint: CountVectorizer sorts words alphabetically [OK]

Common Mistakes:

• Assuming original word order instead of alphabetical
• Mixing up counts for 'love' and 'hate'
• Forgetting to convert sparse matrix to array

4. Identify the error in this sentiment analysis code snippet:

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.linear_model import LogisticRegression
texts = ['good', 'bad']
labels = [1, 0]
vectorizer = CountVectorizer()
X = vectorizer.transform(texts)
model = LogisticRegression()
model.fit(X, labels)

medium

A. Labels should be strings, not integers

B. LogisticRegression cannot be used for sentiment analysis

C. Using transform before fit_transform on vectorizer

D. Texts list should be empty

Solution

1. Step 1: Check vectorizer usage

   CountVectorizer must be fit with fit_transform before transform to learn vocabulary.

2. Step 2: Identify error in code

   Code calls transform without fitting first, causing error because vocabulary is unknown.

3. Final Answer:

   Using transform before fit_transform on vectorizer -> Option C

4. Quick Check:

   Vectorizer must fit before transform [OK]

Hint: Always fit vectorizer before transform [OK]

Common Mistakes:

• Calling transform without fitting vectorizer
• Thinking labels must be strings
• Believing LogisticRegression is unsuitable here

5. You trained a sentiment analysis model with scikit-learn and got 90% accuracy on training data but only 60% on new reviews. What should you do to improve real-world performance?

hard

A. Use more diverse training data and tune model parameters

B. Reduce training data size to avoid overfitting

C. Remove stop words from test data only

D. Switch to a regression model instead of classification

Solution

1. Step 1: Understand overfitting signs

   High training accuracy but low test accuracy means model learned training data too well but fails on new data.

2. Step 2: Choose improvement strategy

   Using more diverse data and tuning parameters helps model generalize better. Reducing data or removing stop words only on test is wrong. Regression is not suitable for classification tasks.

3. Final Answer:

   Use more diverse training data and tune model parameters -> Option A

4. Quick Check:

   Overfitting fix = More data + tuning [OK]

Hint: Fix overfitting with more data and tuning [OK]

Common Mistakes:

• Reducing training data size worsens model
• Altering test data only is incorrect
• Confusing classification with regression