Sentiment analysis pipeline in NLP - Model Metrics & Evaluation

Metrics & Evaluation - Sentiment analysis pipeline

Which metric matters for Sentiment analysis pipeline and WHY

In sentiment analysis, we want to know how well the model can correctly identify positive, negative, or neutral feelings in text. The key metrics are Accuracy, Precision, Recall, and F1-score. Accuracy tells us overall correctness, but because some sentiments might be rare, precision and recall help us understand how well the model finds each sentiment without too many mistakes or misses. F1-score balances precision and recall, giving a single number to compare models.

Confusion matrix for Sentiment analysis

       Predicted
       Pos  Neg  Neu
    P  50   5    10
    N  3    40   7
    U  8    6    60

    Legend:
    P = Positive actual
    N = Negative actual
    U = Neutral actual
    Numbers = counts of predictions
    

This matrix shows how many texts were correctly or incorrectly labeled for each sentiment. For example, 50 positive texts were correctly predicted as positive, 5 were wrongly predicted as negative, and 10 as neutral.

Precision vs Recall tradeoff with examples

Imagine a company uses sentiment analysis to spot unhappy customers (negative sentiment) quickly. Here, recall is very important because missing unhappy customers means lost chances to help them. But if the model marks too many happy customers as unhappy (low precision), it wastes time.

On the other hand, if the company only wants to be sure about unhappy customers before acting, precision matters more to avoid false alarms.

Balancing precision and recall depends on the goal: catching all negatives (high recall) or being very sure about negatives (high precision).

What good vs bad metric values look like

Good: Accuracy above 85%, precision and recall above 80% for each sentiment class, and F1-score close to these values. This means the model correctly finds most sentiments and makes few mistakes.

Bad: Accuracy around 50-60%, precision or recall below 50%, or very unbalanced scores (e.g., high precision but very low recall). This means the model misses many sentiments or wrongly labels many texts.

Common pitfalls in Sentiment analysis metrics

• Accuracy paradox: If one sentiment is very common, a model guessing only that sentiment can have high accuracy but poor usefulness.
• Data leakage: If test data leaks into training, metrics look unrealistically high.
• Overfitting: Very high training accuracy but low test accuracy means the model memorizes training data but fails on new texts.
• Ignoring class imbalance: Not checking precision and recall per class can hide poor performance on rare sentiments.

Self-check question

Your sentiment analysis model has 98% accuracy but only 12% recall on negative sentiment. Is it good for production? Why or why not?

Answer: No, it is not good. The model misses most negative sentiments (low recall), which means unhappy customers might not be detected. High accuracy is misleading if the negative class is rare. Improving recall for negative sentiment is important.

Key Result

For sentiment analysis, balanced precision and recall per sentiment class are key to reliable predictions.

Practice

1. What is the main purpose of a sentiment analysis pipeline in natural language processing?

easy

A. To automatically detect feelings or opinions in text

B. To translate text from one language to another

C. To count the number of words in a sentence

D. To generate new text based on input

Solution

1. Step 1: Understand the goal of sentiment analysis

   Sentiment analysis is about finding emotions or opinions in text data.

2. Step 2: Identify the pipeline's role

   A sentiment analysis pipeline automates this process to detect feelings like positive or negative.

3. Final Answer:

   To automatically detect feelings or opinions in text -> Option A

4. Quick Check:

   Sentiment analysis = detect feelings [OK]

Hint: Sentiment analysis finds emotions in text fast [OK]

Common Mistakes:

• Confusing sentiment analysis with translation
• Thinking it counts words instead of feelings
• Assuming it generates new text

2. Which of the following is the correct way to create a sentiment analysis pipeline using the Hugging Face Transformers library in Python?

easy

A. pipeline = Pipeline('text-classification')

B. pipeline = create_pipeline('sentiment')

C. pipeline = sentiment_pipeline()

D. pipeline = pipeline('sentiment-analysis')

Solution

1. Step 1: Recall the Hugging Face pipeline syntax

   The correct function is pipeline with the task name as a string.

2. Step 2: Match the exact task name for sentiment analysis

   The task name is 'sentiment-analysis', so pipeline('sentiment-analysis') is correct.

3. Final Answer:

   pipeline = pipeline('sentiment-analysis') -> Option D

4. Quick Check:

   Use pipeline('sentiment-analysis') to create sentiment pipeline [OK]

Hint: Use pipeline('sentiment-analysis') exactly [OK]

Common Mistakes:

• Using wrong function names like create_pipeline
• Missing quotes around task name
• Using incorrect task names like 'sentiment'

3. What will be the output of this Python code using Hugging Face's sentiment analysis pipeline?

from transformers import pipeline
sentiment = pipeline('sentiment-analysis')
result = sentiment('I love sunny days!')
print(result)

medium

A. [{'label': 'NEGATIVE', 'score': 0.99}]

B. [{'label': 'POSITIVE', 'score': 0.99}]

C. SyntaxError

D. []

Solution

1. Step 1: Understand the input text sentiment

   The sentence 'I love sunny days!' expresses a positive feeling.

2. Step 2: Predict output from sentiment pipeline

   The pipeline returns a list with a dictionary containing label 'POSITIVE' and a high confidence score.

3. Final Answer:

   [{'label': 'POSITIVE', 'score': 0.99}] -> Option B

4. Quick Check:

   Positive sentence = POSITIVE label [OK]

Hint: Positive words give POSITIVE label with high score [OK]

Common Mistakes:

• Expecting NEGATIVE label for positive text
• Thinking output is a string, not a list of dict
• Confusing syntax errors with runtime output

4. You wrote this code but get an error: NameError: name 'pipeline' is not defined. What is the likely fix?

sentiment = pipeline('sentiment-analysis')
result = sentiment('I hate rain.')
print(result)

medium

A. Add from transformers import pipeline before using pipeline

B. Change 'sentiment-analysis' to 'sentiment'

C. Replace pipeline with sentiment_pipeline

D. Remove parentheses from pipeline call

Solution

1. Step 1: Identify cause of NameError

   The error means Python does not know what pipeline is because it was not imported.

2. Step 2: Fix by importing pipeline function

   Adding from transformers import pipeline defines pipeline so the code runs correctly.

3. Final Answer:

   Add from transformers import pipeline before using pipeline -> Option A

4. Quick Check:

   Import missing = NameError fixed [OK]

Hint: Always import pipeline before using it [OK]

Common Mistakes:

• Changing task name instead of importing
• Assuming pipeline is built-in without import
• Removing parentheses causing syntax errors

5. You want to analyze customer reviews but some reviews are empty strings or just spaces. How should you modify your sentiment analysis pipeline to handle this before prediction?

hard

A. Replace empty reviews with the word 'neutral' and analyze

B. Pass all reviews directly to the pipeline without changes

C. Filter out empty or whitespace-only reviews before passing to the pipeline

D. Use a different pipeline for empty reviews

Solution

1. Step 1: Understand the problem with empty inputs

   Empty or whitespace-only texts do not contain sentiment and can cause errors or meaningless results.

2. Step 2: Apply filtering before analysis

   Removing or skipping these empty reviews ensures the pipeline only processes valid text, improving accuracy and avoiding errors.

3. Final Answer:

   Filter out empty or whitespace-only reviews before passing to the pipeline -> Option C

4. Quick Check:

   Remove empty inputs before analysis [OK]

Hint: Skip empty reviews to avoid errors [OK]

Common Mistakes:

• Passing empty strings causing errors
• Replacing empty with unrelated words
• Using multiple pipelines unnecessarily