Aspect-based sentiment analysis in NLP - Model Metrics & Evaluation

Metrics & Evaluation - Aspect-based sentiment analysis

Which metric matters for Aspect-based Sentiment Analysis and WHY

Aspect-based sentiment analysis finds feelings about parts of a product or service, like "battery" or "screen" in a phone review. We want to know if the model correctly finds these parts and their feelings.

The key metrics are Precision, Recall, and F1-score for each aspect and sentiment class (positive, negative, neutral). These show how well the model finds correct aspects and their feelings without missing or wrongly labeling them.

Precision tells us how many predicted aspects and sentiments are right. Recall tells us how many real aspects and sentiments the model found. F1-score balances both, giving a clear picture of overall quality.

Confusion Matrix Example

Imagine the model predicts sentiment for the "battery" aspect. Here is a confusion matrix for positive sentiment detection:

      | Predicted Positive | Predicted Not Positive |
      |--------------------|------------------------|
      | True Positive (TP) = 40 | False Positive (FP) = 5 |
      | False Negative (FN) = 10 | True Negative (TN) = 45 |
    

Total samples = 40 + 10 + 5 + 45 = 100

From this, we calculate:

• Precision = TP / (TP + FP) = 40 / (40 + 5) = 0.89
• Recall = TP / (TP + FN) = 40 / (40 + 10) = 0.80
• F1-score = 2 * (Precision * Recall) / (Precision + Recall) ≈ 0.84

Precision vs Recall Tradeoff with Examples

In aspect-based sentiment analysis, sometimes we want to be very sure about the sentiments we predict (high precision). For example, if a company wants to respond only to very certain negative feedback, high precision avoids false alarms.

Other times, we want to catch as many relevant sentiments as possible (high recall). For example, if a brand wants to find all possible complaints about "battery", missing any could hurt customer satisfaction.

Balancing precision and recall depends on the goal. F1-score helps find a good middle ground.

What Good vs Bad Metric Values Look Like

Good values:

• Precision and recall above 0.80 for each aspect and sentiment class.
• F1-score close to or above 0.80, showing balanced performance.
• Consistent results across different aspects (battery, screen, service).

Bad values:

• Precision or recall below 0.50, meaning many wrong or missed predictions.
• Very high precision but very low recall, or vice versa, showing imbalance.
• Large differences in metrics between aspects, indicating model struggles with some parts.

Common Pitfalls in Metrics

• Ignoring class imbalance: Some aspects or sentiments appear less often. Accuracy can be misleading if the model just guesses the common class.
• Data leakage: If test data leaks into training, metrics look too good but model fails in real use.
• Overfitting: Very high training metrics but low test metrics mean the model memorizes training data, not generalizing well.
• Not evaluating per aspect: Overall metrics hide poor performance on specific aspects.

Self Check

Your aspect-based sentiment model has 98% accuracy but only 12% recall on negative sentiments about "battery". Is it good for production?

Answer: No. The high accuracy is misleading because most data is not negative battery sentiment. The very low recall means the model misses most negative battery feedback, which is critical to catch. This model needs improvement before production.

Key Result

Precision, recall, and F1-score per aspect and sentiment are key to evaluate aspect-based sentiment analysis models effectively.

Practice

1. What is the main goal of aspect-based sentiment analysis?

easy

A. To translate text from one language to another

B. To count the number of words in a sentence

C. To find feelings about specific parts or features in text

D. To generate new text based on input

Solution

1. Step 1: Understand the concept of aspect-based sentiment analysis

   It focuses on identifying opinions about specific parts or features, not the whole text.

2. Step 2: Compare options with this concept

   Only To find feelings about specific parts or features in text matches this goal; others describe unrelated tasks.

3. Final Answer:

   To find feelings about specific parts or features in text -> Option C

4. Quick Check:

   Aspect-based sentiment = specific parts sentiment [OK]

Hint: Focus on 'specific parts' in the question to find the goal [OK]

Common Mistakes:

• Confusing overall sentiment with aspect-level sentiment
• Thinking it translates or generates text
• Mixing sentiment analysis with word counting

2. Which Python library is commonly used for simple aspect-based sentiment analysis?

easy

A. Matplotlib

B. NumPy

C. Flask

D. TextBlob

Solution

1. Step 1: Identify libraries related to text sentiment

   TextBlob is a simple library for sentiment analysis and text processing.

2. Step 2: Eliminate unrelated libraries

   NumPy is for numbers, Matplotlib for plotting, Flask for web apps, so they don't fit.

3. Final Answer:

   TextBlob -> Option D

4. Quick Check:

   TextBlob = simple sentiment tool [OK]

Hint: Pick the library known for text sentiment, not numbers or web [OK]

Common Mistakes:

• Choosing NumPy or Matplotlib which are not for sentiment
• Confusing Flask as a sentiment tool
• Not knowing TextBlob's purpose

3. Given this Python code snippet for aspect sentiment, what is the output?

from textblob import TextBlob
text = "The battery life is great but the screen is dull."
aspects = ['battery life', 'screen']
results = {}
for aspect in aspects:
    blob = TextBlob(text)
    if aspect in text:
        sentiment = blob.sentiment.polarity
        results[aspect] = 'positive' if sentiment > 0 else 'negative'
print(results)

medium

A. {'battery life': 'positive', 'screen': 'positive'}

B. {'battery life': 'positive', 'screen': 'negative'}

C. {'battery life': 'negative', 'screen': 'negative'}

D. SyntaxError

Solution

1. Step 1: Understand the sentiment polarity calculation

   TextBlob calculates overall sentiment polarity of the whole text, which is positive because 'battery life is great' outweighs 'screen is dull'.

2. Step 2: Check how results are assigned

   For each aspect found in text, sentiment polarity is checked once (overall), so both aspects get 'positive'.

3. Final Answer:

   {'battery life': 'positive', 'screen': 'positive'} -> Option A

4. Quick Check:

   Overall sentiment positive -> both aspects positive [OK]

Hint: TextBlob sentiment is overall, so all aspects get same polarity [OK]

Common Mistakes:

• Assuming aspect-specific sentiment without extra processing
• Expecting different sentiment for each aspect
• Thinking code has syntax errors

4. Identify the error in this aspect-based sentiment analysis code snippet:

from textblob import TextBlob
text = "The food was tasty but the service was slow."
aspects = ['food', 'service']
results = {}
for aspect in aspects:
    blob = TextBlob(aspect)
    sentiment = blob.sentiment.polarity
    results[aspect] = 'positive' if sentiment > 0 else 'negative'
print(results)

medium

A. The aspects list is empty

B. TextBlob is applied to aspect text, not the full sentence

C. The results dictionary is not initialized

D. The print statement is missing parentheses

Solution

1. Step 1: Check what text is analyzed by TextBlob

   TextBlob is called on the aspect word only, not the full sentence, so sentiment is meaningless.

2. Step 2: Identify correct usage

   TextBlob should analyze the full sentence or relevant sentence part, not just the aspect word.

3. Final Answer:

   TextBlob is applied to aspect text, not the full sentence -> Option B

4. Quick Check:

   TextBlob needs full text, not just aspect [OK]

Hint: Check what text TextBlob analyzes--aspect or full sentence? [OK]

Common Mistakes:

• Thinking aspects list is empty
• Ignoring that results dict is initialized
• Assuming print syntax error in Python 3

5. You want to improve aspect-based sentiment analysis by focusing on sentences mentioning each aspect separately. Which approach is best?

hard

A. Split text into sentences, analyze sentiment only for sentences containing the aspect

B. Analyze the entire text sentiment once and assign it to all aspects

C. Ignore sentences and analyze only aspect words with TextBlob

D. Use random sentiment values for each aspect

Solution

1. Step 1: Understand the problem with overall sentiment

   Overall sentiment mixes all opinions, so it can't tell which aspect is positive or negative.

2. Step 2: Use sentence-level analysis for precision

   Splitting text into sentences and analyzing only those mentioning the aspect gives accurate aspect sentiment.

3. Final Answer:

   Split text into sentences, analyze sentiment only for sentences containing the aspect -> Option A

4. Quick Check:

   Sentence-level sentiment = better aspect accuracy [OK]

Hint: Analyze sentences mentioning aspect, not whole text [OK]

Common Mistakes:

• Using overall sentiment for all aspects
• Analyzing only aspect words without context
• Assigning random sentiment values