Aspect-based sentiment analysis in NLP - Model Metrics & Evaluation

Metrics & Evaluation - Aspect-based sentiment analysis

Which metric matters for Aspect-based Sentiment Analysis and WHY

Aspect-based sentiment analysis finds feelings about parts of a product or service, like "battery" or "screen" in a phone review. We want to know if the model correctly finds these parts and their feelings.

The key metrics are Precision, Recall, and F1-score for each aspect and sentiment class (positive, negative, neutral). These show how well the model finds correct aspects and their feelings without missing or wrongly labeling them.

Precision tells us how many predicted aspects and sentiments are right. Recall tells us how many real aspects and sentiments the model found. F1-score balances both, giving a clear picture of overall quality.

Confusion Matrix Example

Imagine the model predicts sentiment for the "battery" aspect. Here is a confusion matrix for positive sentiment detection:

      | Predicted Positive | Predicted Not Positive |
      |--------------------|------------------------|
      | True Positive (TP) = 40 | False Positive (FP) = 5 |
      | False Negative (FN) = 10 | True Negative (TN) = 45 |
    

Total samples = 40 + 10 + 5 + 45 = 100

From this, we calculate:

• Precision = TP / (TP + FP) = 40 / (40 + 5) = 0.89
• Recall = TP / (TP + FN) = 40 / (40 + 10) = 0.80
• F1-score = 2 * (Precision * Recall) / (Precision + Recall) ≈ 0.84

Precision vs Recall Tradeoff with Examples

In aspect-based sentiment analysis, sometimes we want to be very sure about the sentiments we predict (high precision). For example, if a company wants to respond only to very certain negative feedback, high precision avoids false alarms.

Other times, we want to catch as many relevant sentiments as possible (high recall). For example, if a brand wants to find all possible complaints about "battery", missing any could hurt customer satisfaction.

Balancing precision and recall depends on the goal. F1-score helps find a good middle ground.

What Good vs Bad Metric Values Look Like

Good values:

• Precision and recall above 0.80 for each aspect and sentiment class.
• F1-score close to or above 0.80, showing balanced performance.
• Consistent results across different aspects (battery, screen, service).

Bad values:

• Precision or recall below 0.50, meaning many wrong or missed predictions.
• Very high precision but very low recall, or vice versa, showing imbalance.
• Large differences in metrics between aspects, indicating model struggles with some parts.

Common Pitfalls in Metrics

• Ignoring class imbalance: Some aspects or sentiments appear less often. Accuracy can be misleading if the model just guesses the common class.
• Data leakage: If test data leaks into training, metrics look too good but model fails in real use.
• Overfitting: Very high training metrics but low test metrics mean the model memorizes training data, not generalizing well.
• Not evaluating per aspect: Overall metrics hide poor performance on specific aspects.

Self Check

Your aspect-based sentiment model has 98% accuracy but only 12% recall on negative sentiments about "battery". Is it good for production?

Answer: No. The high accuracy is misleading because most data is not negative battery sentiment. The very low recall means the model misses most negative battery feedback, which is critical to catch. This model needs improvement before production.

Key Result

Precision, recall, and F1-score per aspect and sentiment are key to evaluate aspect-based sentiment analysis models effectively.