Abstractive summarization in NLP - Model Metrics & Evaluation

Metrics & Evaluation - Abstractive summarization

Which metric matters for abstractive summarization and WHY

For abstractive summarization, the main metrics are ROUGE scores. ROUGE measures how much the model's summary overlaps with a human-written summary. It checks matching words, phrases, and sentence structures. This is important because abstractive summarization creates new sentences, so exact matches are rare. ROUGE helps us see if the summary keeps the main ideas and important details.

Besides ROUGE, sometimes BLEU is used, but ROUGE is preferred because it focuses on recall (capturing all important info) rather than precision.

Confusion matrix or equivalent visualization

Abstractive summarization is not a simple yes/no classification, so confusion matrices don't apply directly. Instead, we use overlap-based metrics like ROUGE.

Example ROUGE-1 scores (word overlap):
Reference summary: "The cat sat on the mat."
Model summary: "A cat is sitting on a mat."

ROUGE-1 Precision = (Number of overlapping words) / (Total words in model summary)
ROUGE-1 Recall = (Number of overlapping words) / (Total words in reference summary)

If overlapping words = 3, model summary words = 7, reference words = 6:
Precision = 3/7 ≈ 0.43
Recall = 3/6 = 0.50
F1 = 2 * (0.43 * 0.50) / (0.43 + 0.50) ≈ 0.46

Precision vs Recall tradeoff with concrete examples

In summarization, Recall means how much important info from the original text is captured in the summary. Precision means how much of the summary is relevant and not extra or wrong info.

High recall but low precision: The summary includes almost all important points but also adds unrelated or repeated info. It might be too long or confusing.

High precision but low recall: The summary is very concise and accurate but misses some key points, so it may not fully inform the reader.

For example, a news summary that misses a key event (low recall) is less useful, while a summary that repeats facts unnecessarily (low precision) wastes reader time.

Good summarization balances both, often measured by the F1 score of ROUGE.

What "good" vs "bad" metric values look like for abstractive summarization

Good metrics:

• ROUGE-1 F1 score above 0.4 to 0.5 usually means the summary captures important content well.
• ROUGE-L (longest common subsequence) above 0.4 shows good sentence structure similarity.
• Balanced precision and recall scores indicate the summary is both relevant and complete.

Bad metrics:

• ROUGE scores below 0.2 suggest the summary misses many key points or is very different from the reference.
• Very high precision but very low recall means the summary is too short or incomplete.
• Very high recall but very low precision means the summary is too long or noisy.

Common pitfalls in metrics for abstractive summarization

• Over-reliance on ROUGE: ROUGE measures word overlap but not meaning. A summary can have good ROUGE but be confusing or incorrect.
• Ignoring human evaluation: Sometimes metrics don't capture fluency or coherence, so human checks are important.
• Data leakage: If the model sees test summaries during training, metrics will be unrealistically high.
• Length bias: Longer summaries tend to have higher recall but may be less concise.
• Not considering diversity: Metrics don't measure if the summary is repetitive or dull.

Self-check question

Your abstractive summarization model has a ROUGE-1 F1 score of 0.45 but a ROUGE-2 (two-word phrase) recall of 0.2. Is this good? Why or why not?

Answer: The ROUGE-1 F1 of 0.45 is decent, showing the model captures many important words. But the low ROUGE-2 recall of 0.2 means it misses many important phrases or word pairs, indicating the summary may lack fluency or detailed meaning. So, the model is okay but could improve in capturing meaningful phrases for better quality.

Key Result

ROUGE scores, especially ROUGE-1 and ROUGE-L F1, are key to evaluating how well abstractive summaries capture important content and structure.