Summarization in Prompt Engineering / GenAI - Model Metrics & Evaluation

Metrics & Evaluation - Summarization

Which metric matters for Summarization and WHY

For summarization, we want to check how well the summary captures the important parts of the original text. The main metrics are ROUGE scores, especially ROUGE-1, ROUGE-2, and ROUGE-L. These compare the overlap of words and phrases between the generated summary and a human-written summary.

ROUGE-1 measures overlap of single words, ROUGE-2 looks at pairs of words, and ROUGE-L checks the longest matching sequence. Higher ROUGE scores mean the summary is closer to the reference, so it is better.

Confusion matrix or equivalent visualization

Summarization is not a classification task, so it does not use a confusion matrix. Instead, we use overlap-based metrics like ROUGE.

Example ROUGE-1 calculation:

Reference summary: "The cat sat on the mat."
Generated summary: "Cat sat on mat."

Overlap words: cat, sat, on, mat
Total words in reference: 6
ROUGE-1 recall = 4 / 6 = 0.67
    

Precision vs Recall tradeoff with examples

ROUGE metrics can be calculated as precision, recall, or F1 score.

• Precision: How many words in the generated summary appear in the reference? High precision means the summary is mostly relevant.
• Recall: How many words in the reference summary appear in the generated summary? High recall means the summary covers most important points.

For example, a very short summary might have high precision (few words, all relevant) but low recall (misses many points). A very long summary might have high recall but low precision (includes many irrelevant words).

We want a balance, often measured by the F1 score, to get a summary that is both relevant and covers key points.

What "good" vs "bad" metric values look like for Summarization

Good ROUGE scores depend on the dataset and task, but generally:

• ROUGE-1 F1 > 0.4 is considered decent for many summarization tasks.
• ROUGE-2 F1 > 0.2 shows good phrase matching.
• ROUGE-L F1 > 0.35 means good sequence matching.

Bad scores are close to zero, meaning the summary barely matches the reference. Scores near 0.1 or less show poor quality summaries.

Common pitfalls in Summarization metrics

• Overfitting to ROUGE: Models might learn to copy phrases to boost ROUGE but produce less natural summaries.
• Ignoring meaning: ROUGE measures word overlap, not if the summary truly captures meaning.
• Reference bias: Using only one reference summary can limit evaluation fairness.
• Length bias: Very short or very long summaries can skew precision or recall.

Self-check question

Your summarization model has a ROUGE-1 recall of 0.85 but ROUGE-1 precision of 0.3. Is this good? Why or why not?

Answer: This means the summary covers most important words (high recall) but includes many extra words not in the reference (low precision). The summary might be too long or noisy. You want to improve precision to make the summary more concise and relevant.

Key Result

ROUGE scores (especially ROUGE-1, ROUGE-2, ROUGE-L) are key to measure how well a summary matches the reference in content and phrasing.