Why summarization condenses information in NLP - Why Metrics Matter

Metrics & Evaluation - Why summarization condenses information

Which metric matters for this concept and WHY

For summarization, the key metric is ROUGE. ROUGE measures how well the generated summary captures the important parts by comparing overlapping words or phrases with reference summaries. It matters because summarization aims to keep the main ideas while cutting down length. A high ROUGE score means the summary keeps important info without losing meaning.

Confusion matrix or equivalent visualization (ASCII)

Reference summary: 30 words (important info)
Generated summary: 30 words (condensed info)

Overlap (matching words): 25 words

ROUGE-1 recall (word overlap) = Overlap / Reference words = 25 / 30 = 0.83

This shows the generated summary captures 83% of the important words from the reference summary.

Precision vs Recall tradeoff with concrete examples

In summarization, precision means how many words in the summary are actually important per reference summaries. Recall means how many important words from the reference summaries appear in the summary.

Example 1: High precision, low recall summary:
A very short summary with only a few words, all important. It misses many key points (low recall) but what it has is relevant (high precision).

Example 2: High recall, low precision summary:
A longer summary that includes most important words but also many unimportant ones. It covers many points (high recall) but adds noise (low precision).

Good summarization balances both to keep main ideas (high recall) and avoid extra fluff (high precision).

What "good" vs "bad" metric values look like for this use case

Good summary: ROUGE scores above 0.7 show the summary keeps most important info clearly and concisely.

Bad summary: ROUGE scores below 0.4 mean the summary misses many key points or adds irrelevant info, losing meaning.

Metrics pitfalls

• Overfitting: Model memorizes training summaries, scoring high ROUGE but poor on new texts.
• Length bias: Very short summaries may get high precision but low recall, misleading metric interpretation.
• Ignoring meaning: ROUGE counts word overlap but not if summary truly captures meaning or context.
• Data leakage: Using test summaries during training inflates scores unfairly.

Your model has 98% accuracy but 12% recall on fraud. Is it good?

This question is about fraud detection, not summarization. But it shows why recall matters: 12% recall means the model misses 88% of fraud cases, which is very bad. High accuracy can be misleading if the data is mostly non-fraud.

For summarization, similarly, a high ROUGE precision but very low recall means the summary misses many important points, so it is not good.

Key Result

ROUGE score best measures how well a summary keeps important information while condensing text.