Abstractive summarization in NLP - Model Metrics & Evaluation

Metrics & Evaluation - Abstractive summarization

Which metric matters for abstractive summarization and WHY

For abstractive summarization, the main metrics are ROUGE scores. ROUGE measures how much the model's summary overlaps with a human-written summary. It checks matching words, phrases, and sentence structures. This is important because abstractive summarization creates new sentences, so exact matches are rare. ROUGE helps us see if the summary keeps the main ideas and important details.

Besides ROUGE, sometimes BLEU is used, but ROUGE is preferred because it focuses on recall (capturing all important info) rather than precision.

Confusion matrix or equivalent visualization

Abstractive summarization is not a simple yes/no classification, so confusion matrices don't apply directly. Instead, we use overlap-based metrics like ROUGE.

Example ROUGE-1 scores (word overlap):
Reference summary: "The cat sat on the mat."
Model summary: "A cat is sitting on a mat."

ROUGE-1 Precision = (Number of overlapping words) / (Total words in model summary)
ROUGE-1 Recall = (Number of overlapping words) / (Total words in reference summary)

If overlapping words = 3, model summary words = 7, reference words = 6:
Precision = 3/7 ≈ 0.43
Recall = 3/6 = 0.50
F1 = 2 * (0.43 * 0.50) / (0.43 + 0.50) ≈ 0.46

Precision vs Recall tradeoff with concrete examples

In summarization, Recall means how much important info from the original text is captured in the summary. Precision means how much of the summary is relevant and not extra or wrong info.

High recall but low precision: The summary includes almost all important points but also adds unrelated or repeated info. It might be too long or confusing.

High precision but low recall: The summary is very concise and accurate but misses some key points, so it may not fully inform the reader.

For example, a news summary that misses a key event (low recall) is less useful, while a summary that repeats facts unnecessarily (low precision) wastes reader time.

Good summarization balances both, often measured by the F1 score of ROUGE.

What "good" vs "bad" metric values look like for abstractive summarization

Good metrics:

• ROUGE-1 F1 score above 0.4 to 0.5 usually means the summary captures important content well.
• ROUGE-L (longest common subsequence) above 0.4 shows good sentence structure similarity.
• Balanced precision and recall scores indicate the summary is both relevant and complete.

Bad metrics:

• ROUGE scores below 0.2 suggest the summary misses many key points or is very different from the reference.
• Very high precision but very low recall means the summary is too short or incomplete.
• Very high recall but very low precision means the summary is too long or noisy.

Common pitfalls in metrics for abstractive summarization

• Over-reliance on ROUGE: ROUGE measures word overlap but not meaning. A summary can have good ROUGE but be confusing or incorrect.
• Ignoring human evaluation: Sometimes metrics don't capture fluency or coherence, so human checks are important.
• Data leakage: If the model sees test summaries during training, metrics will be unrealistically high.
• Length bias: Longer summaries tend to have higher recall but may be less concise.
• Not considering diversity: Metrics don't measure if the summary is repetitive or dull.

Self-check question

Your abstractive summarization model has a ROUGE-1 F1 score of 0.45 but a ROUGE-2 (two-word phrase) recall of 0.2. Is this good? Why or why not?

Answer: The ROUGE-1 F1 of 0.45 is decent, showing the model captures many important words. But the low ROUGE-2 recall of 0.2 means it misses many important phrases or word pairs, indicating the summary may lack fluency or detailed meaning. So, the model is okay but could improve in capturing meaningful phrases for better quality.

Key Result

ROUGE scores, especially ROUGE-1 and ROUGE-L F1, are key to evaluating how well abstractive summaries capture important content and structure.

Practice

1. What is the main goal of abstractive summarization in natural language processing?

easy

A. To generate a concise summary using new phrases not directly copied from the text

B. To extract exact sentences from the original text without changes

C. To translate text from one language to another

D. To classify text into predefined categories

Solution

1. Step 1: Understand summarization types

   There are two main types: extractive (copying sentences) and abstractive (generating new phrases).

2. Step 2: Identify abstractive summarization goal

   Abstractive summarization creates a shorter version using new wording, not just copying.

3. Final Answer:

   To generate a concise summary using new phrases not directly copied from the text -> Option A

4. Quick Check:

   Abstractive summarization = new phrasing summary [OK]

Hint: Abstractive means creating new summary text, not copying [OK]

Common Mistakes:

• Confusing abstractive with extractive summarization
• Thinking summarization is just sentence extraction
• Mixing summarization with translation

2. Which of the following is the correct way to load a pretrained abstractive summarization model using Hugging Face Transformers in Python?

easy

A. from transformers import SummarizationModel; model = SummarizationModel.load()

B. from transformers import Summarizer; summarizer = Summarizer()

C. import transformers; summarizer = transformers.load('abstractive')

D. from transformers import pipeline; summarizer = pipeline('summarization')

Solution

1. Step 1: Recall Hugging Face pipeline usage

   The correct way to load a summarization model is using pipeline('summarization').

2. Step 2: Check each option

   from transformers import pipeline; summarizer = pipeline('summarization') uses the correct import and function. Others use incorrect classes or methods.

3. Final Answer:

   from transformers import pipeline; summarizer = pipeline('summarization') -> Option D

4. Quick Check:

   Use pipeline('summarization') to load model [OK]

Hint: Use pipeline('summarization') to load models easily [OK]

Common Mistakes:

• Using non-existent classes like Summarizer
• Trying to load models with wrong method names
• Importing whole transformers without pipeline

3. Given the following Python code using Hugging Face Transformers, what will be the output summary length approximately?

from transformers import pipeline
summarizer = pipeline('summarization')
text = "Machine learning is a method of data analysis that automates analytical model building. It is a branch of artificial intelligence based on the idea that systems can learn from data, identify patterns and make decisions with minimal human intervention."
summary = summarizer(text, max_length=30, min_length=10, do_sample=False)
print(len(summary[0]['summary_text'].split()))

medium

A. Between 10 and 30 words

B. Exactly 30 words

C. More than 50 words

D. Less than 5 words

Solution

1. Step 1: Understand max_length and min_length parameters

   The summarizer generates summaries with length between min_length and max_length words.

2. Step 2: Analyze the code output

   The summary length will be between 10 and 30 words, as specified by the parameters.

3. Final Answer:

   Between 10 and 30 words -> Option A

4. Quick Check:

   Summary length constrained by min_length and max_length [OK]

Hint: max_length and min_length set summary word count range [OK]

Common Mistakes:

• Assuming summary length equals max_length exactly
• Ignoring min_length parameter
• Expecting very short or very long summaries regardless of parameters

4. You wrote this code to summarize text but get an error:

from transformers import pipeline
summarizer = pipeline('summarization')
summary = summarizer(12345)

What is the likely cause of the error?

medium

A. The pipeline name 'summarization' is incorrect

B. Input to summarizer must be a string, not an integer

C. Missing model download before using pipeline

D. The summarizer requires a list of strings, not a single string

Solution

1. Step 1: Check input type for summarizer

   The summarizer expects a string or list of strings as input, not an integer.

2. Step 2: Identify error cause

   Passing an integer causes a type error because the model cannot process non-text input.

3. Final Answer:

   Input to summarizer must be a string, not an integer -> Option B

4. Quick Check:

   Summarizer input = string [OK]

Hint: Always pass text strings to summarizer, not numbers [OK]

Common Mistakes:

• Passing numbers or other non-string types
• Assuming pipeline name is wrong without checking
• Thinking model must be downloaded manually

5. You want to build an abstractive summarization system that handles very long documents (over 10,000 words). Which approach is best to handle this challenge effectively?

hard

A. Use extractive summarization only, ignoring abstractive methods

B. Feed the entire document directly into a standard transformer summarization model

C. Split the document into smaller chunks, summarize each, then combine summaries

D. Train a model from scratch on short documents only

Solution

1. Step 1: Understand model input limits

   Standard transformer models have input length limits (usually a few hundred tokens), so very long texts cannot be processed directly.

2. Step 2: Choose a practical approach

   Splitting long documents into smaller parts, summarizing each, then combining results is a common and effective method.

3. Final Answer:

   Split the document into smaller chunks, summarize each, then combine summaries -> Option C

4. Quick Check:

   Chunking long text enables summarization beyond model limits [OK]

Hint: Chunk long texts before summarizing to avoid input limits [OK]

Common Mistakes:

• Trying to input entire long text at once
• Ignoring abstractive summarization benefits
• Training only on short documents without chunking