Summarization in Prompt Engineering / GenAI - Model Metrics & Evaluation

Metrics & Evaluation - Summarization

Which metric matters for Summarization and WHY

For summarization, we want to check how well the summary captures the important parts of the original text. The main metrics are ROUGE scores, especially ROUGE-1, ROUGE-2, and ROUGE-L. These compare the overlap of words and phrases between the generated summary and a human-written summary.

ROUGE-1 measures overlap of single words, ROUGE-2 looks at pairs of words, and ROUGE-L checks the longest matching sequence. Higher ROUGE scores mean the summary is closer to the reference, so it is better.

Confusion matrix or equivalent visualization

Summarization is not a classification task, so it does not use a confusion matrix. Instead, we use overlap-based metrics like ROUGE.

Example ROUGE-1 calculation:

Reference summary: "The cat sat on the mat."
Generated summary: "Cat sat on mat."

Overlap words: cat, sat, on, mat
Total words in reference: 6
ROUGE-1 recall = 4 / 6 = 0.67
    

Precision vs Recall tradeoff with examples

ROUGE metrics can be calculated as precision, recall, or F1 score.

• Precision: How many words in the generated summary appear in the reference? High precision means the summary is mostly relevant.
• Recall: How many words in the reference summary appear in the generated summary? High recall means the summary covers most important points.

For example, a very short summary might have high precision (few words, all relevant) but low recall (misses many points). A very long summary might have high recall but low precision (includes many irrelevant words).

We want a balance, often measured by the F1 score, to get a summary that is both relevant and covers key points.

What "good" vs "bad" metric values look like for Summarization

Good ROUGE scores depend on the dataset and task, but generally:

• ROUGE-1 F1 > 0.4 is considered decent for many summarization tasks.
• ROUGE-2 F1 > 0.2 shows good phrase matching.
• ROUGE-L F1 > 0.35 means good sequence matching.

Bad scores are close to zero, meaning the summary barely matches the reference. Scores near 0.1 or less show poor quality summaries.

Common pitfalls in Summarization metrics

• Overfitting to ROUGE: Models might learn to copy phrases to boost ROUGE but produce less natural summaries.
• Ignoring meaning: ROUGE measures word overlap, not if the summary truly captures meaning.
• Reference bias: Using only one reference summary can limit evaluation fairness.
• Length bias: Very short or very long summaries can skew precision or recall.

Self-check question

Your summarization model has a ROUGE-1 recall of 0.85 but ROUGE-1 precision of 0.3. Is this good? Why or why not?

Answer: This means the summary covers most important words (high recall) but includes many extra words not in the reference (low precision). The summary might be too long or noisy. You want to improve precision to make the summary more concise and relevant.

Key Result

ROUGE scores (especially ROUGE-1, ROUGE-2, ROUGE-L) are key to measure how well a summary matches the reference in content and phrasing.

Practice

1. What is the main purpose of text summarization in AI?

easy

A. To count the number of words in a text

B. To translate text into another language

C. To generate new text from scratch

D. To make long text shorter and easier to understand

Solution

1. Step 1: Understand the goal of summarization

   Summarization aims to reduce the length of text while keeping the main ideas clear.

2. Step 2: Compare options with the goal

   Only To make long text shorter and easier to understand describes making text shorter and easier to understand, which matches summarization.

3. Final Answer:

   To make long text shorter and easier to understand -> Option D

4. Quick Check:

   Summarization = shorten text [OK]

Hint: Summarization shortens text for quick understanding [OK]

Common Mistakes:

• Confusing summarization with translation
• Thinking summarization creates new text
• Mixing summarization with word counting

2. Which of the following is the correct way to call a summarization model in Python using a fictional API?

easy

A. summary = model.summarize(text)

B. summary = model.translate(text)

C. summary = model.generate(text)

D. summary = model.count_words(text)

Solution

1. Step 1: Identify the function for summarization

   The function to get a summary should be named something like 'summarize' to match the task.

2. Step 2: Match function names to tasks

   Only 'model.summarize(text)' fits the summarization task; others do translation, generation, or counting.

3. Final Answer:

   summary = model.summarize(text) -> Option A

4. Quick Check:

   Summarize function call = summary = model.summarize(text) [OK]

Hint: Look for 'summarize' function for summarization calls [OK]

Common Mistakes:

• Using translate() instead of summarize()
• Using generate() which creates new text
• Using count_words() which is unrelated

3. Given the code below, what will be the output?

text = "AI helps us by making complex tasks easier."
summary = model.summarize(text)
print(summary)

Assuming the model works correctly, what is the likely output?

medium

A. "AI simplifies complex tasks."

B. "AI translates text."

C. "AI helps us by making complex tasks easier."

D. "AI counts words in text."

Solution

1. Step 1: Understand summarization output

   The summary should be a shorter version of the original text keeping the main idea.

2. Step 2: Compare options to expected summary

   "AI simplifies complex tasks." shortens the sentence while keeping meaning; "AI helps us by making complex tasks easier." is original text, others unrelated.

3. Final Answer:

   "AI simplifies complex tasks." -> Option A

4. Quick Check:

   Summary shortens text = "AI simplifies complex tasks." [OK]

Hint: Summary is shorter but keeps main idea [OK]

Common Mistakes:

• Thinking summary is the same as original text
• Confusing summarization with translation
• Expecting unrelated outputs like word count

4. The following code throws an error. What is the likely cause?

text = "Summarize this text."
summary = model.summarize_text(text)
print(summary)

medium

A. The variable 'text' is not defined

B. The method name 'summarize_text' is incorrect

C. The print statement is missing parentheses

D. The model object is not created

Solution

1. Step 1: Check method name correctness

   The correct method to summarize is likely 'summarize', not 'summarize_text'.

2. Step 2: Verify other code parts

   The variable 'text' is defined, print has parentheses, and model object assumed created.

3. Final Answer:

   The method name 'summarize_text' is incorrect -> Option B

4. Quick Check:

   Method name must be correct = The method name 'summarize_text' is incorrect [OK]

Hint: Check method names carefully for typos [OK]

Common Mistakes:

• Assuming variable 'text' is undefined
• Forgetting print needs parentheses
• Ignoring if model object exists

5. You want to summarize a long article but keep important keywords intact. Which approach is best?

hard

A. Use translation model to convert text language

B. Use generative summarization to rewrite text freely

C. Use extractive summarization to select key sentences

D. Use word count to find important words

Solution

1. Step 1: Understand extractive vs generative summarization

   Extractive picks actual sentences from text, preserving keywords; generative rewrites freely.

2. Step 2: Choose method to keep keywords intact

   Extractive summarization keeps original sentences and keywords, so it fits the need best.

3. Final Answer:

   Use extractive summarization to select key sentences -> Option C

4. Quick Check:

   Keep keywords = extractive summarization [OK]

Hint: Extractive keeps original words; generative rewrites [OK]

Common Mistakes:

• Confusing generative with extractive summarization
• Using translation instead of summarization
• Relying on word count alone for keywords