ROUGE evaluation metrics in NLP - Model Metrics & Evaluation

Metrics & Evaluation - ROUGE evaluation metrics

Which metric matters for ROUGE and WHY

ROUGE stands for Recall-Oriented Understudy for Gisting Evaluation. It measures how well a computer summary matches a human summary by counting overlapping words or phrases. The main ROUGE metrics are ROUGE-N (overlapping n-grams), ROUGE-L (longest common subsequence), and ROUGE-S (skip-bigrams). ROUGE focuses on recall because it checks how much of the human summary is captured by the machine summary. This helps us know if the important parts are included.

Confusion matrix or equivalent visualization

ROUGE does not use a confusion matrix like classification. Instead, it counts overlapping units between summaries.

Human summary: "The cat sat on the mat"
Machine summary: "The cat is on the mat"

ROUGE-1 (unigrams) overlap: "The", "cat", "on", "the", "mat" = 5
Total human unigrams: 6

ROUGE-1 Recall = Overlap / Human unigrams = 5 / 6 ≈ 0.83
ROUGE-1 Precision = Overlap / Machine unigrams = 5 / 5 = 1.0
ROUGE-1 F1 = 2 * (Precision * Recall) / (Precision + Recall) ≈ 0.90
    

Precision vs Recall tradeoff with examples

ROUGE recall measures how much of the human summary is covered by the machine summary. High recall means the machine summary includes most important info.

ROUGE precision measures how much of the machine summary is relevant to the human summary. High precision means the machine summary is focused and not adding unrelated info.

Example: If a machine summary is very long and repeats many words, recall may be high but precision low. If it is very short, precision may be high but recall low.

For summarization, recall is often more important to ensure key info is not missed, but precision helps keep summaries concise.

What good vs bad ROUGE values look like

Good ROUGE scores are closer to 1.0, meaning strong overlap with human summary.

• ROUGE-1 F1 above 0.5 is decent for many tasks.
• ROUGE-L above 0.4 shows good sequence matching.
• Scores below 0.3 usually mean poor summary quality.

However, very high ROUGE (near 1.0) may mean the machine summary is copying the human summary exactly, which is not always desired.

Common pitfalls with ROUGE metrics

• Overfitting: Models may memorize training summaries, inflating ROUGE scores but not generalizing.
• Ignoring meaning: ROUGE counts words but does not understand meaning, so paraphrased good summaries may score low.
• Length bias: Longer summaries tend to have higher recall but lower precision.
• Data leakage: Using test summaries in training can falsely boost ROUGE.
• Single reference: Using only one human summary limits ROUGE's reliability; multiple references improve it.

Self-check question

Your summarization model has ROUGE-1 recall of 0.95 but precision of 0.3. Is it good for production? Why or why not?

Answer: This means the model includes almost all important words (high recall) but also adds many unrelated words (low precision). The summary may be too long or noisy. It is not ideal for production because users want concise, relevant summaries. You should improve precision while keeping recall high.

Key Result

ROUGE metrics measure overlap between machine and human summaries, focusing on recall to ensure key info is captured.

Practice

1. What does the ROUGE metric primarily measure in natural language processing?

easy

A. The sentiment of the generated text

B. The speed of text generation

C. The overlap between generated text and reference text

D. The grammatical correctness of text

Solution

1. Step 1: Understand ROUGE's purpose

   ROUGE is designed to compare generated text with a reference to check similarity.

2. Step 2: Identify what ROUGE measures

   It measures how much the generated text overlaps with the reference text in terms of words or sequences.

3. Final Answer:

   The overlap between generated text and reference text -> Option C

4. Quick Check:

   ROUGE = overlap measure [OK]

Hint: ROUGE checks text similarity, not speed or grammar [OK]

Common Mistakes:

• Confusing ROUGE with grammar checkers
• Thinking ROUGE measures sentiment
• Assuming ROUGE measures generation speed

2. Which of the following is the correct way to calculate ROUGE-1 recall?

easy

A. Number of overlapping unigrams divided by total unigrams in generated text

B. Number of overlapping unigrams divided by total unigrams in reference text

C. Number of overlapping bigrams divided by total bigrams in generated text

D. Number of overlapping bigrams divided by total bigrams in reference text

Solution

1. Step 1: Recall definition in ROUGE-1

   Recall measures how much of the reference text's unigrams appear in the generated text.

2. Step 2: Apply recall formula

   Recall = overlapping unigrams / total unigrams in reference text.

3. Final Answer:

   Number of overlapping unigrams divided by total unigrams in reference text -> Option B

4. Quick Check:

   Recall = overlap/reference [OK]

Hint: Recall divides by reference text count, not generated [OK]

Common Mistakes:

• Mixing up recall with precision
• Using generated text count in recall
• Confusing unigrams with bigrams

3. Given the reference text: "the cat sat on the mat" and generated text: "the cat lay on rug", what is the ROUGE-1 precision score?

medium

A. 0.6

B. 0.5

C. 0.4

D. 0.7

Solution

1. Step 1: Identify overlapping unigrams

   Common words: "the", "cat", "on". Overlapping unigrams = 3: "the", "cat", "on".

2. Step 2: Calculate precision

   Precision = overlapping unigrams / total unigrams in generated text = 3 / 5 = 0.6.

3. Final Answer:

   0.6 -> Option A

4. Quick Check:

   Precision = 3/5 = 0.6 [OK]

Hint: Precision = overlap / generated text words count [OK]

Common Mistakes:

• Counting duplicates incorrectly
• Using reference text length for precision
• Ignoring repeated words in calculation

4. You wrote code to compute ROUGE-L but the scores are always zero. Which of these is the most likely bug?

medium

A. Calculating precision instead of recall

B. Using ROUGE-1 instead of ROUGE-L

C. Using lowercase text for both inputs

D. Not tokenizing the texts before comparison

Solution

1. Step 1: Understand ROUGE-L calculation

   ROUGE-L depends on longest common subsequence of tokens, so tokenization is essential.

2. Step 2: Identify impact of missing tokenization

   If texts are not tokenized, comparison fails, resulting in zero scores.

3. Final Answer:

   Not tokenizing the texts before comparison -> Option D

4. Quick Check:

   Tokenization missing = zero ROUGE-L [OK]

Hint: Always tokenize texts before ROUGE-L calculation [OK]

Common Mistakes:

• Skipping tokenization step
• Confusing ROUGE types
• Ignoring case normalization impact

5. You want to evaluate a summarization model using ROUGE scores. The model produces very short summaries missing many reference words. Which ROUGE metric and score should you focus on to best understand coverage?

hard

A. ROUGE-1 recall, because it shows how many reference words are captured

B. ROUGE-1 precision, because it shows how many generated words are correct

C. ROUGE-L F1, because it balances precision and recall on longest sequences

D. ROUGE-2 precision, because it focuses on bigram accuracy

Solution

1. Step 1: Understand the problem context

   The summaries are short and miss many reference words, so coverage of reference is low.

2. Step 2: Choose metric that measures coverage

   Recall measures how much of the reference text is captured by the summary, so ROUGE-1 recall is best.

3. Final Answer:

   ROUGE-1 recall, because it shows how many reference words are captured -> Option A

4. Quick Check:

   Coverage = recall = ROUGE-1 recall [OK]

Hint: Use ROUGE-1 recall to check coverage of reference words [OK]

Common Mistakes:

• Focusing on precision instead of recall
• Using ROUGE-2 which is stricter
• Ignoring recall's role in coverage