ROUGE evaluation metrics in NLP - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to import the ROUGE metric from the datasets library.

NLP

from datasets import [1]

Drag options to blanks, or click blank then click option'

Aload_dataset

Bload_metric

Cload_rouge

Dload_eval

Attempts:

3 left

💡 Hint

Common Mistakes

Using load_dataset instead of load_metric

Trying to import a non-existing function like load_rouge

The load_metric function is used to load evaluation metrics like ROUGE.

2fill in blank

medium

Complete the code to compute ROUGE scores given predictions and references.

NLP

rouge = load_metric('rouge')
results = rouge.[1](predictions=preds, references=refs)

Drag options to blanks, or click blank then click option'

Ascore

Bevaluate

Ccompute

Dcalculate

Attempts:

3 left

💡 Hint

Common Mistakes

Using evaluate or score instead of compute

Passing wrong argument names

The compute method calculates the ROUGE scores from predictions and references.

3fill in blank

hard

Fix the error in the code to correctly prepare the predictions for ROUGE evaluation by removing extra spaces.

NLP

clean_preds = [pred.strip() for pred in [1]]

Drag options to blanks, or click blank then click option'

Apredictions

Brouge

Cresults

Drefs

Attempts:

3 left

💡 Hint

Common Mistakes

Cleaning references instead of predictions

Trying to clean the results or rouge object

We clean the predictions by stripping spaces before evaluation.

4fill in blank

hard

Fill both blanks to compute ROUGE-L F1 score from the results dictionary.

NLP

rouge_l_f1 = results['rougeL'].[1].[2]

Drag options to blanks, or click blank then click option'

Amid

Bfmeasure

Cprecision

Drecall

Attempts:

3 left

💡 Hint

Common Mistakes

Using precision or recall instead of fmeasure

Using mean instead of mid

The mid key holds median scores, and fmeasure is the F1 score for ROUGE-L.

5fill in blank

hard

Fill all three blanks to create a dictionary of ROUGE-1, ROUGE-2, and ROUGE-L F1 scores.

NLP

scores = {
    'rouge1': results['rouge1'].[1].[2],
    'rouge2': results['rouge2'].[1].[3],
    'rougeL': results['rougeL'].[1].[2]
}

Drag options to blanks, or click blank then click option'

Amid

Bfmeasure

Crecall

Dprecision

Attempts:

3 left

💡 Hint

Common Mistakes

Mixing precision and recall incorrectly

Using mean instead of mid

We use mid for median, fmeasure for F1 score, and fmeasure for ROUGE-2 F1 score as well.

Practice

1. What does the ROUGE metric primarily measure in natural language processing?

easy

A. The sentiment of the generated text

B. The speed of text generation

C. The overlap between generated text and reference text

D. The grammatical correctness of text

Solution

1. Step 1: Understand ROUGE's purpose

   ROUGE is designed to compare generated text with a reference to check similarity.

2. Step 2: Identify what ROUGE measures

   It measures how much the generated text overlaps with the reference text in terms of words or sequences.

3. Final Answer:

   The overlap between generated text and reference text -> Option C

4. Quick Check:

   ROUGE = overlap measure [OK]

Hint: ROUGE checks text similarity, not speed or grammar [OK]

Common Mistakes:

• Confusing ROUGE with grammar checkers
• Thinking ROUGE measures sentiment
• Assuming ROUGE measures generation speed

2. Which of the following is the correct way to calculate ROUGE-1 recall?

easy

A. Number of overlapping unigrams divided by total unigrams in generated text

B. Number of overlapping unigrams divided by total unigrams in reference text

C. Number of overlapping bigrams divided by total bigrams in generated text

D. Number of overlapping bigrams divided by total bigrams in reference text

Solution

1. Step 1: Recall definition in ROUGE-1

   Recall measures how much of the reference text's unigrams appear in the generated text.

2. Step 2: Apply recall formula

   Recall = overlapping unigrams / total unigrams in reference text.

3. Final Answer:

   Number of overlapping unigrams divided by total unigrams in reference text -> Option B

4. Quick Check:

   Recall = overlap/reference [OK]

Hint: Recall divides by reference text count, not generated [OK]

Common Mistakes:

• Mixing up recall with precision
• Using generated text count in recall
• Confusing unigrams with bigrams

3. Given the reference text: "the cat sat on the mat" and generated text: "the cat lay on rug", what is the ROUGE-1 precision score?

medium

A. 0.6

B. 0.5

C. 0.4

D. 0.7

Solution

1. Step 1: Identify overlapping unigrams

   Common words: "the", "cat", "on". Overlapping unigrams = 3: "the", "cat", "on".

2. Step 2: Calculate precision

   Precision = overlapping unigrams / total unigrams in generated text = 3 / 5 = 0.6.

3. Final Answer:

   0.6 -> Option A

4. Quick Check:

   Precision = 3/5 = 0.6 [OK]

Hint: Precision = overlap / generated text words count [OK]

Common Mistakes:

• Counting duplicates incorrectly
• Using reference text length for precision
• Ignoring repeated words in calculation

4. You wrote code to compute ROUGE-L but the scores are always zero. Which of these is the most likely bug?

medium

A. Calculating precision instead of recall

B. Using ROUGE-1 instead of ROUGE-L

C. Using lowercase text for both inputs

D. Not tokenizing the texts before comparison

Solution

1. Step 1: Understand ROUGE-L calculation

   ROUGE-L depends on longest common subsequence of tokens, so tokenization is essential.

2. Step 2: Identify impact of missing tokenization

   If texts are not tokenized, comparison fails, resulting in zero scores.

3. Final Answer:

   Not tokenizing the texts before comparison -> Option D

4. Quick Check:

   Tokenization missing = zero ROUGE-L [OK]

Hint: Always tokenize texts before ROUGE-L calculation [OK]

Common Mistakes:

• Skipping tokenization step
• Confusing ROUGE types
• Ignoring case normalization impact

5. You want to evaluate a summarization model using ROUGE scores. The model produces very short summaries missing many reference words. Which ROUGE metric and score should you focus on to best understand coverage?

hard

A. ROUGE-1 recall, because it shows how many reference words are captured

B. ROUGE-1 precision, because it shows how many generated words are correct

C. ROUGE-L F1, because it balances precision and recall on longest sequences

D. ROUGE-2 precision, because it focuses on bigram accuracy

Solution

1. Step 1: Understand the problem context

   The summaries are short and miss many reference words, so coverage of reference is low.

2. Step 2: Choose metric that measures coverage

   Recall measures how much of the reference text is captured by the summary, so ROUGE-1 recall is best.

3. Final Answer:

   ROUGE-1 recall, because it shows how many reference words are captured -> Option A

4. Quick Check:

   Coverage = recall = ROUGE-1 recall [OK]

Hint: Use ROUGE-1 recall to check coverage of reference words [OK]

Common Mistakes:

• Focusing on precision instead of recall
• Using ROUGE-2 which is stricter
• Ignoring recall's role in coverage