Evaluating generated text (BLEU, ROUGE) in NLP - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to import the BLEU score function from nltk.

NLP

from nltk.translate.bleu_score import [1]

Drag options to blanks, or click blank then click option'

Asentence_bleu

Bbleu_score

Ccorpus_bleu

Dbleu

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'bleu_score' which is the module, not the function.

Using 'corpus_bleu' which is for multiple sentences.

The function sentence_bleu is used to calculate BLEU score for a single sentence.

2fill in blank

medium

Complete the code to calculate the BLEU score for a candidate sentence.

NLP

score = sentence_bleu([reference], [1])

Drag options to blanks, or click blank then click option'

Acandidate_tokens

Breferences

Ccandidate_sentence

Dcandidate

Attempts:

3 left

💡 Hint

Common Mistakes

Passing the reference list again instead of candidate.

Using a variable name that does not exist.

The candidate sentence tokens should be passed as the second argument to sentence_bleu.

3fill in blank

hard

Fix the error in the ROUGE score calculation by completing the missing import.

NLP

from rouge_score import [1]

Drag options to blanks, or click blank then click option'

ARougeScorer

Brouge_scorer

Cscore_rouge

Drouge

Attempts:

3 left

💡 Hint

Common Mistakes

Using lowercase 'rouge_scorer' which is a module, not the class.

Using 'score_rouge' which does not exist.

The class RougeScorer is imported from the rouge_score package to calculate ROUGE scores.

4fill in blank

hard

Fill both blanks to create a ROUGE scorer and calculate scores for two texts.

NLP

scorer = [1](['rouge1', 'rougeL'])
scores = scorer.score([2], generated)

Drag options to blanks, or click blank then click option'

ARougeScorer

Breference_text

Creference

Dscore

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'reference_text' which is not defined.

Using 'score' which is a variable, not a function/class.

RougeScorer is used to create the scorer object. The reference text variable is named reference.

5fill in blank

hard

Fill all three blanks to compute BLEU and ROUGE scores and print them.

NLP

bleu = sentence_bleu([reference], [1])
scorer = [2](['rouge1', 'rougeL'])
rouge_scores = scorer.score(reference, [3])
print(f"BLEU: {bleu:.2f}")
print(f"ROUGE-1 F1: {rouge_scores['rouge1'].fmeasure:.2f}")

Drag options to blanks, or click blank then click option'

Acandidate

BRougeScorer

Cgenerated

Dreference

Attempts:

3 left

💡 Hint

Common Mistakes

Mixing up reference and generated variables.

Using lowercase 'rougescorer' instead of 'RougeScorer'.

The BLEU score uses the candidate tokens, the ROUGE scorer is created with RougeScorer, and ROUGE is scored comparing reference and generated text.

Practice

1. What is the main purpose of BLEU and ROUGE scores in evaluating generated text?

easy

A. To measure how similar the generated text is to human-written text

B. To check the spelling errors in generated text

C. To count the number of words in the generated text

D. To translate text from one language to another

Solution

1. Step 1: Understand the role of BLEU and ROUGE

   Both BLEU and ROUGE are metrics used to compare generated text with reference human text to check similarity.

2. Step 2: Identify the main purpose

   They do not check spelling, count words, or translate text but measure similarity to human text.

3. Final Answer:

   To measure how similar the generated text is to human-written text -> Option A

4. Quick Check:

   BLEU and ROUGE measure similarity [OK]

Hint: Remember: BLEU and ROUGE check similarity, not spelling or translation [OK]

Common Mistakes:

• Confusing BLEU/ROUGE with spell check
• Thinking they count words only
• Assuming they translate text

2. Which of the following is the correct way to calculate BLEU score using Python's nltk library?

easy

A. bleu_score = nltk.bleu_score([candidate], reference)

B. bleu_score = nltk.translate.bleu_score.sentence_bleu([reference], candidate)

C. bleu_score = nltk.translate.bleu_score([candidate], [reference])

D. bleu_score = nltk.score.bleu(candidate, reference)

Solution

1. Step 1: Recall the nltk BLEU function syntax

   The correct function is sentence_bleu from nltk.translate.bleu_score, which takes a list of references and a candidate sentence.

2. Step 2: Match the correct syntax

   bleu_score = nltk.translate.bleu_score.sentence_bleu([reference], candidate) uses sentence_bleu([reference], candidate), which is the correct call format.

3. Final Answer:

   bleu_score = nltk.translate.bleu_score.sentence_bleu([reference], candidate) -> Option B

4. Quick Check:

   Use sentence_bleu with list of references [OK]

Hint: Use sentence_bleu with references as a list [OK]

Common Mistakes:

• Passing candidate as first argument instead of second
• Not wrapping reference in a list
• Using wrong module or function name

3. Given the following code snippet, what will be the printed BLEU score?

from nltk.translate.bleu_score import sentence_bleu
reference = [['the', 'cat', 'is', 'on', 'the', 'mat']]
candidate = ['the', 'cat', 'sat', 'on', 'the', 'mat']
score = sentence_bleu(reference, candidate)
print(round(score, 2))

medium

A. 0.92

B. 0.75

C. 0.58

D. 0.33

Solution

1. Step 1: Understand BLEU calculation basics

   BLEU compares n-gram overlap; here, candidate differs by one word ('sat' vs 'is'), so score is high but not perfect.

2. Step 2: Run or estimate BLEU score

   Running this code yields approximately 0.916, rounded to 0.92.

3. Final Answer:

   0.92 -> Option A

4. Quick Check:

   BLEU score close to 1 means high similarity [OK]

Hint: BLEU near 1 means very similar sentences [OK]

Common Mistakes:

• Assuming exact match needed for high BLEU
• Confusing BLEU with ROUGE
• Ignoring n-gram overlap effect

4. You wrote code to compute ROUGE-L score but get an error: AttributeError: module 'rouge' has no attribute 'Rouge'. What is the likely cause?

medium

A. The input texts are empty strings

B. ROUGE-L score cannot be computed in Python

C. The 'rouge' package is not installed or imported incorrectly

D. You must use BLEU instead of ROUGE-L

Solution

1. Step 1: Analyze the error message

   The error says the module 'rouge' has no attribute 'Rouge', indicating the package or import is missing or incorrect.

2. Step 2: Understand correct usage

   You need to install the correct 'rouge' package and import Rouge class properly to use ROUGE-L.

3. Final Answer:

   The 'rouge' package is not installed or imported incorrectly -> Option C

4. Quick Check:

   AttributeError usually means missing or wrong import [OK]

Hint: Check package installation and import statements first [OK]

Common Mistakes:

• Assuming ROUGE-L can't be computed in Python
• Ignoring installation errors
• Using wrong package names

5. You have two text generation models. Model A has a BLEU score of 0.45 and ROUGE-L score of 0.60. Model B has a BLEU score of 0.55 and ROUGE-L score of 0.50. Which model should you prefer if you want better phrase matching and why?

hard

A. Model A, because lower BLEU means better phrase matching

B. Model A, because higher ROUGE-L means better phrase matching

C. Model B, because lower ROUGE-L means better phrase matching

D. Model B, because higher BLEU means better phrase matching

Solution

1. Step 1: Understand BLEU and ROUGE focus

   BLEU focuses on phrase matching; ROUGE-L focuses on longest common subsequence (word overlap).

2. Step 2: Compare scores for phrase matching

   Model B has higher BLEU (0.55) than Model A (0.45), so Model B is better for phrase matching.

3. Final Answer:

   Model B, because higher BLEU means better phrase matching -> Option D

4. Quick Check:

   Higher BLEU = better phrase matching [OK]

Hint: BLEU = phrase match; ROUGE = word overlap [OK]

Common Mistakes:

• Confusing BLEU and ROUGE meanings
• Choosing model with higher ROUGE for phrase matching
• Ignoring which metric matches the goal