Evaluating generated text (BLEU, ROUGE) in NLP - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the BLEU score output of this code?

Given the reference and candidate sentences below, what is the BLEU score computed by the code?

NLP

from nltk.translate.bleu_score import sentence_bleu
reference = [['the', 'cat', 'is', 'on', 'the', 'mat']]
candidate = ['the', 'cat', 'sat', 'on', 'the', 'mat']
score = sentence_bleu(reference, candidate)
print(round(score, 3))

A 0.840

B 0.759

C 0.667

D 0.500

💡 Hint

BLEU score compares n-gram overlap; small differences in words reduce the score.

The BLEU score here is about 0.759 because the candidate differs by one word ('sat' vs 'is'), reducing the n-gram matches but still having high overlap.

🧠 Conceptual

intermediate

Which metric is best for evaluating summary length and content overlap?

You want to evaluate how well a generated summary matches a reference summary, focusing on content overlap and length. Which metric is most suitable?

A ROUGE

B BLEU

C Accuracy

D Mean Squared Error

💡 Hint

This metric is designed for comparing summaries and measures overlap of sequences.

ROUGE is designed to evaluate summaries by measuring overlap of n-grams, sequences, and longest common subsequences, making it better for summary evaluation than BLEU.

❓ Metrics

advanced

What ROUGE metric measures longest common subsequence overlap?

Among ROUGE-1, ROUGE-2, and ROUGE-L, which one measures the longest common subsequence between generated and reference texts?

A ROUGE-L

B ROUGE-1

C ROUGE-W

D ROUGE-2

💡 Hint

This metric captures sentence-level structure by longest matching sequence.

ROUGE-L measures the longest common subsequence (LCS) between candidate and reference, capturing sentence-level structure beyond simple n-gram overlap.

🔧 Debug

advanced

Why does this BLEU score code raise an error?

What error does this code raise and why? from nltk.translate.bleu_score import sentence_bleu reference = ['the', 'cat', 'is', 'on', 'the', 'mat'] candidate = ['the', 'cat', 'sat', 'on', 'the', 'mat'] score = sentence_bleu(reference, candidate) print(score)

A ValueError: empty reference list

B SyntaxError: invalid syntax

C No error, prints BLEU score

D TypeError: expected list of references, got list

💡 Hint

Check the expected input type for references in sentence_bleu.

sentence_bleu expects a list of reference sentences (each a list of tokens), but here a single list is passed, causing a TypeError.

❓ Model Choice

expert

Which evaluation metric is best for machine translation quality?

You want to evaluate machine translation output quality automatically. Which metric is most widely accepted and specifically designed for this task?

A F1-score

B ROUGE

C BLEU

D Perplexity

💡 Hint

This metric compares n-gram overlap between candidate and reference translations.

BLEU is the standard metric for machine translation evaluation, measuring n-gram precision with brevity penalty to assess translation quality.