BLEU score evaluation in NLP - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding BLEU score basics

What does the BLEU score primarily measure in machine translation?

A The speed at which the translation model produces output

B The grammatical correctness of the translated sentence

C The similarity between a machine-generated translation and one or more reference translations based on overlapping n-grams

D The semantic meaning similarity using word embeddings

💡 Hint

Think about what BLEU compares between the candidate and reference sentences.

BLEU score measures how many n-grams in the candidate translation appear in the reference translations, focusing on overlap rather than grammar or meaning.

❓ Predict Output

intermediate

BLEU score output for simple sentences

What is the BLEU score output of the following Python code?

NLP

from nltk.translate.bleu_score import sentence_bleu
reference = [['the', 'cat', 'is', 'on', 'the', 'mat']]
candidate = ['the', 'cat', 'is', 'on', 'the', 'mat']
score = sentence_bleu(reference, candidate)
print(round(score, 2))

A 1.0

B 0.75

C 0.5

D 0.0

💡 Hint

Consider what happens when the candidate exactly matches the reference.

When the candidate sentence exactly matches the reference, BLEU score is 1.0, meaning perfect overlap.

❓ Model Choice

advanced

Choosing a model for BLEU score evaluation

You want to evaluate a machine translation model's output using BLEU score. Which model output is best suited for BLEU evaluation?

A Probability distributions over vocabulary for each word

B Raw text strings without tokenization

C Vector embeddings of sentences

D A list of tokenized sentences generated by the model

💡 Hint

BLEU score compares n-grams, so think about the input format it requires.

BLEU score requires tokenized sentences to compare n-gram overlaps, so a list of tokenized sentences is needed.

❓ Metrics

advanced

Interpreting BLEU score values

Which statement about BLEU scores is correct?

A A BLEU score of 0.8 means the candidate translation is perfect

B BLEU scores are between 0 and 1, where higher is better

C BLEU scores closer to 0 indicate better translation quality

D BLEU scores measure recall of n-grams only

💡 Hint

Think about the range and meaning of BLEU scores.

BLEU scores range from 0 to 1, with higher scores indicating better n-gram overlap and thus better translation quality.

🔧 Debug

expert

Debugging BLEU score calculation code

What error does the following code raise when calculating BLEU score?

from nltk.translate.bleu_score import sentence_bleu
reference = ['the', 'cat', 'sat']
candidate = ['the', 'cat', 'sat']
score = sentence_bleu(reference, candidate)
print(score)

A TypeError: list of references should be a list of lists

B SyntaxError: invalid syntax

C ValueError: candidate sentence empty

D No error, prints 1.0

💡 Hint

Check the expected input format for references in sentence_bleu.

sentence_bleu expects the reference to be a list of reference sentences, each itself a list of tokens. Passing a single list causes a TypeError.