ROUGE evaluation metrics in NLP - Cheat Sheet & Quick Revision

Recall & Review

beginner

What does ROUGE stand for in NLP evaluation?

ROUGE stands for Recall-Oriented Understudy for Gisting Evaluation. It is a set of metrics used to evaluate automatic summarization and machine translation by comparing system-generated text to reference texts.

beginner

What is the main purpose of ROUGE metrics?

ROUGE metrics measure how much overlap there is between the words or phrases in a machine-generated summary and a human-written reference summary. It helps check the quality of summaries by focusing on recall, precision, and F1 score.

intermediate

Explain ROUGE-N metric.

ROUGE-N measures the overlap of n-grams (continuous sequences of n words) between the candidate summary and the reference summary. For example, ROUGE-1 looks at single words, ROUGE-2 looks at pairs of words.

intermediate

What is ROUGE-L and why is it useful?

ROUGE-L measures the longest common subsequence (LCS) between the candidate and reference summaries. It captures sentence-level structure similarity and is useful because it does not require consecutive matches but keeps word order.

beginner

How are precision, recall, and F1 score used in ROUGE metrics?

Precision measures how many words in the candidate summary appear in the reference. Recall measures how many words in the reference appear in the candidate. F1 score is the balance between precision and recall, giving a single score to evaluate quality.

What does ROUGE primarily measure in text summaries?

A Overlap of words or phrases between candidate and reference summaries

B The grammatical correctness of the summary

C The length of the summary

D The sentiment of the summary

ROUGE measures the overlap of words or phrases to evaluate how well the candidate summary matches the reference.

Which ROUGE metric uses longest common subsequence (LCS)?

A ROUGE-L

B ROUGE-2

C ROUGE-1

D ROUGE-S

ROUGE-L uses the longest common subsequence to capture sentence-level similarity.

ROUGE-2 evaluates overlap of which type of n-grams?

A Single words

B Sentences

C Triplets of words

D Pairs of words

ROUGE-2 measures overlap of 2-word sequences, or pairs of words.

In ROUGE metrics, what does recall measure?

A How many words in candidate appear in reference

B The length of the candidate summary

C How many words in reference appear in candidate

D The number of sentences in the reference

Recall measures how many words from the reference summary appear in the candidate summary.

Why is F1 score important in ROUGE evaluation?

A It measures only precision

B It balances precision and recall into one score

C It measures summary length

D It measures only recall

F1 score balances precision and recall, providing a single metric to evaluate summary quality.

Describe what ROUGE evaluation metrics are and why they are used in NLP.

Think about how we check if a summary made by a computer matches a human summary.

Explain the difference between ROUGE-N and ROUGE-L metrics.

Consider how sequences of words are matched differently.