RAG evaluation metrics in Prompt Engineering / GenAI - Cheat Sheet & Quick Revision

Recall & Review

beginner

What does RAG stand for in machine learning?

RAG stands for Retrieval-Augmented Generation, a method combining retrieval of documents with text generation.

beginner

Why do we need evaluation metrics for RAG models?

Evaluation metrics help us measure how well the RAG model retrieves relevant information and generates accurate, useful answers.

intermediate

Name two common metrics used to evaluate the retrieval part of RAG.

Recall@k and Precision@k are common metrics to check if the model retrieves relevant documents within the top k results.

intermediate

What metric measures the quality of generated text in RAG?

BLEU, ROUGE, and METEOR are popular metrics to compare generated text with reference answers to measure quality.

beginner

How does Exact Match (EM) metric work in RAG evaluation?

Exact Match checks if the generated answer exactly matches the correct answer, giving a simple yes/no score.

Which metric checks if the correct document is among the top retrieved results in RAG?

A Exact Match

B BLEU

C Recall@k

D METEOR

Recall@k measures if the relevant document is found within the top k retrieved documents.

What does BLEU score evaluate in RAG models?

A Generated text quality

B Retrieval accuracy

C Training speed

D Model size

BLEU measures how closely the generated text matches reference answers.

Which metric gives a simple yes/no score if the generated answer matches exactly the correct answer?

A ROUGE

B Exact Match

C Recall@k

D Precision@k

Exact Match checks for exact equality between generated and correct answers.

Precision@k in RAG evaluation measures:

A Training loss

B How many relevant documents are missed

C Quality of generated text

D How many retrieved documents are relevant within top k

Precision@k measures the proportion of relevant documents among the top k retrieved.

Which metric is NOT typically used to evaluate the generation part of RAG?

A Recall@k

B ROUGE

C METEOR

D BLEU

Recall@k is used for retrieval evaluation, not generation.

Explain the difference between retrieval and generation evaluation metrics in RAG.

Think about the two main parts of RAG: finding info and writing answers.

Describe how Exact Match metric works and when it is useful in RAG evaluation.

Consider simple yes/no correctness.