RAG evaluation metrics in Prompt Engineering / GenAI - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding RAG's Retrieval and Generation Metrics

In Retrieval-Augmented Generation (RAG) models, which metric best measures how well the retrieval component finds relevant documents?

A BLEU score measuring text similarity between generated and reference answers

B Perplexity measuring the language model's uncertainty in generating text

C Recall@k measuring the fraction of relevant documents retrieved in the top k results

D F1 score measuring token overlap between generated and reference answers

💡 Hint

Think about which metric evaluates the retrieval step's ability to find relevant documents.

Recall@k is used to evaluate retrieval quality by checking if relevant documents appear in the top k retrieved results. BLEU and F1 focus on generation quality, while perplexity measures language model uncertainty.

❓ Metrics

intermediate

Evaluating Generated Answers in RAG

Which metric is most appropriate to evaluate the quality of the generated answers in a RAG system compared to reference answers?

A ROUGE-L measuring longest common subsequence overlap

B Top-1 accuracy of retrieved documents

C Mean Squared Error between embeddings

D Recall@k for retrieval accuracy

💡 Hint

Focus on metrics that compare generated text to reference text.

ROUGE-L measures the quality of generated text by comparing longest common subsequence overlap with reference answers, making it suitable for generation evaluation. Recall@k and Top-1 accuracy relate to retrieval, and MSE on embeddings is less common for direct text quality.

❓ Predict Output

advanced

Output of Recall@k Calculation Code

What is the output of this Python code that calculates Recall@3 for a RAG retrieval step?

Prompt Engineering / GenAI

relevant_docs = {101, 102, 103}
retrieved_docs = [104, 102, 105, 101, 106]
recall_at_3 = len(set(retrieved_docs[:3]) & relevant_docs) / len(relevant_docs)
print(round(recall_at_3, 2))

A 1.0

B 0.67

C 0.0

D 0.33

💡 Hint

Check which relevant documents appear in the first 3 retrieved docs.

The first 3 retrieved docs are [104, 102, 105]. Only 102 is relevant. So recall@3 = 1/3 ≈ 0.33.

❓ Model Choice

advanced

Choosing a Metric for End-to-End RAG Evaluation

You want to evaluate a RAG model end-to-end, considering both retrieval and generation quality. Which combined metric approach is best?

A Use Recall@k for retrieval and ROUGE-L for generation, then average the two scores

B Use only BLEU score on generated answers ignoring retrieval

C Use perplexity of the generation model only

D Use accuracy of retrieved documents only

💡 Hint

Think about combining retrieval and generation metrics for full evaluation.

Evaluating RAG end-to-end requires assessing both retrieval (Recall@k) and generation (ROUGE-L). Averaging these gives a balanced view. Using only one ignores part of the pipeline.

🔧 Debug

expert

Debugging Incorrect Recall@k Calculation

Given this code snippet for Recall@5, what is the bug causing incorrect recall calculation?

relevant = [10, 20, 30]
retrieved = [20, 10, 40, 50, 60]
recall = len(set(retrieved[:5]) & set(relevant)) / len(retrieved[:5])
print(round(recall, 2))

A Recall calculation is correct, no bug

B Recall denominator should be number of relevant documents, not retrieved documents

C Recall numerator should count all retrieved documents, not intersection

D Recall should use union of sets, not intersection

💡 Hint

Recall measures fraction of relevant items retrieved, so denominator matters.

Recall is defined as (relevant retrieved) / (total relevant). The code incorrectly divides by number of retrieved documents, causing wrong recall value.