Open-domain QA basics in NLP - Model Metrics & Evaluation

Metrics & Evaluation - Open-domain QA basics

Which metric matters for Open-domain QA and WHY

In open-domain question answering (QA), the main goal is to find the correct answer from a large collection of documents. The key metrics are Exact Match (EM) and F1 score. Exact Match checks if the predicted answer exactly matches the true answer. F1 score measures how much the predicted answer overlaps with the true answer in terms of words. These metrics matter because answers can be short phrases or sentences, so partial correctness is important to capture.

Confusion matrix or equivalent visualization

Open-domain QA does not use a classic confusion matrix like classification. Instead, we evaluate predictions per question:

Question 1: Predicted answer = "Paris"; True answer = "Paris" → Exact Match = 1, F1 = 1.0
Question 2: Predicted answer = "Paris, the capital of France"; True answer = "Paris" → Exact Match = 0, F1 > 0 (partial overlap)
Question 3: Predicted answer = "London"; True answer = "Paris" → Exact Match = 0, F1 = 0

We then average these scores over all questions to get overall performance.

Precision vs Recall tradeoff with concrete examples

In open-domain QA, precision and recall relate to how much of the true answer is captured and how much extra irrelevant text is included.

• High precision, low recall: The model gives a very short answer that is correct but misses some details. For example, answer "Paris" when the full answer is "Paris, the capital of France". This is precise but incomplete.
• High recall, low precision: The model gives a long answer that includes the correct answer but also extra unrelated words. For example, "Paris is the capital of France and a beautiful city". This covers the true answer but adds noise.

Balancing precision and recall is important to give answers that are both correct and concise.

What "good" vs "bad" metric values look like for Open-domain QA

• Good: Exact Match above 70% and F1 score above 80% usually means the model finds correct answers most of the time and captures most answer words.
• Bad: Exact Match below 40% and F1 below 50% means the model often misses the correct answer or gives very incomplete answers.

Common pitfalls in Open-domain QA metrics

• Ignoring partial matches: Using only Exact Match can be too strict and miss partially correct answers.
• Data leakage: If the model sees test answers during training, metrics will be unrealistically high.
• Overfitting: Very high training scores but low test scores mean the model memorizes answers instead of understanding.
• Ambiguous answers: Some questions have multiple correct answers, so metrics must allow for synonyms or paraphrases.

Self-check question

Your open-domain QA model has 60% Exact Match but 85% F1 score. Is it good? Why or why not?

Answer: This means the model gets the exact answer right less often, but captures most of the answer content via high F1. It is good overall but could improve exact matching and phrasing for better precision balance.

Key Result

Exact Match and F1 score are key metrics; Exact Match measures full correctness, F1 captures partial correctness in open-domain QA.

Practice

1. What is the main goal of open-domain question answering (QA)?

easy

A. To summarize a single document

B. To translate text from one language to another

C. To find answers to any question from a large collection of texts

D. To generate new text based on a prompt

Solution

1. Step 1: Understand the definition of open-domain QA

   Open-domain QA aims to answer questions using a wide range of texts, not limited to a specific topic.

2. Step 2: Compare options with this definition

   Only To find answers to any question from a large collection of texts matches this goal; others describe different NLP tasks.

3. Final Answer:

   To find answers to any question from a large collection of texts -> Option C

4. Quick Check:

   Open-domain QA = Finding answers from many texts [OK]

Hint: Open-domain QA means answering questions from many texts [OK]

Common Mistakes:

• Confusing QA with translation
• Thinking QA only summarizes text
• Mixing QA with text generation

2. Which of the following is the correct sequence of steps in an open-domain QA system?

easy

A. Classify questions, then ignore documents

B. Generate answers first, then find documents

C. Summarize documents, then translate answers

D. Retrieve relevant documents, then read and extract answers

Solution

1. Step 1: Recall the typical open-domain QA pipeline

   It first retrieves relevant documents, then reads them to find answers.

2. Step 2: Match options to this pipeline

   Only Retrieve relevant documents, then read and extract answers correctly describes this order; others are incorrect or unrelated.

3. Final Answer:

   Retrieve relevant documents, then read and extract answers -> Option D

4. Quick Check:

   QA steps = Retrieve then read [OK]

Hint: QA first finds texts, then reads for answers [OK]

Common Mistakes:

• Thinking answer generation happens before retrieval
• Confusing summarization with QA
• Ignoring the retrieval step

3. Given this Python snippet using a pretrained QA model:

from transformers import pipeline
qa = pipeline('question-answering')
context = "The Eiffel Tower is in Paris."
question = "Where is the Eiffel Tower located?"
result = qa(question=question, context=context)
print(result['answer'])

What will be printed?

medium

A. Paris

B. Eiffel Tower

C. question-answering

D. The Eiffel Tower

Solution

1. Step 1: Understand the QA pipeline usage

   The pipeline takes a question and context, then returns the answer span from the context.

2. Step 2: Identify the answer span in the context

   The question asks for location; context says "The Eiffel Tower is in Paris." The answer is "Paris".

3. Final Answer:

   Paris -> Option A

4. Quick Check:

   Answer extracted = Paris [OK]

Hint: QA model returns the answer span from context [OK]

Common Mistakes:

• Printing the question instead of answer
• Confusing the model name with output
• Selecting the full sentence instead of answer span

4. You have this code snippet for open-domain QA:

from transformers import pipeline
qa = pipeline('question-answering')
context = "Mount Everest is the highest mountain."
question = "What is the highest mountain?"
result = qa(question=question, context=context)
print(result['answer'])

But it raises a KeyError: 'answer'. What is the likely cause?

medium

A. The context is empty

B. The pipeline was not properly initialized for question-answering

C. The question is not a string

D. The print statement is incorrect

Solution

1. Step 1: Analyze the error KeyError: 'answer'

   This error means the result dictionary does not have the key 'answer'.

2. Step 2: Check pipeline initialization

   If the pipeline is not correctly set for 'question-answering', the output format differs and lacks 'answer'.

3. Final Answer:

   The pipeline was not properly initialized for question-answering -> Option B

4. Quick Check:

   Wrong pipeline type causes missing 'answer' key [OK]

Hint: Ensure pipeline type matches task to get correct keys [OK]

Common Mistakes:

• Assuming context is empty without checking
• Ignoring pipeline initialization errors
• Misreading error as print statement issue

5. You want to improve an open-domain QA system that sometimes returns wrong answers because it reads irrelevant documents. Which approach helps most?

hard

A. Improve the document retrieval step to find more relevant texts

B. Use a smaller pretrained model to speed up reading

C. Remove the retrieval step and read all documents

D. Translate questions to another language before answering

Solution

1. Step 1: Identify the problem cause

   Wrong answers happen because the system reads irrelevant documents.

2. Step 2: Choose the best fix

   Improving retrieval to get relevant documents reduces wrong answers effectively.

3. Final Answer:

   Improve the document retrieval step to find more relevant texts -> Option A

4. Quick Check:

   Better retrieval = better answer relevance [OK]

Hint: Better retrieval means better answers [OK]

Common Mistakes:

• Thinking smaller models improve accuracy
• Removing retrieval causes overload and noise
• Translating questions doesn't fix relevance