Extractive QA concept in NLP - Model Metrics & Evaluation

Metrics & Evaluation - Extractive QA concept

Which metric matters for Extractive QA and WHY

In Extractive Question Answering (QA), the goal is to find the exact part of a text that answers a question. The key metrics are Exact Match (EM) and F1 score.

Exact Match (EM) checks if the predicted answer exactly matches the true answer. It is strict and shows how often the model gets the answer perfectly right.

F1 score measures overlap between the predicted and true answer words. It balances precision (how many predicted words are correct) and recall (how many true answer words were found). This helps when answers are partially correct.

These metrics matter because Extractive QA needs precise text spans. EM shows perfect hits, while F1 shows partial correctness, giving a fuller picture.

Confusion matrix or equivalent visualization

Extractive QA does not use a classic confusion matrix like classification. Instead, we compare predicted answer spans to true answer spans.

True Answer: "the Eiffel Tower"
Predicted Answer: "Eiffel Tower"

- Exact Match: No (because of missing "the")
- Precision: 2/2 = 1.0 (all predicted words are correct)
- Recall: 2/3 ≈ 0.67 (missed "the")
- F1: 2 * (1.0 * 0.67) / (1.0 + 0.67) ≈ 0.8
    

This shows how partial matches are scored.

Precision vs Recall tradeoff with examples

In Extractive QA:

• High Precision: The model's answers are mostly correct words but might miss some parts. Good when you want very accurate answers.
• High Recall: The model finds most of the true answer words but may include extra words. Good when you want to catch all relevant info, even if some noise is included.

Example: For a question about "Where is the Eiffel Tower?" (true answer: "in Paris")

• High precision, low recall: "Paris" (correct words, but missing "in")
• High recall, low precision: "in Paris, France" (includes extra words)

Balancing precision and recall with F1 helps find the best middle ground.

What "good" vs "bad" metric values look like for Extractive QA

Good values:

• Exact Match (EM) above 70% means the model often finds the exact answer.
• F1 score above 80% means the model finds most answer words correctly, even if not exact.

Bad values:

• EM below 40% means the model rarely gets exact answers.
• F1 below 50% means the model often misses many answer words or adds wrong words.

Good scores mean users get reliable answers. Bad scores mean answers are often wrong or incomplete.

Common pitfalls in Extractive QA metrics

• Ignoring partial credit: Only using Exact Match misses partially correct answers, which F1 captures.
• Answer variations: Different but correct answer forms (e.g., "U.S." vs "United States") can lower EM unfairly.
• Data leakage: If test questions appear in training, metrics look better but model won't generalize.
• Overfitting: Very high EM on training but low on test means model memorizes rather than understands.

Self-check question

Your Extractive QA model has 50% Exact Match but 85% F1 score. Is it good?

Answer: Yes, this is typical because F1 is usually equal or higher than EM. A high F1 with low EM suggests the model gets many partial matches right but perfect exact answers less often. Overall quality is good. You should investigate answer variations to improve EM.

Key Result

Exact Match shows perfect answer hits; F1 balances partial correctness, both key to judge Extractive QA quality.

Practice

1. What is the main goal of extractive question answering (QA)?

easy

A. To translate the question into another language

B. To generate a new answer not present in the text

C. To summarize the entire text into a short paragraph

D. To find the exact answer span within a given text

Solution

1. Step 1: Understand extractive QA purpose

   Extractive QA aims to locate the exact part of the text that answers the question.

2. Step 2: Compare options with definition

   Only To find the exact answer span within a given text describes finding the exact answer span inside the text, which matches extractive QA.

3. Final Answer:

   To find the exact answer span within a given text -> Option D

4. Quick Check:

   Extractive QA = find exact answer span [OK]

Hint: Extractive QA picks text parts, not creates new answers [OK]

Common Mistakes:

• Confusing extractive QA with generative QA
• Thinking extractive QA summarizes text
• Assuming extractive QA translates questions

2. Which of the following is the correct way to represent an extractive QA model's output?

easy

A. Span of text indices indicating the answer start and end

B. Single integer representing the answer length

C. List of unrelated keywords from the text

D. Boolean value indicating if the answer exists

Solution

1. Step 1: Recall extractive QA output format

   Extractive QA models output the start and end positions of the answer span in the text.

2. Step 2: Match options to output format

   Only Span of text indices indicating the answer start and end correctly describes output as text span indices.

3. Final Answer:

   Span of text indices indicating the answer start and end -> Option A

4. Quick Check:

   Output = start and end indices [OK]

Hint: Extractive QA outputs answer span positions, not just length [OK]

Common Mistakes:

• Choosing answer length instead of span indices
• Confusing keywords with answer span
• Thinking output is just true/false

3. Given the context: 'The Eiffel Tower is located in Paris.' and the question: 'Where is the Eiffel Tower?', what would an extractive QA model most likely output?

medium

A. "Eiffel Tower"

B. "located"

C. "Paris"

D. "The Eiffel Tower is located"

Solution

1. Step 1: Understand question and context

   The question asks for the location of the Eiffel Tower, which is stated as "Paris" in the context.

2. Step 2: Identify exact answer span

   The extractive QA model selects the exact text span answering the question, which is "Paris".

3. Final Answer:

   "Paris" -> Option C

4. Quick Check:

   Answer = "Paris" [OK]

Hint: Extractive QA picks exact answer phrase from context [OK]

Common Mistakes:

• Selecting part of the question as answer
• Choosing unrelated words from context
• Picking longer phrases than needed

4. Consider this extractive QA model output code snippet:

start_idx = 10
end_idx = 5
answer = context[start_idx:end_idx]

What is the main issue here?

medium

A. The end index is smaller than the start index, causing an empty answer

B. The indices are correct and will extract the answer properly

C. The code is missing a question input

D. The context variable is undefined

Solution

1. Step 1: Analyze index values

   The start index is 10 and the end index is 5, which is smaller than start.

2. Step 2: Understand slicing behavior

   In Python, slicing with start > end returns an empty string, so no answer is extracted.

3. Final Answer:

   The end index is smaller than the start index, causing an empty answer -> Option A

4. Quick Check:

   End index < start index = empty slice [OK]

Hint: End index must be >= start index for valid slice [OK]

Common Mistakes:

• Ignoring index order in slicing
• Assuming code runs without error
• Overlooking empty string result

5. You want to improve an extractive QA model to handle questions where the answer might not be present in the context. Which approach is best?

hard

A. Use a generative model instead of extractive QA

B. Add a 'no answer' prediction option so the model can say answer is missing

C. Train the model only on questions with guaranteed answers

D. Force the model to always select some text span regardless

Solution

1. Step 1: Understand the problem of missing answers

   Extractive QA models can fail if forced to select an answer when none exists in context.

2. Step 2: Evaluate solution options

   Adding a 'no answer' option lets the model explicitly indicate no answer is found, improving reliability.

3. Final Answer:

   Add a 'no answer' prediction option so the model can say answer is missing -> Option B

4. Quick Check:

   Handle missing answers = add 'no answer' option [OK]

Hint: Allow model to say 'no answer' when context lacks answer [OK]

Common Mistakes:

• Forcing answer selection even if none exists
• Ignoring questions without answers during training
• Switching to generative models unnecessarily