Answer span extraction in NLP - Model Metrics & Evaluation

Metrics & Evaluation - Answer span extraction

Which metric matters for Answer span extraction and WHY

Answer span extraction means finding the exact part of text that answers a question. The main metric to check is Exact Match (EM). It tells us how often the model finds the answer exactly right. Another key metric is F1 score, which measures how much the predicted answer overlaps with the true answer. These metrics matter because in real life, getting the exact answer or a very close one is what counts.

Confusion matrix or equivalent visualization

For answer span extraction, we don't use a classic confusion matrix like in classification. Instead, we compare predicted spans to true spans:

True answer span: "the quick brown fox"
Predicted span:  "quick brown"

Overlap tokens: 2
Total tokens in true answer: 4
Total tokens in predicted answer: 2

F1 = 2 * (Precision * Recall) / (Precision + Recall)
Precision = Overlap / Predicted tokens = 2/2 = 1.0
Recall = Overlap / True tokens = 2/4 = 0.5
F1 = 2 * (1.0 * 0.5) / (1.0 + 0.5) = 0.67
Exact Match = 0 (because spans are not exactly the same)
    

This shows how F1 captures partial correctness, while Exact Match is strict.

Precision vs Recall tradeoff with examples

In answer span extraction, precision means how much of the predicted answer is correct, and recall means how much of the true answer the model found.

High precision, low recall: The model gives short answers that are always correct but miss some parts. For example, predicting "brown fox" when the true answer is "the quick brown fox". This is safe but incomplete.

High recall, low precision: The model gives long answers that include the true answer but also extra words. For example, predicting "the quick brown fox jumps" when the true answer is "quick brown fox". This covers the answer but adds noise.

Good models balance precision and recall to get a high F1 score, meaning answers are mostly correct and mostly complete.

What "good" vs "bad" metric values look like for answer span extraction

Good: Exact Match above 70% and F1 score above 80% usually mean the model finds answers correctly and mostly exactly. This is great for applications like chatbots or search engines.

Bad: Exact Match below 40% and F1 below 50% show the model struggles to find correct answers or only finds partial or wrong spans. This leads to poor user experience.

Common pitfalls in metrics for answer span extraction

• Ignoring partial matches: Only using Exact Match misses cases where the answer is mostly right but not exact.
• Overfitting: High Exact Match on training data but low on new data means the model memorizes answers instead of understanding.
• Data leakage: If test questions appear in training, metrics look better but don't reflect real performance.
• Ignoring answer length: Very short or very long predicted spans can skew precision or recall.

Self-check question

Your answer span extraction model has 85% Exact Match but only 60% F1 score. Is it good? Why or why not?

Answer: This means the model often finds exact answers but sometimes misses partial overlaps. It might be too strict or miss some answer parts. Improving recall to raise F1 would help make answers more complete and useful.

Key Result

Exact Match and F1 score are key metrics; Exact Match checks exact answers, F1 balances partial correctness.

Practice

1. What is the main goal of answer span extraction in NLP?

easy

A. To generate new text based on a prompt

B. To find the exact part of text that answers a question

C. To summarize long documents into short sentences

D. To translate text from one language to another

Solution

1. Step 1: Understand the purpose of answer span extraction

   Answer span extraction focuses on locating the exact segment in a text that directly answers a question.

2. Step 2: Compare with other NLP tasks

   Unlike translation, summarization, or text generation, answer span extraction pinpoints a specific text span as the answer.

3. Final Answer:

   To find the exact part of text that answers a question -> Option B

4. Quick Check:

   Answer span extraction = find exact answer span [OK]

Hint: Answer span extraction locates exact text answers [OK]

Common Mistakes:

• Confusing answer span extraction with translation
• Thinking it summarizes text instead of extracting spans
• Assuming it generates new text

2. Which of the following is the correct way to represent the start and end positions for answer span extraction in code?

easy

A. start_index and end_index as integers

B. start_word and end_word as strings

C. start_time and end_time as floats

D. start_char and end_char as booleans

Solution

1. Step 1: Identify typical data types for positions

   Positions in text are usually represented by integer indices marking start and end locations.

2. Step 2: Evaluate options

   Strings or booleans do not represent positions well; floats for time are unrelated to text spans.

3. Final Answer:

   start_index and end_index as integers -> Option A

4. Quick Check:

   Positions = integer indices [OK]

Hint: Positions in text are integer indices [OK]

Common Mistakes:

• Using strings instead of integer indices
• Confusing character positions with time values
• Using booleans for position markers

3. Given the text: 'The cat sat on the mat.' and predicted start index = 1, end index = 4, what is the extracted answer span?

medium

A. 'cat sat on'

B. 'sat on the'

C. 'on the mat'

D. 'The cat sat'

Solution

1. Step 1: Identify tokens and their indices

   Tokenizing the sentence: ['The'(0), 'cat'(1), 'sat'(2), 'on'(3), 'the'(4), 'mat.'(5)]. The indices given (1 to 4) refer to 0-based token positions.

2. Step 2: Extract tokens from start to end index

   In standard extraction, take tokens[start:end] (end exclusive): tokens[1:4] = ['cat'(1), 'sat'(2), 'on'(3)] = 'cat sat on'.

3. Final Answer:

   'cat sat on' -> Option A

4. Quick Check:

   Extract tokens from start to end index = 'cat sat on' [OK]

Hint: Match indices to tokens carefully [OK]

Common Mistakes:

• Confusing character indices with token indices
• Off-by-one errors in slicing
• Ignoring punctuation in tokens

4. You have a model that predicts start and end indices for answer spans but sometimes the end index is smaller than the start index. What is the best way to fix this bug?

medium

A. Ignore the prediction and return an empty answer

B. Always set end index to start index plus one

C. Swap the start and end indices if end < start

D. Use only the start index as the answer

Solution

1. Step 1: Understand the problem with indices

   End index smaller than start index is invalid because answer spans must go forward in text.

2. Step 2: Choose a fix that preserves valid spans

   Swapping start and end indices corrects the order and keeps the predicted span meaningful.

3. Final Answer:

   Swap the start and end indices if end < start -> Option C

4. Quick Check:

   Fix invalid spans by swapping indices [OK]

Hint: Swap indices if end < start to fix spans [OK]

Common Mistakes:

• Ignoring invalid spans instead of fixing
• Forcing fixed span length blindly
• Using only one index loses answer context

5. In a question-answering system, the model outputs start logits and end logits for each token. How should you combine these to find the best answer span?

hard

A. Choose random start and end indices

B. Pick the token with the highest start logit only

C. Pick the token with the highest end logit only

D. Find the pair of start and end indices with the highest sum of start and end logits where start ≤ end

Solution

1. Step 1: Understand logits for start and end tokens

   Start and end logits represent scores for each token being the start or end of the answer span.

2. Step 2: Combine logits to find best span

   We look for the pair (start, end) with the highest combined score, ensuring start ≤ end to form a valid span.

3. Final Answer:

   Find the pair of start and end indices with the highest sum of start and end logits where start ≤ end -> Option D

4. Quick Check:

   Combine start and end logits to find best span [OK]

Hint: Sum start and end logits, ensure start ≤ end [OK]

Common Mistakes:

• Ignoring end logits and using start only
• Choosing invalid spans where end < start
• Picking random indices without scores