QA with Hugging Face pipeline in NLP - Model Metrics & Evaluation

Metrics & Evaluation - QA with Hugging Face pipeline

Which metric matters for QA with Hugging Face pipeline and WHY

For question answering (QA) tasks using Hugging Face pipelines, the key metrics are Exact Match (EM) and F1 score. Exact Match measures how often the model's answer exactly matches the correct answer. F1 score measures the overlap between the predicted and true answer words, balancing precision and recall. These metrics matter because QA answers can be short phrases or sentences, so exact matches are strict, while F1 allows partial credit for close answers.

Confusion matrix or equivalent visualization

QA tasks do not use a traditional confusion matrix because answers are text, not classes. Instead, evaluation compares predicted answers to true answers using token-level overlap.

True answer: "Paris"
Predicted answer: "Paris"
Exact Match: 1 (correct)

True answer: "Paris"
Predicted answer: "the city of Paris"
Exact Match: 0 (not exact)
F1 score: calculated from word overlap
    

Precision vs Recall tradeoff with concrete examples

In QA, precision means how many words in the predicted answer are correct, and recall means how many words from the true answer were found. For example:

• Predicted: "Paris"
  • True: "Paris"
    Precision = 1, Recall = 1 (perfect)
• Predicted: "the city"
  • True: "Paris"
    Precision = 0 (no correct words), Recall = 0 (missed true answer)
• Predicted: "city of Paris"
  • True: "Paris"
    Precision = 1/3, Recall = 1 (all true words found but extra words included)

Good QA models balance precision and recall to get high F1 scores.

What "good" vs "bad" metric values look like for QA

Good QA models have Exact Match scores above 70% and F1 scores above 80% on standard datasets. Bad models have low EM (below 40%) and low F1 (below 50%), meaning answers are often wrong or incomplete.

Example:

• Good: EM = 75%, F1 = 85% (answers mostly correct and complete)
• Bad: EM = 30%, F1 = 45% (answers often wrong or missing key info)

Common pitfalls in QA metrics

• Exact Match too strict: Small differences like punctuation or articles cause zero score even if answer is close.
• Ignoring partial credit: Relying only on EM misses partial correct answers; F1 helps here.
• Data leakage: Training on test questions inflates scores falsely.
• Overfitting: High training scores but low test scores mean model memorizes answers, not generalizes.
• Ambiguous questions: Multiple correct answers can confuse metric calculations.

Self-check question

Your QA model has 85% Exact Match but only 50% F1 score. Is it good? Why or why not?

Answer: This is unusual because F1 should be equal or higher than EM. A low F1 suggests the model's answers are often exact but very short or missing parts, or there may be an error in calculation. You should check the evaluation method and ensure answers are fully captured. Generally, both EM and F1 should be high for a good QA model.

Key Result

Exact Match and F1 score are key metrics for QA; they measure exact correctness and partial answer overlap respectively.