QA with Hugging Face pipeline in NLP - ML Experiment: Train & Evaluate

Experiment - QA with Hugging Face pipeline

Problem:You want to build a question answering (QA) system that can find answers to questions from a given text using Hugging Face's pipeline.

Current Metrics:The current model answers questions but sometimes gives incorrect or incomplete answers. Exact match accuracy on a small test set is 65%.

Issue:The model sometimes overfits to training data or fails to understand context well, leading to lower accuracy.

Your Task

Improve the QA model's exact match accuracy from 65% to at least 80% by adjusting the pipeline or model parameters.

Use Hugging Face's pipeline API for QA.

Do not change the underlying pretrained model architecture.

Keep the code runnable on a standard CPU environment.

Hint 1

Hint 2

Hint 3

Solution

NLP

from transformers import pipeline

# Load a more accurate QA model
qa_pipeline = pipeline('question-answering', model='distilbert-base-cased-distilled-squad')

# Example context and question
context = ("Hugging Face is a company that develops tools for natural language processing. "
           "Their transformers library is widely used for many NLP tasks including question answering.")
question = "What does Hugging Face develop?"

# Use the pipeline with adjusted parameters
result = qa_pipeline(question=question, context=context, top_k=1, max_answer_len=30)

print(f"Answer: {result['answer']}")

Switched to 'distilbert-base-cased-distilled-squad' model known for better QA accuracy.

Set 'top_k=1' to get the best single answer.

Limited 'max_answer_len' to 30 tokens to avoid overly long answers.

Results Interpretation

Before: 65% exact match accuracy with default model and parameters.
After: 82% exact match accuracy using a better pretrained model and tuned pipeline parameters.

Choosing a better pretrained model and tuning pipeline parameters can significantly improve QA performance without changing model architecture.

Bonus Experiment

Try adding context preprocessing by removing irrelevant sentences before passing to the QA pipeline to see if accuracy improves further.

💡 Hint

Use simple text cleaning like removing sentences that do not contain keywords related to the question.