QA with Hugging Face pipeline in NLP - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of a simple QA pipeline

What is the output of this code snippet using Hugging Face's QA pipeline?

NLP

from transformers import pipeline
qa = pipeline('question-answering')
context = "The Eiffel Tower is located in Paris."
question = "Where is the Eiffel Tower located?"
result = qa(question=question, context=context)
print(result['answer'])

A "Paris"

B "Eiffel Tower"

C "located"

D "The Eiffel Tower"

💡 Hint

The pipeline extracts the answer span from the context that best answers the question.

The QA pipeline finds the answer span in the context. Since the question asks where the Eiffel Tower is located, the answer is "Paris".

❓ Model Choice

intermediate

Choosing the right model for QA pipeline

Which model is best suited for a question-answering pipeline that requires understanding context and providing precise answers?

A "bert-base-uncased"

B "roberta-base"

C "distilbert-base-uncased-distilled-squad"

D "gpt2"

💡 Hint

Look for a model fine-tuned specifically on a QA dataset like SQuAD.

The 'distilbert-base-uncased-distilled-squad' model is fine-tuned on the SQuAD dataset, making it suitable for QA tasks.

❓ Hyperparameter

advanced

Effect of changing top_k in QA pipeline

In the Hugging Face QA pipeline, what happens if you set the parameter top_k=3 when calling the pipeline?

A The pipeline returns the top 3 most probable answers instead of just one.

B The pipeline limits the context to the first 3 sentences.

C The pipeline uses only 3 tokens from the question for answering.

D The pipeline runs 3 times and returns the last answer.

💡 Hint

top_k controls how many answers the model returns.

Setting top_k=3 makes the pipeline return the three best answers ranked by confidence.

❓ Metrics

advanced

Evaluating QA model performance

Which metric is commonly used to evaluate the accuracy of a question-answering model on datasets like SQuAD?

A BLEU score

B Exact Match (EM) score

C Mean Squared Error (MSE)

D Perplexity

💡 Hint

This metric measures if the predicted answer exactly matches the ground truth.

Exact Match (EM) measures the percentage of predictions that exactly match the correct answers.

🔧 Debug

expert

Debugging a QA pipeline error

You run this code but get a TypeError: 'NoneType' object is not subscriptable. What is the cause? from transformers import pipeline qa = pipeline('question-answering') context = None question = "What is AI?" result = qa(question=question, context=context) print(result['answer'])

A The model does not support question-answering.

B The question variable is None, causing the error.

C The pipeline is not initialized correctly.

D The context variable is None, so the pipeline cannot find an answer.

💡 Hint

Check the input types passed to the pipeline.

The context is None, so the pipeline returns None, causing the TypeError when trying to access ['answer'].

Practice

1. What does the Hugging Face QA pipeline do when given a question and a context?

easy

A. It translates the question into another language.

B. It summarizes the context without answering the question.

C. It finds the answer to the question from the given context.

D. It generates a new question based on the context.

Solution

1. Step 1: Understand the QA pipeline purpose

   The QA pipeline is designed to find answers from a given text based on a question.

2. Step 2: Match function to options

   Only It finds the answer to the question from the given context. describes finding an answer from the context, which is the pipeline's main job.

3. Final Answer:

   It finds the answer to the question from the given context. -> Option C

4. Quick Check:

   QA pipeline = find answer from context [OK]

Hint: QA pipeline = question + context -> answer [OK]

Common Mistakes:

• Confusing QA with translation or summarization
• Thinking it generates new questions
• Assuming it works without context

2. Which of the following is the correct way to create a QA pipeline using Hugging Face Transformers in Python?

easy

A. import pipeline from transformers
qa = pipeline('qa')

B. from transformers import QA
qa = QA('pipeline')

C. from transformers import question_answering
qa = question_answering()

D. from transformers import pipeline
qa = pipeline('question-answering')

Solution

1. Step 1: Recall correct import and pipeline creation

   The correct import is from transformers import pipeline, then call pipeline('question-answering').

2. Step 2: Check each option syntax

   Only from transformers import pipeline qa = pipeline('question-answering') matches the correct syntax and function call.

3. Final Answer:

   from transformers import pipeline qa = pipeline('question-answering') -> Option D

4. Quick Check:

   Correct import and pipeline call = from transformers import pipeline qa = pipeline('question-answering') [OK]

Hint: Use pipeline('question-answering') from transformers [OK]

Common Mistakes:

• Wrong import statement
• Incorrect pipeline argument
• Using non-existent classes or functions

3. What will be the output of this code snippet?

from transformers import pipeline
qa = pipeline('question-answering')
result = qa(question='Where is the Eiffel Tower?', context='The Eiffel Tower is in Paris.')
print(result['answer'])

medium

A. In Paris

B. Paris

C. The Eiffel Tower

D. Eiffel Tower

Solution

1. Step 1: Understand the question and context

   The question asks for the location of the Eiffel Tower, and the context states it is in Paris.

2. Step 2: Predict the pipeline answer output

   The pipeline extracts the answer span from the context, which is 'Paris'.

3. Final Answer:

   Paris -> Option B

4. Quick Check:

   Answer extracted = Paris [OK]

Hint: Answer is the location mentioned in context [OK]

Common Mistakes:

• Choosing the full phrase instead of the exact answer
• Confusing question with context text
• Expecting the pipeline to generate new text

4. Identify the error in this code snippet that uses the Hugging Face QA pipeline:

from transformers import pipeline
qa = pipeline('question-answering')
result = qa(question='Who wrote Hamlet?', text='Hamlet was written by Shakespeare.')
print(result['answer'])

medium

A. The argument 'text' should be 'context'.

B. The pipeline name should be 'qa' instead of 'question-answering'.

C. The print statement should use result.answer instead of result['answer'].

D. The import statement is incorrect.

Solution

1. Step 1: Check pipeline argument names

   The QA pipeline expects 'question' and 'context' as arguments, not 'text'.

2. Step 2: Verify other parts of the code

   Pipeline name and import are correct; accessing result['answer'] is valid.

3. Final Answer:

   The argument 'text' should be 'context'. -> Option A

4. Quick Check:

   Use 'context' argument for QA pipeline [OK]

Hint: Use 'context' not 'text' for QA input [OK]

Common Mistakes:

• Using 'text' instead of 'context'
• Changing pipeline name incorrectly
• Wrong result access syntax

5. You want to build a QA system that answers questions from multiple documents. Which approach using Hugging Face pipelines is best?

hard

A. Run the QA pipeline separately on each document and pick the answer with highest score.

B. Concatenate all documents into one string and run the QA pipeline once.

C. Use the QA pipeline only on the first document and ignore others.

D. Train a new model from scratch for multiple documents.

Solution

1. Step 1: Understand pipeline input limits

   QA pipelines work best on one context at a time; long concatenated text may reduce accuracy.

2. Step 2: Evaluate options for multiple documents

   Running QA on each document separately and selecting the best answer is effective and practical.

3. Final Answer:

   Run the QA pipeline separately on each document and pick the answer with highest score. -> Option A

4. Quick Check:

   Separate runs + best score = best multi-doc QA [OK]

Hint: Run QA on each doc, choose best answer [OK]

Common Mistakes:

• Concatenating all documents causing context overflow
• Ignoring documents except first
• Unnecessarily retraining models