Extractive QA concept in NLP - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What is the main goal of extractive question answering?

In extractive question answering, what is the system primarily designed to do?

A Generate a completely new answer not present in the text

B Classify the question into predefined categories

C Summarize the entire passage into a short paragraph

D Select a span of text from the given passage as the answer

💡 Hint

Think about whether the answer is created or taken directly from the text.

Extractive QA systems find and return a specific part of the text that answers the question, rather than generating new text.

❓ Predict Output

intermediate

Output of extractive QA model span prediction

Given a passage and a question, an extractive QA model outputs start and end indices of the answer span. If the passage is "The sky is blue and clear today." and the model predicts start index 3 and end index 5, what is the extracted answer?

NLP

passage = "The sky is blue and clear today."
start_idx = 3
end_idx = 5
words = passage.split()
answer = ' '.join(words[start_idx:end_idx+1])
print(answer)

A "and clear today."

B "is blue and"

C "blue and clear"

D "sky is blue"

💡 Hint

Check the words between indices 3 and 5 inclusive.

Splitting the passage into words, indices 3 to 5 correspond to 'blue', 'and', 'clear'.

❓ Model Choice

advanced

Best model architecture for extractive QA

Which model architecture is most suitable for extractive question answering tasks?

A Pretrained Transformer model fine-tuned to predict start and end token positions

B Sequence-to-sequence model trained for translation tasks

C Generative Transformer model like GPT that generates answers from scratch

D Convolutional Neural Network for image classification

💡 Hint

Extractive QA requires locating answer spans, not generating new text.

Models like BERT fine-tuned for extractive QA predict start and end positions of the answer span in the input text.

❓ Metrics

advanced

Which metric best evaluates extractive QA performance?

For extractive question answering, which metric is commonly used to measure how well the predicted answer matches the true answer span?

A BLEU score measuring n-gram overlap

B Exact Match (EM) score checking exact span match

C Mean Squared Error between predicted and true spans

D Accuracy of classifying question types

💡 Hint

Think about a metric that checks if the predicted answer exactly matches the true answer.

Exact Match (EM) measures the percentage of predictions that exactly match the ground truth answer span.

🔧 Debug

expert

Why does this extractive QA model fail to find the answer span?

A fine-tuned extractive QA model always predicts the start and end indices as 0, regardless of input. What is the most likely cause?

NLP

def predict_span(model, input_ids):
    start_scores, end_scores = model(input_ids)
    start_index = start_scores.argmax()
    end_index = end_scores.argmax()
    return start_index, end_index

# Model always returns start_index=0 and end_index=0

A The model was not fine-tuned and outputs default scores favoring index 0

B The input_ids are empty, so model cannot predict spans

C The argmax function is used incorrectly causing wrong indices

D The model architecture is incompatible with extractive QA

💡 Hint

Consider what happens if the model is not trained on the task.

Without fine-tuning, the model outputs uniform or default scores, causing argmax to pick index 0 always.