Open-domain QA basics in NLP - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What is the main difference between open-domain QA and closed-domain QA?

Open-domain QA systems answer questions from any topic using a large knowledge source. Closed-domain QA systems focus on a specific topic or dataset.

Which statement best describes the main difference?

A Closed-domain QA uses large external databases, while open-domain QA only uses small, fixed datasets.

B Open-domain QA requires no training, but closed-domain QA always requires training on labeled data.

C Open-domain QA can answer questions about any topic, while closed-domain QA is limited to a specific subject area.

D Closed-domain QA systems are always faster than open-domain QA systems.

💡 Hint

Think about the scope of questions each system can handle.

Open-domain QA systems are designed to answer questions on any topic, often using large collections like Wikipedia. Closed-domain QA systems focus on a narrow area, such as medical records or legal documents.

❓ Predict Output

intermediate

What is the output of this simple retrieval step in open-domain QA?

Given a list of documents and a query, the code below finds documents containing the query word.

documents = ["The sky is blue.", "Grass is green.", "The sun is bright."]
query = "sky"
retrieved = [doc for doc in documents if query in doc]
print(retrieved)

What is printed?

A ["The sky is blue."]

B ["Grass is green."]

C ["The sun is bright."]

D []

💡 Hint

Check which document contains the word 'sky'.

The list comprehension filters documents containing the exact substring 'sky'. Only the first document contains it.

❓ Model Choice

advanced

Which model architecture is best suited for open-domain QA tasks?

Open-domain QA often requires understanding questions and retrieving answers from large text collections.

Which model architecture is most appropriate?

A A simple linear regression model.

B A convolutional neural network trained on image classification.

C A recurrent neural network trained for speech recognition.

D A sequence-to-sequence transformer model fine-tuned for question answering.

💡 Hint

Consider models designed for text understanding and generation.

Sequence-to-sequence transformer models like BERT or T5 are commonly fine-tuned for QA tasks because they understand context and generate answers. CNNs and RNNs for other domains are not suitable here.

❓ Metrics

advanced

Which metric is most appropriate to evaluate an open-domain QA system's answer quality?

Open-domain QA systems produce text answers to questions. Which metric below best measures answer correctness?

A Accuracy of classifying images into categories.

B Exact Match (EM) score measuring if predicted answer exactly matches the ground truth.

C BLEU score used for machine translation quality.

D Mean Squared Error (MSE) between predicted and true numerical values.

💡 Hint

Think about metrics that compare predicted text answers to correct answers.

Exact Match (EM) is widely used in QA to check if the predicted answer exactly matches the reference answer. MSE and image accuracy are unrelated, and BLEU is less precise for short QA answers.

🔧 Debug

expert

Why does this open-domain QA retrieval code return an empty list?

Consider this code snippet for retrieving documents containing a query word:

documents = ["The sky is blue.", "Grass is green.", "The sun is bright."]
query = "Sky"
retrieved = [doc for doc in documents if query in doc]
print(retrieved)

Why is the output an empty list?

A Because the query 'Sky' has uppercase 'S' but documents contain lowercase 'sky', so the match fails due to case sensitivity.

B Because the documents list is empty.

C Because the query word is not a string.

D Because the list comprehension syntax is incorrect.

💡 Hint

Check if the query matches the document text exactly including letter case.

String containment checks in Python are case sensitive. 'Sky' does not match 'sky' in the documents, so no documents are retrieved.