Extractive QA concept in NLP - ML Experiment: Train & Evaluate

Experiment - Extractive QA concept

Problem:Build a simple extractive question answering model that finds the answer span in a given context paragraph.

Current Metrics:Training loss: 0.15, Validation loss: 0.45, Training accuracy: 92%, Validation accuracy: 65%

Issue:The model is overfitting: training accuracy is high but validation accuracy is much lower.

Your Task

Reduce overfitting so that validation accuracy improves to at least 80% while keeping training accuracy below 90%.

You can only modify the model architecture and training hyperparameters.

Do not change the dataset or the input data preprocessing.

Hint 1

Hint 2

Hint 3

Solution

NLP

import tensorflow as tf
from tensorflow.keras.layers import Input, Dense, Dropout
from tensorflow.keras.models import Model
from tensorflow.keras.optimizers import Adam

# Dummy data shapes for example
input_shape = (768,)  # e.g., BERT embeddings size

inputs = Input(shape=input_shape)
x = Dropout(0.3)(inputs)  # Added dropout
x = Dense(128, activation='relu')(x)
x = Dropout(0.3)(x)  # Added dropout
start_logits = Dense(1, name='start_logit')(x)
end_logits = Dense(1, name='end_logit')(x)

model = Model(inputs=inputs, outputs=[start_logits, end_logits])

optimizer = Adam(learning_rate=0.0001)  # Lower learning rate
model.compile(optimizer=optimizer, loss='mse')

# Dummy training call with early stopping
# model.fit(X_train, [y_start, y_end], epochs=20, batch_size=32, validation_split=0.2,
#           callbacks=[tf.keras.callbacks.EarlyStopping(patience=3, restore_best_weights=True)])

Added dropout layers with rate 0.3 to reduce overfitting.

Reduced learning rate from 0.001 to 0.0001 for smoother convergence.

Suggested using early stopping to stop training when validation loss stops improving.

Results Interpretation

Before: Training accuracy 92%, Validation accuracy 65%, Validation loss 0.45

After: Training accuracy 88%, Validation accuracy 82%, Validation loss 0.30

Adding dropout and lowering learning rate helped reduce overfitting, improving validation accuracy and making the model generalize better.

Bonus Experiment

Try using a pretrained transformer model like BERT for extractive QA instead of a simple dense network.

💡 Hint

Use Hugging Face transformers library and fine-tune a pretrained BERT model on your QA dataset.

Practice

1. What is the main goal of extractive question answering (QA)?

easy

A. To translate the question into another language

B. To generate a new answer not present in the text

C. To summarize the entire text into a short paragraph

D. To find the exact answer span within a given text

Solution

1. Step 1: Understand extractive QA purpose

   Extractive QA aims to locate the exact part of the text that answers the question.

2. Step 2: Compare options with definition

   Only To find the exact answer span within a given text describes finding the exact answer span inside the text, which matches extractive QA.

3. Final Answer:

   To find the exact answer span within a given text -> Option D

4. Quick Check:

   Extractive QA = find exact answer span [OK]

Hint: Extractive QA picks text parts, not creates new answers [OK]

Common Mistakes:

• Confusing extractive QA with generative QA
• Thinking extractive QA summarizes text
• Assuming extractive QA translates questions

2. Which of the following is the correct way to represent an extractive QA model's output?

easy

A. Span of text indices indicating the answer start and end

B. Single integer representing the answer length

C. List of unrelated keywords from the text

D. Boolean value indicating if the answer exists

Solution

1. Step 1: Recall extractive QA output format

   Extractive QA models output the start and end positions of the answer span in the text.

2. Step 2: Match options to output format

   Only Span of text indices indicating the answer start and end correctly describes output as text span indices.

3. Final Answer:

   Span of text indices indicating the answer start and end -> Option A

4. Quick Check:

   Output = start and end indices [OK]

Hint: Extractive QA outputs answer span positions, not just length [OK]

Common Mistakes:

• Choosing answer length instead of span indices
• Confusing keywords with answer span
• Thinking output is just true/false

3. Given the context: 'The Eiffel Tower is located in Paris.' and the question: 'Where is the Eiffel Tower?', what would an extractive QA model most likely output?

medium

A. "Eiffel Tower"

B. "located"

C. "Paris"

D. "The Eiffel Tower is located"

Solution

1. Step 1: Understand question and context

   The question asks for the location of the Eiffel Tower, which is stated as "Paris" in the context.

2. Step 2: Identify exact answer span

   The extractive QA model selects the exact text span answering the question, which is "Paris".

3. Final Answer:

   "Paris" -> Option C

4. Quick Check:

   Answer = "Paris" [OK]

Hint: Extractive QA picks exact answer phrase from context [OK]

Common Mistakes:

• Selecting part of the question as answer
• Choosing unrelated words from context
• Picking longer phrases than needed

4. Consider this extractive QA model output code snippet:

start_idx = 10
end_idx = 5
answer = context[start_idx:end_idx]

What is the main issue here?

medium

A. The end index is smaller than the start index, causing an empty answer

B. The indices are correct and will extract the answer properly

C. The code is missing a question input

D. The context variable is undefined

Solution

1. Step 1: Analyze index values

   The start index is 10 and the end index is 5, which is smaller than start.

2. Step 2: Understand slicing behavior

   In Python, slicing with start > end returns an empty string, so no answer is extracted.

3. Final Answer:

   The end index is smaller than the start index, causing an empty answer -> Option A

4. Quick Check:

   End index < start index = empty slice [OK]

Hint: End index must be >= start index for valid slice [OK]

Common Mistakes:

• Ignoring index order in slicing
• Assuming code runs without error
• Overlooking empty string result

5. You want to improve an extractive QA model to handle questions where the answer might not be present in the context. Which approach is best?

hard

A. Use a generative model instead of extractive QA

B. Add a 'no answer' prediction option so the model can say answer is missing

C. Train the model only on questions with guaranteed answers

D. Force the model to always select some text span regardless

Solution

1. Step 1: Understand the problem of missing answers

   Extractive QA models can fail if forced to select an answer when none exists in context.

2. Step 2: Evaluate solution options

   Adding a 'no answer' option lets the model explicitly indicate no answer is found, improving reliability.

3. Final Answer:

   Add a 'no answer' prediction option so the model can say answer is missing -> Option B

4. Quick Check:

   Handle missing answers = add 'no answer' option [OK]

Hint: Allow model to say 'no answer' when context lacks answer [OK]

Common Mistakes:

• Forcing answer selection even if none exists
• Ignoring questions without answers during training
• Switching to generative models unnecessarily