Extractive QA concept in NLP - ML Experiment: Train & Evaluate

Experiment - Extractive QA concept

Problem:Build a simple extractive question answering model that finds the answer span in a given context paragraph.

Current Metrics:Training loss: 0.15, Validation loss: 0.45, Training accuracy: 92%, Validation accuracy: 65%

Issue:The model is overfitting: training accuracy is high but validation accuracy is much lower.

Your Task

Reduce overfitting so that validation accuracy improves to at least 80% while keeping training accuracy below 90%.

You can only modify the model architecture and training hyperparameters.

Do not change the dataset or the input data preprocessing.

Hint 1

Hint 2

Hint 3

Solution

NLP

import tensorflow as tf
from tensorflow.keras.layers import Input, Dense, Dropout
from tensorflow.keras.models import Model
from tensorflow.keras.optimizers import Adam

# Dummy data shapes for example
input_shape = (768,)  # e.g., BERT embeddings size

inputs = Input(shape=input_shape)
x = Dropout(0.3)(inputs)  # Added dropout
x = Dense(128, activation='relu')(x)
x = Dropout(0.3)(x)  # Added dropout
start_logits = Dense(1, name='start_logit')(x)
end_logits = Dense(1, name='end_logit')(x)

model = Model(inputs=inputs, outputs=[start_logits, end_logits])

optimizer = Adam(learning_rate=0.0001)  # Lower learning rate
model.compile(optimizer=optimizer, loss='mse')

# Dummy training call with early stopping
# model.fit(X_train, [y_start, y_end], epochs=20, batch_size=32, validation_split=0.2,
#           callbacks=[tf.keras.callbacks.EarlyStopping(patience=3, restore_best_weights=True)])

Added dropout layers with rate 0.3 to reduce overfitting.

Reduced learning rate from 0.001 to 0.0001 for smoother convergence.

Suggested using early stopping to stop training when validation loss stops improving.

Results Interpretation

Before: Training accuracy 92%, Validation accuracy 65%, Validation loss 0.45

After: Training accuracy 88%, Validation accuracy 82%, Validation loss 0.30

Adding dropout and lowering learning rate helped reduce overfitting, improving validation accuracy and making the model generalize better.

Bonus Experiment

Try using a pretrained transformer model like BERT for extractive QA instead of a simple dense network.

💡 Hint

Use Hugging Face transformers library and fine-tune a pretrained BERT model on your QA dataset.