Custom QA model fine-tuning in NLP - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding the purpose of fine-tuning a QA model

Why do we fine-tune a pre-trained question answering (QA) model on a custom dataset?

A To convert the QA model into a text generation model

B To reduce the model size so it runs faster on any device

C To remove all pre-trained knowledge and start training from scratch

D To adapt the model to the specific language and style of the custom dataset for better accuracy

💡 Hint

Think about why a model trained on general data might not perform well on your specific questions.

Fine-tuning adjusts the model's knowledge to the specific language, domain, and style of your custom dataset, improving its accuracy on your tasks.

❓ Predict Output

intermediate

Output of fine-tuning training loop snippet

What will be the printed output after running this training loop snippet for 1 epoch?

NLP

for epoch in range(1):
    total_loss = 0
    for batch in [[{'input_ids': [1,2], 'labels': [1,2]}], [{'input_ids': [3,4], 'labels': [3,4]}]]:
        loss = sum(batch[0]['input_ids']) * 0.1
        total_loss += loss
    print(f"Epoch {epoch+1} loss: {total_loss:.2f}")

A Epoch 1 loss: 1.0\nEpoch 2 loss: 2.0

B Epoch 1 loss: 1.00

C Epoch 1 loss: 1.5

D Epoch 1 loss: 1.4

💡 Hint

Calculate the sum of input_ids for each batch and multiply by 0.1, then add them.

First batch: sum([1,2])=3 * 0.1 = 0.3; second batch: sum([3,4])=7 * 0.1 = 0.7; total_loss=1.0, formatted as 1.00.

❓ Hyperparameter

advanced

Choosing the best learning rate for fine-tuning

You are fine-tuning a QA model on a small dataset. Which learning rate is most appropriate to avoid overfitting and unstable training?

A 0.00001 (1e-5)

B 0.5

C 1.0

D 0.1

💡 Hint

Think about how large learning rates affect small datasets and model stability.

Very small learning rates like 1e-5 help fine-tune pre-trained models gently, preventing overfitting and unstable updates on small datasets.

❓ Metrics

advanced

Evaluating fine-tuned QA model performance

Which metric best measures the quality of answers generated by a fine-tuned extractive QA model?

A BLEU score

B Mean Squared Error (MSE)

C Exact Match (EM) score

D Perplexity

💡 Hint

Consider metrics that compare predicted answer spans to ground truth answers exactly.

Exact Match measures the percentage of predictions that exactly match the correct answer span, making it ideal for extractive QA evaluation.

🔧 Debug

expert

Debugging a fine-tuning script error

Given this snippet from a fine-tuning script, what error will it raise?

NLP

from transformers import AutoModelForQuestionAnswering
model = AutoModelForQuestionAnswering.from_pretrained('bert-base-uncased')
inputs = {'input_ids': [101, 2009, 2003, 1037, 2204, 2154, 102], 'attention_mask': [1,1,1,1,1,1,1]}
outputs = model(**inputs)
print(outputs.loss)

A No error, prints loss value

B TypeError: forward() missing 1 required positional argument: 'start_positions'

C RuntimeError: CUDA out of memory

D AttributeError: 'QuestionAnsweringModelOutput' object has no attribute 'loss'

💡 Hint

Check what inputs the model expects during training vs inference.

Without start_positions and end_positions, the model performs inference, returning logits with loss=None. No error; prints None.