RoBERTa and DistilBERT in NLP - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Difference in Pretraining Objectives

Which of the following best describes the main difference in the pretraining objectives between RoBERTa and DistilBERT?

A RoBERTa is trained only on next sentence prediction, while DistilBERT uses masked language modeling.

B RoBERTa uses dynamic masking with a masked language model objective, while DistilBERT uses a distillation loss to mimic BERT's outputs.

C RoBERTa uses autoregressive language modeling, while DistilBERT uses masked language modeling.

D RoBERTa uses a sequence-to-sequence objective, while DistilBERT uses a classification objective.

💡 Hint

Think about how RoBERTa improved BERT's training and how DistilBERT reduces model size.

RoBERTa improves BERT by using dynamic masking and removing next sentence prediction. DistilBERT is trained by distilling knowledge from BERT using a loss that mimics BERT's outputs.

❓ Predict Output

intermediate

Output Shape of RoBERTa Model

Given the following code snippet using Hugging Face Transformers, what is the shape of the last_hidden_state tensor?

NLP

from transformers import RobertaModel, RobertaTokenizer
import torch

tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
model = RobertaModel.from_pretrained('roberta-base')
inputs = tokenizer('Hello world!', return_tensors='pt')
outputs = model(**inputs)
last_hidden_state = outputs.last_hidden_state
print(last_hidden_state.shape)

A torch.Size([1, 5, 768])

B torch.Size([1, 3, 768])

C torch.Size([1, 4, 768])

D torch.Size([1, 4, 512])

💡 Hint

Count the tokens after tokenization including special tokens.

The tokenizer adds special tokens, so 'Hello world!' becomes 4 tokens. The hidden size of roberta-base is 768.

❓ Model Choice

advanced

Choosing a Model for Low-Latency Applications

You want to deploy a transformer model for real-time text classification on a mobile device with limited memory and CPU. Which model is the best choice?

A DistilBERT-base

B RoBERTa-base

C BERT-large

D RoBERTa-large

💡 Hint

Consider model size and speed for mobile deployment.

DistilBERT is a smaller, faster version of BERT designed for resource-constrained environments, making it suitable for mobile devices.

❓ Hyperparameter

advanced

Effect of Sequence Length on RoBERTa Training

When fine-tuning RoBERTa on a text classification task, increasing the maximum sequence length from 128 to 512 will most likely:

A Have no effect on training time or accuracy.

B Decrease training time because longer sequences are processed faster.

C Reduce memory usage by truncating sequences.

D Increase training time and memory usage but may improve accuracy on longer texts.

💡 Hint

Think about how sequence length affects computation in transformers.

Longer sequences require more memory and computation, increasing training time, but can capture more context improving accuracy on longer inputs.

❓ Metrics

expert

Comparing Model Performance Metrics

You fine-tune both RoBERTa-base and DistilBERT-base on the same sentiment analysis dataset. After evaluation, you get these results:

• RoBERTa-base: Accuracy=0.92, F1-score=0.91, Inference time=120ms
• DistilBERT-base: Accuracy=0.89, F1-score=0.88, Inference time=70ms

Which statement best summarizes the trade-off between these models?

A RoBERTa-base is more accurate but slower; DistilBERT is faster but slightly less accurate.

B DistilBERT is both more accurate and faster than RoBERTa-base.

C RoBERTa-base is faster and more accurate than DistilBERT.

D Both models have the same speed and accuracy.

💡 Hint

Look at both accuracy and inference time values.

RoBERTa-base achieves higher accuracy and F1 but has longer inference time. DistilBERT trades some accuracy for faster inference.