BERT for text classification in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output shape of BERT's last hidden state?

Given the following PyTorch code snippet using a pretrained BERT model, what is the shape of the tensor last_hidden_state?

PyTorch

from transformers import BertModel, BertTokenizer
import torch

model = BertModel.from_pretrained('bert-base-uncased')
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

inputs = tokenizer('Hello world!', return_tensors='pt')
outputs = model(**inputs)
last_hidden_state = outputs.last_hidden_state
print(last_hidden_state.shape)

A torch.Size([1, 3, 768])

B torch.Size([1, 2, 768])

C torch.Size([1, 5, 768])

D torch.Size([1, 4, 768])

💡 Hint

Count the number of tokens including special tokens added by the tokenizer.

The tokenizer adds special tokens [CLS] and [SEP]. The text 'Hello world!' is tokenized into 'hello', 'world', and '!', for a total sequence length of 5 tokens: [CLS] hello world ! [SEP]. The batch size is 1, and hidden size for bert-base-uncased is 768.

❓ Model Choice

intermediate

Which BERT output is best for sentence classification?

When using BERT for text classification, which output from the model is typically used as input to the classification head?

A The pooled output corresponding to the [CLS] token

B The last hidden state of all tokens concatenated

C The embedding of the first token in the vocabulary

D The sum of all token embeddings

💡 Hint

Think about which token is designed to represent the whole sentence.

The [CLS] token's pooled output is designed to represent the entire input sequence and is commonly used for classification tasks.

❓ Hyperparameter

advanced

Choosing learning rate for fine-tuning BERT

Which learning rate is generally recommended when fine-tuning a pretrained BERT model for text classification?

A 5e-5

B 0.1

C 1.0

D 0.000001

💡 Hint

Fine-tuning large pretrained models usually requires a small learning rate.

Typical fine-tuning learning rates for BERT are in the range 2e-5 to 5e-5. Larger rates like 0.1 or 1.0 are too high and cause training instability.

❓ Metrics

advanced

Which metric is best for imbalanced text classification?

For a text classification task with highly imbalanced classes, which evaluation metric gives the most reliable performance measure?

A Precision

B F1-score

C Accuracy

D Mean Squared Error

💡 Hint

Consider a metric that balances false positives and false negatives.

F1-score balances precision and recall, making it better for imbalanced datasets than accuracy which can be misleading.

🔧 Debug

expert

Why does this BERT fine-tuning code raise a RuntimeError?

Consider this PyTorch code snippet for fine-tuning BERT. It raises a RuntimeError: 'element 0 of tensors does not require grad and does not have a grad_fn'. What is the cause?

PyTorch

from transformers import BertForSequenceClassification, BertTokenizer
import torch

model = BertForSequenceClassification.from_pretrained('bert-base-uncased')
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

inputs = tokenizer('Test input', return_tensors='pt')
labels = torch.tensor([1])

outputs = model(**inputs, labels=labels)
loss = outputs.loss
loss.backward()

A The inputs dictionary keys are incorrect

B The labels tensor is not on the same device as model inputs

C The tokenizer output is missing attention_mask

D The model is in eval mode instead of train mode

💡 Hint

Check if gradients are enabled during loss computation.

If the model is in evaluation mode (model.eval()), gradients are disabled, so loss does not track gradients causing this error.