BERT for text classification in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output shape of BERT's last hidden state?

Given the following PyTorch code snippet using a pretrained BERT model, what is the shape of the tensor last_hidden_state?

PyTorch

from transformers import BertModel, BertTokenizer
import torch

model = BertModel.from_pretrained('bert-base-uncased')
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

inputs = tokenizer('Hello world!', return_tensors='pt')
outputs = model(**inputs)
last_hidden_state = outputs.last_hidden_state
print(last_hidden_state.shape)

A torch.Size([1, 3, 768])

B torch.Size([1, 2, 768])

C torch.Size([1, 5, 768])

D torch.Size([1, 4, 768])

💡 Hint

Count the number of tokens including special tokens added by the tokenizer.

The tokenizer adds special tokens [CLS] and [SEP]. The text 'Hello world!' is tokenized into 'hello', 'world', and '!', for a total sequence length of 5 tokens: [CLS] hello world ! [SEP]. The batch size is 1, and hidden size for bert-base-uncased is 768.

❓ Model Choice

intermediate

Which BERT output is best for sentence classification?

When using BERT for text classification, which output from the model is typically used as input to the classification head?

A The pooled output corresponding to the [CLS] token

B The last hidden state of all tokens concatenated

C The embedding of the first token in the vocabulary

D The sum of all token embeddings

💡 Hint

Think about which token is designed to represent the whole sentence.

The [CLS] token's pooled output is designed to represent the entire input sequence and is commonly used for classification tasks.

❓ Hyperparameter

advanced

Choosing learning rate for fine-tuning BERT

Which learning rate is generally recommended when fine-tuning a pretrained BERT model for text classification?

A 5e-5

B 0.1

C 1.0

D 0.000001

💡 Hint

Fine-tuning large pretrained models usually requires a small learning rate.

Typical fine-tuning learning rates for BERT are in the range 2e-5 to 5e-5. Larger rates like 0.1 or 1.0 are too high and cause training instability.

❓ Metrics

advanced

Which metric is best for imbalanced text classification?

For a text classification task with highly imbalanced classes, which evaluation metric gives the most reliable performance measure?

A Precision

B F1-score

C Accuracy

D Mean Squared Error

💡 Hint

Consider a metric that balances false positives and false negatives.

F1-score balances precision and recall, making it better for imbalanced datasets than accuracy which can be misleading.

🔧 Debug

expert

Why does this BERT fine-tuning code raise a RuntimeError?

Consider this PyTorch code snippet for fine-tuning BERT. It raises a RuntimeError: 'element 0 of tensors does not require grad and does not have a grad_fn'. What is the cause?

PyTorch

from transformers import BertForSequenceClassification, BertTokenizer
import torch

model = BertForSequenceClassification.from_pretrained('bert-base-uncased')
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

inputs = tokenizer('Test input', return_tensors='pt')
labels = torch.tensor([1])

outputs = model(**inputs, labels=labels)
loss = outputs.loss
loss.backward()

A The inputs dictionary keys are incorrect

B The labels tensor is not on the same device as model inputs

C The tokenizer output is missing attention_mask

D The model is in eval mode instead of train mode

💡 Hint

Check if gradients are enabled during loss computation.

If the model is in evaluation mode (model.eval()), gradients are disabled, so loss does not track gradients causing this error.

Practice

1. What is the main advantage of using BERT for text classification compared to traditional bag-of-words models?

easy

A. BERT understands the context of words in a sentence.

B. BERT uses fewer parameters than bag-of-words models.

C. BERT requires no preprocessing of text data.

D. BERT only works with short texts.

Solution

1. Step 1: Understand BERT's design

   BERT reads words in context, meaning it looks at surrounding words to understand meaning better.

2. Step 2: Compare with bag-of-words

   Bag-of-words treats words independently, ignoring order and context, which limits understanding.

3. Final Answer:

   BERT understands the context of words in a sentence. -> Option A

4. Quick Check:

   Contextual understanding = A [OK]

Hint: Remember BERT reads words with context, unlike bag-of-words [OK]

Common Mistakes:

• Thinking BERT uses fewer parameters
• Assuming BERT needs no preprocessing
• Believing BERT only works on short texts

2. Which of the following is the correct way to load a pretrained BERT model for sequence classification in PyTorch?

easy

A. model = torch.load('bert-base-uncased')

B. model = BertTokenizer.from_pretrained('bert-base-uncased')

C. model = BertForSequenceClassification.from_pretrained('bert-base-uncased')

D. model = nn.Sequential('bert-base-uncased')

Solution

1. Step 1: Identify correct class for model loading

   BertForSequenceClassification is the class to load pretrained BERT models for classification tasks.

2. Step 2: Check method usage

   from_pretrained is the method to load pretrained weights by model name.

3. Final Answer:

   model = BertForSequenceClassification.from_pretrained('bert-base-uncased') -> Option C

4. Quick Check:

   Use BertForSequenceClassification.from_pretrained = A [OK]

Hint: Use from_pretrained with BertForSequenceClassification for classification models [OK]

Common Mistakes:

• Using BertTokenizer to load model weights
• Trying to load model with torch.load without saving first
• Using nn.Sequential incorrectly for BERT

3. Given the following code snippet, what will be the shape of the logits output?

from transformers import BertTokenizer, BertForSequenceClassification
import torch

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=3)

inputs = tokenizer(['Hello world!', 'BERT is great.'], padding=True, return_tensors='pt')
outputs = model(**inputs)
logits = outputs.logits
print(logits.shape)

medium

A. torch.Size([3, 2])

B. torch.Size([2, 3])

C. torch.Size([2, 768])

D. torch.Size([1, 3])

Solution

1. Step 1: Understand batch size and labels

   Input has 2 sentences, so batch size is 2. num_labels=3 means 3 classes.

2. Step 2: Check output logits shape

   Logits shape is (batch_size, num_labels), so (2, 3).

3. Final Answer:

   torch.Size([2, 3]) -> Option B

4. Quick Check:

   Batch size 2 and 3 labels = (2, 3) [OK]

Hint: Logits shape = (batch size, number of classes) [OK]

Common Mistakes:

• Confusing batch size and label dimension order
• Assuming output shape matches hidden size (768)
• Ignoring batch size in output shape

4. You run this code but get an error: TypeError: forward() got an unexpected keyword argument 'input_ids'. What is the likely cause?

outputs = model(input_ids=inputs['input_ids'], attention_mask=inputs['attention_mask'])

medium

A. The model variable is a BertTokenizer, not BertForSequenceClassification.

B. The inputs dictionary is missing 'input_ids' key.

C. The model was not loaded with from_pretrained.

D. The attention_mask argument is not supported by BERT models.

Solution

1. Step 1: Analyze error message

   The error says forward() got unexpected keyword 'input_ids', meaning the model does not accept this argument.

2. Step 2: Identify model type mismatch

   BertTokenizer does not have a forward method and does not accept input_ids; only BertForSequenceClassification does.

3. Final Answer:

   The model variable is a BertTokenizer, not BertForSequenceClassification. -> Option A

4. Quick Check:

   Wrong model type causes unexpected argument error = D [OK]

Hint: Check if model is BertForSequenceClassification, not tokenizer [OK]

Common Mistakes:

• Confusing tokenizer with model
• Assuming attention_mask is unsupported
• Thinking inputs dictionary is missing keys

5. You want to fine-tune BERT for a 4-class text classification task with imbalanced classes. Which approach helps improve model performance?

hard

A. Set num_labels=2 since 4 classes is too many for BERT.

B. Remove the attention_mask to speed up training.

C. Freeze all BERT layers and train only the final classifier layer.

D. Use weighted loss function to give more importance to minority classes.

Solution

1. Step 1: Understand class imbalance problem

   Imbalanced classes cause the model to favor majority classes, hurting minority class accuracy.

2. Step 2: Choose appropriate solution

   Weighted loss assigns higher penalty to errors on minority classes, helping balance learning.

3. Final Answer:

   Use weighted loss function to give more importance to minority classes. -> Option D

4. Quick Check:

   Weighted loss helps with imbalance = C [OK]

Hint: Apply weighted loss to handle class imbalance [OK]

Common Mistakes:

• Removing attention_mask harms input understanding
• Reducing num_labels loses class info
• Freezing all layers limits fine-tuning benefits