BERT for text classification in PyTorch - Cheat Sheet & Quick Revision

Recall & Review

beginner

What does BERT stand for and what is its main purpose?

BERT stands for Bidirectional Encoder Representations from Transformers. It is a model designed to understand the context of words in a sentence by looking at both left and right sides, helping machines understand language better.

beginner

Why is BERT called 'bidirectional'?

Because BERT reads the text from both directions (left to right and right to left) at the same time, it understands the full context of a word based on all surrounding words.

intermediate

What is the role of the [CLS] token in BERT for text classification?

The [CLS] token is added at the start of every input sentence. After processing, its output embedding is used as a summary representation of the whole sentence for classification tasks.

intermediate

How do you fine-tune BERT for a text classification task?

You add a simple classification layer on top of BERT's output (usually on the [CLS] token embedding) and train the whole model on your labeled data, adjusting weights to improve prediction accuracy.

beginner

What metric is commonly used to evaluate BERT's performance on text classification?

Accuracy is commonly used to measure how many texts BERT correctly classifies out of all examples. Other metrics like F1-score can also be used for imbalanced data.

What does the [SEP] token do in BERT input?

A Represents unknown words

B Separates two sentences in input

C Indicates padding tokens

D Marks the start of the sentence

The [SEP] token is used to separate two sentences or segments in BERT input, helping the model understand sentence boundaries.

Which part of BERT's output is used for classification tasks?

A Output embedding of the [CLS] token

B Output embedding of the last word

C Sum of all token embeddings

D Input token embeddings

The [CLS] token's output embedding is used as a summary representation for classification.

What is the main advantage of fine-tuning BERT instead of training from scratch?

A Only works for images

B Needs more data and time

C Does not improve accuracy

D Requires less data and time

Fine-tuning uses pre-trained knowledge, so it needs less data and training time to perform well.

Which optimizer is commonly used when fine-tuning BERT?

A RMSProp

B SGD

C AdamW

D Adagrad

AdamW optimizer is commonly used because it handles weight decay properly during fine-tuning.

What does 'tokenization' mean in BERT preprocessing?

A Splitting text into smaller pieces called tokens

B Converting text to uppercase

C Removing punctuation only

D Translating text to another language

Tokenization splits text into tokens (words or subwords) that BERT can understand.

Explain how BERT processes input text for classification, including tokenization, special tokens, and output usage.

Think about how BERT reads and prepares sentences before predicting.

Describe the steps to fine-tune a pre-trained BERT model on a new text classification dataset.

Focus on model modification, data preparation, training, and evaluation.

Practice

1. What is the main advantage of using BERT for text classification compared to traditional bag-of-words models?

easy

A. BERT understands the context of words in a sentence.

B. BERT uses fewer parameters than bag-of-words models.

C. BERT requires no preprocessing of text data.

D. BERT only works with short texts.

Solution

1. Step 1: Understand BERT's design

   BERT reads words in context, meaning it looks at surrounding words to understand meaning better.

2. Step 2: Compare with bag-of-words

   Bag-of-words treats words independently, ignoring order and context, which limits understanding.

3. Final Answer:

   BERT understands the context of words in a sentence. -> Option A

4. Quick Check:

   Contextual understanding = A [OK]

Hint: Remember BERT reads words with context, unlike bag-of-words [OK]

Common Mistakes:

• Thinking BERT uses fewer parameters
• Assuming BERT needs no preprocessing
• Believing BERT only works on short texts

2. Which of the following is the correct way to load a pretrained BERT model for sequence classification in PyTorch?

easy

A. model = torch.load('bert-base-uncased')

B. model = BertTokenizer.from_pretrained('bert-base-uncased')

C. model = BertForSequenceClassification.from_pretrained('bert-base-uncased')

D. model = nn.Sequential('bert-base-uncased')

Solution

1. Step 1: Identify correct class for model loading

   BertForSequenceClassification is the class to load pretrained BERT models for classification tasks.

2. Step 2: Check method usage

   from_pretrained is the method to load pretrained weights by model name.

3. Final Answer:

   model = BertForSequenceClassification.from_pretrained('bert-base-uncased') -> Option C

4. Quick Check:

   Use BertForSequenceClassification.from_pretrained = A [OK]

Hint: Use from_pretrained with BertForSequenceClassification for classification models [OK]

Common Mistakes:

• Using BertTokenizer to load model weights
• Trying to load model with torch.load without saving first
• Using nn.Sequential incorrectly for BERT

3. Given the following code snippet, what will be the shape of the logits output?

from transformers import BertTokenizer, BertForSequenceClassification
import torch

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=3)

inputs = tokenizer(['Hello world!', 'BERT is great.'], padding=True, return_tensors='pt')
outputs = model(**inputs)
logits = outputs.logits
print(logits.shape)

medium

A. torch.Size([3, 2])

B. torch.Size([2, 3])

C. torch.Size([2, 768])

D. torch.Size([1, 3])

Solution

1. Step 1: Understand batch size and labels

   Input has 2 sentences, so batch size is 2. num_labels=3 means 3 classes.

2. Step 2: Check output logits shape

   Logits shape is (batch_size, num_labels), so (2, 3).

3. Final Answer:

   torch.Size([2, 3]) -> Option B

4. Quick Check:

   Batch size 2 and 3 labels = (2, 3) [OK]

Hint: Logits shape = (batch size, number of classes) [OK]

Common Mistakes:

• Confusing batch size and label dimension order
• Assuming output shape matches hidden size (768)
• Ignoring batch size in output shape

4. You run this code but get an error: TypeError: forward() got an unexpected keyword argument 'input_ids'. What is the likely cause?

outputs = model(input_ids=inputs['input_ids'], attention_mask=inputs['attention_mask'])

medium

A. The model variable is a BertTokenizer, not BertForSequenceClassification.

B. The inputs dictionary is missing 'input_ids' key.

C. The model was not loaded with from_pretrained.

D. The attention_mask argument is not supported by BERT models.

Solution

1. Step 1: Analyze error message

   The error says forward() got unexpected keyword 'input_ids', meaning the model does not accept this argument.

2. Step 2: Identify model type mismatch

   BertTokenizer does not have a forward method and does not accept input_ids; only BertForSequenceClassification does.

3. Final Answer:

   The model variable is a BertTokenizer, not BertForSequenceClassification. -> Option A

4. Quick Check:

   Wrong model type causes unexpected argument error = D [OK]

Hint: Check if model is BertForSequenceClassification, not tokenizer [OK]

Common Mistakes:

• Confusing tokenizer with model
• Assuming attention_mask is unsupported
• Thinking inputs dictionary is missing keys

5. You want to fine-tune BERT for a 4-class text classification task with imbalanced classes. Which approach helps improve model performance?

hard

A. Set num_labels=2 since 4 classes is too many for BERT.

B. Remove the attention_mask to speed up training.

C. Freeze all BERT layers and train only the final classifier layer.

D. Use weighted loss function to give more importance to minority classes.

Solution

1. Step 1: Understand class imbalance problem

   Imbalanced classes cause the model to favor majority classes, hurting minority class accuracy.

2. Step 2: Choose appropriate solution

   Weighted loss assigns higher penalty to errors on minority classes, helping balance learning.

3. Final Answer:

   Use weighted loss function to give more importance to minority classes. -> Option D

4. Quick Check:

   Weighted loss helps with imbalance = C [OK]

Hint: Apply weighted loss to handle class imbalance [OK]

Common Mistakes:

• Removing attention_mask harms input understanding
• Reducing num_labels loses class info
• Freezing all layers limits fine-tuning benefits