BERT fine-tuning for classification in NLP - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is BERT in the context of natural language processing?

BERT stands for Bidirectional Encoder Representations from Transformers. It is a model that understands language by looking at words before and after a target word, helping it grasp context better.

beginner

Why do we fine-tune BERT for classification tasks?

Fine-tuning adjusts BERT's pre-trained knowledge to a specific task, like classifying text, by training it on labeled examples so it learns to make predictions for that task.

intermediate

What is the role of the [CLS] token in BERT fine-tuning for classification?

The [CLS] token is a special token added at the start of input text. Its output embedding is used as a summary representation of the whole input for classification decisions.

intermediate

How is the output layer structured in BERT fine-tuning for a binary classification task?

A simple linear layer is added on top of BERT's [CLS] output embedding, followed by a sigmoid activation to predict the probability of the positive class.

beginner

What metrics are commonly used to evaluate BERT classification models?

Accuracy, precision, recall, and F1-score are common metrics. They measure how well the model predicts correct classes and balances false positives and negatives.

What does fine-tuning BERT involve?

A Training BERT from scratch on a large dataset

B Adjusting BERT's weights on a specific labeled dataset

C Using BERT without any changes

D Only changing the tokenizer

Fine-tuning means adjusting the pre-trained BERT model weights on a specific task dataset to improve performance.

Which token's output embedding is used for classification in BERT?

A [CLS]

B [PAD]

C [SEP]

D Last word token

The [CLS] token's output embedding summarizes the input and is used for classification.

What activation function is commonly used for binary classification output in BERT fine-tuning?

A Softmax

B ReLU

C Tanh

D Sigmoid

Sigmoid activation outputs a probability between 0 and 1 for binary classification.

Which metric is NOT typically used to evaluate classification models?

A Mean Squared Error

B Recall

C Accuracy

D F1-score

Mean Squared Error is used for regression, not classification.

What is the main advantage of BERT's bidirectional training?

A It reads text only from left to right

B It reads text only from right to left

C It understands context from both directions

D It ignores word order

BERT reads text in both directions to better understand context.

Explain the steps to fine-tune BERT for a text classification task.

Think about starting with BERT, adding a layer, training on examples, and checking results.

Describe why the [CLS] token is important in BERT fine-tuning for classification.

Consider how BERT summarizes input for decision making.

Practice

1. What is the main purpose of fine-tuning BERT for a classification task?

easy

A. To adapt BERT's knowledge to classify specific categories in your data

B. To train BERT from scratch on a large dataset

C. To reduce the size of the BERT model for faster inference

D. To convert text into images for classification

Solution

1. Step 1: Understand BERT's pretraining

   BERT is pretrained on general language tasks and needs adjustment for specific tasks like classification.

2. Step 2: Purpose of fine-tuning

   Fine-tuning adapts BERT's learned language understanding to classify categories in your dataset.

3. Final Answer:

   To adapt BERT's knowledge to classify specific categories in your data -> Option A

4. Quick Check:

   Fine-tuning = adapt BERT for classification [OK]

Hint: Fine-tuning means adjusting BERT for your task, not training from scratch [OK]

Common Mistakes:

• Thinking fine-tuning trains BERT from zero
• Confusing fine-tuning with model compression
• Assuming BERT outputs images

2. Which of the following is the correct way to tokenize text before feeding it to BERT in Python?

easy

A. tokens = text.split(' ')

B. tokens = tokenizer.encode_plus(text, return_tensors='pt')

C. tokens = tokenizer.tokenize(text)

D. tokens = text.lower()

Solution

1. Step 1: Identify proper BERT tokenization method

   BERT uses tokenizer.encode_plus to convert text into token IDs and attention masks.

2. Step 2: Compare options

   tokens = tokenizer.encode_plus(text, return_tensors='pt') uses encode_plus with return_tensors='pt' for PyTorch tensors, which is correct for BERT input.

3. Final Answer:

   tokens = tokenizer.encode_plus(text, return_tensors='pt') -> Option B

4. Quick Check:

   Use encode_plus for BERT tokenization [OK]

Hint: Use tokenizer.encode_plus or tokenizer() for BERT input [OK]

Common Mistakes:

• Using simple split instead of tokenizer
• Only tokenizing without encoding IDs
• Not returning tensors for model input

3. Given this code snippet for fine-tuning BERT, what will be the output of print(predictions.argmax(dim=1)) if the model predicts logits [[2.0, 1.0], [0.5, 1.5]] for two samples?

logits = torch.tensor([[2.0, 1.0], [0.5, 1.5]])
predictions = logits
print(predictions.argmax(dim=1))

medium

A. tensor([2, 1])

B. tensor([1, 0])

C. tensor([1, 1])

D. tensor([0, 1])

Solution

1. Step 1: Understand argmax(dim=1)

   Argmax along dim=1 finds the index of max value in each row (sample).

2. Step 2: Calculate argmax for each sample

   First row: max is 2.0 at index 0; second row: max is 1.5 at index 1.

3. Final Answer:

   tensor([0, 1]) -> Option D

4. Quick Check:

   Argmax per row = [0, 1] [OK]

Hint: Argmax dim=1 picks max index per sample row [OK]

Common Mistakes:

• Confusing dim=0 with dim=1
• Mixing up indices and values
• Expecting values instead of indices

4. You run this training loop snippet but get a runtime error: TypeError: forward() missing 1 required positional argument: 'labels'. What is the likely fix?

outputs = model(input_ids, attention_mask)
loss = outputs.loss
loss.backward()

medium

A. Pass labels to the model call: model(input_ids, attention_mask, labels=labels)

B. Remove loss.backward() call

C. Change input_ids to input_id

D. Call model with only input_ids

Solution

1. Step 1: Understand error cause

   The model expects labels to compute loss but they are missing in the call.

2. Step 2: Fix by passing labels

   Include labels argument in model call to get loss: model(input_ids, attention_mask, labels=labels).

3. Final Answer:

   Pass labels to the model call: model(input_ids, attention_mask, labels=labels) -> Option A

4. Quick Check:

   Missing labels argument causes loss error [OK]

Hint: Always pass labels to get loss during training [OK]

Common Mistakes:

• Ignoring the missing labels argument
• Removing backward call instead of fixing input
• Changing variable names incorrectly

5. You want to fine-tune BERT on a small dataset for sentiment classification. Which strategy helps avoid overfitting during training?

hard

A. Train BERT without tokenization to save time

B. Increase batch size to maximum and train longer

C. Use a small learning rate and add dropout layers

D. Remove the classification head and train only embeddings

Solution

1. Step 1: Identify overfitting risks

   Small datasets can cause the model to memorize instead of generalize.

2. Step 2: Apply regularization techniques

   Using a small learning rate and dropout helps the model learn smoothly and avoid overfitting.

3. Final Answer:

   Use a small learning rate and add dropout layers -> Option C

4. Quick Check:

   Small LR + dropout reduces overfitting [OK]

Hint: Small learning rate + dropout helps generalize on small data [OK]

Common Mistakes:

• Training longer without regularization
• Skipping tokenization
• Removing classification head incorrectly