BERT fine-tuning for classification in NLP - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is BERT in the context of natural language processing?

BERT stands for Bidirectional Encoder Representations from Transformers. It is a model that understands language by looking at words before and after a target word, helping it grasp context better.

beginner

Why do we fine-tune BERT for classification tasks?

Fine-tuning adjusts BERT's pre-trained knowledge to a specific task, like classifying text, by training it on labeled examples so it learns to make predictions for that task.

intermediate

What is the role of the [CLS] token in BERT fine-tuning for classification?

The [CLS] token is a special token added at the start of input text. Its output embedding is used as a summary representation of the whole input for classification decisions.

intermediate

How is the output layer structured in BERT fine-tuning for a binary classification task?

A simple linear layer is added on top of BERT's [CLS] output embedding, followed by a sigmoid activation to predict the probability of the positive class.

beginner

What metrics are commonly used to evaluate BERT classification models?

Accuracy, precision, recall, and F1-score are common metrics. They measure how well the model predicts correct classes and balances false positives and negatives.

What does fine-tuning BERT involve?

A Training BERT from scratch on a large dataset

B Adjusting BERT's weights on a specific labeled dataset

C Using BERT without any changes

D Only changing the tokenizer

Fine-tuning means adjusting the pre-trained BERT model weights on a specific task dataset to improve performance.

Which token's output embedding is used for classification in BERT?

A [CLS]

B [PAD]

C [SEP]

D Last word token

The [CLS] token's output embedding summarizes the input and is used for classification.

What activation function is commonly used for binary classification output in BERT fine-tuning?

A Softmax

B ReLU

C Tanh

D Sigmoid

Sigmoid activation outputs a probability between 0 and 1 for binary classification.

Which metric is NOT typically used to evaluate classification models?

A Mean Squared Error

B Recall

C Accuracy

D F1-score

Mean Squared Error is used for regression, not classification.

What is the main advantage of BERT's bidirectional training?

A It reads text only from left to right

B It reads text only from right to left

C It understands context from both directions

D It ignores word order

BERT reads text in both directions to better understand context.

Explain the steps to fine-tune BERT for a text classification task.

Think about starting with BERT, adding a layer, training on examples, and checking results.

Describe why the [CLS] token is important in BERT fine-tuning for classification.

Consider how BERT summarizes input for decision making.