BERT for text classification in PyTorch - Cheat Sheet & Quick Revision

Recall & Review

beginner

What does BERT stand for and what is its main purpose?

BERT stands for Bidirectional Encoder Representations from Transformers. It is a model designed to understand the context of words in a sentence by looking at both left and right sides, helping machines understand language better.

beginner

Why is BERT called 'bidirectional'?

Because BERT reads the text from both directions (left to right and right to left) at the same time, it understands the full context of a word based on all surrounding words.

intermediate

What is the role of the [CLS] token in BERT for text classification?

The [CLS] token is added at the start of every input sentence. After processing, its output embedding is used as a summary representation of the whole sentence for classification tasks.

intermediate

How do you fine-tune BERT for a text classification task?

You add a simple classification layer on top of BERT's output (usually on the [CLS] token embedding) and train the whole model on your labeled data, adjusting weights to improve prediction accuracy.

beginner

What metric is commonly used to evaluate BERT's performance on text classification?

Accuracy is commonly used to measure how many texts BERT correctly classifies out of all examples. Other metrics like F1-score can also be used for imbalanced data.

What does the [SEP] token do in BERT input?

A Represents unknown words

B Separates two sentences in input

C Indicates padding tokens

D Marks the start of the sentence

The [SEP] token is used to separate two sentences or segments in BERT input, helping the model understand sentence boundaries.

Which part of BERT's output is used for classification tasks?

A Output embedding of the [CLS] token

B Output embedding of the last word

C Sum of all token embeddings

D Input token embeddings

The [CLS] token's output embedding is used as a summary representation for classification.

What is the main advantage of fine-tuning BERT instead of training from scratch?

A Only works for images

B Needs more data and time

C Does not improve accuracy

D Requires less data and time

Fine-tuning uses pre-trained knowledge, so it needs less data and training time to perform well.

Which optimizer is commonly used when fine-tuning BERT?

A RMSProp

B SGD

C AdamW

D Adagrad

AdamW optimizer is commonly used because it handles weight decay properly during fine-tuning.

What does 'tokenization' mean in BERT preprocessing?

A Splitting text into smaller pieces called tokens

B Converting text to uppercase

C Removing punctuation only

D Translating text to another language

Tokenization splits text into tokens (words or subwords) that BERT can understand.

Explain how BERT processes input text for classification, including tokenization, special tokens, and output usage.

Think about how BERT reads and prepares sentences before predicting.

Describe the steps to fine-tune a pre-trained BERT model on a new text classification dataset.

Focus on model modification, data preparation, training, and evaluation.