BERT pre-training concept in NLP - Cheat Sheet & Quick Revision

Recall & Review

beginner

What does BERT stand for in NLP?

BERT stands for Bidirectional Encoder Representations from Transformers. It is a model designed to understand the context of words in both directions in a sentence.

beginner

What are the two main tasks used in BERT pre-training?

The two main tasks are:<br>1. Masked Language Modeling (MLM): Randomly hides some words and trains the model to predict them.<br>2. Next Sentence Prediction (NSP): Trains the model to understand if one sentence logically follows another.

intermediate

Why is BERT called 'bidirectional'?

Because BERT looks at the words before and after a target word at the same time during training. This helps it understand the full context, unlike older models that read text only left-to-right or right-to-left.

beginner

Explain Masked Language Modeling (MLM) in simple terms.

MLM is like a fill-in-the-blank game. Some words in a sentence are hidden, and BERT tries to guess those missing words using the surrounding words. This helps BERT learn word meanings and context.

intermediate

What is the purpose of Next Sentence Prediction (NSP) in BERT pre-training?

NSP teaches BERT to understand relationships between sentences. It learns to predict if one sentence naturally follows another, which helps in tasks like question answering and text summarization.

What does BERT use to understand the context of words?

A Bidirectional reading of sentences

B Only left-to-right reading

C Only right-to-left reading

D Random word order

BERT reads sentences in both directions to understand full context.

In Masked Language Modeling, what does BERT try to predict?

A The topic of the text

B The next sentence

C Hidden words in a sentence

D The length of the sentence

MLM hides some words and BERT predicts those missing words.

What is the goal of Next Sentence Prediction in BERT?

A Predict the next word in a sentence

B Predict if one sentence follows another

C Predict the sentiment of a sentence

D Predict the length of a paragraph

NSP helps BERT learn if one sentence logically follows another.

Why is BERT pre-trained before fine-tuning on specific tasks?

A To avoid training

B To memorize answers

C To reduce model size

D To learn general language understanding

Pre-training helps BERT learn language patterns useful for many tasks.

Which architecture does BERT use?

A Transformer Encoder

B Convolutional Neural Network

C Support Vector Machine

D Recurrent Neural Network

BERT is based on the Transformer encoder architecture.

Describe the two main pre-training tasks of BERT and why they are important.

Think about how BERT learns words and sentence order.

Explain why BERT's bidirectional approach helps it understand language better than previous models.

Consider how knowing words before and after helps guess meaning.