BERT pre-training concept in NLP - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What is the main goal of BERT's Masked Language Model (MLM) during pre-training?

BERT uses a special pre-training task called Masked Language Model (MLM). What is the main goal of MLM?

A Predict randomly masked words in a sentence using context from both sides

B Classify the sentiment of a sentence as positive or negative

C Predict the next word in a sentence given all previous words

D Translate a sentence from one language to another

💡 Hint

Think about how BERT learns from words hidden in the middle of sentences.

MLM trains BERT to predict missing words by looking at words before and after the masked word, helping it understand context deeply.

🧠 Conceptual

intermediate

What is the purpose of the Next Sentence Prediction (NSP) task in BERT pre-training?

Besides MLM, BERT uses Next Sentence Prediction (NSP) during pre-training. What does NSP help BERT learn?

A To predict the sentiment of a sentence

B To determine if one sentence logically follows another

C To translate sentences between languages

D To generate new sentences from scratch

💡 Hint

Think about how BERT understands relationships between two sentences.

NSP trains BERT to decide if a second sentence naturally follows the first, helping it learn sentence relationships useful for tasks like question answering.

❓ Model Choice

advanced

Which architecture component enables BERT to use context from both left and right sides during MLM pre-training?

BERT can look at words before and after a masked word simultaneously. Which part of BERT's architecture allows this?

A Unidirectional LSTM layers

B Recurrent neural networks with attention

C Convolutional neural networks

D Bidirectional Transformer encoder layers

💡 Hint

Think about which architecture processes all words at once with attention.

BERT uses Transformer encoder layers that attend to all words in the input simultaneously, enabling bidirectional context understanding.

❓ Metrics

advanced

During BERT pre-training, which metric best indicates how well the model predicts masked tokens?

Which metric is commonly used to measure BERT's performance on the Masked Language Model task during pre-training?

A Mean squared error of token embeddings

B BLEU score for sentence generation

C Accuracy of predicting masked tokens

D F1 score for next sentence prediction

💡 Hint

Focus on how well the model guesses the hidden words correctly.

Accuracy measures the percentage of masked tokens correctly predicted, directly reflecting MLM task performance.

🔧 Debug

expert

What error will occur if BERT's input tokens are not properly masked during MLM pre-training?

Suppose you accidentally feed BERT input sequences without masking any tokens during MLM pre-training. What is the most likely outcome?

A The loss will be very low but the model won't learn to predict masked words

B The model will train normally with no issues

C A runtime error will occur due to missing mask tokens

D The model will overfit quickly and produce random predictions

💡 Hint

Think about what happens if the model never has to guess missing words.

If no tokens are masked, the model always sees the full input and does not learn to predict missing words, resulting in low loss but poor MLM learning.