BERT pre-training concept in NLP - Model Pipeline Trace

Model Pipeline - BERT pre-training concept

BERT pre-training teaches a language model to understand words and sentences by guessing missing words and checking if sentences follow each other. This helps the model learn language patterns before using it for tasks like answering questions or translating.

Data Flow - 7 Stages

1Input Text

1000 sentences x variable length tokens→Collect raw sentences from a large text corpus→1000 sentences x variable length tokens

"The cat sat on the mat."

↓

2Tokenization and Masking

1000 sentences x variable length tokens→Split sentences into tokens and randomly mask 15% of tokens→1000 sentences x variable length tokens (with 15% tokens replaced by [MASK])

"The cat [MASK] on the mat."

↓

3Next Sentence Pairing

1000 sentences→Create pairs of sentences; 50% are actual next sentences, 50% random→1000 sentence pairs (sentence A + sentence B)

["The cat sat on the mat.", "It was sunny outside."] (random pair)

↓

4Input Embeddings

1000 sentence pairs x tokens→Convert tokens to vectors including position and segment info→1000 sentence pairs x tokens x embedding size (e.g., 768)

Vector representation of "The cat [MASK] on the mat."

↓

5Transformer Encoder Layers

1000 sentence pairs x tokens x embedding size→Process embeddings through multiple transformer layers to learn context→1000 sentence pairs x tokens x embedding size

Contextualized vectors for each token

↓

6Masked Language Model (MLM) Prediction

1000 sentence pairs x tokens x embedding size→Predict original tokens for masked positions→1000 sentence pairs x masked tokens x vocabulary size

Prediction probabilities for masked token '[MASK]'

↓

7Next Sentence Prediction (NSP)

1000 sentence pairs x embedding size→Predict if sentence B follows sentence A→1000 sentence pairs x 2 classes (IsNext, NotNext)

Probability that sentence B follows sentence A

Training Trace - Epoch by Epoch

Loss
1.2 |****
1.0 |***
0.8 |**
0.6 |*
0.4 | 
    +----------------
     1 2 3 4 5 Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	1.2	0.55	Model starts learning to predict masked words and sentence order
2	0.9	0.65	Loss decreases as model improves predictions
3	0.7	0.75	Accuracy steadily increases, model understands context better
4	0.55	0.82	Model converges, good balance between MLM and NSP tasks
5	0.45	0.87	Final epoch shows strong language understanding

Prediction Trace - 5 Layers

Layer 1: Input Tokenization and Masking

Layer 2: Embedding Layer

Layer 3: Transformer Encoder Layers

Layer 4: Masked Language Model Prediction

Layer 5: Next Sentence Prediction

Model Quiz - 3 Questions

What does the masked language model task teach BERT?

A To predict missing words in sentences

B To translate sentences into another language

C To summarize long paragraphs

D To classify images

Key Insight

BERT pre-training uses two simple but powerful tasks—guessing missing words and checking sentence order—to help the model learn deep language understanding. This foundation allows BERT to perform well on many language tasks after fine-tuning.