BERT pre-training concept in NLP - Model Metrics & Evaluation

Metrics & Evaluation - BERT pre-training concept

Which metric matters for BERT pre-training and WHY

BERT pre-training uses two main tasks: Masked Language Modeling (MLM) and Next Sentence Prediction (NSP). For MLM, the key metric is cross-entropy loss, which measures how well the model predicts missing words. Lower loss means better understanding of language context.

For NSP, accuracy is important because it shows how well the model predicts if one sentence follows another. High accuracy means the model learns sentence relationships well.

These metrics matter because BERT learns language patterns without labeled data. Good MLM loss and NSP accuracy mean BERT can understand words and sentence order, which helps later tasks like question answering.

Confusion matrix for Next Sentence Prediction (NSP)

    | Predicted Yes | Predicted No |
    |---------------|--------------|
    | True Yes  (TP)| False No (FN)|
    | False Yes (FP)| True No  (TN)|

    Example:
    TP = 8000, FP = 2000
    FN = 1500, TN = 8500

    Total samples = 8000 + 2000 + 1500 + 8500 = 20000
    Accuracy = (TP + TN) / Total = (8000 + 8500) / 20000 = 0.825 or 82.5%
    

This matrix helps check how well BERT predicts if one sentence follows another during pre-training.

Precision vs Recall tradeoff in NSP task

In NSP, precision means when BERT says "sentence B follows sentence A," how often it is correct.

Recall means how many actual sentence pairs BERT correctly identifies as following each other.

High precision but low recall means BERT is careful but misses many true pairs.

High recall but low precision means BERT finds most pairs but makes many mistakes.

For BERT pre-training, balanced precision and recall are best to learn good sentence relations.

What good vs bad metrics look like for BERT pre-training

• Good MLM loss: Low cross-entropy loss (e.g., below 1.0) means BERT predicts masked words well.
• Bad MLM loss: High loss (e.g., above 2.0) means poor word prediction.
• Good NSP accuracy: Above 80% accuracy shows BERT understands sentence order well.
• Bad NSP accuracy: Around 50% means random guessing, no learning.

Common pitfalls in BERT pre-training metrics

• Ignoring MLM loss: Focusing only on NSP accuracy can miss poor word understanding.
• Data leakage: Using test data in pre-training can inflate metrics falsely.
• Overfitting: Very low MLM loss but poor downstream task performance means BERT memorized training data.
• Accuracy paradox: High NSP accuracy on unbalanced data can be misleading if one class dominates.

Self-check question

Your BERT pre-training shows 98% NSP accuracy but MLM loss is very high (3.5). Is this good?

Answer: No, because high NSP accuracy alone is not enough. The high MLM loss means BERT is not learning to predict masked words well. This hurts its language understanding. Both tasks must have good metrics for effective pre-training.

Key Result

BERT pre-training success depends on low MLM loss and balanced NSP accuracy to learn language context and sentence order.