Transformer decoder in PyTorch - Model Metrics & Evaluation

Metrics & Evaluation - Transformer decoder

Which metric matters for Transformer decoder and WHY

The Transformer decoder is often used in tasks like language generation or translation. Here, perplexity is a key metric. It measures how well the model predicts the next word. Lower perplexity means better predictions.

For classification tasks using Transformer decoders, accuracy, precision, and recall matter depending on the goal. For example, in text generation, accuracy of predicted tokens is important. In tasks like summarization, metrics like BLEU or ROUGE are used but are outside this scope.

Confusion matrix or equivalent visualization

For classification tasks, confusion matrix shows true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN). For example, if the Transformer decoder classifies tokens into categories:

      | Predicted Positive | Predicted Negative |
      |--------------------|--------------------|
      | TP                 | FP                 |
      | FN                 | TN                 |
    

Sum of TP + FP + TN + FN equals total tokens classified.

For language generation, confusion matrix is less common. Instead, we look at token-level accuracy or perplexity.

Precision vs Recall tradeoff with concrete examples

Imagine a Transformer decoder used for detecting spam in messages:

• High precision: Most messages marked as spam really are spam. Few good messages are wrongly flagged.
• High recall: Most spam messages are caught, but some good messages might be wrongly flagged.

If you want to avoid annoying users by wrongly blocking good messages, prioritize precision.

If you want to catch as much spam as possible, prioritize recall.

Transformer decoders can be tuned to balance this tradeoff by adjusting thresholds or training data.

What "good" vs "bad" metric values look like for Transformer decoder

For language generation:

• Good perplexity: Close to 1 (perfect prediction). Typical good models have perplexity between 10 and 50 depending on task.
• Bad perplexity: Very high (100+), meaning poor prediction.

For classification tasks:

• Good accuracy: High (above 90%) on balanced data.
• Bad accuracy: Close to random guessing (e.g., 50% for binary).
• Good precision and recall: Both above 80% usually indicate a balanced, effective model.
• Bad precision or recall: Very low values (below 50%) show model struggles.

Common pitfalls in metrics for Transformer decoder

• Accuracy paradox: High accuracy can be misleading if data is imbalanced (e.g., many non-spam messages).
• Data leakage: If training data leaks into test, metrics look better but model won't generalize.
• Overfitting indicators: Training loss very low but validation loss high means model memorizes training data.
• Ignoring sequence context: Evaluating token accuracy without considering sequence coherence can mislead.
• Using wrong metric: For generation, accuracy is less meaningful than perplexity or BLEU.

Self-check question

Your Transformer decoder model has 98% accuracy but only 12% recall on spam detection. Is it good for production? Why or why not?

Answer: No, it is not good. The model misses 88% of spam messages (low recall), so many spam messages get through. High accuracy is misleading because most messages are not spam, so the model guesses non-spam often. You need to improve recall to catch more spam.

Key Result

Perplexity is key for Transformer decoder language tasks; precision and recall balance is critical for classification uses.