Transformer decoder in PyTorch - Model Metrics & Evaluation

Metrics & Evaluation - Transformer decoder

Which metric matters for Transformer decoder and WHY

The Transformer decoder is often used in tasks like language generation or translation. Here, perplexity is a key metric. It measures how well the model predicts the next word. Lower perplexity means better predictions.

For classification tasks using Transformer decoders, accuracy, precision, and recall matter depending on the goal. For example, in text generation, accuracy of predicted tokens is important. In tasks like summarization, metrics like BLEU or ROUGE are used but are outside this scope.

Confusion matrix or equivalent visualization

For classification tasks, confusion matrix shows true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN). For example, if the Transformer decoder classifies tokens into categories:

      | Predicted Positive | Predicted Negative |
      |--------------------|--------------------|
      | TP                 | FP                 |
      | FN                 | TN                 |
    

Sum of TP + FP + TN + FN equals total tokens classified.

For language generation, confusion matrix is less common. Instead, we look at token-level accuracy or perplexity.

Precision vs Recall tradeoff with concrete examples

Imagine a Transformer decoder used for detecting spam in messages:

• High precision: Most messages marked as spam really are spam. Few good messages are wrongly flagged.
• High recall: Most spam messages are caught, but some good messages might be wrongly flagged.

If you want to avoid annoying users by wrongly blocking good messages, prioritize precision.

If you want to catch as much spam as possible, prioritize recall.

Transformer decoders can be tuned to balance this tradeoff by adjusting thresholds or training data.

What "good" vs "bad" metric values look like for Transformer decoder

For language generation:

• Good perplexity: Close to 1 (perfect prediction). Typical good models have perplexity between 10 and 50 depending on task.
• Bad perplexity: Very high (100+), meaning poor prediction.

For classification tasks:

• Good accuracy: High (above 90%) on balanced data.
• Bad accuracy: Close to random guessing (e.g., 50% for binary).
• Good precision and recall: Both above 80% usually indicate a balanced, effective model.
• Bad precision or recall: Very low values (below 50%) show model struggles.

Common pitfalls in metrics for Transformer decoder

• Accuracy paradox: High accuracy can be misleading if data is imbalanced (e.g., many non-spam messages).
• Data leakage: If training data leaks into test, metrics look better but model won't generalize.
• Overfitting indicators: Training loss very low but validation loss high means model memorizes training data.
• Ignoring sequence context: Evaluating token accuracy without considering sequence coherence can mislead.
• Using wrong metric: For generation, accuracy is less meaningful than perplexity or BLEU.

Self-check question

Your Transformer decoder model has 98% accuracy but only 12% recall on spam detection. Is it good for production? Why or why not?

Answer: No, it is not good. The model misses 88% of spam messages (low recall), so many spam messages get through. High accuracy is misleading because most messages are not spam, so the model guesses non-spam often. You need to improve recall to catch more spam.

Key Result

Perplexity is key for Transformer decoder language tasks; precision and recall balance is critical for classification uses.

Practice

1. What is the main role of the Transformer decoder in a sequence-to-sequence model?

easy

A. To reduce the dimensionality of input data

B. To encode input sequences into fixed-size vectors

C. To perform image classification tasks

D. To generate output tokens step-by-step using previous outputs and encoder info

Solution

1. Step 1: Understand the Transformer decoder purpose

   The Transformer decoder generates output sequences by attending to previous outputs and encoder outputs.

2. Step 2: Compare options with decoder role

   Only To generate output tokens step-by-step using previous outputs and encoder info describes generating output tokens step-by-step using previous outputs and encoder info, which matches the decoder's role.

3. Final Answer:

   To generate output tokens step-by-step using previous outputs and encoder info -> Option D

4. Quick Check:

   Decoder generates outputs stepwise = D [OK]

Hint: Decoder creates outputs using past outputs and encoder data [OK]

Common Mistakes:

• Confusing encoder and decoder roles
• Thinking decoder encodes inputs
• Assuming decoder reduces data size

2. Which of the following is the correct way to create a Transformer decoder layer in PyTorch?

easy

A. torch.nn.TransformerDecoderLayer(d_model=512, nhead=8)

B. torch.nn.TransformerEncoderLayer(d_model=512, nhead=8)

C. torch.nn.TransformerDecoder(d_model=512, nhead=8)

D. torch.nn.TransformerLayer(d_model=512, nhead=8)

Solution

1. Step 1: Identify the correct PyTorch class for a decoder layer

   PyTorch provides torch.nn.TransformerDecoderLayer to create a single decoder layer.

2. Step 2: Check options for correct class and parameters

   torch.nn.TransformerDecoderLayer(d_model=512, nhead=8) uses torch.nn.TransformerDecoderLayer with correct parameters. Others are either encoder layers or invalid classes.

3. Final Answer:

   torch.nn.TransformerDecoderLayer(d_model=512, nhead=8) -> Option A

4. Quick Check:

   Decoder layer class is TransformerDecoderLayer = A [OK]

Hint: Decoder layer class ends with 'DecoderLayer' in PyTorch [OK]

Common Mistakes:

• Using EncoderLayer instead of DecoderLayer
• Trying to instantiate TransformerDecoder directly
• Using non-existent class names

3. Given the code below, what is the shape of the output tensor?

import torch
from torch.nn import TransformerDecoder, TransformerDecoderLayer

layer = TransformerDecoderLayer(d_model=256, nhead=8)
decoder = TransformerDecoder(layer, num_layers=3)

memory = torch.rand(10, 32, 256)  # (source_seq_len, batch_size, d_model)
tgt = torch.rand(20, 32, 256)     # (target_seq_len, batch_size, d_model)

output = decoder(tgt, memory)

medium

A. (32, 20, 256)

B. (10, 32, 256)

C. (20, 32, 256)

D. (20, 256, 32)

Solution

1. Step 1: Understand input and output shapes for TransformerDecoder

   The decoder input shape is (target_seq_len, batch_size, d_model). The output shape matches the target input shape.

2. Step 2: Check given shapes and output

   Target input 'tgt' has shape (20, 32, 256), so output will have the same shape.

3. Final Answer:

   (20, 32, 256) -> Option C

4. Quick Check:

   Output shape matches target input shape = C [OK]

Hint: Output shape matches target input shape in decoder [OK]

Common Mistakes:

• Confusing source and target sequence lengths
• Mixing batch size and feature dimensions
• Assuming output shape changes after decoding

4. What is wrong with this PyTorch Transformer decoder usage?

import torch
from torch.nn import TransformerDecoder, TransformerDecoderLayer

layer = TransformerDecoderLayer(d_model=128, nhead=4)
decoder = TransformerDecoder(layer, num_layers=2)

memory = torch.rand(15, 16, 128)
tgt = torch.rand(15, 16, 64)  # Incorrect feature size

output = decoder(tgt, memory)

medium

A. The memory tensor has wrong batch size

B. The target tensor 'tgt' has wrong feature dimension

C. The number of decoder layers is too low

D. The nhead parameter should be equal to batch size

Solution

1. Step 1: Check feature dimensions of inputs

   The decoder expects target and memory tensors to have the same feature dimension (d_model=128).

2. Step 2: Identify mismatch in target tensor

   Target tensor 'tgt' has feature size 64, which does not match d_model=128, causing an error.

3. Final Answer:

   The target tensor 'tgt' has wrong feature dimension -> Option B

4. Quick Check:

   Feature size mismatch in target tensor = A [OK]

Hint: Check if target and memory feature sizes match d_model [OK]

Common Mistakes:

• Ignoring feature dimension mismatch
• Assuming batch size must match nhead
• Thinking number of layers causes this error

5. You want to build a Transformer decoder that generates text step-by-step using previous outputs and encoder memory. Which of the following is the best approach to implement the decoder's forward pass in PyTorch?

hard

A. Use a TransformerDecoder with masked target input to prevent attending to future tokens

B. Use a TransformerDecoder without masking so it sees all target tokens

C. Use only a TransformerEncoder for generation

D. Use a TransformerDecoder but feed the entire output sequence at once without masking

Solution

1. Step 1: Understand the need for masking in decoding

   During generation, the decoder must not look at future tokens to avoid cheating, so masking is required.

2. Step 2: Identify correct usage in PyTorch

   Using TransformerDecoder with a mask on target input prevents attending to future tokens, enabling step-by-step generation.

3. Final Answer:

   Use a TransformerDecoder with masked target input to prevent attending to future tokens -> Option A

4. Quick Check:

   Mask future tokens in decoder input = B [OK]

Hint: Mask future tokens in decoder input for stepwise generation [OK]

Common Mistakes:

• Not using masks and allowing future token attention
• Using encoder only for generation
• Feeding full output sequence without masking