Transformer decoder in PyTorch - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the main role of the Transformer decoder in a sequence-to-sequence model?

The Transformer decoder generates the output sequence step-by-step by attending to the encoder's output and previously generated tokens, enabling tasks like translation or text generation.

beginner

Name the three main sub-layers inside a Transformer decoder block.

1. Masked multi-head self-attention<br>2. Multi-head cross-attention (attends to encoder output)<br>3. Position-wise feed-forward network

intermediate

Why does the Transformer decoder use masked self-attention?

Masked self-attention prevents the decoder from 'seeing' future tokens during training, ensuring predictions are made only from past and current tokens, preserving the autoregressive property.

beginner

In PyTorch, which class is commonly used to implement a Transformer decoder layer?

torch.nn.TransformerDecoderLayer is used to build a single decoder layer, and torch.nn.TransformerDecoder stacks multiple such layers.

intermediate

What is the purpose of the cross-attention sub-layer in the Transformer decoder?

Cross-attention allows the decoder to focus on relevant parts of the encoder's output, helping it generate context-aware outputs based on the input sequence.

Which sub-layer in the Transformer decoder prevents the model from attending to future tokens?

A Feed-forward network

B Cross-attention

C Masked multi-head self-attention

D Layer normalization

Masked multi-head self-attention blocks future tokens to maintain autoregressive generation.

What does the cross-attention layer in the Transformer decoder attend to?

A Encoder outputs

B Future decoder tokens

C Previous decoder outputs only

D Input embeddings

Cross-attention attends to encoder outputs to incorporate input context.

In PyTorch, which class stacks multiple Transformer decoder layers?

A torch.nn.TransformerEncoder

B torch.nn.MultiheadAttention

C torch.nn.TransformerDecoderLayer

D torch.nn.TransformerDecoder

torch.nn.TransformerDecoder stacks multiple TransformerDecoderLayer instances.

Why is masking important in the Transformer decoder's self-attention?

A To prevent attending to future tokens

B To speed up training

C To reduce model size

D To normalize inputs

Masking ensures the decoder only uses past and current tokens for prediction.

Which of these is NOT a sub-layer in a Transformer decoder block?

A Masked multi-head self-attention

B Convolutional layer

C Position-wise feed-forward network

D Multi-head cross-attention

Transformer decoders do not use convolutional layers in their standard architecture.

Explain the architecture and function of a Transformer decoder layer.

Think about how the decoder processes previous outputs and encoder information step-by-step.

Describe how masking works in the Transformer decoder and why it is necessary.

Consider the order of token generation in language models.