Transformer decoder in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output shape of Transformer decoder layer

Given the following PyTorch code snippet for a Transformer decoder layer, what is the shape of the output tensor?

PyTorch

import torch
import torch.nn as nn

decoder_layer = nn.TransformerDecoderLayer(d_model=512, nhead=8)
memory = torch.rand(10, 32, 512)  # (sequence_length, batch_size, d_model)
tgt = torch.rand(20, 32, 512)     # (target_sequence_length, batch_size, d_model)
output = decoder_layer(tgt, memory)
print(output.shape)

A torch.Size([20, 32, 512])

B torch.Size([10, 32, 512])

C torch.Size([32, 20, 512])

D torch.Size([20, 512, 32])

💡 Hint

Remember the output shape matches the target input shape for the decoder layer.

The Transformer decoder layer outputs a tensor with the same shape as the target input: (target_sequence_length, batch_size, d_model). Here, tgt has shape (20, 32, 512), so output shape is (20, 32, 512).

🧠 Conceptual

intermediate

Purpose of the memory input in Transformer decoder

In a Transformer decoder, what is the role of the 'memory' input?

A It stores the previous decoder outputs for autoregressive generation.

B It provides the encoded information from the encoder to guide decoding.

C It contains the positional encodings for the target sequence.

D It is used to compute the loss during training.

💡 Hint

Think about how the decoder uses information from the encoder.

The 'memory' input is the output of the encoder. The decoder attends to this memory to incorporate encoded source information when generating outputs.

❓ Hyperparameter

advanced

Effect of increasing nhead in Transformer decoder

What is the effect of increasing the 'nhead' parameter in a Transformer decoder layer?

A It increases the batch size during training.

B It increases the size of the feedforward network inside the decoder layer.

C It increases the number of decoder layers stacked.

D It increases the number of attention heads, allowing the model to focus on more representation subspaces.

💡 Hint

Recall what 'nhead' controls in multi-head attention.

'nhead' specifies how many parallel attention heads the model uses. More heads allow the model to attend to different parts of the input simultaneously.

🔧 Debug

advanced

Identifying error in Transformer decoder usage

What error will this code raise when running the Transformer decoder layer?

PyTorch

import torch
import torch.nn as nn

decoder_layer = nn.TransformerDecoderLayer(d_model=256, nhead=4)
memory = torch.rand(15, 16, 256)
tgt = torch.rand(15, 16, 128)  # Incorrect d_model size
output = decoder_layer(tgt, memory)

A No error, code runs successfully

B SyntaxError because of missing parentheses

C RuntimeError due to mismatched feature dimensions between tgt and memory

D TypeError because tgt is not a tensor

💡 Hint

Check the last dimension sizes of tgt and memory tensors.

The decoder layer expects tgt and memory to have the same feature dimension (d_model). Here, tgt has 128 features but memory has 256, causing a RuntimeError.

❓ Metrics

expert

Choosing the best metric for Transformer decoder language generation

Which metric is most appropriate to evaluate the quality of text generated by a Transformer decoder in a language translation task?

A BLEU score measuring n-gram overlap with reference translations

B Mean Squared Error between generated and reference token embeddings

C Accuracy of predicting the next token in the sequence

D F1 score computed on binary classification labels

💡 Hint

Think about how translation quality is commonly measured.

BLEU score compares the generated text to reference translations by measuring n-gram overlaps, which is standard for translation tasks.