Transformer encoder in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output shape of Transformer encoder layer

Given the following PyTorch code snippet, what is the shape of the output tensor after passing through the Transformer encoder layer?

PyTorch

import torch
import torch.nn as nn

batch_size = 4
seq_length = 10
embedding_dim = 32

x = torch.rand(batch_size, seq_length, embedding_dim)
encoder_layer = nn.TransformerEncoderLayer(d_model=embedding_dim, nhead=4, batch_first=True)
output = encoder_layer(x)
print(output.shape)

A torch.Size([10, 4, 32])

B torch.Size([4, 10, 32])

C torch.Size([4, 32, 10])

D torch.Size([10, 32, 4])

💡 Hint

Remember that nn.TransformerEncoderLayer expects input shape (batch_size, seq_length, embedding_dim) and returns the same shape.

The Transformer encoder layer in PyTorch expects input of shape (batch_size, seq_length, embedding_dim) and outputs the same shape. So the output shape matches the input shape.

❓ Model Choice

intermediate

Choosing the number of attention heads

You want to create a Transformer encoder layer with embedding dimension 64. Which choice of number of attention heads is valid?

A 8

B 9

C 10

D 7

💡 Hint

The embedding dimension must be divisible by the number of attention heads.

The number of attention heads must divide the embedding dimension evenly. 64 divided by 8 is 8, which is an integer. Other options do not divide 64 evenly.

❓ Hyperparameter

advanced

Effect of increasing dropout in Transformer encoder

What is the most likely effect of increasing the dropout rate in a Transformer encoder layer during training?

A It speeds up training by skipping computations.

B It increases model capacity by adding more neurons.

C It reduces overfitting by randomly zeroing some activations, improving generalization.

D It always causes the model to underfit and perform worse.

💡 Hint

Dropout randomly disables parts of the network during training.

Increasing dropout helps prevent overfitting by forcing the model to not rely on any single neuron, which can improve generalization.

🔧 Debug

advanced

Identifying error in Transformer encoder input shape

What error will this code raise when running the Transformer encoder layer?

PyTorch

import torch
import torch.nn as nn

x = torch.rand(10, 4, 32)  # shape (seq_length, batch_size, embedding_dim)
encoder_layer = nn.TransformerEncoderLayer(d_model=32, nhead=4, batch_first=True)
output = encoder_layer(x)
print(output.shape)

A RuntimeError: Expected input of shape (batch_size, seq_length, embedding_dim)

B No error, output shape is torch.Size([10, 4, 32])

C TypeError: input tensor must be 2D

D ValueError: nhead must divide d_model

💡 Hint

Check the expected input shape for nn.TransformerEncoderLayer in PyTorch.

PyTorch's nn.TransformerEncoderLayer expects input shape (batch_size, seq_length, embedding_dim). The code provides (seq_length, batch_size, embedding_dim), causing a runtime error.

🧠 Conceptual

expert

Why use multi-head attention in Transformer encoder?

What is the main advantage of using multi-head attention instead of a single attention head in a Transformer encoder?

A It reduces the total number of parameters in the model.

B It prevents the model from learning positional information.

C It makes the model faster by parallelizing computations across heads.

D It allows the model to jointly attend to information from different representation subspaces at different positions.

💡 Hint

Think about how multiple attention heads help the model understand different aspects of the input.

Multi-head attention lets the model look at the input from multiple perspectives simultaneously, capturing diverse features and relationships.