Transformer encoder in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output shape of Transformer encoder layer

Given the following PyTorch code snippet, what is the shape of the output tensor after passing through the Transformer encoder layer?

PyTorch

import torch
import torch.nn as nn

batch_size = 4
seq_length = 10
embedding_dim = 32

x = torch.rand(batch_size, seq_length, embedding_dim)
encoder_layer = nn.TransformerEncoderLayer(d_model=embedding_dim, nhead=4, batch_first=True)
output = encoder_layer(x)
print(output.shape)

A torch.Size([10, 4, 32])

B torch.Size([4, 10, 32])

C torch.Size([4, 32, 10])

D torch.Size([10, 32, 4])

💡 Hint

Remember that nn.TransformerEncoderLayer expects input shape (batch_size, seq_length, embedding_dim) and returns the same shape.

The Transformer encoder layer in PyTorch expects input of shape (batch_size, seq_length, embedding_dim) and outputs the same shape. So the output shape matches the input shape.

❓ Model Choice

intermediate

Choosing the number of attention heads

You want to create a Transformer encoder layer with embedding dimension 64. Which choice of number of attention heads is valid?

A 8

B 9

C 10

D 7

💡 Hint

The embedding dimension must be divisible by the number of attention heads.

The number of attention heads must divide the embedding dimension evenly. 64 divided by 8 is 8, which is an integer. Other options do not divide 64 evenly.

❓ Hyperparameter

advanced

Effect of increasing dropout in Transformer encoder

What is the most likely effect of increasing the dropout rate in a Transformer encoder layer during training?

A It speeds up training by skipping computations.

B It increases model capacity by adding more neurons.

C It reduces overfitting by randomly zeroing some activations, improving generalization.

D It always causes the model to underfit and perform worse.

💡 Hint

Dropout randomly disables parts of the network during training.

Increasing dropout helps prevent overfitting by forcing the model to not rely on any single neuron, which can improve generalization.

🔧 Debug

advanced

Identifying error in Transformer encoder input shape

What error will this code raise when running the Transformer encoder layer?

PyTorch

import torch
import torch.nn as nn

x = torch.rand(10, 4, 32)  # shape (seq_length, batch_size, embedding_dim)
encoder_layer = nn.TransformerEncoderLayer(d_model=32, nhead=4, batch_first=True)
output = encoder_layer(x)
print(output.shape)

A RuntimeError: Expected input of shape (batch_size, seq_length, embedding_dim)

B No error, output shape is torch.Size([10, 4, 32])

C TypeError: input tensor must be 2D

D ValueError: nhead must divide d_model

💡 Hint

Check the expected input shape for nn.TransformerEncoderLayer in PyTorch.

PyTorch's nn.TransformerEncoderLayer expects input shape (batch_size, seq_length, embedding_dim). The code provides (seq_length, batch_size, embedding_dim), causing a runtime error.

🧠 Conceptual

expert

Why use multi-head attention in Transformer encoder?

What is the main advantage of using multi-head attention instead of a single attention head in a Transformer encoder?

A It reduces the total number of parameters in the model.

B It prevents the model from learning positional information.

C It makes the model faster by parallelizing computations across heads.

D It allows the model to jointly attend to information from different representation subspaces at different positions.

💡 Hint

Think about how multiple attention heads help the model understand different aspects of the input.

Multi-head attention lets the model look at the input from multiple perspectives simultaneously, capturing diverse features and relationships.

Practice

1. What is the primary purpose of a Transformer encoder in PyTorch?

easy

A. To classify images using convolutional layers

B. To generate new sequences from random noise

C. To process sequences by attending to all positions simultaneously

D. To reduce the dimensionality of input data using convolution

Solution

1. Step 1: Understand the Transformer encoder role

   The Transformer encoder looks at all parts of the input sequence at once using self-attention, unlike sequential models.

2. Step 2: Compare options with this role

   Only To process sequences by attending to all positions simultaneously describes this behavior correctly. Other options describe unrelated tasks.

3. Final Answer:

   To process sequences by attending to all positions simultaneously -> Option C

4. Quick Check:

   Transformer encoder = attends all positions [OK]

Hint: Remember: Transformer encoder uses self-attention on sequences [OK]

Common Mistakes:

• Confusing encoder with decoder
• Thinking it generates sequences
• Mixing with convolutional networks

2. Which of the following is the correct way to create a Transformer encoder layer in PyTorch?

easy

A. torch.nn.Conv1d(in_channels=512, out_channels=8)

B. torch.nn.TransformerEncoderLayer(d_model=512, nhead=8)

C. torch.nn.TransformerDecoderLayer(d_model=512, nhead=8)

D. torch.nn.Linear(512, 8)

Solution

1. Step 1: Identify the correct PyTorch class for Transformer encoder layer

   The class to create a Transformer encoder layer is torch.nn.TransformerEncoderLayer.

2. Step 2: Check the parameters

   d_model and nhead are required parameters for this class, matching torch.nn.TransformerEncoderLayer(d_model=512, nhead=8).

3. Final Answer:

   torch.nn.TransformerEncoderLayer(d_model=512, nhead=8) -> Option B

4. Quick Check:

   Encoder layer class = TransformerEncoderLayer [OK]

Hint: Look for TransformerEncoderLayer class with d_model and nhead [OK]

Common Mistakes:

• Using DecoderLayer instead of EncoderLayer
• Confusing convolution or linear layers with Transformer layers
• Missing required parameters

3. Given the following code, what is the shape of the output tensor?

import torch
from torch.nn import TransformerEncoderLayer

layer = TransformerEncoderLayer(d_model=16, nhead=4)
input_tensor = torch.rand(10, 32, 16)  # (seq_len, batch_size, features)
output = layer(input_tensor)

medium

A. (32, 10, 16)

B. (32, 16, 10)

C. (10, 16, 32)

D. (10, 32, 16)

Solution

1. Step 1: Understand input shape format

   The input tensor shape is (sequence length, batch size, features) = (10, 32, 16).

2. Step 2: TransformerEncoderLayer output shape

   The output shape matches the input shape for TransformerEncoderLayer.

3. Final Answer:

   (10, 32, 16) -> Option D

4. Quick Check:

   Output shape = input shape (seq_len, batch, features) [OK]

Hint: Output shape matches input shape for Transformer encoder layers [OK]

Common Mistakes:

• Swapping batch size and sequence length
• Confusing feature dimension order
• Assuming output shape changes

4. What is wrong with this code snippet that tries to create a Transformer encoder with 2 layers?

import torch
from torch.nn import TransformerEncoder, TransformerEncoderLayer

layer = TransformerEncoderLayer(d_model=64, nhead=8)
encoder = TransformerEncoder(layer, num_layers=2)
input_tensor = torch.rand(20, 16, 64)
output = encoder(input_tensor)

medium

A. The input tensor shape is correct; no error

B. The number of heads (nhead) must be 1

C. The input tensor shape should be (batch_size, seq_len, features)

D. TransformerEncoderLayer requires an additional dropout parameter

Solution

1. Step 1: Check input tensor shape

   Input shape is (seq_len=20, batch_size=16, features=64), which is correct for TransformerEncoder.

2. Step 2: Verify other parameters

   nhead=8 is valid as 64 is divisible by 8; dropout is optional; no error in code.

3. Final Answer:

   The input tensor shape is correct; no error -> Option A

4. Quick Check:

   Input shape (seq_len, batch, features) is correct [OK]

Hint: Remember input shape is (seq_len, batch, features) for TransformerEncoder [OK]

Common Mistakes:

• Swapping batch and sequence dimensions
• Thinking dropout is mandatory
• Assuming nhead must be 1

5. You want to build a Transformer encoder in PyTorch that processes sequences of length 15 with batch size 8 and feature size 32. You want 3 encoder layers, each with 4 attention heads. Which code snippet correctly creates this model and processes a random input?

hard

A. import torch
from torch.nn import TransformerEncoder, TransformerEncoderLayer

layer = TransformerEncoderLayer(d_model=32, nhead=4)
encoder = TransformerEncoder(layer, num_layers=3)
input_tensor = torch.rand(15, 8, 32)
output = encoder(input_tensor)

B. import torch
from torch.nn import TransformerEncoder, TransformerEncoderLayer

layer = TransformerEncoderLayer(d_model=32, nhead=3)
encoder = TransformerEncoder(layer, num_layers=4)
input_tensor = torch.rand(8, 15, 32)
output = encoder(input_tensor)

C. import torch
from torch.nn import TransformerEncoder, TransformerEncoderLayer

layer = TransformerEncoderLayer(d_model=15, nhead=4)
encoder = TransformerEncoder(layer, num_layers=3)
input_tensor = torch.rand(15, 8, 32)
output = encoder(input_tensor)

D. import torch
from torch.nn import TransformerEncoder, TransformerEncoderLayer

layer = TransformerEncoderLayer(d_model=32, nhead=4)
encoder = TransformerEncoder(layer, num_layers=3)
input_tensor = torch.rand(8, 15, 32)
output = encoder(input_tensor)

Solution

1. Step 1: Check d_model and nhead parameters

   d_model must match feature size (32), and nhead must divide d_model (4 divides 32). import torch from torch.nn import TransformerEncoder, TransformerEncoderLayer layer = TransformerEncoderLayer(d_model=32, nhead=4) encoder = TransformerEncoder(layer, num_layers=3) input_tensor = torch.rand(15, 8, 32) output = encoder(input_tensor) matches this.

2. Step 2: Verify input tensor shape and num_layers

   Input shape should be (seq_len=15, batch=8, features=32). import torch from torch.nn import TransformerEncoder, TransformerEncoderLayer layer = TransformerEncoderLayer(d_model=32, nhead=4) encoder = TransformerEncoder(layer, num_layers=3) input_tensor = torch.rand(15, 8, 32) output = encoder(input_tensor) uses this shape and 3 layers as required.

3. Final Answer:

   Code snippet in option A correctly creates and processes the input -> Option A

4. Quick Check:

   d_model=32, nhead=4, input shape correct [OK]

Hint: Match d_model to features and input shape (seq_len, batch, features) [OK]

Common Mistakes:

• Swapping batch and sequence dimensions
• Using nhead that doesn't divide d_model
• Mismatching d_model and input feature size