Transformer encoder in PyTorch - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the main purpose of a Transformer encoder in machine learning?

A Transformer encoder processes input data by capturing relationships between all parts of the input simultaneously, helping models understand context and meaning without relying on sequence order alone.

beginner

What is 'self-attention' in the context of a Transformer encoder?

Self-attention is a mechanism where the model looks at all parts of the input to decide which parts are important to focus on when encoding each word or token.

intermediate

Name the two main sub-layers inside a Transformer encoder block.

The two main sub-layers are: 1) Multi-head self-attention layer, 2) Position-wise feed-forward neural network.

intermediate

Why do Transformer encoders use 'positional encoding'?

Because Transformers process all input tokens at once, positional encoding adds information about the order of tokens so the model knows the sequence position of each token.

beginner

In PyTorch, which class can be used to create a Transformer encoder layer?

You can use torch.nn.TransformerEncoderLayer to create a single encoder layer, and torch.nn.TransformerEncoder to stack multiple layers.

What does the self-attention mechanism in a Transformer encoder help the model do?

A Reduce the size of the input data

B Focus on important parts of the input sequence

C Sort the input tokens alphabetically

D Generate output tokens directly

Self-attention helps the model focus on important parts of the input sequence by weighing the relevance of each token to others.

Which component adds information about token order in a Transformer encoder?

A Positional encoding

B Layer normalization

C Feed-forward network

D Multi-head attention

Positional encoding adds information about the position of tokens since the Transformer processes tokens in parallel.

What is the role of the feed-forward network inside a Transformer encoder layer?

A To apply a simple neural network to each position independently

B To add positional information

C To combine outputs from multiple attention heads

D To normalize the input data

The feed-forward network applies a small neural network to each token's representation independently to add complexity and non-linearity.

In PyTorch, which class stacks multiple Transformer encoder layers?

A torch.nn.TransformerEncoderLayer

B torch.nn.MultiheadAttention

C torch.nn.TransformerEncoder

D torch.nn.Linear

torch.nn.TransformerEncoder stacks multiple TransformerEncoderLayer instances to build a full encoder.

Why do Transformer encoders process all tokens simultaneously instead of one by one?

A Because they cannot handle sequences

B To generate output faster

C To sort tokens before processing

D To reduce training time and capture global context

Processing all tokens simultaneously allows the model to learn relationships between all tokens and speeds up training.

Explain how self-attention works inside a Transformer encoder and why it is important.

Think about how the model decides what parts of the input to focus on.

Describe the main components of a Transformer encoder layer and their roles.

Consider the flow of data through the encoder block.

Practice

1. What is the primary purpose of a Transformer encoder in PyTorch?

easy

A. To classify images using convolutional layers

B. To generate new sequences from random noise

C. To process sequences by attending to all positions simultaneously

D. To reduce the dimensionality of input data using convolution

Solution

1. Step 1: Understand the Transformer encoder role

   The Transformer encoder looks at all parts of the input sequence at once using self-attention, unlike sequential models.

2. Step 2: Compare options with this role

   Only To process sequences by attending to all positions simultaneously describes this behavior correctly. Other options describe unrelated tasks.

3. Final Answer:

   To process sequences by attending to all positions simultaneously -> Option C

4. Quick Check:

   Transformer encoder = attends all positions [OK]

Hint: Remember: Transformer encoder uses self-attention on sequences [OK]

Common Mistakes:

• Confusing encoder with decoder
• Thinking it generates sequences
• Mixing with convolutional networks

2. Which of the following is the correct way to create a Transformer encoder layer in PyTorch?

easy

A. torch.nn.Conv1d(in_channels=512, out_channels=8)

B. torch.nn.TransformerEncoderLayer(d_model=512, nhead=8)

C. torch.nn.TransformerDecoderLayer(d_model=512, nhead=8)

D. torch.nn.Linear(512, 8)

Solution

1. Step 1: Identify the correct PyTorch class for Transformer encoder layer

   The class to create a Transformer encoder layer is torch.nn.TransformerEncoderLayer.

2. Step 2: Check the parameters

   d_model and nhead are required parameters for this class, matching torch.nn.TransformerEncoderLayer(d_model=512, nhead=8).

3. Final Answer:

   torch.nn.TransformerEncoderLayer(d_model=512, nhead=8) -> Option B

4. Quick Check:

   Encoder layer class = TransformerEncoderLayer [OK]

Hint: Look for TransformerEncoderLayer class with d_model and nhead [OK]

Common Mistakes:

• Using DecoderLayer instead of EncoderLayer
• Confusing convolution or linear layers with Transformer layers
• Missing required parameters

3. Given the following code, what is the shape of the output tensor?

import torch
from torch.nn import TransformerEncoderLayer

layer = TransformerEncoderLayer(d_model=16, nhead=4)
input_tensor = torch.rand(10, 32, 16)  # (seq_len, batch_size, features)
output = layer(input_tensor)

medium

A. (32, 10, 16)

B. (32, 16, 10)

C. (10, 16, 32)

D. (10, 32, 16)

Solution

1. Step 1: Understand input shape format

   The input tensor shape is (sequence length, batch size, features) = (10, 32, 16).

2. Step 2: TransformerEncoderLayer output shape

   The output shape matches the input shape for TransformerEncoderLayer.

3. Final Answer:

   (10, 32, 16) -> Option D

4. Quick Check:

   Output shape = input shape (seq_len, batch, features) [OK]

Hint: Output shape matches input shape for Transformer encoder layers [OK]

Common Mistakes:

• Swapping batch size and sequence length
• Confusing feature dimension order
• Assuming output shape changes

4. What is wrong with this code snippet that tries to create a Transformer encoder with 2 layers?

import torch
from torch.nn import TransformerEncoder, TransformerEncoderLayer

layer = TransformerEncoderLayer(d_model=64, nhead=8)
encoder = TransformerEncoder(layer, num_layers=2)
input_tensor = torch.rand(20, 16, 64)
output = encoder(input_tensor)

medium

A. The input tensor shape is correct; no error

B. The number of heads (nhead) must be 1

C. The input tensor shape should be (batch_size, seq_len, features)

D. TransformerEncoderLayer requires an additional dropout parameter

Solution

1. Step 1: Check input tensor shape

   Input shape is (seq_len=20, batch_size=16, features=64), which is correct for TransformerEncoder.

2. Step 2: Verify other parameters

   nhead=8 is valid as 64 is divisible by 8; dropout is optional; no error in code.

3. Final Answer:

   The input tensor shape is correct; no error -> Option A

4. Quick Check:

   Input shape (seq_len, batch, features) is correct [OK]

Hint: Remember input shape is (seq_len, batch, features) for TransformerEncoder [OK]

Common Mistakes:

• Swapping batch and sequence dimensions
• Thinking dropout is mandatory
• Assuming nhead must be 1

5. You want to build a Transformer encoder in PyTorch that processes sequences of length 15 with batch size 8 and feature size 32. You want 3 encoder layers, each with 4 attention heads. Which code snippet correctly creates this model and processes a random input?

hard

A. import torch
from torch.nn import TransformerEncoder, TransformerEncoderLayer

layer = TransformerEncoderLayer(d_model=32, nhead=4)
encoder = TransformerEncoder(layer, num_layers=3)
input_tensor = torch.rand(15, 8, 32)
output = encoder(input_tensor)

B. import torch
from torch.nn import TransformerEncoder, TransformerEncoderLayer

layer = TransformerEncoderLayer(d_model=32, nhead=3)
encoder = TransformerEncoder(layer, num_layers=4)
input_tensor = torch.rand(8, 15, 32)
output = encoder(input_tensor)

C. import torch
from torch.nn import TransformerEncoder, TransformerEncoderLayer

layer = TransformerEncoderLayer(d_model=15, nhead=4)
encoder = TransformerEncoder(layer, num_layers=3)
input_tensor = torch.rand(15, 8, 32)
output = encoder(input_tensor)

D. import torch
from torch.nn import TransformerEncoder, TransformerEncoderLayer

layer = TransformerEncoderLayer(d_model=32, nhead=4)
encoder = TransformerEncoder(layer, num_layers=3)
input_tensor = torch.rand(8, 15, 32)
output = encoder(input_tensor)

Solution

1. Step 1: Check d_model and nhead parameters

   d_model must match feature size (32), and nhead must divide d_model (4 divides 32). import torch from torch.nn import TransformerEncoder, TransformerEncoderLayer layer = TransformerEncoderLayer(d_model=32, nhead=4) encoder = TransformerEncoder(layer, num_layers=3) input_tensor = torch.rand(15, 8, 32) output = encoder(input_tensor) matches this.

2. Step 2: Verify input tensor shape and num_layers

   Input shape should be (seq_len=15, batch=8, features=32). import torch from torch.nn import TransformerEncoder, TransformerEncoderLayer layer = TransformerEncoderLayer(d_model=32, nhead=4) encoder = TransformerEncoder(layer, num_layers=3) input_tensor = torch.rand(15, 8, 32) output = encoder(input_tensor) uses this shape and 3 layers as required.

3. Final Answer:

   Code snippet in option A correctly creates and processes the input -> Option A

4. Quick Check:

   d_model=32, nhead=4, input shape correct [OK]

Hint: Match d_model to features and input shape (seq_len, batch, features) [OK]

Common Mistakes:

• Swapping batch and sequence dimensions
• Using nhead that doesn't divide d_model
• Mismatching d_model and input feature size