Positional encoding in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of Positional Encoding Tensor Shape

What is the shape of the positional encoding tensor generated by the following PyTorch code snippet?

PyTorch

import torch
import math

def positional_encoding(seq_len, d_model):
    pe = torch.zeros(seq_len, d_model)
    position = torch.arange(0, seq_len, dtype=torch.float).unsqueeze(1)
    div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))
    pe[:, 0::2] = torch.sin(position * div_term)
    pe[:, 1::2] = torch.cos(position * div_term)
    return pe

pe_tensor = positional_encoding(50, 512)
print(pe_tensor.shape)

A (50, 512)

B (512, 50)

C (50, 256)

D (512, 50, 2)

💡 Hint

Think about how the positional encoding is created for each position and each dimension.

The positional encoding tensor has shape (sequence length, model dimension). Here, sequence length is 50 and model dimension is 512, so the shape is (50, 512).

🧠 Conceptual

intermediate

Purpose of Positional Encoding in Transformers

Why do transformer models use positional encoding?

A To normalize the input data before feeding it to the model.

B To add information about the order of tokens since transformers have no built-in sequence order awareness.

C To randomly shuffle the input tokens to improve generalization.

D To reduce the size of the input embeddings for faster computation.

💡 Hint

Transformers process all tokens simultaneously without sequence order. How do they know token positions?

Transformers do not have recurrence or convolution, so they cannot inherently understand token order. Positional encoding adds unique position information to each token embedding.

❓ Hyperparameter

advanced

Effect of Changing Model Dimension in Positional Encoding

If you increase the model dimension (d_model) in the positional encoding function, what is the expected effect on the positional encoding vectors?

A The positional encoding vectors will become sparse with many zeros.

B The positional encoding vectors will become shorter, losing positional information.

C The positional encoding vectors will have more dimensions, allowing finer granularity in encoding positions.

D The positional encoding vectors will remain the same size but with different values.

💡 Hint

Consider what d_model controls in the positional encoding tensor shape.

Increasing d_model increases the number of dimensions in the positional encoding, allowing the model to encode position information with more detail.

🔧 Debug

advanced

Identify the Error in Positional Encoding Code

What error will the following PyTorch code raise when executed?

PyTorch

import torch
import math

def positional_encoding(seq_len, d_model):
    pe = torch.zeros(seq_len, d_model)
    position = torch.arange(0, seq_len).float().unsqueeze(1)
    div_term = torch.exp(torch.arange(0, d_model, 2) * (-math.log(10000.0) / d_model))
    pe[:, 0::2] = torch.sin(position * div_term)
    pe[:, 1::2] = torch.cos(position * div_term)
    return pe

pe_tensor = positional_encoding(50, 512)

A RuntimeError due to shape mismatch in assignment to pe[:, 0::2]

B SyntaxError due to missing colon in function definition

C TypeError because torch.arange returns a list, not a tensor

D No error, code runs successfully

💡 Hint

Check the data types and shapes of tensors used in multiplication and assignment.

torch.arange returns a tensor of integers by default. Multiplying integer tensor with float tensor without casting causes shape mismatch in assignment.

❓ Model Choice

expert

Choosing Positional Encoding Type for a Transformer Model

You want to build a transformer model for a task with very long sequences (e.g., 10,000 tokens). Which positional encoding approach is best to handle this scenario?

A Use learned positional embeddings with fixed maximum length of 512 tokens.

B Use random positional embeddings initialized at each training batch.

C Use no positional encoding and rely on attention alone.

D Use sinusoidal positional encoding which can generalize to longer sequences.

💡 Hint

Consider which encoding can extrapolate beyond training sequence lengths.

Sinusoidal positional encoding uses fixed mathematical functions that can generalize to any sequence length, unlike learned embeddings limited by training length.