Positional encoding in PyTorch

Introduction

Positional encoding helps a model know the order of words in a sentence. It adds position information to word data so the model can understand sequence.

When building a transformer model for language tasks like translation or text generation.

When you want the model to understand the order of words without using recurrent layers.

When processing sequences where position matters, like time series or sentences.

When you need a fixed way to add position info to input embeddings.

When training models that handle variable-length sequences but need position context.

Syntax

PyTorch

import torch
import torch.nn as nn
import math

class PositionalEncoding(nn.Module):
    def __init__(self, d_model, max_len=5000):
        super().__init__()
        pe = torch.zeros(max_len, d_model)
        position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)
        div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))
        pe[:, 0::2] = torch.sin(position * div_term)
        pe[:, 1::2] = torch.cos(position * div_term)
        pe = pe.unsqueeze(0)  # shape (1, max_len, d_model)
        self.register_buffer('pe', pe)

    def forward(self, x):
        x = x + self.pe[:, :x.size(1), :]
        return x

The positional encoding matrix uses sine and cosine functions of different frequencies.

It is added to the input embeddings to give each position a unique representation.

Examples

This example creates a positional encoder for embeddings of size 512 and applies it to a zero tensor representing a batch of 1 sequence of length 10.

PyTorch

pos_encoder = PositionalEncoding(d_model=512)
input_tensor = torch.zeros(1, 10, 512)  # batch_size=1, seq_len=10, embedding_dim=512
output = pos_encoder(input_tensor)

Here, the positional encoder is set for max length 100 and embedding size 256, applied to a random tensor with batch size 2 and sequence length 50.

PyTorch

pos_encoder = PositionalEncoding(d_model=256, max_len=100)
input_tensor = torch.randn(2, 50, 256)  # batch_size=2, seq_len=50
output = pos_encoder(input_tensor)

Sample Model

This program defines the positional encoding class, creates a small example input, applies positional encoding, and prints the result. The output shows the added sine and cosine values for each position.

PyTorch

import torch
import torch.nn as nn
import math

class PositionalEncoding(nn.Module):
    def __init__(self, d_model, max_len=5000):
        super().__init__()
        pe = torch.zeros(max_len, d_model)
        position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)
        div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))
        pe[:, 0::2] = torch.sin(position * div_term)
        pe[:, 1::2] = torch.cos(position * div_term)
        pe = pe.unsqueeze(0)  # shape (1, max_len, d_model)
        self.register_buffer('pe', pe)

    def forward(self, x):
        x = x + self.pe[:, :x.size(1), :]
        return x

# Create positional encoding for embedding size 6 and sequence length 4
pos_encoder = PositionalEncoding(d_model=6, max_len=10)

# Create a dummy input tensor (batch_size=1, seq_len=4, embedding_dim=6)
input_tensor = torch.zeros(1, 4, 6)

# Apply positional encoding
output = pos_encoder(input_tensor)

# Print the output tensor
print(output)

Important Notes

Positional encoding is fixed and does not change during training.

It allows the model to distinguish positions without using recurrent or convolutional layers.

Summary

Positional encoding adds position info to input embeddings using sine and cosine waves.

It helps models understand the order of words or tokens in sequences.

It is commonly used in transformer models for language and sequence tasks.

Practice

1. What is the main purpose of positional encoding in transformer models?

easy

A. To increase the number of layers in the model

B. To reduce the size of the input data

C. To add information about the order of tokens in a sequence

D. To normalize the input embeddings

Solution

1. Step 1: Understand the role of positional encoding

   Positional encoding adds information about the position of each token in the input sequence, which helps the model understand order. The other options relate to unrelated tasks such as increasing the number of layers, reducing input size, or normalizing embeddings.

2. Final Answer:

   To add information about the order of tokens in a sequence -> Option C

3. Quick Check:

   Positional encoding = order info [OK]

Hint: Positional encoding = position info added to embeddings [OK]

Common Mistakes:

• Thinking positional encoding reduces data size
• Confusing positional encoding with normalization
• Assuming it increases model layers

2. Which of the following is the correct way to compute the positional encoding for position pos and dimension i using sine and cosine functions in PyTorch?

easy

A. pos_enc[pos, i] = sin(pos * 10000^(i/d_model))

B. pos_enc[pos, 2*i] = sin(pos / 10000^(2*i/d_model)); pos_enc[pos, 2*i+1] = cos(pos / 10000^(2*i/d_model))

C. pos_enc[pos, i] = cos(pos / 10000^(i/d_model))

D. pos_enc[pos, 2*i] = cos(pos / 10000^(2*i/d_model)); pos_enc[pos, 2*i+1] = sin(pos / 10000^(2*i/d_model))

Solution

1. Step 1: Recall the positional encoding formula

   Positional encoding uses sine for even indices and cosine for odd indices with the formula: pos_enc[pos, 2*i] = sin(pos / 10000^(2*i/d_model)); pos_enc[pos, 2*i+1] = cos(pos / 10000^(2*i/d_model)).

2. Final Answer:

   pos_enc[pos, 2*i] = sin(pos / 10000^(2*i/d_model)); pos_enc[pos, 2*i+1] = cos(pos / 10000^(2*i/d_model)) -> Option B

3. Quick Check:

   Sine even, cosine odd = pos_enc[pos, 2*i] = sin(pos / 10000^(2*i/d_model)); pos_enc[pos, 2*i+1] = cos(pos / 10000^(2*i/d_model)) [OK]

Hint: Sine for even dims, cosine for odd dims in positional encoding [OK]

Common Mistakes:

• Swapping sine and cosine roles
• Using multiplication instead of division in exponent
• Applying sine or cosine to all dimensions only

3. Given the following PyTorch code snippet for positional encoding, what is the shape of the output tensor pos_encoding?

import torch
import math

def get_positional_encoding(seq_len, d_model):
    pos_encoding = torch.zeros(seq_len, d_model)
    for pos in range(seq_len):
        for i in range(0, d_model, 2):
            pos_encoding[pos, i] = math.sin(pos / (10000 ** (i / d_model)))
            if i + 1 < d_model:
                pos_encoding[pos, i + 1] = math.cos(pos / (10000 ** (i / d_model)))
    return pos_encoding

pos_encoding = get_positional_encoding(50, 512)

medium

A. [50, 512]

B. [512, 50]

C. [50, 256]

D. [512, 512]

Solution

1. Step 1: Analyze function input parameters and output shape

   The function takes seq_len=50 and d_model=512, creating a tensor of zeros with shape (50, 512). The loops fill values in this tensor without changing its shape, so output shape remains (50, 512).

2. Final Answer:

   [50, 512] -> Option A

3. Quick Check:

   Output shape = (seq_len, d_model) = [50, 512] [OK]

Hint: Output shape matches (sequence length, embedding dim) [OK]

Common Mistakes:

• Confusing dimensions order
• Thinking output is half the embedding size
• Assuming square matrix output

4. Identify the error in this PyTorch positional encoding code and select the fix:

import torch
import math

def positional_encoding(seq_len, d_model):
    pos_enc = torch.zeros(seq_len, d_model)
    for pos in range(seq_len):
        for i in range(d_model):
            pos_enc[pos, i] = math.sin(pos / 10000 ** (i / d_model))
    return pos_enc

pos_enc = positional_encoding(10, 16)

medium

A. Use sine for even indices and cosine for odd indices instead of sine only

B. Change math.sin to torch.sin for GPU compatibility

C. Initialize pos_enc with ones instead of zeros

D. Remove the inner loop and use vectorized operations

Solution

1. Step 1: Review positional encoding formula and identify fix

   Positional encoding uses sine for even dimensions and cosine for odd dimensions, but code uses sine only. Use sine for even indices and cosine for odd indices instead of sine only correctly fixes this.

2. Final Answer:

   Use sine for even indices and cosine for odd indices instead of sine only -> Option A

3. Quick Check:

   Sine and cosine alternate = Use sine for even indices and cosine for odd indices instead of sine only [OK]

Hint: Positional encoding alternates sine and cosine per dimension [OK]

Common Mistakes:

• Using sine only for all dimensions
• Ignoring cosine for odd indices
• Thinking initialization value causes error

5. You want to add positional encoding to a batch of input embeddings with shape (batch_size, seq_len, d_model) in PyTorch. Which approach correctly applies positional encoding to each sequence in the batch?

hard

A. Create a positional encoding tensor of shape (batch_size, d_model) and multiply element-wise with input embeddings

B. Create a positional encoding tensor of shape (seq_len,) and add it to input embeddings directly

C. Create a positional encoding tensor of shape (batch_size, seq_len, 1) and concatenate it to input embeddings

D. Create a positional encoding tensor of shape (seq_len, d_model) and add it to input embeddings using broadcasting

Solution

1. Step 1: Understand input and positional encoding shapes

   Input embeddings have shape (batch_size, seq_len, d_model). Positional encoding is per position and dimension, so shape (seq_len, d_model) fits. Adding it broadcasts over batch dimension correctly.

2. Final Answer:

   Create a positional encoding tensor of shape (seq_len, d_model) and add it to input embeddings using broadcasting -> Option D

3. Quick Check:

   Broadcast positional encoding over batch = Create a positional encoding tensor of shape (seq_len, d_model) and add it to input embeddings using broadcasting [OK]

Hint: Broadcast positional encoding (seq_len, d_model) over batch [OK]

Common Mistakes:

• Using positional encoding without seq_len dimension
• Multiplying instead of adding positional encoding
• Concatenating positional encoding incorrectly