Positional encoding in PyTorch - Deep Dive

Overview - Positional encoding

What is it?

Positional encoding is a way to add information about the order of words or tokens in a sequence to a model. Since some models, like transformers, do not process data in order, positional encoding helps them understand the position of each token. It creates a unique pattern for each position that the model can learn from. This allows the model to use the order of words to make better predictions.

Why it matters

Without positional encoding, models that process sequences all at once would treat the input tokens as if they were unordered, like a bag of words. This would make it impossible to understand sentences or time series where order matters. Positional encoding solves this by giving the model a sense of position, enabling it to learn relationships that depend on order, such as grammar or time dependencies.

Where it fits

Before learning positional encoding, you should understand basic neural networks and the transformer architecture. After mastering positional encoding, you can explore advanced transformer models, attention mechanisms, and sequence modeling tasks like language translation or time series forecasting.

Mental Model

Core Idea

Positional encoding adds unique position information to each token so models can understand the order in sequences that process data all at once.

Think of it like...

It's like adding a numbered label to each book on a shelf so you know their order, even if you look at all books at once.

Sequence: [Token1] [Token2] [Token3] ...
Positions:   1        2        3   ...

Positional Encoding:
  Token1 + Pos1 → Vector with unique pattern
  Token2 + Pos2 → Vector with unique pattern
  Token3 + Pos3 → Vector with unique pattern

Model input = Token embedding + Positional encoding

Build-Up - 7 Steps

1

Foundation

Why order matters in sequences

Concept: Sequences have order, and understanding this order is key to meaning.

Imagine the sentence 'I love cats' versus 'Cats love I'. The words are the same but the meaning changes because of order. Models need to know this order to understand language or time series data.

Result

Recognizing that order changes meaning helps us see why models must know token positions.

Understanding that sequence order carries meaning is the foundation for why positional encoding is necessary.

2

Foundation

Limitations of transformer input processing

Concept: Transformers process all tokens simultaneously, losing natural order information.

Unlike RNNs that read tokens one by one, transformers look at all tokens at once. This parallel processing is fast but means the model doesn't know which token came first unless we add position info.

Result

Seeing that transformers lack built-in order awareness shows the need for positional encoding.

Knowing how transformers process input reveals why positional encoding is essential to restore order information.

3

Intermediate

How positional encoding represents positions

🤔Before reading on: do you think positional encoding uses learned numbers or fixed math formulas? Commit to your answer.

Concept: Positional encoding creates unique vectors for each position using sine and cosine functions at different frequencies.

Instead of learning position vectors, the original transformer uses fixed sine and cosine waves to encode positions. Each dimension of the positional vector uses a different frequency, creating unique patterns for each position.

Result

Each position gets a unique, continuous vector that the model can add to token embeddings.

Understanding the use of sine and cosine functions explains how positional encoding can represent infinite positions without learning each one.

4

Intermediate

Adding positional encoding to token embeddings

🤔Before reading on: do you think positional encoding replaces token embeddings or is combined with them? Commit to your answer.

Concept: Positional encoding vectors are added to token embeddings to create input vectors with both content and position info.

For each token, its embedding vector is summed with its positional encoding vector. This combined vector is then fed into the transformer layers, allowing the model to use both word meaning and position.

Result

The model input contains both what the token is and where it is in the sequence.

Knowing that positional encoding is added, not replaced, preserves token meaning while adding order information.

5

Intermediate

Implementing positional encoding in PyTorch

🤔Before reading on: do you think positional encoding is a fixed tensor or learned parameter in PyTorch? Commit to your answer.

Concept: Positional encoding can be implemented as a fixed tensor using sine and cosine functions, added to embeddings during training.

Here is a PyTorch example creating positional encoding: import torch import math def get_positional_encoding(seq_len, d_model): pe = torch.zeros(seq_len, d_model) position = torch.arange(0, seq_len, dtype=torch.float).unsqueeze(1) div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model)) pe[:, 0::2] = torch.sin(position * div_term) pe[:, 1::2] = torch.cos(position * div_term) return pe pos_encoding = get_positional_encoding(50, 512) # 50 tokens, 512 dims This tensor is added to token embeddings before input to the model.

Result

A fixed positional encoding tensor is created that can be reused for any input batch.

Seeing the code demystifies how positional encoding vectors are generated and applied in practice.

6

Advanced

Learned vs fixed positional encoding tradeoffs

🤔Before reading on: do you think learned positional encodings always perform better than fixed ones? Commit to your answer.

Concept: Positional encoding can be fixed (sine/cosine) or learned parameters, each with pros and cons.

Fixed encodings provide smooth, continuous position info and generalize to longer sequences. Learned encodings let the model adapt position info but may not generalize well beyond training length. Some models combine both or use relative position encodings.

Result

Choosing encoding type affects model flexibility and generalization.

Understanding these tradeoffs helps in designing models for different tasks and sequence lengths.

7

Expert

Positional encoding in relative attention models

🤔Before reading on: do you think relative positional encoding encodes absolute positions or differences? Commit to your answer.

Concept: Relative positional encoding encodes the distance between tokens rather than their absolute positions.

Instead of adding fixed position vectors, relative positional encoding modifies attention scores based on how far apart tokens are. This allows models to focus on relative order, improving performance on longer or variable-length sequences.

Result

Models better capture relationships independent of absolute position, improving generalization.

Knowing relative positional encoding reveals how models can flexibly understand order beyond fixed positions, a key advance in transformer design.

Under the Hood

Positional encoding works by creating a unique vector for each position using sine and cosine waves at different frequencies. These vectors are added to token embeddings, so the model input contains both token meaning and position info. The model's attention mechanism then uses these combined vectors to learn relationships that depend on token order. Internally, the sine and cosine functions produce smooth, continuous signals that help the model distinguish positions and interpolate between them.

Why designed this way?

The original transformer used fixed sine and cosine functions to avoid learning position embeddings, which could overfit or fail to generalize to longer sequences. The choice of different frequencies allows encoding positions uniquely and continuously. Alternatives like learned embeddings were possible but risked losing generalization. This design balances simplicity, generalization, and performance.

Input tokens → Token embeddings ─┐
                               │
                               +→ Add positional encoding → Model input

Positional encoding:
Position index → Sine & Cosine functions at multiple frequencies → Position vector

Model input = Token embedding + Position vector

Transformer layers process combined input preserving order info.

Myth Busters - 4 Common Misconceptions

Quick: Does positional encoding change the token meaning vectors themselves? Commit yes or no.

Common Belief: Positional encoding replaces the token embeddings with position information.

Reality: Positional encoding is added to token embeddings, not replaced, so token meaning is preserved alongside position info.

Why it matters: Replacing embeddings would lose the original token meaning, harming model understanding and performance.

Quick: Do learned positional encodings always outperform fixed ones? Commit yes or no.

Common Belief: Learned positional encodings are always better because the model can adapt them.

Reality: Learned encodings can overfit and fail to generalize to longer sequences, while fixed encodings generalize better.

Why it matters: Choosing learned encodings blindly can reduce model robustness on unseen sequence lengths.

Quick: Is positional encoding only needed for transformers? Commit yes or no.

Common Belief: Only transformer models need positional encoding because other models process sequences differently.

Reality: Any model that processes sequences without inherent order awareness, like some convolutional models, may benefit from positional encoding.

Why it matters: Ignoring positional encoding in other architectures can limit their ability to learn order-dependent patterns.

Quick: Does positional encoding encode absolute positions only? Commit yes or no.

Common Belief: Positional encoding always encodes absolute positions of tokens.

Reality: Some models use relative positional encoding, which encodes distances between tokens rather than absolute positions.

Why it matters: Understanding relative encoding is key to grasping advanced transformer improvements and better sequence generalization.

Expert Zone

1

Positional encoding vectors form a continuous space allowing interpolation for unseen positions, which helps models generalize beyond training lengths.

2

The choice of frequencies in sine/cosine functions affects how well the model can distinguish close versus distant positions.

3

Relative positional encoding modifies attention scores directly, which can reduce memory usage and improve efficiency in long sequences.

When NOT to use

Positional encoding is less useful in models that inherently process sequences in order, like RNNs or LSTMs, which track position through their recurrent steps. For tasks where absolute position is irrelevant, such as bag-of-words models, positional encoding is unnecessary.

Production Patterns

In production, fixed positional encoding is common for standard transformers due to simplicity and generalization. Learned positional embeddings are used when training data is large and fixed sequence length is guaranteed. Relative positional encoding is popular in large language models and speech recognition systems to handle variable-length inputs efficiently.

Connections

Fourier Transform

Positional encoding uses sine and cosine waves similar to Fourier basis functions.

Understanding Fourier transforms helps grasp why sine and cosine functions can uniquely represent positions as continuous signals.

Time Series Analysis

Both positional encoding and time series methods rely on order and temporal patterns in data.

Knowing how time series models use time steps clarifies why positional encoding is crucial for sequence models to capture temporal dependencies.

Music Notation

Positional encoding is like musical notes having timing and order to create melody.

Recognizing that order and timing create meaning in music helps understand why models need position info to interpret sequences.

Common Pitfalls

#1 Using learned positional embeddings without enough data or sequence length variety.

Wrong approach:

pos_embedding = torch.nn.Embedding(max_len, d_model)
# Model uses pos_embedding learned from limited data only

Correct approach:

pos_encoding = get_positional_encoding(max_len, d_model)
# Use fixed sine/cosine positional encoding for better generalization

Root cause:

Assuming learned embeddings always outperform fixed ones ignores overfitting and poor generalization risks.

#2 Replacing token embeddings with positional encoding vectors.

Wrong approach:

model_input = positional_encoding # ignoring token embeddings

Correct approach:

model_input = token_embeddings + positional_encoding # combine both

Root cause:

Misunderstanding that positional encoding supplements rather than replaces token meaning.

#3 Applying positional encoding to unordered data where position is irrelevant.

Wrong approach:

Adding positional encoding to bag-of-words model inputs

Correct approach:

Skip positional encoding for unordered data or use models designed for unordered inputs

Root cause:

Not recognizing when sequence order is meaningful leads to unnecessary complexity.

Key Takeaways

Positional encoding gives models a way to understand the order of tokens in sequences that are processed all at once.

It works by adding unique position vectors, often created with sine and cosine functions, to token embeddings.

This combined input lets models learn relationships that depend on token order, essential for language and time series tasks.

There are fixed and learned positional encodings, each with tradeoffs in generalization and flexibility.

Advanced models use relative positional encoding to focus on distances between tokens, improving performance on variable-length sequences.