Positional encoding in PyTorch - Model Pipeline Trace

Model Pipeline - Positional encoding

This pipeline shows how positional encoding adds position information to input data so a model can understand order, especially in sequences like sentences.

Data Flow - 3 Stages

1Input Embeddings

10 tokens x 512 dimensions→Convert tokens to vector embeddings→10 tokens x 512 dimensions

[[0.1, 0.3, ..., 0.2], [0.05, 0.4, ..., 0.1], ..., [0.2, 0.1, ..., 0.3]]

↓

2Generate Positional Encoding

10 positions x 512 dimensions→Calculate sine and cosine values for each position and dimension→10 positions x 512 dimensions

[[sin(0/10000^(0/512)), cos(0/10000^(1/512)), ..., sin(0/10000^(510/512)), cos(0/10000^(511/512))], ..., [sin(9/10000^(0/512)), cos(9/10000^(1/512)), ..., sin(9/10000^(510/512)), cos(9/10000^(511/512))]]

↓

3Add Positional Encoding to Embeddings

10 tokens x 512 dimensions→Element-wise addition of embeddings and positional encoding→10 tokens x 512 dimensions

Embedding vector + positional encoding vector for each token

Training Trace - Epoch by Epoch

Loss
1.2 |****
1.0 |*** 
0.8 |**  
0.6 |*   
0.4 |    
     1  2  3  4  5  Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	1.2	0.45	Model starts learning with random weights and positional info
2	0.9	0.60	Loss decreases as model uses positional encoding to understand order
3	0.7	0.72	Model improves predictions by combining token meaning and position
4	0.55	0.80	Clear improvement showing positional encoding helps sequence tasks
5	0.45	0.85	Training converges with positional info aiding context understanding

Prediction Trace - 3 Layers

Layer 1: Input Embedding

Layer 2: Positional Encoding Calculation

Layer 3: Add Positional Encoding

Model Quiz - 3 Questions

Why do we add positional encoding to token embeddings?

A To increase the size of the embeddings randomly

B To remove noise from the input data

C To give the model information about the order of tokens

D To convert tokens into numbers

Key Insight

Positional encoding helps models understand the order of tokens in sequences by adding unique position information to embeddings, which improves learning and prediction accuracy in sequence tasks.