Positional encoding in PyTorch - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is positional encoding in the context of machine learning models like Transformers?

Positional encoding is a way to add information about the order or position of elements in a sequence, so the model knows where each element is located since models like Transformers do not process data in order by default.

beginner

Why do Transformers need positional encoding?

Transformers process all input tokens simultaneously without any inherent order. Positional encoding helps the model understand the sequence order, which is important for tasks like language understanding.

intermediate

How is sinusoidal positional encoding calculated?

It uses sine and cosine functions of different frequencies to create unique position vectors. For position pos and dimension i: PE(pos, 2i) = sin(pos / 10000^(2i/d_model)) and PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model)). This helps the model learn relative positions.

intermediate

What is the shape of the positional encoding tensor for a batch of sequences?

The positional encoding tensor usually has shape (sequence_length, embedding_dimension). When added to input embeddings, it matches the input shape (batch_size, sequence_length, embedding_dimension) by broadcasting over the batch.

beginner

How does adding positional encoding affect the input embeddings in a Transformer?

Positional encoding is added element-wise to the input embeddings. This combined input carries both the token meaning and its position, allowing the Transformer to use position information during training and prediction.

Why can't Transformers rely on sequence order without positional encoding?

A Because they ignore input data

B Because they only work with images

C Because they use recurrent layers

D Because they process all tokens simultaneously without order

Transformers process tokens in parallel, so they need positional encoding to know the order of tokens.

Which functions are used in sinusoidal positional encoding?

A Sine and cosine

B Linear and quadratic

C Tangent and cotangent

D Exponential and logarithm

Sinusoidal positional encoding uses sine and cosine functions to encode positions.

What does the positional encoding vector depend on?

A The batch size

B Only the token value

C The position in the sequence and embedding dimension

D The model's output

Positional encoding depends on the position index and the embedding dimension to create unique vectors.

How is positional encoding combined with input embeddings?

A By adding them element-wise

B By multiplying them

C By concatenating them

D By ignoring positional encoding

Positional encoding is added element-wise to input embeddings to include position information.

What is the main benefit of sinusoidal positional encoding?

A It speeds up training

B It allows the model to learn relative positions

C It reduces model size

D It removes the need for embeddings

Sinusoidal encoding helps the model understand relative positions between tokens.

Explain in your own words why positional encoding is important for Transformer models.

Think about how Transformers see all words at once.

Describe how sinusoidal positional encoding is calculated and why sine and cosine functions are used.

Focus on the math functions and their role in encoding position.