Positional encoding in PyTorch - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to create a positional encoding tensor of zeros with shape (max_len, d_model).

PyTorch

pos_encoding = torch.zeros([1], d_model)

Drag options to blanks, or click blank then click option'

Aseq_len

Bd_model

Cbatch_size

Dmax_len

Attempts:

3 left

💡 Hint

Common Mistakes

Using d_model as the first dimension instead of max_len.

Confusing batch size with sequence length.

We create a positional encoding tensor with shape (max_len, d_model) to hold positional information for each position up to max_len.

2fill in blank

medium

Complete the code to compute the position indices tensor for max_len positions.

PyTorch

position = torch.arange([1]).unsqueeze(1)

Drag options to blanks, or click blank then click option'

Ad_model

Bbatch_size

Cmax_len

Dseq_len

Attempts:

3 left

💡 Hint

Common Mistakes

Using d_model instead of max_len for position range.

Not unsqueezing to get a column vector.

We use torch.arange(max_len) to get position indices from 0 to max_len-1, then unsqueeze to make it a column vector.

3fill in blank

hard

Fix the error in the code to compute the angle rates for positional encoding.

PyTorch

angle_rates = 1 / torch.pow(10000, (2 * ([1] // 2)) / d_model)

Drag options to blanks, or click blank then click option'

Ai

Bpos

Cdim

Dposition

Attempts:

3 left

💡 Hint

Common Mistakes

Using position instead of dimension index.

Using undefined variable pos.

The variable i represents the dimension index used to calculate the angle rates for each dimension.

4fill in blank

hard

Fill both blanks to assign sine to even indices and cosine to odd indices in the positional encoding.

PyTorch

pos_encoding[:, [1]] = torch.sin(angle_rads[:, [2]])

Drag options to blanks, or click blank then click option'

A::2

B1::2

C0::2

D2::2

Attempts:

3 left

💡 Hint

Common Mistakes

Using odd indices for sine instead of even.

Mismatching slices between pos_encoding and angle_rads.

Even indices start at 0 and step by 2, so 0::2 selects even dims for sine. The angle_rads uses the same indexing.

5fill in blank

hard

Complete the code to assign cosine to odd indices in the positional encoding.

PyTorch

pos_encoding[:, [1]] = torch.cos(angle_rads[:, [2]])

Drag options to blanks, or click blank then click option'

A::2

B1::2

C0::2

D2::2

Attempts:

3 left

💡 Hint

Common Mistakes

Using even indices (0::2 or ::2) for cosine.

Mismatching slices between pos_encoding and angle_rads.

Odd indices start at 1 and step by 2, so 1::2 selects odd dims for cosine. Use the same slice for angle_rads.