Bird
Raised Fist0

Implement Positional Encoding - Sinusoidal and Learned

CODINGadvancedInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
▶
Execution Trace
Step 1/5
Initialize sinusoidal encoding

Sinusoidal encoding matrix initialized with shape (6, 512)

🎯 Understanding the initialization of sinusoidal encodings based on sequence length and model dimension.
import numpy as np
def get_sinusoidal_encoding(n_position, d_model):
    position = np.arange(n_position)[:, np.newaxis]
    div_term = np.exp(np.arange(0, d_model, 2) * -(np.log(10000.0) / d_model))
    pe = np.zeros((n_position, d_model))
    pe[:, 0::2] = np.sin(position * div_term)
    pe[:, 1::2] = np.cos(position * div_term)
Compute sinusoidal encoding for position 1

Position 1 encoding: [0.841, 0.540, 0.909, 0.416, ...]

🎯 How sinusoidal functions are applied to generate encodings for each position.
position = 1
encoding = pe[position]
print(encoding)
Initialize learned encoding

Learned encoding matrix initialized with random values, shape (6, 512)

🎯 Difference between fixed and learned encodings in terms of initialization.
import torch
learned_pe = torch.nn.Parameter(torch.zeros(6, 512))
torch.nn.init.xavier_uniform_(learned_pe)
Update learned encoding during training

Learned encoding for position 1 updated to [0.23, 0.45, 0.67, 0.89, ...]

🎯 How learned encodings are updated during training.
optimizer = torch.optim.Adam([learned_pe], lr=0.01)
loss.backward()
optimizer.step()
Apply positional encoding to input sequence

Input sequence transformed with positional encodings applied.

🎯 Integration of positional encodings into the input sequence for transformer models.
input_sequence = torch.tensor([[1, 2, 3, 4, 5, 6]])
encoded_sequence = input_sequence + learned_pe
print(encoded_sequence)
📊
Diagram
Diagram
┌──────────────────────────────────────────────────────────────┐
│ Positional Encoding Mechanism                                │
├──────────────────────────────────────────────────────────────┤
│ Input Sequence: [ 'The', 'cat', 'sat', 'on', 'the', 'mat' ]  │
│                                                              │
│ Sinusoidal Encoding:                                         │
│   pos=0: [0.0, 1.0, 0.0, 1.0, ...]                           │
│   pos=1: [0.841, 0.540, 0.909, 0.416, ...]                   │
│                                                              │
│ Learned Encoding:                                            │
│   pos=0: [0.12, 0.34, 0.56, 0.78, ...]                       │
│   pos=1: [0.23, 0.45, 0.67, 0.89, ...]                       │
└──────────────────────────────────────────────────────────────┘
Labels
  • Input Sequence: Tokens to be encoded
  • Sinusoidal Encoding: Fixed function-based encoding
  • Learned Encoding: Trainable parameter-based encoding
🔑
Key Moments
▲
Q: Why are sinusoidal encodings fixed and not learned?
A: Sinusoidal encodings provide a consistent way to encode positions without learning, ensuring that the model can generalize to sequences longer than those seen during training.
Q: How does the learned encoding update during training?
A: Learned encodings are updated through backpropagation, allowing the model to optimize positional representations based on the task-specific data.
Q: What is the advantage of using positional encodings in transformers?
A: Positional encodings allow transformers to incorporate sequence order information, which is crucial for tasks like language modeling where word order affects meaning.
Trace Summary

This diagram and trace explain the implementation of positional encoding in transformers, focusing on both sinusoidal and learned methods. Sinusoidal encodings are fixed and provide a consistent positional representation, while learned encodings are trainable and adapt to the data. The trace steps through the initialization, computation, and application of these encodings, highlighting their role in enabling transformers to understand sequence order. Key insights include the difference in initialization and update mechanisms between sinusoidal and learned encodings, and their integration into the input sequence for effective model training.

Primary Focus

✓ Implementing and understanding positional encoding in transformers

Practice

(1/5)
1. 
import numpy as np
from sklearn.metrics import confusion_matrix

y_true = [0, 1, 0, 1, 0, 1, 1]
y_pred = [0, 0, 0, 1, 1, 1, 1]

cm = confusion_matrix(y_true, y_pred)
print(cm)
What does the code output?
easy
A. [[2, 1], [1, 3]]
B. [[3, 1], [1, 2]]
C. [[2, 2], [1, 2]]
D. [[3, 0], [2, 2]]

Solution

  1. Step 1: Understand confusion matrix structure

    The confusion matrix is structured as [[TN, FP], [FN, TP]].

  2. Step 2: Calculate values

    TN=2 (true negatives), FP=1 (false positives), FN=1 (false negatives), TP=3 (true positives).

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Confusion matrix is [[TN, FP], [FN, TP]]. [OK]
Hint: Confusion matrix is [[TN, FP], [FN, TP]]. [OK]
Common Mistakes:
2. What does the following Python code snippet output when calculating the Intersection over Union (IoU) for two bounding boxes? 
import numpy as np

def iou(box1, box2):
    x1 = max(box1[0], box2[0])
    y1 = max(box1[1], box2[1])
    x2 = min(box1[2], box2[2])
    y2 = min(box1[3], box2[3])
    intersection = max(0, x2 - x1) * max(0, y2 - y1)
    area1 = (box1[2] - box1[0]) * (box1[3] - box1[1])
    area2 = (box2[2] - box2[0]) * (box2[3] - box2[1])
    union = area1 + area2 - intersection
    return intersection / union

box1 = [0, 0, 2, 2]
box2 = [1, 1, 3, 3]
print(iou(box1, box2))
easy
A. 0.14285714285714285
B. 0.3333333333333333
C. 0.5
D. 0.25

Solution

  1. Step 1: Calculate Intersection and Union

    The intersection area is 1 (from (1,1) to (2,2)). The union area is 7 (4 from box1 + 4 from box2 - 1 intersection).

  2. Step 2: Calculate IoU

    IoU = 1/7 = 0.14285714285714285.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Intersection over Union formula [OK]
Hint: Intersection over Union formula [OK]
Common Mistakes:
3. 
import numpy as np
from sklearn.model_selection import train_test_split

X = np.array([1, 2, 3, 4, 5, 6])
y = np.array([0, 0, 1, 1, 0, 1])

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, stratify=y)
print(y_train)
easy
A. [0, 0, 1]
B. [0, 1, 0]
C. [1, 0, 1]
D. [0, 1, 1]

Solution

  1. Step 1: Understand stratification

    Stratification ensures that the class distribution in the train and test sets is similar to the original dataset.

  2. Step 2: Calculate class distribution

    Original y has 3 zeros and 3 ones. With stratification, y_train should have 1 zero and 2 ones.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Stratify maintains class distribution. [OK]
Hint: Stratify maintains class distribution. [OK]
Common Mistakes:
4. How would you modify the K-Means algorithm to handle a dataset with categorical features?
medium
A. Use Euclidean distance for all features.
B. Use a different clustering algorithm like DBSCAN.
C. Ignore categorical features during clustering.
D. Convert categorical features to numerical using one-hot encoding.

Solution

  1. Step 1: Handle categorical features

    One-hot encoding converts categorical features into a numerical format suitable for K-Means.

  2. Step 2: Eliminate wrong options

    Use Euclidean distance for all features. is incorrect because Euclidean distance isn't suitable for categorical data. Ignore categorical features during clustering. is incorrect as it loses information. Use a different clustering algorithm like DBSCAN. suggests a different algorithm, not a modification.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Use one-hot encoding for categorical data. [OK]
Hint: Use one-hot encoding for categorical data. [OK]
Common Mistakes:
5. How can the calculation of precision, recall, and F1-score be optimized for a large confusion matrix?
hard
A. Use a more efficient data structure than a matrix.
B. Pre-compute and store results for common confusion matrices.
C. Use parallel processing to compute metrics for each class.
D. Approximate metrics using a subset of the data.

Solution

  1. Step 1: Identify bottlenecks

    Calculating metrics for each class can be parallelized.

  2. Step 2: Choose the best optimization strategy

    Parallel processing allows simultaneous computation, reducing time complexity.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Parallelize class-wise computations. [OK]
Hint: Parallelize class-wise computations. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard