Bird
Raised Fist0

Transformer Architecture - Self-Attention, Positional Encoding, Scaling Laws

THEORYadvancedInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
▶
Execution Trace
Step 1/5
Input Sequence

Tokens 'The', 'cat', 'sat', 'on', 'the', 'mat' are fed into the model.

🎯 Understanding the role of input sequences in transformers.
input_sequence = ['The', 'cat', 'sat', 'on', 'the', 'mat']
Positional Encoding

Positional encodings [P0, P1, P2, P3, P4, P5] are added to tokens.

🎯 Why positional encoding is crucial for sequence order.
positional_encoding = get_positional_encoding(len(input_sequence))
Self-Attention Mechanism

Attention scores calculated for 'cat' peak at 'sat' (0.76) and 'mat' (0.87).

🎯 How self-attention captures dependencies between tokens.
attention_scores = calculate_attention(Q, K, V)
Weighted Values

Weighted values are computed using attention scores.

🎯 The impact of attention scores on the final output.
weighted_values = apply_attention_weights(attention_scores, V)
Output Sequence

The output sequence is generated from weighted values.

🎯 How the transformer outputs a new sequence from the input.
output_sequence = generate_output_sequence(weighted_values)
📊
Diagram
Diagram
┌──────────────────────────────────────────────────────────────┐
│ Input Sequence: [The, cat, sat, on, the, mat]               │
├──────────────────────────────────────────────────────────────┤
│ Positional Encoding: [P0, P1, P2, P3, P4, P5]               │
├──────────────────────────────────────────────────────────────┤
│ Self-Attention Mechanism                                    │
│ ┌───────────────┐  ┌───────────────┐  ┌───────────────┐     │
│ │ Query (Q)     │  │ Key (K)       │  │ Value (V)     │     │
│ └───────────────┘  └───────────────┘  └───────────────┘     │
│        │                  │                  │              │
│        └───────┬──────────┴──────────┬───────┘              │
│                ▼                     ▼                      │
│          Attention Scores       Weighted Values             │
│                │                     │                      │
│                └──────────────┬──────┘                      │
│                               ▼                             │
│                        Output Sequence                      │
└──────────────────────────────────────────────────────────────┘
Labels
  • Input Sequence: The initial tokens fed into the transformer.
  • Positional Encoding: Adds position information to each token.
  • Self-Attention Mechanism: Computes attention scores and weighted values.
  • Query (Q), Key (K), Value (V): Components used to calculate attention.
  • Output Sequence: The final transformed sequence after attention.
🔑
Key Moments
▲
Q: Why does the attention score for 'cat' peak at 'sat' and 'mat'?
A: The attention mechanism captures semantic and positional relevance, highlighting 'sat' and 'mat' due to their contextual importance.
Q: How does positional encoding affect the attention mechanism?
A: Positional encoding provides necessary sequence order information, allowing the model to distinguish between tokens with similar content but different positions.
Q: What would happen if positional encoding was omitted?
A: Without positional encoding, the model would treat the sequence as a bag of words, losing the ability to understand order and context.
Trace Summary

This diagram and trace explain the self-attention mechanism in transformers, highlighting how input sequences are processed with positional encoding to maintain order. The self-attention mechanism calculates attention scores, allowing the model to focus on relevant tokens. Weighted values derived from these scores are used to generate the output sequence. Key insights include the importance of positional encoding for sequence order and the role of attention scores in capturing token dependencies.

Primary Focus

✓ Understanding the mechanism of self-attention and positional encoding in transformers

Practice

(1/5)
1. In a deep learning model, when is it more appropriate to use Layer Normalization instead of Batch Normalization?
easy
A. When the model is trained on a very large batch size.
B. When the model is trained on a very small batch size or online learning.
C. When the model has a very deep architecture.
D. When the model is trained on a balanced dataset.

Solution

  1. Step 1: Understand the context of normalization techniques.

    Batch Normalization relies on batch statistics, which can be unstable with small batch sizes. Layer Normalization uses layer statistics, making it more stable for small batches or online learning.

  2. Step 2: Eliminate wrong options.

    When the model is trained on a very large batch size. is incorrect because large batch sizes are suitable for Batch Normalization. When the model has a very deep architecture. is incorrect because depth doesn't directly affect the choice between these normalizations. When the model is trained on a balanced dataset. is irrelevant to normalization choice.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Layer Norm for small batches or online learning. [OK]
Hint: Layer Norm for small batches or online learning. [OK]
Common Mistakes:
2. In a regression problem where outliers are present, which metric is generally more robust to these outliers?
easy
A. Mean Absolute Percentage Error (MAPE)
B. Mean Squared Error (MSE)
C. Root Mean Squared Error (RMSE)
D. Mean Absolute Error (MAE)

Solution

  1. Step 1: Understand the impact of outliers on metrics.

    MAE is less sensitive to outliers compared to MSE and RMSE because it doesn't square the errors.

  2. Step 2: Eliminate wrong options.

    MSE and RMSE are sensitive to outliers due to squaring errors. MAPE is not suitable for zero or near-zero values.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    MAE handles outliers better than MSE. [OK]
Hint: MAE is less sensitive to outliers than MSE. [OK]
Common Mistakes:
3. 
import numpy as np
from sklearn.cluster import KMeans

X = np.array([[1, 2], [1, 4], [1, 0],
              [4, 2], [4, 4], [4, 0]])
kmeans = KMeans(n_clusters=2, random_state=0).fit(X)
print(kmeans.labels_)
What does the code output?
easy
A. [0 0 0 1 1 1]
B. [1 1 1 0 0 0]
C. [0 1 0 1 0 1]
D. [1 0 1 0 1 0]

Solution

  1. Step 1: Understand KMeans clustering.

    KMeans assigns labels based on proximity to cluster centers.

  2. Step 2: Analyze the data points.

    Points [1, 2], [1, 4], [1, 0] are closer to one center, and [4, 2], [4, 4], [4, 0] to another.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Cluster proximity determines labels [OK]
Hint: Cluster proximity determines labels [OK]
Common Mistakes:
4. For a dataset with significant outliers, which scaling method is more appropriate and why?
medium
A. Standardization, because it scales data to a fixed range.
B. Normalization, because it scales data to a fixed range.
C. Normalization, because it handles outliers better.
D. Standardization, because it is less affected by outliers.

Solution

  1. Step 1: Consider the effect of outliers on scaling methods.

    Normalization is sensitive to outliers as it scales data to a fixed range.

  2. Step 2: Choose the method less affected by outliers.

    Standardization is less affected because it centers data around the mean and scales by standard deviation.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Standardize for outliers, less sensitive. [OK]
Hint: Standardize for outliers, less sensitive. [OK]
Common Mistakes:
5. For a regression task with a large number of outliers, which metric would be more appropriate to use?
medium
A. Mean Squared Error (MSE)
B. Huber Loss
C. Mean Absolute Percentage Error (MAPE)
D. Root Mean Squared Error (RMSE)

Solution

  1. Step 1: Consider the impact of outliers on metrics.

    Huber Loss is less sensitive to outliers than MSE and RMSE.

  2. Step 2: Eliminate wrong options.

    MSE and RMSE amplify outliers due to squaring. MAPE is not suitable for zero or near-zero values.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Huber Loss balances sensitivity to outliers. [OK]
Hint: Huber Loss balances sensitivity to outliers. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard