Self-attention mechanism in PyTorch - Model Metrics & Evaluation

Metrics & Evaluation - Self-attention mechanism

Which metric matters for this concept and WHY

For self-attention mechanisms, the main goal is to improve how well the model understands relationships in data, especially sequences like sentences. Metrics like loss (how far predictions are from true answers) and accuracy (how often predictions are correct) matter most during training. When self-attention is used in tasks like language translation or text classification, metrics such as BLEU score or F1 score become important to measure quality. These metrics show if the model is paying attention to the right parts of the input.

Confusion matrix or equivalent visualization (ASCII)

Self-attention itself is a mechanism inside models, so it doesn't have a confusion matrix directly. But when used in classification tasks, the confusion matrix helps us see how well the model predicts each class.

      Confusion Matrix Example:

          Predicted
          Pos   Neg
      Pos  TP    FN
      Neg  FP    TN

      TP = True Positive
      FP = False Positive
      TN = True Negative
      FN = False Negative
    

From this, we calculate precision, recall, and F1 score to understand model performance.

Precision vs Recall tradeoff with concrete examples

When self-attention helps a model classify text, precision and recall tell us different things:

• Precision means: When the model says "this is positive," how often is it right? High precision means few false alarms.
• Recall means: Of all actual positive cases, how many did the model find? High recall means few misses.

Example: In spam detection, high precision means few good emails are marked spam (important to avoid annoyance). High recall means catching most spam emails (important to keep inbox clean). Self-attention helps by focusing on important words to balance this tradeoff.

What "good" vs "bad" metric values look like for this use case

Good metrics when using self-attention in classification:

• Loss steadily decreases during training, showing the model learns.
• Accuracy above 80% on test data for simple tasks.
• Precision and recall both above 75%, indicating balanced performance.
• F1 score close to precision and recall, showing no big tradeoff.

Bad metrics:

• Loss stays high or fluctuates, meaning poor learning.
• Accuracy near random chance (e.g., 50% for two classes).
• Very high precision but very low recall, or vice versa, showing imbalance.
• F1 score much lower than precision or recall, indicating poor overall performance.

Metrics pitfalls (accuracy paradox, data leakage, overfitting indicators)

• Accuracy paradox: High accuracy can be misleading if classes are imbalanced. For example, if 90% of data is one class, predicting that class always gives 90% accuracy but no real learning.
• Data leakage: If test data leaks into training, metrics look too good but model fails in real use.
• Overfitting: Training loss very low but test loss high means model memorizes training data but doesn't generalize.
• Ignoring recall: In some tasks, missing important cases (low recall) is worse than false alarms.

Self-check question

Your model with self-attention has 98% accuracy but only 12% recall on fraud detection. Is it good for production? Why not?

Answer: No, it is not good. Even though accuracy is high, the model misses 88% of fraud cases (low recall). In fraud detection, missing fraud is very costly, so recall is more important. The model needs improvement to catch more fraud.

Key Result

Self-attention models should be evaluated with balanced precision and recall, not just accuracy, to ensure meaningful understanding of data relationships.

Practice

1. What is the main purpose of the self-attention mechanism in models like Transformers?

easy

A. To convert input data into images

B. To increase the size of the input data

C. To reduce the number of layers in the model

D. To help the model focus on important parts of the input data

Solution

1. Step 1: Understand self-attention's role

   Self-attention helps the model decide which parts of the input are important by comparing each part to others.

2. Step 2: Identify the main goal

   The main goal is to focus on important parts, not to change input size or model layers.

3. Final Answer:

   To help the model focus on important parts of the input data -> Option D

4. Quick Check:

   Self-attention = focus on important parts [OK]

Hint: Self-attention means focusing on important input parts [OK]

Common Mistakes:

• Thinking self-attention changes input size
• Confusing self-attention with model depth
• Assuming it converts data formats

2. Which of the following is the correct PyTorch code snippet to compute scaled dot-product attention scores given query Q and key K tensors?

easy

A. scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(Q.size(-1))

B. scores = torch.matmul(K, Q.transpose(-2, -1)) * math.sqrt(Q.size(-1))

C. scores = torch.matmul(Q, K) / Q.size(0)

D. scores = torch.matmul(Q.transpose(-2, -1), K) / math.sqrt(K.size(-1))

Solution

1. Step 1: Recall scaled dot-product attention formula

   Attention scores = (Q x K^T) / sqrt(d_k), where d_k is the dimension of keys.

2. Step 2: Match formula with code

   scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(Q.size(-1)) correctly transposes K, multiplies Q and K^T, and divides by sqrt of last dimension size.

3. Final Answer:

   scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(Q.size(-1)) -> Option A

4. Quick Check:

   Scaled dot-product = QxK^T / sqrt(d_k) [OK]

Hint: Transpose keys and divide by sqrt of key dimension [OK]

Common Mistakes:

• Multiplying K by Q instead of Q by K^T
• Forgetting to transpose keys
• Using multiplication instead of division by sqrt

3. Given the following PyTorch code for self-attention, what is the shape of the output tensor attn_output?

import torch
Q = torch.randn(2, 4, 8)  # batch=2, seq_len=4, dim=8
K = torch.randn(2, 4, 8)
V = torch.randn(2, 4, 8)

scores = torch.matmul(Q, K.transpose(-2, -1)) / (8 ** 0.5)
weights = torch.nn.functional.softmax(scores, dim=-1)
attn_output = torch.matmul(weights, V)

medium

A. (4, 2, 8)

B. (2, 4, 8)

C. (2, 8, 4)

D. (4, 8, 2)

Solution

1. Step 1: Analyze input tensor shapes

   Q, K, V have shape (batch=2, seq_len=4, dim=8).

2. Step 2: Understand matrix multiplications

   Scores shape: (2,4,4), weights shape: (2,4,4), multiplying weights (2,4,4) by V (2,4,8) results in (2,4,8).

3. Final Answer:

   (2, 4, 8) -> Option B

4. Quick Check:

   Output shape = (batch, seq_len, dim) = (2,4,8) [OK]

Hint: Output shape matches Q and V batch and seq_len dims [OK]

Common Mistakes:

• Confusing batch and sequence dimensions
• Mixing up transpose effects on shape
• Assuming output shape changes dimension size

4. Identify the error in this PyTorch self-attention code snippet:

import torch
Q = torch.randn(1, 3, 5)
K = torch.randn(1, 3, 5)
V = torch.randn(1, 3, 5)

scores = torch.matmul(Q, K) / (5 ** 0.5)
weights = torch.nn.functional.softmax(scores, dim=-1)
output = torch.matmul(weights, V)

medium

A. Q and V should have the same shape

B. Softmax should be applied on dim=0 instead of dim=-1

C. K should be transposed before multiplication with Q

D. Division by sqrt(5) is incorrect

Solution

1. Step 1: Check matrix multiplication for scores

   Q shape is (1,3,5), K shape is (1,3,5). To multiply Q by K^T, K must be transposed on last two dims.

2. Step 2: Identify missing transpose

   Code multiplies Q by K directly, causing shape mismatch or wrong attention scores.

3. Final Answer:

   K should be transposed before multiplication with Q -> Option C

4. Quick Check:

   QxK^T needed, missing transpose causes error [OK]

Hint: Always transpose keys before multiplying with queries [OK]

Common Mistakes:

• Skipping transpose of K
• Applying softmax on wrong dimension
• Confusing shapes of Q and V

5. You want to implement multi-head self-attention in PyTorch with 4 heads. If your input tensor has shape (batch=2, seq_len=6, dim=32), what should be the dimension size of each head's query, key, and value vectors?

hard

A. 8

B. 4

C. 16

D. 32

Solution

1. Step 1: Understand multi-head attention dimension split

   Total dimension 32 is split evenly across 4 heads, so each head gets 32 / 4 = 8 dimensions.

2. Step 2: Confirm dimension per head

   Each head's query, key, and value vectors have dimension 8 to keep total dimension consistent.

3. Final Answer:

   8 -> Option A

4. Quick Check:

   32 total dim / 4 heads = 8 per head [OK]

Hint: Divide total dim by number of heads for each head size [OK]

Common Mistakes:

• Using total dimension for each head
• Dividing by wrong number of heads
• Confusing batch size with dimension