Attention mechanism in depth in NLP - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to compute the attention scores using dot product.

NLP

import torch

query = torch.randn(1, 5)
key = torch.randn(1, 5)

attention_scores = torch.matmul(query, [1].T)
print(attention_scores)

Drag options to blanks, or click blank then click option'

Avalue

Bkey

Cweights

Dquery

Attempts:

3 left

💡 Hint

Common Mistakes

Using query instead of key for the dot product.

Not transposing the key matrix before multiplication.

The attention scores are computed by the dot product of the query and the transpose of the key vectors.

2fill in blank

medium

Complete the code to apply softmax to the attention scores to get attention weights.

NLP

import torch.nn.functional as F

attention_scores = torch.tensor([[1.0, 2.0, 3.0]])
attention_weights = F.[1](attention_scores, dim=-1)
print(attention_weights)

Drag options to blanks, or click blank then click option'

Arelu

Bsigmoid

Csoftmax

Dtanh

Attempts:

3 left

💡 Hint

Common Mistakes

Using sigmoid instead of softmax, which does not normalize across the dimension.

Applying activation functions like relu or tanh which do not produce probabilities.

Softmax converts raw attention scores into probabilities that sum to 1.

3fill in blank

hard

Fix the error in the code to compute the weighted sum of values using attention weights.

NLP

import torch

attention_weights = torch.tensor([[0.1, 0.7, 0.2]])
values = torch.tensor([[1.0, 0.0, 0.0], [0.0, 1.0, 0.0], [0.0, 0.0, 1.0]])
weighted_sum = torch.matmul([1], values)
print(weighted_sum)

Drag options to blanks, or click blank then click option'

Avalues

Bvalues.T

Cattention_weights.T

Dattention_weights

Attempts:

3 left

💡 Hint

Common Mistakes

Transposing attention weights causing shape mismatch.

Multiplying values by values instead of attention weights.

The weighted sum is the matrix multiplication of attention weights and values.

4fill in blank

hard

Fill both blanks to scale the attention scores and apply softmax.

NLP

import torch
import torch.nn.functional as F

query = torch.randn(1, 64)
key = torch.randn(10, 64)

scores = torch.matmul(query, key.T) / [1]
attention_weights = F.[2](scores, dim=-1)
print(attention_weights)

Drag options to blanks, or click blank then click option'

A64**0.5

B64

Csoftmax

Dsigmoid

Attempts:

3 left

💡 Hint

Common Mistakes

Using sigmoid instead of softmax for attention weights.

Not scaling scores causing unstable training.

Attention scores are scaled by the square root of the key dimension before softmax to stabilize gradients.

5fill in blank

hard

Fill all three blanks to implement multi-head attention output concatenation and projection.

NLP

import torch
import torch.nn as nn

class MultiHeadAttention(nn.Module):
    def __init__(self, embed_dim, num_heads):
        super().__init__()
        self.num_heads = num_heads
        self.head_dim = embed_dim // num_heads
        self.linear_out = nn.Linear(embed_dim, embed_dim)

    def forward(self, x):
        batch_size, seq_len, embed_dim = x.size()
        # Assume x is already split into heads and attention applied
        concat_heads = x.reshape(batch_size, seq_len, [1])
        output = self.linear_out([2])
        return output

attention = MultiHeadAttention(embed_dim=128, num_heads=8)
x = torch.randn(2, 10, 128)
result = attention(x)
print(result.shape)

Drag options to blanks, or click blank then click option'

Aembed_dim

Bconcat_heads

Dx

Attempts:

3 left

💡 Hint

Common Mistakes

Using wrong dimension in reshape causing size mismatch.

Passing wrong variable to linear_out layer.

The heads are concatenated back to the original embedding dimension before passing through the output linear layer.

Practice

1. What is the main purpose of the attention mechanism in NLP models?

easy

A. To increase the size of the input data

B. To reduce the number of layers in the model

C. To help the model focus on important parts of the input data

D. To randomly shuffle the input tokens

Solution

1. Step 1: Understand attention's role

   Attention helps models decide which parts of the input are most important for the task.

2. Step 2: Compare options

   Only To help the model focus on important parts of the input data correctly describes this focus mechanism; others describe unrelated actions.

3. Final Answer:

   To help the model focus on important parts of the input data -> Option C

4. Quick Check:

   Attention = Focus on important input [OK]

Hint: Remember: attention means focusing on key input parts [OK]

Common Mistakes:

• Thinking attention changes input size
• Confusing attention with model depth
• Assuming attention shuffles data

2. Which of the following correctly represents the formula for attention weights using queries (Q), keys (K), and softmax?

easy

A. softmax(Q x K^T)

B. Q + K

C. softmax(Q - K)

D. Q x K

Solution

1. Step 1: Recall attention weight calculation

   Attention weights are computed by multiplying queries with keys transposed, then applying softmax.

2. Step 2: Evaluate options

   Only softmax(Q x K^T) matches the correct formula softmax(Q x K^T). Others are incorrect operations.

3. Final Answer:

   softmax(Q x K^T) -> Option A

4. Quick Check:

   Attention weights = softmax(Q x K^T) [OK]

Hint: Attention weights = softmax of query-key dot product [OK]

Common Mistakes:

• Using addition instead of multiplication
• Forgetting to transpose keys
• Skipping softmax normalization

3. Given queries Q = [[1, 0]], keys K = [[1, 0], [-10, 1]], and values V = [[10, 20], [30, 40]], what is the output of the attention mechanism (using dot product and softmax)?

medium

A. [[10, 20]]

B. [[20, 30]]

C. [[20, 40]]

D. [[30, 40]]

Solution

1. Step 1: Calculate dot products Q x K^T

   Q = [1,0], K = [[1,0],[-10,1]]; dot products: [1*1+0*0=1, 1*(-10)+0*1=-10]

2. Step 2: Apply softmax to scores

   softmax([1,-10]) ≈ [1, 0] (e^{-10} negligible)

3. Step 3: Compute weighted sum of values

   Output ≈ 1*[10,20] + 0*[30,40] = [[10, 20]]

4. Step 4: Match option

   [[10, 20]] matches exactly.

5. Final Answer:

   [[10, 20]] -> Option A

6. Quick Check:

   Weighted sum of values = [[10, 20]] [OK]

Hint: Calculate dot, softmax, then weighted sum of values [OK]

Common Mistakes:

• Skipping softmax normalization
• Using keys instead of values for output
• Incorrect dot product calculation

4. Identify the error in this attention weight calculation code snippet:

import numpy as np
Q = np.array([[1, 0]])
K = np.array([[1, 0], [-10, 1]])
scores = np.dot(Q, K)
weights = np.exp(scores) / np.sum(np.exp(scores))

medium

A. Values are missing in the calculation

B. Softmax is applied incorrectly

C. Queries and keys have incompatible shapes

D. Keys should be transposed before dot product

Solution

1. Step 1: Check dot product operation

   Dot product should be between Q and K transposed to align dimensions correctly.

2. Step 2: Analyze code

   Code uses np.dot(Q, K) without transposing K, causing wrong shape and incorrect scores.

3. Final Answer:

   Keys should be transposed before dot product -> Option D

4. Quick Check:

   Transpose keys before dot product [OK]

Hint: Always transpose keys before dot product with queries [OK]

Common Mistakes:

• Forgetting to transpose keys
• Misapplying softmax formula
• Ignoring shape compatibility

5. In a transformer model, why is scaling the dot product by the square root of the key dimension important before applying softmax?

hard

A. To increase the dot product values for better attention

B. To prevent large dot product values causing very small gradients

C. To normalize the values between 0 and 1

D. To reduce the number of keys used in attention

Solution

1. Step 1: Understand dot product scaling

   Large dot products can cause softmax to produce very small gradients, slowing learning.

2. Step 2: Role of scaling by sqrt of key dimension

   Scaling reduces dot product magnitude, stabilizing gradients and improving training.

3. Final Answer:

   To prevent large dot product values causing very small gradients -> Option B

4. Quick Check:

   Scaling avoids tiny gradients in softmax [OK]

Hint: Scale dot product to keep gradients healthy [OK]

Common Mistakes:

• Thinking scaling increases dot product
• Confusing scaling with normalization to [0,1]
• Assuming scaling reduces keys count