Attention mechanism in depth in NLP - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of scaled dot-product attention calculation

What is the output of the following scaled dot-product attention calculation code snippet?

NLP

import torch
import torch.nn.functional as F

query = torch.tensor([[1.0, 0.0, 1.0]])  # shape (1, 3)
key = torch.tensor([[1.0, 0.0, 1.0], [0.0, 1.0, 0.0]])  # shape (2, 3)
value = torch.tensor([[1.0, 2.0], [3.0, 4.0]])  # shape (2, 2)

# Calculate attention scores
scores = torch.matmul(query, key.T) / (key.shape[1] ** 0.5)  # scale by sqrt(d_k)
weights = F.softmax(scores, dim=1)

# Weighted sum of values
output = torch.matmul(weights, value)
print(output)

A [[1.4795 2.4795]]

B [[3.0 4.0]]

C [[1.5 2.5]]

D [[2.0 3.0]]

💡 Hint

Recall that scaled dot-product attention uses softmax on scaled scores to get weights.

The query matches the first key perfectly, so the attention weight is higher for the first value. The scores are scaled by sqrt of key dimension (sqrt(3)) before softmax. The output is the weighted sum of values using these weights.

🧠 Conceptual

intermediate

Purpose of the 'key' in attention mechanism

In the attention mechanism, what is the main role of the 'key' vectors?

A They normalize the query vectors before similarity calculation.

B They are used to generate the final output directly without matching.

C They represent the information to be retrieved by matching with the query.

D They store the positional encoding of the input sequence.

💡 Hint

Think about how queries find relevant information in keys.

Keys act like labels or addresses that queries compare against to find relevant information. The attention scores come from comparing queries to keys.

❓ Hyperparameter

advanced

Effect of increasing number of attention heads

What is the main effect of increasing the number of attention heads in a multi-head attention model?

A It eliminates the need for positional encoding.

B It reduces the total number of parameters in the model.

C It increases the sequence length the model can process without changing computation.

D It allows the model to jointly attend to information from different representation subspaces at different positions.

💡 Hint

Think about how multiple heads help the model see different aspects of the input.

Multiple attention heads let the model focus on different parts or features of the input simultaneously, improving learning capacity.

❓ Metrics

advanced

Interpreting attention weights distribution

If an attention layer outputs very uniform attention weights across all positions, what does this typically indicate about the model's focus?

A The model is confidently focusing on a single important position.

B The model is uncertain and is distributing focus evenly, possibly due to lack of strong signals.

C The model has overfitted and memorized the training data.

D The model is ignoring the input sequence entirely.

💡 Hint

Uniform weights mean no position stands out more than others.

When attention weights are uniform, the model does not find any position more relevant than others, indicating uncertainty or weak signals.

🔧 Debug

expert

Identifying error in custom attention implementation

Consider this simplified custom attention code snippet. What error will it raise when run? ```python import torch import torch.nn.functional as F def custom_attention(query, key, value): scores = torch.matmul(query, key) # shape mismatch possible weights = F.softmax(scores, dim=-1) output = torch.matmul(weights, value) return output q = torch.randn(1, 3) k = torch.randn(2, 3) v = torch.randn(2, 4) result = custom_attention(q, k, v) print(result) ```

A RuntimeError due to shape mismatch in torch.matmul(query, key)

B RuntimeError due to shape mismatch in torch.matmul(weights, value)

C No error; outputs a tensor of shape (1, 4)

D SyntaxError due to missing colon in function definition

💡 Hint

Check the shapes of query and key tensors before multiplication.

The query has shape (1, 3) and key has shape (2, 3). Multiplying (1,3) by (2,3) is invalid because inner dimensions do not match. The key should be transposed.