Attention mechanism basics in NLP - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What is the main purpose of the attention mechanism in neural networks?

Choose the best explanation for why attention mechanisms are used in models like transformers.

A To allow the model to focus on different parts of the input sequence when producing each output element.

B To reduce the size of the input data by compressing it into a fixed vector.

C To increase the number of layers in the neural network for deeper learning.

D To randomly drop some neurons during training to prevent overfitting.

💡 Hint

Think about how the model decides which words or tokens are important when generating output.

The attention mechanism helps the model weigh different parts of the input differently for each output, allowing it to focus on relevant information dynamically.

❓ Predict Output

intermediate

What is the output shape of the attention scores matrix?

Given a query matrix Q of shape (batch_size, seq_len_q, d_k) and a key matrix K of shape (batch_size, seq_len_k, d_k), what is the shape of the attention scores computed as Q × Kᵀ?

NLP

import torch
batch_size = 2
seq_len_q = 3
seq_len_k = 4
d_k = 5
Q = torch.randn(batch_size, seq_len_q, d_k)
K = torch.randn(batch_size, seq_len_k, d_k)
attention_scores = torch.bmm(Q, K.transpose(1, 2))
print(attention_scores.shape)

A (2, 3, 4)

B (2, 5, 5)

C (3, 4, 5)

D (2, 4, 3)

💡 Hint

Remember that matrix multiplication between Q and Kᵀ involves the last two dimensions.

The batch matrix multiplication of Q (batch_size, seq_len_q, d_k) and Kᵀ (batch_size, d_k, seq_len_k) results in (batch_size, seq_len_q, seq_len_k).

❓ Model Choice

advanced

Which model architecture introduced the scaled dot-product attention?

Identify the model that first used scaled dot-product attention as a core component.

A Autoencoder

B Recurrent Neural Network (RNN)

C Convolutional Neural Network (CNN)

D Transformer

💡 Hint

This model replaced recurrence with attention mechanisms.

The Transformer model introduced scaled dot-product attention to efficiently compute attention scores and improve training stability.

❓ Hyperparameter

advanced

What is the effect of the scaling factor 1/√d_k in scaled dot-product attention?

Why do we multiply the dot products by 1 divided by the square root of the key dimension (d_k) in attention calculations?

A To increase the dot product values so the model focuses more on important tokens.

B To reduce the sequence length by scaling down the keys.

C To prevent the dot products from growing too large and causing softmax to have extremely small gradients.

D To normalize the input embeddings before attention calculation.

💡 Hint

Think about how large dot products affect the softmax function.

Scaling by 1/√d_k keeps the dot products at a scale that prevents softmax from saturating, which helps maintain meaningful gradients during training.

❓ Metrics

expert

How does attention mechanism impact model interpretability metrics?

Which statement best describes the relationship between attention weights and model interpretability?

A Attention weights reduce model accuracy but improve interpretability.

B Attention weights can offer insights but do not guarantee faithful explanations of model behavior.

C Attention weights are unrelated to interpretability and only affect training speed.

D Attention weights always provide a perfect explanation of model decisions.

💡 Hint

Consider recent research findings on attention as explanation.

While attention weights highlight important input parts, they do not always align perfectly with the model's true decision process, so interpretability is limited.