Transformer architecture in NLP - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What is the main purpose of the self-attention mechanism in a Transformer?

In the Transformer model, the self-attention mechanism helps the model to:

A Focus on different parts of the input sequence to understand relationships between words.

B Reduce the size of the input data by compressing it into a smaller vector.

C Generate random noise to improve model robustness during training.

D Sort the input words in alphabetical order before processing.

💡 Hint

Think about how the model learns connections between words regardless of their position.

Self-attention allows the Transformer to weigh the importance of each word in the input sequence relative to others, capturing context effectively.

❓ Predict Output

intermediate

Output shape after multi-head attention layer

Given the following code snippet using PyTorch, what is the shape of the output tensor?

NLP

import torch
import torch.nn as nn

batch_size = 2
seq_len = 5
embed_dim = 16
num_heads = 4

x = torch.rand(batch_size, seq_len, embed_dim)
mha = nn.MultiheadAttention(embed_dim=embed_dim, num_heads=num_heads)

# PyTorch MultiheadAttention expects input shape (seq_len, batch_size, embed_dim)
x_t = x.transpose(0, 1)
out, _ = mha(x_t, x_t, x_t)

output_shape = out.shape

A (2, 16, 5)

B (5, 2, 16)

C (2, 5, 16)

D (5, 16, 2)

💡 Hint

Check the input and output shapes expected by PyTorch's MultiheadAttention.

PyTorch's MultiheadAttention expects input shape (sequence length, batch size, embedding dimension) and returns output of the same shape.

❓ Hyperparameter

advanced

Choosing the number of attention heads

Which of the following is a valid reason to increase the number of attention heads in a Transformer model?

A To reduce the total number of parameters in the model for faster training.

B To convert the model from a Transformer to a convolutional neural network.

C To ensure the model only focuses on the first few words of the input sequence.

D To allow the model to attend to information from multiple representation subspaces at different positions.

💡 Hint

Think about how multiple heads help the model understand different aspects of the input.

Multiple attention heads allow the model to jointly attend to information from different representation subspaces, improving learning capacity.

❓ Metrics

advanced

Interpreting training loss in Transformer models

During training a Transformer for language modeling, the loss decreases steadily but the validation loss starts increasing after some epochs. What does this indicate?

A The model is overfitting the training data and not generalizing well to new data.

B The model is underfitting and needs more training epochs.

C The training data is corrupted and causing unstable loss values.

D The optimizer is not updating the model weights correctly.

💡 Hint

Consider what it means when training loss improves but validation loss worsens.

When validation loss increases while training loss decreases, the model is memorizing training data but failing to generalize, a sign of overfitting.

🔧 Debug

expert

Identifying error in Transformer positional encoding implementation

Consider this simplified code snippet for positional encoding in a Transformer. What error will this code raise when run?

NLP

import torch
import math

def positional_encoding(seq_len, d_model):
    pe = torch.zeros(seq_len, d_model)
    position = torch.arange(0, seq_len).unsqueeze(1)
    div_term = torch.exp(torch.arange(0, d_model, 2) * (-math.log(10000.0) / d_model))
    pe[:, 0::2] = torch.sin(position * div_term)
    pe[:, 1::2] = torch.cos(position * div_term)
    return pe

pos_enc = positional_encoding(10, 7)

A SyntaxError because of missing colon in function definition

B TypeError because torch.arange returns a list, not a tensor

C RuntimeError due to shape mismatch when assigning to pe[:, 1::2]

D No error, code runs correctly and returns positional encoding tensor

💡 Hint

Check the shapes of slices pe[:, 0::2] and pe[:, 1::2] when d_model is odd.

When d_model is odd (7), pe[:, 1::2] has one more column than div_term, causing a shape mismatch error during assignment.