Transformer architecture overview in Prompt Engineering / GenAI - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What is the main purpose of the self-attention mechanism in a Transformer?

The Transformer model uses a self-attention mechanism. What does this mechanism mainly do?

A It helps the model focus on different parts of the input sequence to understand context.

B It reduces the size of the input data by compressing it into a smaller vector.

C It generates random noise to improve model robustness.

D It sorts the input tokens in order of importance before processing.

💡 Hint

Think about how the model understands relationships between words in a sentence.

Self-attention allows the model to weigh the importance of each word relative to others, helping it understand context and meaning.

❓ Predict Output

intermediate

Output shape after multi-head attention layer

Given an input tensor of shape (batch_size=2, seq_len=5, embedding_dim=64) passed through a multi-head attention layer with 8 heads and output dimension 64, what is the shape of the output tensor?

Prompt Engineering / GenAI

import torch
import torch.nn as nn

batch_size = 2
seq_len = 5
embedding_dim = 64
num_heads = 8

x = torch.rand(batch_size, seq_len, embedding_dim)
mha = nn.MultiheadAttention(embed_dim=embedding_dim, num_heads=num_heads, batch_first=True)
output, _ = mha(x, x, x)
output.shape

A (2, 8, 8)

B (5, 2, 64)

C (2, 5, 512)

D (2, 5, 64)

💡 Hint

Remember the output shape matches the input sequence length and embedding dimension.

The multi-head attention layer outputs a tensor with the same batch size, sequence length, and embedding dimension as the input.

❓ Hyperparameter

advanced

Choosing the number of attention heads in a Transformer

Why might increasing the number of attention heads in a Transformer model improve performance?

A Because more heads increase the embedding dimension automatically without extra computation.

B Because more heads reduce the total number of parameters, making training faster.

C Because more heads allow the model to attend to information from different representation subspaces at different positions.

D Because more heads guarantee the model will not overfit on training data.

💡 Hint

Think about how multiple heads help the model see different aspects of the input.

Multiple attention heads let the model focus on different parts or features of the input simultaneously, improving learning capacity.

❓ Metrics

advanced

Interpreting Transformer training loss curves

During training of a Transformer model, the training loss decreases steadily but the validation loss starts increasing after some epochs. What does this indicate?

A The model is underfitting and needs more training epochs.

B The model is overfitting the training data and not generalizing well to new data.

C The learning rate is too low and should be increased.

D The batch size is too large causing unstable training.

💡 Hint

Think about what it means when validation loss worsens but training loss improves.

When validation loss increases while training loss decreases, the model fits training data too closely and fails to generalize, a sign of overfitting.

🔧 Debug

expert

Identifying cause of NaN values in Transformer training

While training a Transformer model, the loss suddenly becomes NaN after a few epochs. Which of the following is the most likely cause?

A The learning rate is too high, causing unstable gradients and exploding values.

B The batch size is too small, causing insufficient gradient updates.

C The model has too few layers, limiting its capacity.

D The input data is normalized, which causes NaN values.

💡 Hint

Consider what can cause gradients or loss to become infinite or undefined.

A very high learning rate can cause gradients to explode, leading to NaN loss values during training.