Encoder-decoder with attention in NLP - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What is the main purpose of the attention mechanism in an encoder-decoder model?

In an encoder-decoder model for sequence-to-sequence tasks, what does the attention mechanism primarily help with?

A It replaces the decoder with a simpler feedforward network.

B It allows the decoder to focus on different parts of the input sequence dynamically during generation.

C It speeds up training by skipping the encoder step.

D It reduces the size of the input sequence by compressing it into a fixed-length vector.

💡 Hint

Think about how the decoder decides which parts of the input to use when producing each output word.

The attention mechanism helps the decoder look at different parts of the input sequence at each step, instead of relying on a single fixed vector. This improves translation and other sequence tasks.

❓ Predict Output

intermediate

Output shape of attention weights in a simple encoder-decoder

Given the following PyTorch code snippet for scaled dot-product attention weights calculation, what is the shape of attention_weights?

NLP

import torch

batch_size = 2
seq_len_enc = 5
seq_len_dec = 3
hidden_dim = 4

encoder_outputs = torch.rand(batch_size, seq_len_enc, hidden_dim)
decoder_hidden = torch.rand(batch_size, seq_len_dec, hidden_dim)

# Compute attention scores
scores = torch.bmm(decoder_hidden, encoder_outputs.transpose(1, 2)) / (hidden_dim ** 0.5)

# Apply softmax to get attention weights
attention_weights = torch.softmax(scores, dim=2)

print(attention_weights.shape)

A (3, 2, 5)

B (2, 5, 3)

C (2, 3, 5)

D (2, 3, 4)

💡 Hint

Recall that torch.bmm batch-multiplies matrices of shape (batch, n, m) and (batch, m, p) resulting in (batch, n, p).

The scores tensor has shape (batch_size, seq_len_dec, seq_len_enc). After softmax over encoder sequence length, attention_weights shape remains (2, 3, 5).

❓ Model Choice

advanced

Choosing the correct attention type for long input sequences

You want to build an encoder-decoder model for translating very long sentences. Which attention mechanism is best to handle long input sequences efficiently?

A Self-attention only in the decoder without encoder-decoder attention.

B No attention, just use the last encoder hidden state as context.

C Global attention that attends to all encoder outputs at every decoding step.

D Local attention that attends only to a small window of encoder outputs near the current decoding position.

💡 Hint

Consider the computational cost and relevance of distant input tokens for very long sequences.

Local attention reduces computation by focusing on a small relevant window, making it more efficient for long inputs while still capturing important context.

❓ Hyperparameter

advanced

Effect of increasing attention head count in multi-head attention

In a transformer encoder-decoder model, what is the effect of increasing the number of attention heads in multi-head attention?

A It allows the model to jointly attend to information from different representation subspaces at different positions.

B It reduces the model's capacity by splitting the hidden dimension into smaller parts.

C It always decreases training time because each head runs independently.

D It removes the need for positional encoding.

💡 Hint

Think about why multiple attention heads might help the model understand different aspects of the input.

Multiple heads let the model focus on different parts or features of the input simultaneously, improving learning and representation.

🔧 Debug

expert

Identifying the cause of NaN loss in an encoder-decoder with attention training

During training of an encoder-decoder model with attention, the loss suddenly becomes NaN after a few epochs. Which of the following is the most likely cause?

A The attention scores are not normalized properly before softmax, causing numerical instability.

B The optimizer learning rate is too low, causing the loss to diverge to NaN.

C The model uses dropout layers, which always cause NaN values during training.

D The input sequences are too short, causing the model to overfit and produce NaN.

💡 Hint

Consider what can cause softmax to produce invalid values and how attention scores are computed.

If attention scores are very large or not scaled, softmax can produce NaNs due to overflow. Proper scaling (e.g., dividing by sqrt of hidden size) prevents this.