Why transformers revolutionized NLP - Challenge Your Understanding

Challenge - 5 Problems

🧠 Conceptual

intermediate

Key innovation of transformers in NLP

Which feature of transformers most directly allows them to handle long-range dependencies in text better than previous models?

A Convolutional layers that capture local patterns in fixed windows

B Use of recurrent connections to process sequences step-by-step

C Predefined fixed-length context windows for input sequences

D Self-attention mechanism that weighs all words in a sentence simultaneously

💡 Hint

Think about how the model can look at all words at once instead of one by one.

Transformers use self-attention to consider all words in a sentence at the same time, enabling them to capture relationships between distant words effectively.

❓ Model Choice

intermediate

Choosing a model architecture for NLP tasks

You want to build a model that understands context in long documents for summarization. Which model architecture is best suited?

A Recurrent Neural Network (RNN) with LSTM cells

B Transformer with self-attention layers

C Convolutional Neural Network (CNN) with small kernels

D Simple feedforward neural network

💡 Hint

Consider which model can capture relationships across long text spans.

Transformers excel at capturing context over long sequences due to their self-attention mechanism, unlike RNNs or CNNs which have limitations with long-range dependencies.

❓ Metrics

advanced

Evaluating transformer model performance

After training a transformer for language translation, which metric best measures how well the model's output matches human translations?

A BLEU score comparing generated and reference sentences

B Accuracy of predicted next word

C Mean squared error between word embeddings

D Confusion matrix of predicted classes

💡 Hint

Think about a metric designed for comparing sentences in translation tasks.

BLEU score measures the overlap between machine-generated and human reference translations, making it suitable for evaluating translation quality.

🔧 Debug

advanced

Identifying a common transformer training issue

You trained a transformer model but notice the training loss does not decrease and stays very high. Which issue is most likely causing this?

A Using batch size too large causing slow convergence

B Not using dropout layers causing overfitting

C Using a learning rate that is too high causing unstable updates

D Applying layer normalization after the output layer

💡 Hint

Consider what happens if the model weights update too aggressively.

A learning rate that is too high can cause the model to overshoot minima, preventing loss from decreasing.

❓ Predict Output

expert

Output shape of transformer attention scores

Given the following PyTorch code snippet for a transformer attention layer, what is the shape of the 'attention_scores' tensor?

NLP

import torch
batch_size = 2
seq_len = 5
embed_dim = 16
num_heads = 4

# Query, Key tensors
Q = torch.rand(batch_size, num_heads, seq_len, embed_dim // num_heads)
K = torch.rand(batch_size, num_heads, seq_len, embed_dim // num_heads)

# Compute attention scores
attention_scores = torch.matmul(Q, K.transpose(-2, -1))

A torch.Size([2, 4, 5, 5])

B torch.Size([2, 4, 16, 16])

C torch.Size([2, 16, 5, 5])

D torch.Size([2, 5, 4, 4])

💡 Hint

Recall that attention scores are computed by multiplying queries and keys along the embedding dimension.

Q has shape (batch_size, num_heads, seq_len, head_dim). Multiplying Q by K^T along the last two dims results in (batch_size, num_heads, seq_len, seq_len).