Sequence-to-sequence architecture in NLP - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What is the main purpose of a sequence-to-sequence model?

Sequence-to-sequence models are widely used in tasks like language translation. What is their main purpose?

A To cluster data points into groups without labels

B To classify images into fixed categories

C To map an input sequence to an output sequence of possibly different length

D To reduce the dimensionality of input data

💡 Hint

Think about tasks like translating a sentence from English to French.

Sequence-to-sequence models take a sequence as input and produce another sequence as output, often of different length, such as translating sentences or summarizing text.

❓ Predict Output

intermediate

Output shape of encoder and decoder in a seq2seq model

Consider a seq2seq model with an LSTM encoder and decoder. The encoder processes input sequences of length 10 with 16 features, and the decoder outputs sequences of length 12 with 20 features. What is the shape of the encoder's final hidden state and the decoder's output?

NLP

import torch
import torch.nn as nn

encoder = nn.LSTM(input_size=16, hidden_size=32, batch_first=True)
decoder = nn.LSTM(input_size=20, hidden_size=32, batch_first=True)

inputs = torch.randn(5, 10, 16)  # batch_size=5
encoder_outputs, (h_n, c_n) = encoder(inputs)

decoder_inputs = torch.randn(5, 12, 20)
decoder_outputs, _ = decoder(decoder_inputs, (h_n, c_n))

print(h_n.shape, decoder_outputs.shape)

A torch.Size([5, 32]) torch.Size([5, 12, 20])

B torch.Size([5, 1, 32]) torch.Size([12, 5, 32])

C torch.Size([1, 5, 16]) torch.Size([5, 10, 32])

D torch.Size([1, 5, 32]) torch.Size([5, 12, 32])

💡 Hint

Remember LSTM hidden states have shape (num_layers * num_directions, batch, hidden_size).

The encoder's hidden state shape is (1, batch_size, hidden_size) because it has 1 layer and is unidirectional. The decoder output shape is (batch_size, sequence_length, hidden_size).

❓ Hyperparameter

advanced

Choosing the right hidden size for seq2seq LSTM

You are training a sequence-to-sequence model for machine translation. Which hidden size choice is most likely to improve model capacity without causing excessive overfitting?

A Increase hidden size from 128 to 512 with dropout and early stopping

B Decrease hidden size from 128 to 32 to reduce overfitting

C Keep hidden size at 128 and remove dropout layers

D Increase hidden size to 1024 without any regularization

💡 Hint

Think about balancing model complexity and regularization.

Increasing hidden size can improve capacity, but must be paired with regularization like dropout and early stopping to avoid overfitting.

❓ Metrics

advanced

Evaluating seq2seq model with BLEU score

You trained a seq2seq model for text summarization. Which metric best measures how well the model output matches human summaries?

A Confusion matrix of predicted classes

B BLEU score measuring n-gram overlap between model and reference summaries

C Mean Squared Error between input and output sequences

D Accuracy of predicting the next word in the input sequence

💡 Hint

Think about metrics used in natural language generation tasks.

BLEU score compares n-gram overlap between generated and reference text, making it suitable for summarization evaluation.

🔧 Debug

expert

Why does this seq2seq training loop cause exploding gradients?

Consider this simplified training loop for a seq2seq model. Why might the gradients explode?

NLP

for input_seq, target_seq in dataloader:
    optimizer.zero_grad()
    output_seq = model(input_seq, target_seq)
    loss = loss_fn(output_seq.view(-1, vocab_size), target_seq.view(-1))
    loss.backward()
    optimizer.step()

A No gradient clipping is applied, so gradients can grow too large during backpropagation

B The loss function is incorrect and returns zero, causing no gradient updates

C The optimizer.zero_grad() is called after loss.backward(), causing accumulation

D The model input and target sequences have mismatched batch sizes

💡 Hint

Think about common causes of exploding gradients in RNNs.

Without gradient clipping, RNNs like seq2seq models can have gradients that grow exponentially, causing instability during training.