Sequence-to-sequence architecture in NLP - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What is the main purpose of a sequence-to-sequence model?

Sequence-to-sequence models are widely used in tasks like language translation. What is their main purpose?

A To cluster data points into groups without labels

B To classify images into fixed categories

C To map an input sequence to an output sequence of possibly different length

D To reduce the dimensionality of input data

💡 Hint

Think about tasks like translating a sentence from English to French.

Sequence-to-sequence models take a sequence as input and produce another sequence as output, often of different length, such as translating sentences or summarizing text.

❓ Predict Output

intermediate

Output shape of encoder and decoder in a seq2seq model

Consider a seq2seq model with an LSTM encoder and decoder. The encoder processes input sequences of length 10 with 16 features, and the decoder outputs sequences of length 12 with 20 features. What is the shape of the encoder's final hidden state and the decoder's output?

NLP

import torch
import torch.nn as nn

encoder = nn.LSTM(input_size=16, hidden_size=32, batch_first=True)
decoder = nn.LSTM(input_size=20, hidden_size=32, batch_first=True)

inputs = torch.randn(5, 10, 16)  # batch_size=5
encoder_outputs, (h_n, c_n) = encoder(inputs)

decoder_inputs = torch.randn(5, 12, 20)
decoder_outputs, _ = decoder(decoder_inputs, (h_n, c_n))

print(h_n.shape, decoder_outputs.shape)

A torch.Size([5, 32]) torch.Size([5, 12, 20])

B torch.Size([5, 1, 32]) torch.Size([12, 5, 32])

C torch.Size([1, 5, 16]) torch.Size([5, 10, 32])

D torch.Size([1, 5, 32]) torch.Size([5, 12, 32])

💡 Hint

Remember LSTM hidden states have shape (num_layers * num_directions, batch, hidden_size).

The encoder's hidden state shape is (1, batch_size, hidden_size) because it has 1 layer and is unidirectional. The decoder output shape is (batch_size, sequence_length, hidden_size).

❓ Hyperparameter

advanced

Choosing the right hidden size for seq2seq LSTM

You are training a sequence-to-sequence model for machine translation. Which hidden size choice is most likely to improve model capacity without causing excessive overfitting?

A Increase hidden size from 128 to 512 with dropout and early stopping

B Decrease hidden size from 128 to 32 to reduce overfitting

C Keep hidden size at 128 and remove dropout layers

D Increase hidden size to 1024 without any regularization

💡 Hint

Think about balancing model complexity and regularization.

Increasing hidden size can improve capacity, but must be paired with regularization like dropout and early stopping to avoid overfitting.

❓ Metrics

advanced

Evaluating seq2seq model with BLEU score

You trained a seq2seq model for text summarization. Which metric best measures how well the model output matches human summaries?

A Confusion matrix of predicted classes

B BLEU score measuring n-gram overlap between model and reference summaries

C Mean Squared Error between input and output sequences

D Accuracy of predicting the next word in the input sequence

💡 Hint

Think about metrics used in natural language generation tasks.

BLEU score compares n-gram overlap between generated and reference text, making it suitable for summarization evaluation.

🔧 Debug

expert

Why does this seq2seq training loop cause exploding gradients?

Consider this simplified training loop for a seq2seq model. Why might the gradients explode?

NLP

for input_seq, target_seq in dataloader:
    optimizer.zero_grad()
    output_seq = model(input_seq, target_seq)
    loss = loss_fn(output_seq.view(-1, vocab_size), target_seq.view(-1))
    loss.backward()
    optimizer.step()

A No gradient clipping is applied, so gradients can grow too large during backpropagation

B The loss function is incorrect and returns zero, causing no gradient updates

C The optimizer.zero_grad() is called after loss.backward(), causing accumulation

D The model input and target sequences have mismatched batch sizes

💡 Hint

Think about common causes of exploding gradients in RNNs.

Without gradient clipping, RNNs like seq2seq models can have gradients that grow exponentially, causing instability during training.

Practice

1. What is the main role of the encoder in a sequence-to-sequence model?

easy

A. To generate the output sequence directly

B. To read and understand the input sequence

C. To evaluate the model's accuracy

D. To preprocess the data before training

Solution

1. Step 1: Understand the encoder's function

   The encoder processes the input sequence and converts it into a meaningful representation.

2. Step 2: Differentiate encoder from decoder

   The decoder uses this representation to generate the output sequence, so it does not directly read input.

3. Final Answer:

   To read and understand the input sequence -> Option B

4. Quick Check:

   Encoder = input reader [OK]

Hint: Encoder reads input; decoder writes output [OK]

Common Mistakes:

• Confusing encoder with decoder
• Thinking encoder generates output
• Assuming encoder evaluates accuracy

2. Which of the following is the correct way to describe the decoder in a sequence-to-sequence model?

easy

A. It generates the output sequence from the encoded input

B. It encodes the input sequence into a fixed vector

C. It normalizes the input data before encoding

D. It splits the input sequence into smaller parts

Solution

1. Step 1: Identify decoder's role

   The decoder takes the encoded input and produces the output sequence step-by-step.

2. Step 2: Eliminate incorrect options

   Encoding is done by the encoder, not the decoder; normalization and splitting are preprocessing steps.

3. Final Answer:

   It generates the output sequence from the encoded input -> Option A

4. Quick Check:

   Decoder = output generator [OK]

Hint: Decoder creates output from encoder's info [OK]

Common Mistakes:

• Mixing encoder and decoder roles
• Confusing preprocessing with decoding
• Assuming decoder encodes input

3. Consider this simplified pseudocode for a sequence-to-sequence model:

encoded = encoder(input_sequence)
output = decoder(encoded)
print(len(output))

If the input sequence length is 5 and the model is trained to translate to a sequence of length 7, what will len(output) print?

medium

A. 5

B. Cannot determine without more info

C. 12

D. 7

Solution

1. Step 1: Understand input and output lengths

   The input sequence length is 5, but the model is trained to produce output sequences of length 7.

2. Step 2: Recognize decoder output length

   The decoder generates output sequences based on training, so output length should be 7 regardless of input length.

3. Final Answer:

   7 -> Option D

4. Quick Check:

   Output length = trained target length = 7 [OK]

Hint: Output length matches target, not input length [OK]

Common Mistakes:

• Assuming output length equals input length
• Adding input and output lengths
• Saying output length is unknown

4. You have this code snippet for a sequence-to-sequence model training step:

for input_seq, target_seq in dataset:
    encoded = encoder(input_seq)
    output = decoder(encoded)
    loss = loss_function(output, target_seq)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

What is the likely error in this code?

medium

A. optimizer.zero_grad() should be called before loss.backward()

B. optimizer.step() should be called before loss.backward()

C. Missing call to optimizer.zero_grad() before loss.backward()

D. optimizer.zero_grad() should be called before optimizer.step()

Solution

1. Step 1: Recall training step order

   Gradients must be cleared before computing new gradients with loss.backward().

2. Step 2: Identify correct zero_grad() placement

   optimizer.zero_grad() should be called before loss.backward(), not after optimizer.step().

3. Final Answer:

   Missing call to optimizer.zero_grad() before loss.backward() -> Option C

4. Quick Check:

   Clear grads before backward pass [OK]

Hint: Call zero_grad() before backward() [OK]

Common Mistakes:

• Calling zero_grad() after backward()
• Calling optimizer.step() before backward()
• Skipping zero_grad() entirely

5. In a sequence-to-sequence model for language translation, why might adding an attention mechanism improve performance?

hard

A. It allows the decoder to focus on relevant parts of the input sequence dynamically

B. It reduces the size of the input sequence to a fixed vector

C. It speeds up training by skipping the encoder step

D. It replaces the decoder with a simpler model

Solution

1. Step 1: Understand attention's purpose

   Attention helps the decoder look at different parts of the input sequence when generating each output token.

2. Step 2: Compare with fixed vector encoding

   Without attention, the encoder compresses input into one fixed vector, which can lose details.

3. Step 3: Eliminate incorrect options

   Attention does not reduce input size, skip encoder, or replace decoder; it enhances focus during decoding.

4. Final Answer:

   It allows the decoder to focus on relevant parts of the input sequence dynamically -> Option A

5. Quick Check:

   Attention = dynamic focus on input [OK]

Hint: Attention helps decoder focus on input parts [OK]

Common Mistakes:

• Thinking attention reduces input size
• Believing attention skips encoder
• Assuming attention replaces decoder