Sequence-to-sequence architecture in NLP - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to create an embedding layer for the input sequence.

NLP

embedding_layer = nn.Embedding(num_embeddings=[1], embedding_dim=256)

Drag options to blanks, or click blank then click option'

Avocab_size

Bsequence_length

Cbatch_size

Dhidden_size

Attempts:

3 left

💡 Hint

Common Mistakes

Using sequence length instead of vocabulary size

Using batch size as number of embeddings

The embedding layer requires the vocabulary size as the number of embeddings.

2fill in blank

medium

Complete the code to initialize the encoder LSTM with the correct input size.

NLP

encoder_lstm = nn.LSTM(input_size=[1], hidden_size=512, batch_first=True)

Drag options to blanks, or click blank then click option'

Aembedding_dim

Bsequence_length

Cvocab_size

Dhidden_size

Attempts:

3 left

💡 Hint

Common Mistakes

Using vocabulary size as input size

Using sequence length as input size

The LSTM input size should match the embedding dimension, not the vocabulary size.

3fill in blank

hard

Fix the error in the decoder forward pass by selecting the correct input to the decoder LSTM.

NLP

decoder_output, (hidden, cell) = decoder_lstm([1], (hidden, cell))

Drag options to blanks, or click blank then click option'

Adecoder_hidden

Bencoder_output

Cencoder_input

Ddecoder_input

Attempts:

3 left

💡 Hint

Common Mistakes

Passing encoder output as input

Passing encoder input as input

The decoder LSTM takes the previous decoder input (usually the embedded previous token) as input.

4fill in blank

hard

Fill both blanks to complete the attention score calculation using dot product.

NLP

attention_scores = torch.bmm(encoder_outputs, [1].unsqueeze(2)).squeeze(2)
attention_weights = torch.softmax(attention_scores, dim=[2])

Drag options to blanks, or click blank then click option'

Adecoder_hidden

Bencoder_hidden

C1

D2

Attempts:

3 left

💡 Hint

Common Mistakes

Using encoder hidden instead of decoder hidden

Applying softmax over wrong dimension

The attention scores are computed by dot product of encoder outputs and decoder hidden state. Softmax is applied over the sequence length dimension (dim=1).

5fill in blank

hard

Fill all three blanks to complete the decoder output calculation with attention context.

NLP

context = torch.bmm(attention_weights.unsqueeze(1), [1])
combined = torch.cat((context.squeeze(1), [2]), dim=[3])
output = decoder_fc(combined)

Drag options to blanks, or click blank then click option'

Aencoder_outputs

Bdecoder_output.squeeze(1)

C1

D0

Attempts:

3 left

💡 Hint

Common Mistakes

Using decoder output instead of encoder outputs for context

Concatenating along wrong dimension

The context vector is computed from encoder outputs weighted by attention. It is concatenated with the decoder output along the feature dimension (dim=1) before passing to the final layer.

Practice

1. What is the main role of the encoder in a sequence-to-sequence model?

easy

A. To generate the output sequence directly

B. To read and understand the input sequence

C. To evaluate the model's accuracy

D. To preprocess the data before training

Solution

1. Step 1: Understand the encoder's function

   The encoder processes the input sequence and converts it into a meaningful representation.

2. Step 2: Differentiate encoder from decoder

   The decoder uses this representation to generate the output sequence, so it does not directly read input.

3. Final Answer:

   To read and understand the input sequence -> Option B

4. Quick Check:

   Encoder = input reader [OK]

Hint: Encoder reads input; decoder writes output [OK]

Common Mistakes:

• Confusing encoder with decoder
• Thinking encoder generates output
• Assuming encoder evaluates accuracy

2. Which of the following is the correct way to describe the decoder in a sequence-to-sequence model?

easy

A. It generates the output sequence from the encoded input

B. It encodes the input sequence into a fixed vector

C. It normalizes the input data before encoding

D. It splits the input sequence into smaller parts

Solution

1. Step 1: Identify decoder's role

   The decoder takes the encoded input and produces the output sequence step-by-step.

2. Step 2: Eliminate incorrect options

   Encoding is done by the encoder, not the decoder; normalization and splitting are preprocessing steps.

3. Final Answer:

   It generates the output sequence from the encoded input -> Option A

4. Quick Check:

   Decoder = output generator [OK]

Hint: Decoder creates output from encoder's info [OK]

Common Mistakes:

• Mixing encoder and decoder roles
• Confusing preprocessing with decoding
• Assuming decoder encodes input

3. Consider this simplified pseudocode for a sequence-to-sequence model:

encoded = encoder(input_sequence)
output = decoder(encoded)
print(len(output))

If the input sequence length is 5 and the model is trained to translate to a sequence of length 7, what will len(output) print?

medium

A. 5

B. Cannot determine without more info

C. 12

D. 7

Solution

1. Step 1: Understand input and output lengths

   The input sequence length is 5, but the model is trained to produce output sequences of length 7.

2. Step 2: Recognize decoder output length

   The decoder generates output sequences based on training, so output length should be 7 regardless of input length.

3. Final Answer:

   7 -> Option D

4. Quick Check:

   Output length = trained target length = 7 [OK]

Hint: Output length matches target, not input length [OK]

Common Mistakes:

• Assuming output length equals input length
• Adding input and output lengths
• Saying output length is unknown

4. You have this code snippet for a sequence-to-sequence model training step:

for input_seq, target_seq in dataset:
    encoded = encoder(input_seq)
    output = decoder(encoded)
    loss = loss_function(output, target_seq)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

What is the likely error in this code?

medium

A. optimizer.zero_grad() should be called before loss.backward()

B. optimizer.step() should be called before loss.backward()

C. Missing call to optimizer.zero_grad() before loss.backward()

D. optimizer.zero_grad() should be called before optimizer.step()

Solution

1. Step 1: Recall training step order

   Gradients must be cleared before computing new gradients with loss.backward().

2. Step 2: Identify correct zero_grad() placement

   optimizer.zero_grad() should be called before loss.backward(), not after optimizer.step().

3. Final Answer:

   Missing call to optimizer.zero_grad() before loss.backward() -> Option C

4. Quick Check:

   Clear grads before backward pass [OK]

Hint: Call zero_grad() before backward() [OK]

Common Mistakes:

• Calling zero_grad() after backward()
• Calling optimizer.step() before backward()
• Skipping zero_grad() entirely

5. In a sequence-to-sequence model for language translation, why might adding an attention mechanism improve performance?

hard

A. It allows the decoder to focus on relevant parts of the input sequence dynamically

B. It reduces the size of the input sequence to a fixed vector

C. It speeds up training by skipping the encoder step

D. It replaces the decoder with a simpler model

Solution

1. Step 1: Understand attention's purpose

   Attention helps the decoder look at different parts of the input sequence when generating each output token.

2. Step 2: Compare with fixed vector encoding

   Without attention, the encoder compresses input into one fixed vector, which can lose details.

3. Step 3: Eliminate incorrect options

   Attention does not reduce input size, skip encoder, or replace decoder; it enhances focus during decoding.

4. Final Answer:

   It allows the decoder to focus on relevant parts of the input sequence dynamically -> Option A

5. Quick Check:

   Attention = dynamic focus on input [OK]

Hint: Attention helps decoder focus on input parts [OK]

Common Mistakes:

• Thinking attention reduces input size
• Believing attention skips encoder
• Assuming attention replaces decoder