Sequence-to-sequence basics in TensorFlow - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What is the main purpose of a sequence-to-sequence model?

In simple terms, what does a sequence-to-sequence model do?

A It predicts a single number from a fixed set of input features.

B It converts one sequence of data into another sequence, like translating a sentence from one language to another.

C It classifies images into categories based on their content.

D It clusters data points into groups without labels.

💡 Hint

Think about tasks like language translation or summarizing text.

Sequence-to-sequence models take a sequence as input and produce a sequence as output, useful for tasks like translation or speech recognition.

❓ Predict Output

intermediate

Output shape of encoder LSTM in a seq2seq model

Given the following TensorFlow code for an encoder LSTM, what is the shape of the output 'encoder_outputs'?

TensorFlow

import tensorflow as tf
encoder_inputs = tf.keras.Input(shape=(None, 10))
encoder_lstm = tf.keras.layers.LSTM(16, return_sequences=True, return_state=True)
encoder_outputs, state_h, state_c = encoder_lstm(encoder_inputs)

A (None, None, 16)

B (None, 16)

C (None, 10, 16)

D (None, 16, 16)

💡 Hint

return_sequences=True means output has one vector per input time step.

With return_sequences=True, the output shape is (batch_size, timesteps, units). Here units=16 and timesteps is variable (None).

❓ Model Choice

advanced

Choosing the right decoder output activation for sequence generation

You are building a sequence-to-sequence model to generate text word-by-word from a vocabulary of 5000 words. Which decoder output activation function is most appropriate?

A Softmax activation to output probabilities over the vocabulary

B Linear activation to output raw scores without normalization

C ReLU activation to output positive scores for each word

D Sigmoid activation to output independent probabilities for each word

💡 Hint

Think about how to pick one word from many possible words at each step.

Softmax outputs a probability distribution over all words, allowing selection of the most likely next word.

❓ Hyperparameter

advanced

Effect of increasing LSTM units in a seq2seq model

What is the most likely effect of increasing the number of units in the LSTM layers of a sequence-to-sequence model?

A The model will ignore input sequences and output random predictions

B The model will train faster and use less memory

C The model will always perform worse due to overfitting

D The model can capture more complex patterns but may overfit if data is limited

💡 Hint

More units mean more capacity but also more risk of memorizing training data.

Increasing units increases model capacity, helping learn complex patterns but risking overfitting without enough data.

❓ Metrics

expert

Choosing the best metric for evaluating a seq2seq translation model

You trained a sequence-to-sequence model for language translation. Which metric best measures how close the model's output sentences are to the correct translations?

A Accuracy, counting exact matches of entire output sequences

B Mean Squared Error, measuring average squared difference between predicted and true word indices

C BLEU score, which compares overlapping n-grams between output and reference sentences

D Perplexity, measuring how well the model predicts the next word probabilities

💡 Hint

Think about a metric that compares similarity of sentences rather than exact matches.

BLEU score measures n-gram overlap, capturing partial matches and fluency in translation quality.

Practice

1. What is the main role of the encoder in a sequence-to-sequence model?

easy

A. To calculate the loss during training

B. To process the input sequence and create a summary representation

C. To generate the output sequence directly

D. To shuffle the input data randomly

Solution

1. Step 1: Understand the encoder's function

   The encoder reads the input sequence and compresses its information into a fixed-size context vector.

2. Step 2: Differentiate encoder from decoder

   The decoder uses this context vector to generate the output sequence, so the encoder does not generate output directly.

3. Final Answer:

   To process the input sequence and create a summary representation -> Option B

4. Quick Check:

   Encoder = input summary [OK]

Hint: Encoder summarizes input; decoder creates output [OK]

Common Mistakes:

• Confusing encoder with decoder
• Thinking encoder generates output sequence
• Assuming encoder shuffles data

2. Which TensorFlow layer is commonly used to build the encoder and decoder in sequence-to-sequence models?

easy

A. Dense

B. Dropout

C. LSTM

D. Conv2D

Solution

1. Step 1: Identify sequence processing layers

   LSTM layers are designed to handle sequences by remembering information over time steps.

2. Step 2: Exclude other layers

   Dense layers are fully connected but not specialized for sequences; Conv2D is for images; Dropout is for regularization.

3. Final Answer:

   LSTM -> Option C

4. Quick Check:

   LSTM = sequence layer [OK]

Hint: LSTM handles sequences best in TensorFlow [OK]

Common Mistakes:

• Choosing Dense for sequence memory
• Confusing Conv2D with sequence layers
• Using Dropout as main sequence layer

3. What will be the output shape of the decoder LSTM(128, return_sequences=True) layer if the input shape is (batch_size, 10, 64)?

medium

A. (batch_size, 10, 128)

B. (batch_size, 128)

C. (batch_size, 10, 64)

D. (batch_size, 64)

Solution

1. Step 1: Understand LSTM output with return_sequences=True

   When return_sequences=True, LSTM returns output for each time step, so output shape is (batch_size, time_steps, units).

2. Step 2: Match units and input shape

   The LSTM has 128 units and input time steps are 10, output shape is (batch_size, 10, 128).

3. Final Answer:

   (batch_size, 10, 128) -> Option A

4. Quick Check:

   return_sequences=True keeps time steps [OK]

Hint: return_sequences=True keeps time steps in output [OK]

Common Mistakes:

• Ignoring return_sequences effect
• Confusing units with input feature size
• Assuming output shape drops time steps

4. Identify the error in this TensorFlow sequence-to-sequence decoder snippet:

decoder_lstm = tf.keras.layers.LSTM(64, return_sequences=True, return_state=True)
outputs, state_h = decoder_lstm(decoder_inputs, initial_state=encoder_states)

medium

A. initial_state should be None

B. LSTM units should be 128, not 64

C. return_sequences should be False for decoder

D. Missing unpacking of the second state variable

Solution

1. Step 1: Check LSTM return values

   When return_state=True, LSTM returns three outputs: output, state_h, and state_c.

2. Step 2: Verify unpacking variables

   The code unpacks only two variables, missing the cell state (state_c), causing an error.

3. Final Answer:

   Missing unpacking of the second state variable -> Option D

4. Quick Check:

   return_state=True returns 3 outputs [OK]

Hint: return_state=True returns 3 outputs; unpack all [OK]

Common Mistakes:

• Unpacking only two outputs instead of three
• Changing units without reason
• Misusing return_sequences parameter

5. You want to build a sequence-to-sequence model in TensorFlow to translate sentences. Which approach correctly connects encoder and decoder?

hard

A. Use encoder LSTM to get states, then pass these states as initial_state to decoder LSTM

B. Feed decoder inputs directly to encoder without states

C. Use two separate LSTMs without sharing states or outputs

D. Only use Dense layers for both encoder and decoder

Solution

1. Step 1: Understand encoder-decoder connection

   The encoder processes input and produces states that summarize the input sequence.

2. Step 2: Use encoder states to initialize decoder

   Passing encoder states as initial_state to decoder LSTM allows the decoder to start generating output based on input context.

3. Final Answer:

   Use encoder LSTM to get states, then pass these states as initial_state to decoder LSTM -> Option A

4. Quick Check:

   Encoder states initialize decoder [OK]

Hint: Pass encoder states to decoder initial_state [OK]

Common Mistakes:

• Ignoring encoder states in decoder
• Feeding decoder inputs to encoder
• Using only Dense layers for sequences