Sequence-to-sequence basics in TensorFlow - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is a sequence-to-sequence (seq2seq) model?

A seq2seq model is a type of neural network that takes a sequence as input and produces another sequence as output. It is often used for tasks like language translation or text summarization.

beginner

What are the two main parts of a seq2seq model?

The two main parts are the encoder and the decoder. The encoder reads the input sequence and creates a summary, and the decoder uses this summary to generate the output sequence.

intermediate

Why do seq2seq models use recurrent neural networks (RNNs) or similar layers?

Because RNNs can process sequences of varying lengths by remembering information from previous steps, making them suitable for handling input and output sequences in seq2seq models.

intermediate

What is the role of the 'context vector' in a basic seq2seq model?

The context vector is the fixed-size summary output by the encoder that contains information about the entire input sequence. The decoder uses it to generate the output sequence.

advanced

What is teacher forcing in seq2seq training?

Teacher forcing is a training technique where the true output from the training data is fed as the next input to the decoder instead of the decoder's own previous prediction. This helps the model learn faster.

What does the encoder in a seq2seq model do?

A Generates the output sequence

B Processes the input sequence and creates a summary

C Calculates the loss function

D Feeds data into the training loop

The encoder reads the input sequence and creates a summary (context vector) for the decoder to use.

Which layer type is commonly used in seq2seq models to handle sequences?

A Convolutional layers

B Pooling layers

C Dense layers only

D Recurrent layers (RNN, LSTM, GRU)

Recurrent layers like LSTM or GRU are used because they can remember information across sequence steps.

What is the purpose of the decoder in a seq2seq model?

A To generate the output sequence from the context vector

B To encode the input sequence

C To preprocess the data

D To calculate accuracy

The decoder uses the context vector from the encoder to produce the output sequence.

What does teacher forcing help with during training?

A Removes the need for a decoder

B Slows down training

C Improves model learning by using true previous outputs

D Increases model size

Teacher forcing feeds the true previous output to the decoder, helping the model learn faster and better.

In a basic seq2seq model, what is the 'context vector'?

A A fixed-size summary of the input sequence

B The output sequence

C The input sequence

D The loss value

The context vector is a fixed-size summary produced by the encoder representing the input sequence.

Explain the roles of the encoder and decoder in a sequence-to-sequence model.

Think about how information flows from input to output.

Describe why recurrent layers like LSTM or GRU are important in seq2seq models.

Consider how the model keeps track of previous words or steps.

Practice

1. What is the main role of the encoder in a sequence-to-sequence model?

easy

A. To calculate the loss during training

B. To process the input sequence and create a summary representation

C. To generate the output sequence directly

D. To shuffle the input data randomly

Solution

1. Step 1: Understand the encoder's function

   The encoder reads the input sequence and compresses its information into a fixed-size context vector.

2. Step 2: Differentiate encoder from decoder

   The decoder uses this context vector to generate the output sequence, so the encoder does not generate output directly.

3. Final Answer:

   To process the input sequence and create a summary representation -> Option B

4. Quick Check:

   Encoder = input summary [OK]

Hint: Encoder summarizes input; decoder creates output [OK]

Common Mistakes:

• Confusing encoder with decoder
• Thinking encoder generates output sequence
• Assuming encoder shuffles data

2. Which TensorFlow layer is commonly used to build the encoder and decoder in sequence-to-sequence models?

easy

A. Dense

B. Dropout

C. LSTM

D. Conv2D

Solution

1. Step 1: Identify sequence processing layers

   LSTM layers are designed to handle sequences by remembering information over time steps.

2. Step 2: Exclude other layers

   Dense layers are fully connected but not specialized for sequences; Conv2D is for images; Dropout is for regularization.

3. Final Answer:

   LSTM -> Option C

4. Quick Check:

   LSTM = sequence layer [OK]

Hint: LSTM handles sequences best in TensorFlow [OK]

Common Mistakes:

• Choosing Dense for sequence memory
• Confusing Conv2D with sequence layers
• Using Dropout as main sequence layer

3. What will be the output shape of the decoder LSTM(128, return_sequences=True) layer if the input shape is (batch_size, 10, 64)?

medium

A. (batch_size, 10, 128)

B. (batch_size, 128)

C. (batch_size, 10, 64)

D. (batch_size, 64)

Solution

1. Step 1: Understand LSTM output with return_sequences=True

   When return_sequences=True, LSTM returns output for each time step, so output shape is (batch_size, time_steps, units).

2. Step 2: Match units and input shape

   The LSTM has 128 units and input time steps are 10, output shape is (batch_size, 10, 128).

3. Final Answer:

   (batch_size, 10, 128) -> Option A

4. Quick Check:

   return_sequences=True keeps time steps [OK]

Hint: return_sequences=True keeps time steps in output [OK]

Common Mistakes:

• Ignoring return_sequences effect
• Confusing units with input feature size
• Assuming output shape drops time steps

4. Identify the error in this TensorFlow sequence-to-sequence decoder snippet:

decoder_lstm = tf.keras.layers.LSTM(64, return_sequences=True, return_state=True)
outputs, state_h = decoder_lstm(decoder_inputs, initial_state=encoder_states)

medium

A. initial_state should be None

B. LSTM units should be 128, not 64

C. return_sequences should be False for decoder

D. Missing unpacking of the second state variable

Solution

1. Step 1: Check LSTM return values

   When return_state=True, LSTM returns three outputs: output, state_h, and state_c.

2. Step 2: Verify unpacking variables

   The code unpacks only two variables, missing the cell state (state_c), causing an error.

3. Final Answer:

   Missing unpacking of the second state variable -> Option D

4. Quick Check:

   return_state=True returns 3 outputs [OK]

Hint: return_state=True returns 3 outputs; unpack all [OK]

Common Mistakes:

• Unpacking only two outputs instead of three
• Changing units without reason
• Misusing return_sequences parameter

5. You want to build a sequence-to-sequence model in TensorFlow to translate sentences. Which approach correctly connects encoder and decoder?

hard

A. Use encoder LSTM to get states, then pass these states as initial_state to decoder LSTM

B. Feed decoder inputs directly to encoder without states

C. Use two separate LSTMs without sharing states or outputs

D. Only use Dense layers for both encoder and decoder

Solution

1. Step 1: Understand encoder-decoder connection

   The encoder processes input and produces states that summarize the input sequence.

2. Step 2: Use encoder states to initialize decoder

   Passing encoder states as initial_state to decoder LSTM allows the decoder to start generating output based on input context.

3. Final Answer:

   Use encoder LSTM to get states, then pass these states as initial_state to decoder LSTM -> Option A

4. Quick Check:

   Encoder states initialize decoder [OK]

Hint: Pass encoder states to decoder initial_state [OK]

Common Mistakes:

• Ignoring encoder states in decoder
• Feeding decoder inputs to encoder
• Using only Dense layers for sequences