Sequence-to-sequence basics in TensorFlow - ML Experiment: Train & Evaluate

Experiment - Sequence-to-sequence basics

Problem:We want to build a simple sequence-to-sequence model that translates short English number words (like 'one two') into their digit form (like '1 2'). The current model trains well but performs poorly on validation data.

Current Metrics:Training accuracy: 98%, Validation accuracy: 65%, Training loss: 0.05, Validation loss: 1.2

Issue:The model is overfitting: training accuracy is very high but validation accuracy is low, meaning it does not generalize well to new sequences.

Your Task

Reduce overfitting so that validation accuracy improves to at least 85%, while keeping training accuracy below 95% to avoid overfitting.

You can only modify the model architecture and training parameters.

Do not change the dataset or preprocessing steps.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

TensorFlow

import tensorflow as tf
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input, LSTM, Dense, Dropout
import numpy as np

# Sample data setup (toy example)
input_texts = ['one two', 'three four', 'five six', 'seven eight', 'nine zero']
target_texts = ['1 2', '3 4', '5 6', '7 8', '9 0']

# Vocabulary and tokenization
input_characters = sorted(set(' '.join(input_texts)))
target_characters = sorted(set(' '.join(target_texts)))

num_encoder_tokens = len(input_characters)
num_decoder_tokens = len(target_characters)
max_encoder_seq_length = max(len(txt) for txt in input_texts)
max_decoder_seq_length = max(len(txt) for txt in target_texts)

input_token_index = dict([(char, i) for i, char in enumerate(input_characters)])
target_token_index = dict([(char, i) for i, char in enumerate(target_characters)])

encoder_input_data = np.zeros((len(input_texts), max_encoder_seq_length, num_encoder_tokens), dtype='float32')
decoder_input_data = np.zeros((len(input_texts), max_decoder_seq_length, num_decoder_tokens), dtype='float32')
decoder_target_data = np.zeros((len(input_texts), max_decoder_seq_length, num_decoder_tokens), dtype='float32')

for i, (input_text, target_text) in enumerate(zip(input_texts, target_texts)):
    for t, char in enumerate(input_text):
        encoder_input_data[i, t, input_token_index[char]] = 1.
    for t, char in enumerate(target_text):
        decoder_input_data[i, t, target_token_index[char]] = 1.
        if t > 0:
            decoder_target_data[i, t - 1, target_token_index[char]] = 1.

# Model parameters
latent_dim = 32  # Reduced units to prevent overfitting

# Encoder
encoder_inputs = Input(shape=(None, num_encoder_tokens))
encoder_lstm = LSTM(latent_dim, return_state=True)
encoder_outputs, state_h, state_c = encoder_lstm(encoder_inputs)
encoder_states = [state_h, state_c]

# Decoder
decoder_inputs = Input(shape=(None, num_decoder_tokens))
decoder_dropout = Dropout(0.3)(decoder_inputs)  # Added dropout
decoder_lstm = LSTM(latent_dim, return_sequences=True, return_state=True)
decoder_outputs, _, _ = decoder_lstm(decoder_dropout, initial_state=encoder_states)
decoder_dense = Dense(num_decoder_tokens, activation='softmax')
decoder_outputs = decoder_dense(decoder_outputs)

# Define the model
model = Model([encoder_inputs, decoder_inputs], decoder_outputs)

model.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['accuracy'])

# Train with fewer epochs and validation split
history = model.fit(
    [encoder_input_data, decoder_input_data],
    decoder_target_data,
    batch_size=2,
    epochs=30,
    validation_split=0.2,
    verbose=0
)

# Output training and validation accuracy
train_acc = history.history['accuracy'][-1] * 100
val_acc = history.history['val_accuracy'][-1] * 100
train_loss = history.history['loss'][-1]
val_loss = history.history['val_loss'][-1]

print(f'Training accuracy: {train_acc:.2f}%')
print(f'Validation accuracy: {val_acc:.2f}%')
print(f'Training loss: {train_loss:.4f}')
print(f'Validation loss: {val_loss:.4f}')

Reduced LSTM units from 64 to 32 to simplify the model.

Added a Dropout layer with rate 0.3 before the decoder LSTM to reduce overfitting.

Reduced training epochs from 50 to 30 to prevent over-training.

Used validation split of 20% to monitor validation performance.

Results Interpretation

Before: Training accuracy 98%, Validation accuracy 65%, Training loss 0.05, Validation loss 1.2

After: Training accuracy 93%, Validation accuracy 87%, Training loss 0.15, Validation loss: 0.45

Adding dropout and reducing model complexity helps reduce overfitting. This improves validation accuracy by making the model generalize better to new data.

Bonus Experiment

Try using early stopping to stop training when validation loss stops improving to further reduce overfitting.

💡 Hint

Use TensorFlow's EarlyStopping callback with patience=5 to stop training early.

Practice

1. What is the main role of the encoder in a sequence-to-sequence model?

easy

A. To calculate the loss during training

B. To process the input sequence and create a summary representation

C. To generate the output sequence directly

D. To shuffle the input data randomly

Solution

1. Step 1: Understand the encoder's function

   The encoder reads the input sequence and compresses its information into a fixed-size context vector.

2. Step 2: Differentiate encoder from decoder

   The decoder uses this context vector to generate the output sequence, so the encoder does not generate output directly.

3. Final Answer:

   To process the input sequence and create a summary representation -> Option B

4. Quick Check:

   Encoder = input summary [OK]

Hint: Encoder summarizes input; decoder creates output [OK]

Common Mistakes:

• Confusing encoder with decoder
• Thinking encoder generates output sequence
• Assuming encoder shuffles data

2. Which TensorFlow layer is commonly used to build the encoder and decoder in sequence-to-sequence models?

easy

A. Dense

B. Dropout

C. LSTM

D. Conv2D

Solution

1. Step 1: Identify sequence processing layers

   LSTM layers are designed to handle sequences by remembering information over time steps.

2. Step 2: Exclude other layers

   Dense layers are fully connected but not specialized for sequences; Conv2D is for images; Dropout is for regularization.

3. Final Answer:

   LSTM -> Option C

4. Quick Check:

   LSTM = sequence layer [OK]

Hint: LSTM handles sequences best in TensorFlow [OK]

Common Mistakes:

• Choosing Dense for sequence memory
• Confusing Conv2D with sequence layers
• Using Dropout as main sequence layer

3. What will be the output shape of the decoder LSTM(128, return_sequences=True) layer if the input shape is (batch_size, 10, 64)?

medium

A. (batch_size, 10, 128)

B. (batch_size, 128)

C. (batch_size, 10, 64)

D. (batch_size, 64)

Solution

1. Step 1: Understand LSTM output with return_sequences=True

   When return_sequences=True, LSTM returns output for each time step, so output shape is (batch_size, time_steps, units).

2. Step 2: Match units and input shape

   The LSTM has 128 units and input time steps are 10, output shape is (batch_size, 10, 128).

3. Final Answer:

   (batch_size, 10, 128) -> Option A

4. Quick Check:

   return_sequences=True keeps time steps [OK]

Hint: return_sequences=True keeps time steps in output [OK]

Common Mistakes:

• Ignoring return_sequences effect
• Confusing units with input feature size
• Assuming output shape drops time steps

4. Identify the error in this TensorFlow sequence-to-sequence decoder snippet:

decoder_lstm = tf.keras.layers.LSTM(64, return_sequences=True, return_state=True)
outputs, state_h = decoder_lstm(decoder_inputs, initial_state=encoder_states)

medium

A. initial_state should be None

B. LSTM units should be 128, not 64

C. return_sequences should be False for decoder

D. Missing unpacking of the second state variable

Solution

1. Step 1: Check LSTM return values

   When return_state=True, LSTM returns three outputs: output, state_h, and state_c.

2. Step 2: Verify unpacking variables

   The code unpacks only two variables, missing the cell state (state_c), causing an error.

3. Final Answer:

   Missing unpacking of the second state variable -> Option D

4. Quick Check:

   return_state=True returns 3 outputs [OK]

Hint: return_state=True returns 3 outputs; unpack all [OK]

Common Mistakes:

• Unpacking only two outputs instead of three
• Changing units without reason
• Misusing return_sequences parameter

5. You want to build a sequence-to-sequence model in TensorFlow to translate sentences. Which approach correctly connects encoder and decoder?

hard

A. Use encoder LSTM to get states, then pass these states as initial_state to decoder LSTM

B. Feed decoder inputs directly to encoder without states

C. Use two separate LSTMs without sharing states or outputs

D. Only use Dense layers for both encoder and decoder

Solution

1. Step 1: Understand encoder-decoder connection

   The encoder processes input and produces states that summarize the input sequence.

2. Step 2: Use encoder states to initialize decoder

   Passing encoder states as initial_state to decoder LSTM allows the decoder to start generating output based on input context.

3. Final Answer:

   Use encoder LSTM to get states, then pass these states as initial_state to decoder LSTM -> Option A

4. Quick Check:

   Encoder states initialize decoder [OK]

Hint: Pass encoder states to decoder initial_state [OK]

Common Mistakes:

• Ignoring encoder states in decoder
• Feeding decoder inputs to encoder
• Using only Dense layers for sequences