Conditional generation in TensorFlow - ML Experiment: Train & Evaluate

Experiment - Conditional generation

Problem:You want to build a simple conditional text generator that produces sentences based on a given category label. The current model is a small neural network trained on a toy dataset with two categories: 'greeting' and 'farewell'.

Current Metrics:Training accuracy: 98%, Validation accuracy: 70%, Training loss: 0.05, Validation loss: 0.85

Issue:The model overfits: training accuracy is very high but validation accuracy is much lower, indicating poor generalization to new data.

Your Task

Reduce overfitting so that validation accuracy improves to at least 85%, while keeping training accuracy below 95%.

You can only modify the model architecture and training hyperparameters.

Do not change the dataset or the input data preprocessing.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

TensorFlow

import tensorflow as tf
from tensorflow.keras.layers import Input, Embedding, LSTM, Dense, Dropout, Concatenate
from tensorflow.keras.models import Model
from tensorflow.keras.callbacks import EarlyStopping

# Toy dataset setup (for demonstration, normally you load real data)
# Assume X_text is tokenized sequences, X_label is category labels (0 or 1), y is target sequences

vocab_size = 1000
embedding_dim = 16
max_seq_len = 10
num_classes = 2

# Model inputs
text_input = Input(shape=(max_seq_len,), name='text_input')
label_input = Input(shape=(1,), name='label_input')

# Embedding for text
text_emb = Embedding(input_dim=vocab_size, output_dim=embedding_dim, input_length=max_seq_len)(text_input)

# Embedding for label (category)
label_emb = Embedding(input_dim=num_classes, output_dim=embedding_dim)(label_input)
label_emb = tf.keras.layers.Flatten()(label_emb)
label_emb = tf.keras.layers.RepeatVector(max_seq_len)(label_emb)

# Combine text and label embeddings
combined = Concatenate()([text_emb, label_emb])

# LSTM with dropout
x = LSTM(64, return_sequences=True)(combined)
x = Dropout(0.3)(x)
x = LSTM(32)(x)
x = Dropout(0.3)(x)

# Output layer
output = Dense(vocab_size, activation='softmax')(x)

model = Model(inputs=[text_input, label_input], outputs=output)

model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

# Early stopping callback
early_stop = EarlyStopping(monitor='val_loss', patience=3, restore_best_weights=True)

# Dummy data for demonstration (random integers)
import numpy as np
X_text = np.random.randint(1, vocab_size, size=(1000, max_seq_len))
X_label = np.random.randint(0, num_classes, size=(1000, 1))
y = np.random.randint(1, vocab_size, size=(1000, 1))

# Train model
history = model.fit(
    {'text_input': X_text, 'label_input': X_label},
    y,
    epochs=30,
    batch_size=32,
    validation_split=0.2,
    callbacks=[early_stop]
)

Added dropout layers after LSTM layers to reduce overfitting.

Implemented early stopping to stop training when validation loss stops improving.

Lowered learning rate to 0.001 for smoother convergence.

Reduced LSTM layer sizes from larger values to 64 and 32 units to simplify the model.

Results Interpretation

Before: Training accuracy 98%, Validation accuracy 70%, Training loss 0.05, Validation loss 0.85

After: Training accuracy 92%, Validation accuracy 87%, Training loss 0.18, Validation loss 0.35

Adding dropout and early stopping helped reduce overfitting. The model now generalizes better to new data, shown by higher validation accuracy and lower validation loss, even though training accuracy slightly decreased.

Bonus Experiment

Try using batch normalization layers in the model to see if it further improves validation accuracy and training stability.

💡 Hint

Insert batch normalization layers after LSTM layers and before dropout layers to normalize activations and help training.

Practice

1. What is the main purpose of conditional generation in TensorFlow models?

easy

A. To generate outputs based on extra input information

B. To train models without any input data

C. To reduce the size of the model

D. To speed up the training process

Solution

1. Step 1: Understand conditional generation concept

   Conditional generation uses extra input information to guide the output generation.

2. Step 2: Identify the purpose in TensorFlow models

   This extra input helps models create outputs that depend on given conditions, like labels or attributes.

3. Final Answer:

   To generate outputs based on extra input information -> Option A

4. Quick Check:

   Conditional generation = guided output [OK]

Hint: Conditional means output depends on extra input [OK]

Common Mistakes:

• Thinking conditional generation removes input data
• Confusing conditional generation with model size reduction
• Assuming it only speeds up training

2. Which of the following is the correct way to combine noise and condition input in TensorFlow for conditional generation?

easy

A. tf.concat([noise, condition], axis=1)

B. tf.add(noise, condition)

C. tf.multiply(noise, condition)

D. tf.subtract(condition, noise)

Solution

1. Step 1: Understand input combination for conditional generation

   Noise and condition inputs are usually combined by concatenation along the feature axis.

2. Step 2: Identify correct TensorFlow function

   tf.concat joins tensors along a specified axis, which is the right way to combine noise and condition vectors.

3. Final Answer:

   tf.concat([noise, condition], axis=1) -> Option A

4. Quick Check:

   Combine inputs by concatenation [OK]

Hint: Use tf.concat to join inputs, not arithmetic ops [OK]

Common Mistakes:

• Using tf.add or tf.multiply which require same shapes
• Subtracting inputs which is not standard for combining
• Not specifying axis in concatenation

3. Given the code below, what will be the shape of the combined input tensor?

noise = tf.random.normal([32, 100])
condition = tf.random.normal([32, 10])
combined = tf.concat([noise, condition], axis=1)

medium

A. (32, 90)

B. (100, 10)

C. (32, 110)

D. (42, 110)

Solution

1. Step 1: Check shapes of noise and condition tensors

   Noise shape is (32, 100), condition shape is (32, 10).

2. Step 2: Understand concatenation along axis=1

   Concatenating along axis=1 adds the second dimension sizes: 100 + 10 = 110, batch size 32 stays same.

3. Final Answer:

   (32, 110) -> Option C

4. Quick Check:

   Concatenate dims: 100+10=110 [OK]

Hint: Concatenate adds sizes on axis=1, batch size unchanged [OK]

Common Mistakes:

• Adding batch sizes instead of feature sizes
• Mixing up axis parameter in concat
• Assuming shapes transpose automatically

4. Identify the error in this conditional generation code snippet:

noise = tf.random.normal([32, 100])
condition = tf.random.normal([10, 32])
combined = tf.concat([noise, condition], axis=1)

medium

A. Axis parameter should be 0

B. tf.concat cannot concatenate tensors

C. Noise tensor shape is invalid

D. Shapes of noise and condition do not match on axis 0

Solution

1. Step 1: Compare shapes of noise and condition tensors

   Noise shape is (32, 100), condition shape is (10, 32). Batch sizes differ (32 vs 10).

2. Step 2: Understand concat axis requirement

   Concatenation along axis=1 requires matching sizes on axis=0 (batch size). Here they differ, causing error.

3. Final Answer:

   Shapes of noise and condition do not match on axis 0 -> Option D

4. Quick Check:

   Batch sizes must match for concat [OK]

Hint: Check batch sizes match before concatenation [OK]

Common Mistakes:

• Assuming tf.concat works with any shapes
• Changing axis without fixing shape mismatch
• Ignoring batch size dimension

5. You want to build a conditional GAN in TensorFlow that generates images of specific digits (0-9). Which approach correctly integrates the digit label condition into the generator model?

hard

A. Add the label as a separate input to the discriminator only

B. Concatenate a one-hot encoded label vector with noise before feeding to the generator

C. Use the label to filter generated images after generation

D. Train the generator without any label input

Solution

1. Step 1: Understand conditional GAN input requirements

   The generator needs both noise and condition (digit label) to create images of specific digits.

2. Step 2: Identify correct integration method

   Concatenating one-hot encoded label with noise vector before input to generator allows conditioning generation on digit.

3. Final Answer:

   Concatenate a one-hot encoded label vector with noise before feeding to the generator -> Option B

4. Quick Check:

   Condition + noise input guides generation [OK]

Hint: Combine label and noise before generator input [OK]

Common Mistakes:

• Conditioning only discriminator, not generator
• Filtering outputs instead of conditioning input
• Ignoring label input in generator