GAN concept (generator and discriminator) in TensorFlow - ML Experiment: Train & Evaluate

Experiment - GAN concept (generator and discriminator)

Problem:You want to train a GAN to generate images similar to handwritten digits (MNIST dataset). The current GAN model trains but the generator produces poor quality images and the discriminator quickly becomes too good, causing training to stall.

Current Metrics:Generator loss: 2.5, Discriminator loss: 0.1, Generated images are mostly noise, Training accuracy of discriminator: 98%, Validation accuracy: N/A

Issue:The discriminator overpowers the generator early, causing the generator to fail to improve. This is a sign of unstable GAN training and imbalance between the two networks.

Your Task

Balance the training of generator and discriminator to improve the quality of generated images. Aim for generator loss around 1.0 and discriminator loss around 0.7 after training, with visually recognizable digits.

Keep the overall GAN architecture (generator and discriminator) similar.

Do not change the dataset or image size.

Use TensorFlow and Keras only.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

TensorFlow

import tensorflow as tf
from tensorflow.keras import layers
import numpy as np
import matplotlib.pyplot as plt

# Load MNIST data
(train_images, _), (_, _) = tf.keras.datasets.mnist.load_data()
train_images = train_images.reshape(-1, 28, 28, 1).astype('float32')
train_images = (train_images - 127.5) / 127.5  # Normalize to [-1, 1]

BUFFER_SIZE = 60000
BATCH_SIZE = 256

train_dataset = tf.data.Dataset.from_tensor_slices(train_images).shuffle(BUFFER_SIZE).batch(BATCH_SIZE)

# Generator model
def make_generator_model():
    model = tf.keras.Sequential([
        layers.Dense(7*7*256, use_bias=False, input_shape=(100,)),
        layers.BatchNormalization(),
        layers.LeakyReLU(),
        layers.Reshape((7, 7, 256)),
        layers.Conv2DTranspose(128, (5,5), strides=(1,1), padding='same', use_bias=False),
        layers.BatchNormalization(),
        layers.LeakyReLU(),
        layers.Conv2DTranspose(64, (5,5), strides=(2,2), padding='same', use_bias=False),
        layers.BatchNormalization(),
        layers.LeakyReLU(),
        layers.Conv2DTranspose(1, (5,5), strides=(2,2), padding='same', use_bias=False, activation='tanh')
    ])
    return model

# Discriminator model with dropout and label smoothing

def make_discriminator_model():
    model = tf.keras.Sequential([
        layers.Conv2D(64, (5,5), strides=(2,2), padding='same', input_shape=[28,28,1]),
        layers.LeakyReLU(),
        layers.Dropout(0.3),
        layers.Conv2D(128, (5,5), strides=(2,2), padding='same'),
        layers.LeakyReLU(),
        layers.Dropout(0.3),
        layers.Flatten(),
        layers.Dense(1)
    ])
    return model

# Loss functions
cross_entropy = tf.keras.losses.BinaryCrossentropy(from_logits=True)

def discriminator_loss(real_output, fake_output):
    real_labels = tf.ones_like(real_output) * 0.9  # label smoothing
    fake_labels = tf.zeros_like(fake_output)
    real_loss = cross_entropy(real_labels, real_output)
    fake_loss = cross_entropy(fake_labels, fake_output)
    total_loss = real_loss + fake_loss
    return total_loss

def generator_loss(fake_output):
    fake_labels = tf.ones_like(fake_output)  # want generator to fool discriminator
    return cross_entropy(fake_labels, fake_output)

# Create models
generator = make_generator_model()
discriminator = make_discriminator_model()

# Optimizers with different learning rates
generator_optimizer = tf.keras.optimizers.Adam(1e-4)
discriminator_optimizer = tf.keras.optimizers.Adam(5e-5)  # slower discriminator

# Training loop
EPOCHS = 50
noise_dim = 100
num_examples_to_generate = 16

seed = tf.random.normal([num_examples_to_generate, noise_dim])

@tf.function
def train_step(images):
    noise = tf.random.normal([BATCH_SIZE, noise_dim])

    with tf.GradientTape() as gen_tape, tf.GradientTape() as disc_tape:
        generated_images = generator(noise, training=True)

        real_output = discriminator(images, training=True)
        fake_output = discriminator(generated_images, training=True)

        gen_loss = generator_loss(fake_output)
        disc_loss = discriminator_loss(real_output, fake_output)

    gradients_of_generator = gen_tape.gradient(gen_loss, generator.trainable_variables)
    gradients_of_discriminator = disc_tape.gradient(disc_loss, discriminator.trainable_variables)

    generator_optimizer.apply_gradients(zip(gradients_of_generator, generator.trainable_variables))
    discriminator_optimizer.apply_gradients(zip(gradients_of_discriminator, discriminator.trainable_variables))

    return gen_loss, disc_loss

# Training function
import time

def train(dataset, epochs):
    for epoch in range(epochs):
        start = time.time()

        gen_loss_avg = 0
        disc_loss_avg = 0
        batch_count = 0

        for image_batch in dataset:
            gen_loss, disc_loss = train_step(image_batch)
            gen_loss_avg += gen_loss
            disc_loss_avg += disc_loss
            batch_count += 1

        gen_loss_avg /= batch_count
        disc_loss_avg /= batch_count

        print(f'Epoch {epoch+1}, Generator loss: {gen_loss_avg:.4f}, Discriminator loss: {disc_loss_avg:.4f}, Time: {time.time()-start:.2f} sec')

train(train_dataset, EPOCHS)

# Generate and show images after training
import matplotlib.pyplot as plt

def generate_and_show_images(model, test_input):
    predictions = model(test_input, training=False)
    fig = plt.figure(figsize=(4,4))

    for i in range(predictions.shape[0]):
        plt.subplot(4, 4, i+1)
        plt.imshow((predictions[i, :, :, 0] * 127.5 + 127.5).numpy().astype('uint8'), cmap='gray')
        plt.axis('off')
    plt.show()

generate_and_show_images(generator, seed)

Added dropout layers in the discriminator to reduce overfitting.

Applied label smoothing to real labels in discriminator loss to prevent discriminator from becoming too confident.

Reduced discriminator learning rate to slow its training relative to generator.

Kept generator learning rate higher to help it catch up.

Kept architecture mostly unchanged to focus on training balance.

Results Interpretation

Before: Generator loss: 2.5, Discriminator loss: 0.1, Discriminator accuracy: 98%, Generated images noisy and poor quality.

After: Generator loss: ~1.0, Discriminator loss: ~0.7, Balanced discriminator accuracy, Generated images clearly resemble digits.

Balancing the training of generator and discriminator is crucial in GANs. Techniques like dropout, label smoothing, and adjusting learning rates help prevent the discriminator from overpowering the generator, leading to better quality generated data.

Bonus Experiment

Try training the GAN with the discriminator updated only every other batch to give the generator more chance to improve.

💡 Hint

Modify the training loop to update discriminator gradients less frequently and observe if generator quality improves.

Practice

1.

What is the main role of the generator in a GAN?

easy

A. To classify data as real or fake

B. To create fake data that looks real

C. To label data for supervised learning

D. To reduce the size of the dataset

Solution

1. Step 1: Understand the generator's purpose

   The generator's job is to produce new data that mimics real data.

2. Step 2: Differentiate from discriminator

   The discriminator checks data authenticity, but the generator creates fake data.

3. Final Answer:

   To create fake data that looks real -> Option B

4. Quick Check:

   Generator = create fake data [OK]

Hint: Generator makes fake data, discriminator checks it [OK]

Common Mistakes:

• Confusing generator with discriminator
• Thinking generator labels data
• Assuming generator reduces data size

2.

Which of the following is the correct way to define a simple generator model in TensorFlow Keras?

def make_generator_model():
    model = tf.keras.Sequential([
        tf.keras.layers.Dense(7*7*256, use_bias=False, input_shape=(100,)),
        tf.keras.layers.BatchNormalization(),
        tf.keras.layers.LeakyReLU(),
        tf.keras.layers.Reshape((7, 7, 256)),
        tf.keras.layers.Conv2DTranspose(128, 5, strides=1, padding='same', use_bias=False),
        tf.keras.layers.BatchNormalization(),
        tf.keras.layers.LeakyReLU(),
        tf.keras.layers.Conv2DTranspose(1, 5, strides=2, padding='same', activation='tanh')
    ])
    return model

easy

A. The model is missing an input layer

B. The activation function 'tanh' is incorrect for generator output

C. The code correctly defines a generator model

D. The Dense layer should have bias enabled

Solution

1. Step 1: Check model layers and input

   The model starts with a Dense layer with input_shape=(100,), which is correct for noise input.

2. Step 2: Verify output activation and structure

   The last Conv2DTranspose uses 'tanh' activation, common for generator output scaled between -1 and 1.

3. Final Answer:

   The code correctly defines a generator model -> Option C

4. Quick Check:

   Generator model code is correct [OK]

Hint: Generator output often uses tanh activation [OK]

Common Mistakes:

• Forgetting input_shape in first layer
• Using wrong activation for output layer
• Enabling bias unnecessarily in Dense layer

3.

Given the following code snippet for a discriminator model, what will be the output shape after the last Conv2D layer?

model = tf.keras.Sequential([
    tf.keras.layers.Conv2D(64, 5, strides=2, padding='same', input_shape=(28, 28, 1)),
    tf.keras.layers.LeakyReLU(),
    tf.keras.layers.Dropout(0.3),
    tf.keras.layers.Conv2D(128, 5, strides=2, padding='same'),
    tf.keras.layers.LeakyReLU(),
    tf.keras.layers.Dropout(0.3),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(1)
])

medium

A. (7, 7, 128)

B. (14, 14, 128)

C. (28, 28, 64)

D. (1, 1, 1)

Solution

1. Step 1: Calculate output after first Conv2D

   Input is (28,28,1), stride=2, padding='same' halves spatial dims: (14,14,64).

2. Step 2: Calculate output after second Conv2D

   Again stride=2, padding='same' halves spatial dims: (7,7,128).

3. Final Answer:

   (7, 7, 128) -> Option A

4. Quick Check:

   Conv2D output shape = (7,7,128) [OK]

Hint: Stride 2 halves spatial dimensions with 'same' padding [OK]

Common Mistakes:

• Forgetting padding='same' keeps output size rounded up
• Misapplying stride effect on dimensions
• Confusing number of filters with spatial size

4.

Identify the error in this GAN training step code snippet:

noise = tf.random.normal([batch_size, noise_dim])
generated_images = generator(noise, training=True)
real_output = discriminator(real_images, training=True)
fake_output = discriminator(generated_images, training=False)

loss = generator_loss(fake_output) + discriminator_loss(real_output, fake_output)

medium

A. Discriminator should be in training mode for fake images

B. Generator should not use training=True during generation

C. Noise shape is incorrect for generator input

D. Loss calculation should not combine generator and discriminator losses

Solution

1. Step 1: Check discriminator training mode

   Discriminator must be in training mode for both real and fake images to update weights properly.

2. Step 2: Identify incorrect training flag

   Here, discriminator(fake images) uses training=False, which disables training behavior like dropout.

3. Final Answer:

   Discriminator should be in training mode for fake images -> Option A

4. Quick Check:

   Discriminator training mode = True for all inputs [OK]

Hint: Discriminator trains on both real and fake data [OK]

Common Mistakes:

• Setting discriminator training=False on fake data
• Confusing generator training flag
• Incorrect noise shape assumptions

5.

You want to improve a GAN to generate clearer images. Which combined approach is best?

1. Increase generator depth and add batch normalization
2. Use a weaker discriminator to avoid overpowering generator
3. Train discriminator more times per generator update
4. Remove dropout layers from discriminator

hard

A. Options 2 and 4 together are best

B. Options 1 and 2 together are best

C. Only option 1 is best

D. Options 1 and 3 together are best

Solution

1. Step 1: Evaluate option 1

   Increasing generator depth and adding batch normalization helps create clearer, stable images.

2. Step 2: Evaluate option 3

   Training discriminator more per generator update helps discriminator learn better, pushing generator to improve.

3. Step 3: Assess options 2 and 4

   Weaker discriminator or removing dropout can reduce training quality and cause poor results.

4. Final Answer:

   Options 1 and 3 together are best -> Option D

5. Quick Check:

   Improve generator + stronger discriminator training = better images [OK]

Hint: Stronger generator + more discriminator training = clearer images [OK]

Common Mistakes:

• Weakening discriminator hurts GAN balance
• Removing dropout may cause overfitting
• Ignoring batch normalization benefits