Autoencoders in TensorFlow - ML Experiment: Train & Evaluate

Experiment - Autoencoders

Problem:Train an autoencoder to compress and reconstruct images from the MNIST dataset.

Current Metrics:Training loss: 0.02, Validation loss: 0.15

Issue:The model overfits: training loss is very low but validation loss is much higher, indicating poor generalization.

Your Task

Reduce overfitting by improving validation loss to below 0.08 while keeping training loss below 0.05.

Keep the same autoencoder architecture (encoder and decoder layers).

Do not increase the number of training epochs beyond 50.

Use TensorFlow and Keras only.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

TensorFlow

import tensorflow as tf
from tensorflow.keras import layers, models, regularizers
from tensorflow.keras.datasets import mnist
from tensorflow.keras.callbacks import EarlyStopping

# Load and normalize data
(x_train, _), (x_test, _) = mnist.load_data()
x_train = x_train.astype('float32') / 255.
x_test = x_test.astype('float32') / 255.
x_train = x_train.reshape((len(x_train), 28, 28, 1))
x_test = x_test.reshape((len(x_test), 28, 28, 1))

# Define encoder
input_img = layers.Input(shape=(28, 28, 1))
x = layers.Conv2D(32, (3, 3), activation='relu', padding='same', kernel_regularizer=regularizers.l2(1e-4))(input_img)
x = layers.Dropout(0.2)(x)
x = layers.MaxPooling2D((2, 2), padding='same')(x)
x = layers.Conv2D(16, (3, 3), activation='relu', padding='same', kernel_regularizer=regularizers.l2(1e-4))(x)
x = layers.Dropout(0.2)(x)
x = layers.MaxPooling2D((2, 2), padding='same')(x)
encoded = x

# Define decoder
x = layers.Conv2D(16, (3, 3), activation='relu', padding='same', kernel_regularizer=regularizers.l2(1e-4))(encoded)
x = layers.UpSampling2D((2, 2))(x)
x = layers.Conv2D(32, (3, 3), activation='relu', padding='same', kernel_regularizer=regularizers.l2(1e-4))(x)
x = layers.UpSampling2D((2, 2))(x)
decoded = layers.Conv2D(1, (3, 3), activation='sigmoid', padding='same')(x)

# Build autoencoder
autoencoder = models.Model(input_img, decoded)

# Compile model with lower learning rate
autoencoder.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.001), loss='binary_crossentropy')

# Early stopping callback
early_stop = EarlyStopping(monitor='val_loss', patience=5, restore_best_weights=True)

# Train model
history = autoencoder.fit(
    x_train, x_train,
    epochs=50,
    batch_size=128,
    shuffle=True,
    validation_data=(x_test, x_test),
    callbacks=[early_stop]
)

Added Dropout layers after convolutional layers to reduce overfitting.

Added L2 regularization to convolutional layers to penalize large weights.

Reduced learning rate to 0.001 for smoother training.

Added EarlyStopping callback to stop training when validation loss stops improving.

Results Interpretation

Before: Training loss = 0.02, Validation loss = 0.15 (high overfitting)

After: Training loss = 0.045, Validation loss = 0.075 (reduced overfitting)

Adding dropout, L2 regularization, and early stopping helps reduce overfitting in autoencoders, improving validation performance while keeping training loss low.

Bonus Experiment

Try using a convolutional variational autoencoder (VAE) instead of a standard autoencoder to improve the quality of reconstructed images.

💡 Hint

Implement the VAE loss with reconstruction loss plus KL divergence and use sampling in the encoder.

Practice

1. What is the main purpose of an autoencoder in machine learning?

easy

A. To compress data and then reconstruct it to learn important features

B. To classify images into categories

C. To generate new data from random noise

D. To sort data in ascending order

Solution

1. Step 1: Understand the role of autoencoders

   Autoencoders are designed to learn a compressed representation of data by encoding and then decoding it.

2. Step 2: Identify the main goal

   The goal is to capture important features by reconstructing the input data as closely as possible.

3. Final Answer:

   To compress data and then reconstruct it to learn important features -> Option A

4. Quick Check:

   Autoencoder purpose = compress and reconstruct [OK]

Hint: Autoencoders compress and rebuild data to learn features [OK]

Common Mistakes:

• Confusing autoencoders with classifiers
• Thinking autoencoders generate new random data
• Assuming autoencoders sort or label data

2. Which of the following is the correct way to define the encoder part of a simple autoencoder in TensorFlow using Keras Sequential API?

easy

A. encoder = tf.keras.Sequential([tf.keras.layers.Conv2D(32, (3,3), activation='relu', input_shape=(784,))])

B. encoder = tf.keras.Sequential([tf.keras.layers.Dense(32, activation='softmax', input_shape=(784,))])

C. encoder = tf.keras.Sequential([tf.keras.layers.Dense(32, activation='relu', input_shape=(784,))])

D. encoder = tf.keras.Sequential([tf.keras.layers.Dense(32)])

Solution

1. Step 1: Check the layer type and activation

   The encoder should use a Dense layer with 'relu' activation for feature extraction.

2. Step 2: Verify input shape and activation correctness

   Input shape (784,) matches flattened input, and 'relu' is suitable for encoding.

3. Final Answer:

   encoder = tf.keras.Sequential([tf.keras.layers.Dense(32, activation='relu', input_shape=(784,))]) -> Option C

4. Quick Check:

   Dense + relu + correct input shape = encoder = tf.keras.Sequential([tf.keras.layers.Dense(32, activation='relu', input_shape=(784,))]) [OK]

Hint: Use Dense with relu and correct input shape for encoder [OK]

Common Mistakes:

• Using softmax activation in encoder
• Using Conv2D layer with flat input shape
• Omitting activation function

3. Given the following TensorFlow code for an autoencoder, what will be the shape of the encoded output?

import tensorflow as tf
input_layer = tf.keras.layers.Input(shape=(28, 28, 1))
flat = tf.keras.layers.Flatten()(input_layer)
encoded = tf.keras.layers.Dense(16, activation='relu')(flat)
model = tf.keras.Model(inputs=input_layer, outputs=encoded)

sample_input = tf.random.normal([10, 28, 28, 1])
encoded_output = model(sample_input)
print(encoded_output.shape)

medium

A. (16, 10)

B. (10, 28, 28, 1)

C. (10, 784)

D. (10, 16)

Solution

1. Step 1: Analyze input and flatten layer

   Input shape is (28, 28, 1), flatten converts it to (784,) per sample.

2. Step 2: Check Dense layer output size

   The Dense layer outputs 16 units, so output shape per sample is (16,).

3. Step 3: Consider batch size

   Batch size is 10, so final output shape is (10, 16).

4. Final Answer:

   (10, 16) -> Option D

5. Quick Check:

   Batch 10 x Dense 16 units = (10, 16) [OK]

Hint: Flatten input then Dense units define output shape [OK]

Common Mistakes:

• Confusing input shape with output shape
• Ignoring batch size dimension
• Assuming output shape matches input shape

4. You wrote this autoencoder training code but get a shape mismatch error:

autoencoder.compile(optimizer='adam', loss='mse')
autoencoder.fit(x_train, x_train.reshape(-1, 784), epochs=10)

What is the likely cause?

medium

A. Input and output shapes do not match during training

B. Loss function 'mse' is not suitable for autoencoders

C. Optimizer 'adam' cannot be used with autoencoders

D. Epochs value 10 is too low

Solution

1. Step 1: Check input and output shapes in fit()

   Input is x_train, output is x_train reshaped to (-1, 784), shapes differ if model expects original shape.

2. Step 2: Understand autoencoder training requirement

   Autoencoder input and output shapes must match; reshaping output differently causes mismatch.

3. Final Answer:

   Input and output shapes do not match during training -> Option A

4. Quick Check:

   Input/output shape mismatch causes error [OK]

Hint: Ensure input and output shapes match exactly during training [OK]

Common Mistakes:

• Changing output shape differently from input
• Blaming optimizer or loss function incorrectly
• Ignoring shape requirements in fit()

5. You want to build an autoencoder to denoise images. Which approach best fits this task?

hard

A. Train the autoencoder with clean images as input and noisy images as output

B. Train the autoencoder with noisy images as input and clean images as output

C. Train the autoencoder only on clean images without noise

D. Train the autoencoder with random noise as both input and output

Solution

1. Step 1: Understand denoising autoencoder goal

   Denoising autoencoders learn to remove noise by mapping noisy input to clean output.

2. Step 2: Choose training data setup

   Input should be noisy images, output should be corresponding clean images for learning noise removal.

3. Final Answer:

   Train the autoencoder with noisy images as input and clean images as output -> Option B

4. Quick Check:

   Denoising = noisy input, clean output [OK]

Hint: Noisy input, clean output trains denoising autoencoder [OK]

Common Mistakes:

• Swapping input and output roles
• Training only on clean images without noise
• Using random noise as output