Data augmentation as regularization in TensorFlow - ML Experiment: Train & Evaluate

Experiment - Data augmentation as regularization

Problem:Train a neural network to classify images from the CIFAR-10 dataset. The current model achieves 95% training accuracy but only 70% validation accuracy.

Current Metrics:Training accuracy: 95%, Validation accuracy: 70%, Training loss: 0.15, Validation loss: 1.0

Issue:The model is overfitting: it performs very well on training data but poorly on validation data.

Your Task

Reduce overfitting by using data augmentation as a regularization technique to improve validation accuracy to at least 80% while keeping training accuracy below 90%.

You must keep the same model architecture.

You can only add data augmentation layers or techniques.

Do not change the optimizer or learning rate.

Hint 1

Hint 2

Hint 3

Solution

TensorFlow

import tensorflow as tf
from tensorflow.keras import layers, models

# Load CIFAR-10 dataset
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.cifar10.load_data()

# Normalize pixel values
x_train, x_test = x_train / 255.0, x_test / 255.0

# Define data augmentation pipeline
data_augmentation = tf.keras.Sequential([
    layers.RandomFlip('horizontal'),
    layers.RandomRotation(0.1),
    layers.RandomZoom(0.1),
])

# Define the model architecture (same as before)
inputs = layers.Input(shape=(32, 32, 3))

# Apply data augmentation only during training
x = data_augmentation(inputs)

x = layers.Conv2D(32, (3, 3), activation='relu')(x)
x = layers.MaxPooling2D((2, 2))(x)
x = layers.Conv2D(64, (3, 3), activation='relu')(x)
x = layers.MaxPooling2D((2, 2))(x)
x = layers.Conv2D(64, (3, 3), activation='relu')(x)

x = layers.Flatten()(x)
x = layers.Dense(64, activation='relu')(x)
outputs = layers.Dense(10, activation='softmax')(x)

model = models.Model(inputs=inputs, outputs=outputs)

# Compile the model
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

# Train the model with validation
history = model.fit(x_train, y_train, epochs=20, batch_size=64, validation_data=(x_test, y_test))

Added a data augmentation pipeline using RandomFlip, RandomRotation, and RandomZoom layers.

Inserted the augmentation pipeline as the first layer in the model to augment training images on the fly.

Kept the original model architecture and optimizer unchanged.

Results Interpretation

Before augmentation: Training accuracy was 95%, validation accuracy was 70%, showing strong overfitting.

After augmentation: Training accuracy decreased to 88%, validation accuracy improved to 82%, and validation loss decreased, indicating better generalization.

Data augmentation acts as a regularizer by increasing training data diversity, which helps the model generalize better and reduces overfitting.

Bonus Experiment

Try adding dropout layers after dense layers to further reduce overfitting and compare results.

💡 Hint

Add layers.Dropout(0.5) after the dense layer before the output layer and retrain the model.