Why transfer learning saves time and data in TensorFlow - Experiment to Prove It

Experiment - Why transfer learning saves time and data

Problem:You want to train a model to recognize new types of images but have only a small dataset and limited time.

Current Metrics:Training accuracy: 95%, Validation accuracy: 60%, Training time: 2 hours

Issue:The model overfits due to small data and takes a long time to train from scratch.

Your Task

Use transfer learning to improve validation accuracy to at least 80% and reduce training time to under 30 minutes.

Use a pre-trained model as a base.

Fine-tune only the last layers.

Do not increase dataset size.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

TensorFlow

import tensorflow as tf
from tensorflow.keras import layers, models

# Load pre-trained MobileNetV2 without top layer
base_model = tf.keras.applications.MobileNetV2(input_shape=(128,128,3), include_top=False, weights='imagenet')
base_model.trainable = False  # Freeze base

# Add new classification head
model = models.Sequential([
    base_model,
    layers.GlobalAveragePooling2D(),
    layers.Dense(128, activation='relu'),
    layers.Dropout(0.3),
    layers.Dense(5, activation='softmax')  # Assuming 5 classes
])

model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

# Assume train_ds and val_ds are prepared tf.data datasets with batch size 32
# train_ds, val_ds = ... (user provides their datasets)

history = model.fit(train_ds, epochs=5, validation_data=val_ds)

# Optionally unfreeze some base layers and fine-tune
base_model.trainable = True
fine_tune_at = 100
for layer in base_model.layers[:fine_tune_at]:
    layer.trainable = False

model.compile(optimizer=tf.keras.optimizers.Adam(1e-5), loss='sparse_categorical_crossentropy', metrics=['accuracy'])
fine_tune_history = model.fit(train_ds, epochs=5, validation_data=val_ds)

Replaced training from scratch with a pre-trained MobileNetV2 base model.

Froze base model layers to keep learned features and reduce training time.

Added a small new classification head to adapt to the new task.

Trained only the new layers initially, then fine-tuned some base layers with a low learning rate.

Results Interpretation

Before Transfer Learning: Training accuracy was very high (95%) but validation accuracy was low (60%), showing overfitting. Training took 2 hours.

After Transfer Learning: Validation accuracy improved to 82%, training accuracy is closer at 85%, showing better generalization. Training time reduced to 25 minutes.

Transfer learning uses knowledge from a model trained on a large dataset to help learn new tasks faster and with less data. Freezing base layers keeps useful features, reducing overfitting and training time.

Bonus Experiment

Try unfreezing more layers of the base model and fine-tune with a smaller learning rate to see if validation accuracy improves further.

💡 Hint

Gradually unfreeze layers starting from the top and use a very low learning rate to avoid losing pre-trained knowledge.