Freezing and unfreezing layers in TensorFlow - ML Experiment: Train & Evaluate

Experiment - Freezing and unfreezing layers

Problem:You want to improve a neural network's performance on a small image dataset by using transfer learning. Currently, the model is trained by fine-tuning all layers from the start.

Current Metrics:Training accuracy: 95%, Validation accuracy: 70%, Training loss: 0.15, Validation loss: 0.65

Issue:The model is overfitting: training accuracy is very high but validation accuracy is much lower.

Your Task

Reduce overfitting by freezing the base model layers initially and then unfreezing some layers later. Target validation accuracy > 80% with training accuracy < 90%.

Use TensorFlow and Keras.

Start by freezing the base model layers.

Unfreeze only the top 20% of layers after initial training.

Do not change the dataset or model architecture.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

TensorFlow

import tensorflow as tf
from tensorflow.keras.applications import MobileNetV2
from tensorflow.keras import layers, models, optimizers

# Load base model with pretrained weights
base_model = MobileNetV2(input_shape=(128, 128, 3), include_top=False, weights='imagenet')

# Freeze all base model layers
for layer in base_model.layers:
    layer.trainable = False

# Add classification head
model = models.Sequential([
    base_model,
    layers.GlobalAveragePooling2D(),
    layers.Dense(128, activation='relu'),
    layers.Dropout(0.5),
    layers.Dense(10, activation='softmax')
])

# Compile model
model.compile(optimizer=optimizers.Adam(learning_rate=0.001),
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

# Assume train_ds and val_ds are prepared tf.data datasets
# Phase 1: Train with frozen base
history1 = model.fit(train_ds, epochs=5, validation_data=val_ds)

# Unfreeze top 20% layers of base model
num_layers = len(base_model.layers)
num_to_unfreeze = int(num_layers * 0.2)
for layer in base_model.layers[-num_to_unfreeze:]:
    layer.trainable = True

# Compile again with lower learning rate
model.compile(optimizer=optimizers.Adam(learning_rate=0.0001),
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

# Phase 2: Fine-tune
history2 = model.fit(train_ds, epochs=5, validation_data=val_ds)

# Output final metrics
final_train_acc = history2.history['accuracy'][-1] * 100
final_val_acc = history2.history['val_accuracy'][-1] * 100
final_train_loss = history2.history['loss'][-1]
final_val_loss = history2.history['val_loss'][-1]

print(f'Final training accuracy: {final_train_acc:.2f}%')
print(f'Final validation accuracy: {final_val_acc:.2f}%')
print(f'Final training loss: {final_train_loss:.4f}')
print(f'Final validation loss: {final_val_loss:.4f}')

Initially froze all layers of the pretrained base model to prevent overfitting.

Added a classification head on top of the frozen base.

Trained the model with frozen base layers for 5 epochs.

Unfroze the top 20% of base model layers to allow fine-tuning.

Lowered the learning rate during fine-tuning to avoid large weight updates.

Trained the model again for 5 epochs with partial unfreezing.

Results Interpretation

Before: Training accuracy 95%, Validation accuracy 70%, Training loss 0.15, Validation loss 0.65

After: Training accuracy 88%, Validation accuracy 82%, Training loss 0.25, Validation loss 0.40

Freezing pretrained layers initially helps prevent overfitting by keeping learned features stable. Later unfreezing some layers with a lower learning rate allows the model to adapt better to new data, improving validation accuracy.

Bonus Experiment

Try unfreezing different percentages of the base model layers (e.g., 10%, 50%) and observe how validation accuracy changes.

💡 Hint

Unfreeze fewer layers to reduce overfitting or more layers to increase model flexibility, but watch for training instability.

Practice

1. What does freezing layers in a TensorFlow model do?

easy

A. Stops the layers from updating their weights during training

B. Allows the layers to learn new features

C. Deletes the layers from the model

D. Increases the learning rate for those layers

Solution

1. Step 1: Understand the meaning of freezing layers

   Freezing layers means preventing them from changing their weights during training.

2. Step 2: Identify the effect of freezing

   Since weights do not update, the layers keep their learned knowledge intact.

3. Final Answer:

   Stops the layers from updating their weights during training -> Option A

4. Quick Check:

   Freezing = stop weight updates [OK]

Hint: Freezing means no weight updates during training [OK]

Common Mistakes:

• Thinking freezing deletes layers
• Believing freezing increases learning rate
• Confusing freezing with unfreezing

2. Which is the correct way to freeze a layer named layer in TensorFlow before compiling the model?

easy

A. layer.frozen = True

B. layer.freeze = True

C. layer.trainable = True

D. layer.trainable = False

Solution

1. Step 1: Recall the attribute to control training

   TensorFlow uses the trainable attribute to allow or prevent training on layers.

2. Step 2: Set the attribute to freeze

   Setting trainable = False freezes the layer so it won't update during training.

3. Final Answer:

   layer.trainable = False -> Option D

4. Quick Check:

   Freeze layer = trainable False [OK]

Hint: Use trainable = False to freeze layers before compile [OK]

Common Mistakes:

• Using non-existent attributes like freeze or frozen
• Setting trainable = True to freeze
• Changing trainable after compiling

3. Given the code below, what will be the output of model.layers[0].trainable after compiling?

model.layers[0].trainable = False
model.compile(optimizer='adam', loss='mse')
print(model.layers[0].trainable)

medium

A. None

B. True

C. False

D. Raises an error

Solution

1. Step 1: Set layer.trainable before compiling

   The code sets trainable = False before calling compile.

2. Step 2: Check if trainable attribute changes after compile

   The trainable attribute remains as set before compile; it does not change automatically.

3. Final Answer:

   False -> Option C

4. Quick Check:

   trainable stays False after compile [OK]

Hint: Set trainable before compile; it stays unchanged after [OK]

Common Mistakes:

• Assuming compile resets trainable to True
• Expecting an error from accessing trainable
• Confusing trainable with layer existence

4. You want to unfreeze a previously frozen layer in your TensorFlow model. Which code snippet fixes the error below?

AttributeError: 'Sequential' object has no attribute 'unfreeze'

Current code:

model.unfreeze()
model.compile(optimizer='adam', loss='mse')

medium

A. Set trainable on each layer: for layer in model.layers: layer.trainable = True

B. Set model.trainable = False instead

C. Call model.unfreeze() method

D. Rebuild the model from scratch

Solution

1. Step 1: Understand the error cause

   The error says Sequential model has no unfreeze attribute, meaning calling model.unfreeze() is invalid.

2. Step 2: Correct way to unfreeze layers

   Each layer has a trainable attribute; set it to True for all layers to unfreeze.

3. Final Answer:

   Set trainable on each layer: for layer in model.layers: layer.trainable = True -> Option A

4. Quick Check:

   Unfreeze layers by setting layer.trainable True [OK]

Hint: Set trainable on layers, not on model object [OK]

Common Mistakes:

• Trying to call model.unfreeze()
• Using non-existent unfreeze() method
• Rebuilding model unnecessarily

5. You have a pretrained model and want to freeze all layers except the last two for fine-tuning. Which code correctly achieves this before compiling?

hard

A. for layer in model.layers: layer.trainable = True

B. for layer in model.layers[:-2]: layer.trainable = False
for layer in model.layers[-2:]: layer.trainable = True

C. model.trainable = False
model.layers[-2:].trainable = True

D. model.freeze_layers(except_last=2)

Solution

1. Step 1: Freeze all layers except last two

   Use slicing to set trainable = False for all layers except last two.

2. Step 2: Unfreeze last two layers

   Set trainable = True for the last two layers to allow training.

3. Final Answer:

   for layer in model.layers[:-2]: layer.trainable = False for layer in model.layers[-2:]: layer.trainable = True -> Option B

4. Quick Check:

   Freeze all but last two layers [OK]

Hint: Freeze with slicing; unfreeze last layers explicitly [OK]

Common Mistakes:

• Trying to set trainable on model object
• Using non-existent freeze_layers method
• Setting all layers trainable True