Fine-tuning approach in TensorFlow - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Model Choice

intermediate

Choosing the right layer to fine-tune

You have a pre-trained convolutional neural network for image classification. You want to fine-tune it on a new dataset with similar images but different classes. Which layer is best to fine-tune to adapt the model effectively?

A The last few convolutional layers and the fully connected layers

B All layers including the input layer

C Only the first convolutional layer

D Only the last fully connected (dense) layer

💡 Hint

Think about which layers capture general features versus task-specific features.

The first layers learn general features like edges, which are useful across tasks. The last layers learn task-specific features. Fine-tuning the last few convolutional layers and fully connected layers allows adapting to new classes while keeping general features intact.

❓ Hyperparameter

intermediate

Selecting learning rate for fine-tuning

When fine-tuning a pre-trained model, which learning rate setting is generally recommended to avoid destroying the learned features?

A Use a very high learning rate like 0.1

B Use no learning rate, keep weights frozen

C Use a smaller learning rate than training from scratch, like 0.0001

D Use the same learning rate as training from scratch

💡 Hint

Consider how big changes to weights affect pre-trained knowledge.

A smaller learning rate helps make gentle updates to the pre-trained weights, preserving useful features while adapting to new data. A high learning rate can overwrite learned features too quickly.

❓ Predict Output

advanced

Output of fine-tuning code snippet

What will be the printed output after running this TensorFlow fine-tuning code snippet?

TensorFlow

import tensorflow as tf
from tensorflow.keras.applications import MobileNetV2

base_model = MobileNetV2(weights='imagenet', include_top=False, input_shape=(96,96,3))
base_model.trainable = False

model = tf.keras.Sequential([
    base_model,
    tf.keras.layers.GlobalAveragePooling2D(),
    tf.keras.layers.Dense(10, activation='softmax')
])

model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

print(f"Trainable layers before fine-tuning: {sum([layer.trainable for layer in base_model.layers])}")

base_model.trainable = True

print(f"Trainable layers after setting trainable=True: {sum([layer.trainable for layer in base_model.layers])}")

A

Trainable layers before fine-tuning: 0
Trainable layers after setting trainable=True: 0

B

Trainable layers before fine-tuning: 0
Trainable layers after setting trainable=True: 154

C

Trainable layers before fine-tuning: 154
Trainable layers after setting trainable=True: 0

D

Trainable layers before fine-tuning: 154
Trainable layers after setting trainable=True: 154

💡 Hint

Check the trainable attribute before and after changing it.

Initially, base_model.trainable is set to False, so no layers are trainable (0). After setting base_model.trainable = True, all layers become trainable. MobileNetV2 has 154 layers, so the count is 154.

❓ Metrics

advanced

Interpreting fine-tuning training metrics

You fine-tune a pre-trained model on a small dataset. After 10 epochs, training accuracy is 98% but validation accuracy is 70%. What does this indicate?

A The model has a bug in the code

B The model is perfectly generalized

C The model is underfitting the training data

D The model is overfitting the training data

💡 Hint

Compare training and validation accuracy to assess model behavior.

High training accuracy but much lower validation accuracy means the model learned the training data too well but does not generalize to new data, which is overfitting.

🔧 Debug

expert

Debugging fine-tuning freezing layers issue

You want to fine-tune only the last 10 layers of a pre-trained model in TensorFlow. You set base_model.trainable = True and then set the first layers' trainable attribute to False in a loop. However, after training, you notice all layers are still being updated. What is the likely cause?

TensorFlow

base_model.trainable = True
for layer in base_model.layers[:-10]:
    layer.trainable = False

model.compile(optimizer='adam', loss='sparse_categorical_crossentropy')
model.fit(train_data, epochs=5)

A You must compile the model after changing layer trainable attributes

B Setting base_model.trainable = True overrides individual layer settings

C You need to set model.trainable = False instead

D The optimizer must be reset to apply layer freezing

💡 Hint

Think about when TensorFlow reads layer trainable flags.

In TensorFlow, changing layer trainable flags after compiling the model has no effect until you recompile. The optimizer and trainable variables are fixed at compile time.

Practice

1. What is the main purpose of fine-tuning a pretrained TensorFlow model?

easy

A. To adapt the model to a new task by training some layers

B. To train a model from scratch with random weights

C. To increase the size of the dataset automatically

D. To reduce the number of layers in the model

Solution

1. Step 1: Understand fine-tuning concept

   Fine-tuning means adjusting a pretrained model to work well on a new task by training some of its layers.

2. Step 2: Compare options

   Only To adapt the model to a new task by training some layers correctly describes this process. Other options describe unrelated tasks.

3. Final Answer:

   To adapt the model to a new task by training some layers -> Option A

4. Quick Check:

   Fine-tuning = adapt pretrained model [OK]

Hint: Fine-tuning means training some layers on new data [OK]

Common Mistakes:

• Thinking fine-tuning means training from scratch
• Confusing fine-tuning with data augmentation
• Assuming fine-tuning reduces model size

2. Which TensorFlow code snippet correctly freezes the base model layers before fine-tuning?

easy

A. base_model.layers.freeze = True

B. base_model.trainable = False

C. base_model.freeze()

D. base_model.trainable = True

Solution

1. Step 1: Recall TensorFlow syntax for freezing layers

   In TensorFlow, setting trainable = False freezes the layers.

2. Step 2: Check options

   base_model.trainable = False uses correct syntax. Options A, B, and C are invalid methods/attributes or incorrect (e.g., trainable = True unfreezes layers).

3. Final Answer:

   base_model.trainable = False -> Option B

4. Quick Check:

   Freeze layers = trainable False [OK]

Hint: Freeze layers by setting trainable to False [OK]

Common Mistakes:

• Using incorrect method names like freeze()
• Setting trainable to True to freeze layers
• Trying to freeze layers by modifying layer attributes directly

3. What will be the output shape of the model after adding a new Dense layer with 10 units on top of a base model output shape (None, 1280)?

base_model = tf.keras.applications.MobileNetV2(input_shape=(224,224,3), include_top=False, pooling='avg')
new_output = tf.keras.layers.Dense(10, activation='softmax')(base_model.output)
model = tf.keras.Model(inputs=base_model.input, outputs=new_output)
print(model.output_shape)

medium

A. (None, 128)

B. (None, 224, 224, 3)

C. (None, 10)

D. (10, None)

Solution

1. Step 1: Understand base model output shape

   The base model outputs a tensor with shape (None, 1280) due to pooling='avg'.

2. Step 2: Adding Dense layer changes output shape

   The Dense layer with 10 units changes output shape to (None, 10), where None is batch size.

3. Final Answer:

   (None, 10) -> Option C

4. Quick Check:

   Dense units = output shape last dim [OK]

Hint: Dense layer units define output shape's last dimension [OK]

Common Mistakes:

• Confusing input shape with output shape
• Ignoring batch size dimension None
• Assuming output shape keeps base model shape

4. Identify the error in this fine-tuning code snippet:

base_model = tf.keras.applications.ResNet50(weights='imagenet', include_top=False)
base_model.trainable = False

model = tf.keras.Sequential([
    base_model,
    tf.keras.layers.Dense(5, activation='softmax')
])

model.compile(optimizer='adam', loss='sparse_categorical_crossentropy')
model.fit(train_data, epochs=5)

base_model.trainable = True
model.fit(train_data, epochs=5, learning_rate=0.001)

medium

A. Dense layer should be added before base_model

B. base_model.trainable should be True before first fit

C. Loss function is incorrect for classification

D. learning_rate is not a valid argument in model.fit

Solution

1. Step 1: Check model.fit arguments

   The model.fit method does not accept learning_rate as an argument; it belongs to the optimizer.

2. Step 2: Identify correct way to change learning rate

   To change learning rate, create a new optimizer with desired learning rate and recompile the model.

3. Final Answer:

   learning_rate is not a valid argument in model.fit -> Option D

4. Quick Check:

   learning_rate belongs to optimizer, not fit [OK]

Hint: learning_rate must be set in optimizer, not fit() [OK]

Common Mistakes:

• Passing learning_rate to model.fit
• Not recompiling model after changing trainable layers
• Freezing layers after first training instead of before

5. You want to fine-tune a pretrained model on a small dataset. Which approach is best to avoid overfitting?

# Step 1: Freeze base model
base_model.trainable = False
# Step 2: Add new classifier layers
# Step 3: Train new layers
# Step 4: Unfreeze some base layers
# Step 5: Continue training with low learning rate

Which of these is the best practice for Step 5?

hard

A. Use a very low learning rate like 1e-5 to fine-tune base layers

B. Use a high learning rate like 0.1 to speed up training

C. Keep learning rate same as initial training

D. Stop training immediately after unfreezing layers

Solution

1. Step 1: Understand fine-tuning with small data

   Fine-tuning with small data risks overfitting, so training must be gentle.

2. Step 2: Choose learning rate carefully

   A very low learning rate (e.g., 1e-5) helps adjust pretrained weights slowly to avoid overfitting.

3. Final Answer:

   Use a very low learning rate like 1e-5 to fine-tune base layers -> Option A

4. Quick Check:

   Low learning rate prevents overfitting in fine-tuning [OK]

Hint: Use low learning rate when unfreezing pretrained layers [OK]

Common Mistakes:

• Using high learning rate causing weight damage
• Not unfreezing any layers for fine-tuning
• Stopping training too early after unfreezing