Fine-tuning approach in TensorFlow - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the fine-tuning approach in machine learning?

Fine-tuning is a method where you take a pre-trained model and continue training it on a new, often smaller dataset to adapt it to a specific task.

intermediate

Why do we freeze some layers during fine-tuning?

We freeze layers to keep the learned features intact and only update the later layers to adapt to the new task, which helps prevent overfitting and reduces training time.

beginner

In TensorFlow, how do you freeze layers of a pre-trained model?

You set the layer's 'trainable' attribute to False, for example: model.layers[i].trainable = False.

beginner

What is the benefit of using a pre-trained model for fine-tuning instead of training from scratch?

Pre-trained models have already learned useful features from large datasets, so fine-tuning saves time, requires less data, and often achieves better results.

intermediate

What is a common practice after freezing layers and before fine-tuning the model?

Compile the model again with a low learning rate to carefully adjust weights without large changes.

What does freezing layers in a model mean?

A Removing layers from the model

B Preventing layers from updating during training

C Adding new layers to the model

D Increasing the learning rate for some layers

Freezing layers means setting them so their weights do not change during training.

Why is fine-tuning useful when you have a small dataset?

A Because it trains the model from scratch

B Because it increases the dataset size automatically

C Because it ignores the small dataset

D Because it uses pre-learned features to adapt quickly

Fine-tuning leverages features learned from large datasets to work well on smaller datasets.

In TensorFlow, how do you make a layer non-trainable?

A layer.trainable = False

B layer.freeze()

C layer.trainable = True

D layer.stop_training = True

Setting layer.trainable = False freezes the layer during training.

What should you do after freezing layers before continuing training?

A Increase the batch size drastically

B Delete the optimizer

C Compile the model with a low learning rate

D Remove the frozen layers

Compiling with a low learning rate helps fine-tune the model gently.

Which of these is NOT a benefit of fine-tuning?

A Requires large new datasets

B Better performance on new tasks

C Faster training

D Uses pre-trained knowledge

Fine-tuning usually requires smaller datasets, not large ones.

Explain the steps involved in fine-tuning a pre-trained model using TensorFlow.

Think about what you keep, what you change, and how you train.

Describe why fine-tuning can improve model performance on a new task compared to training from scratch.

Consider the advantage of starting with knowledge already learned.

Practice

1. What is the main purpose of fine-tuning a pretrained TensorFlow model?

easy

A. To adapt the model to a new task by training some layers

B. To train a model from scratch with random weights

C. To increase the size of the dataset automatically

D. To reduce the number of layers in the model

Solution

1. Step 1: Understand fine-tuning concept

   Fine-tuning means adjusting a pretrained model to work well on a new task by training some of its layers.

2. Step 2: Compare options

   Only To adapt the model to a new task by training some layers correctly describes this process. Other options describe unrelated tasks.

3. Final Answer:

   To adapt the model to a new task by training some layers -> Option A

4. Quick Check:

   Fine-tuning = adapt pretrained model [OK]

Hint: Fine-tuning means training some layers on new data [OK]

Common Mistakes:

• Thinking fine-tuning means training from scratch
• Confusing fine-tuning with data augmentation
• Assuming fine-tuning reduces model size

2. Which TensorFlow code snippet correctly freezes the base model layers before fine-tuning?

easy

A. base_model.layers.freeze = True

B. base_model.trainable = False

C. base_model.freeze()

D. base_model.trainable = True

Solution

1. Step 1: Recall TensorFlow syntax for freezing layers

   In TensorFlow, setting trainable = False freezes the layers.

2. Step 2: Check options

   base_model.trainable = False uses correct syntax. Options A, B, and C are invalid methods/attributes or incorrect (e.g., trainable = True unfreezes layers).

3. Final Answer:

   base_model.trainable = False -> Option B

4. Quick Check:

   Freeze layers = trainable False [OK]

Hint: Freeze layers by setting trainable to False [OK]

Common Mistakes:

• Using incorrect method names like freeze()
• Setting trainable to True to freeze layers
• Trying to freeze layers by modifying layer attributes directly

3. What will be the output shape of the model after adding a new Dense layer with 10 units on top of a base model output shape (None, 1280)?

base_model = tf.keras.applications.MobileNetV2(input_shape=(224,224,3), include_top=False, pooling='avg')
new_output = tf.keras.layers.Dense(10, activation='softmax')(base_model.output)
model = tf.keras.Model(inputs=base_model.input, outputs=new_output)
print(model.output_shape)

medium

A. (None, 128)

B. (None, 224, 224, 3)

C. (None, 10)

D. (10, None)

Solution

1. Step 1: Understand base model output shape

   The base model outputs a tensor with shape (None, 1280) due to pooling='avg'.

2. Step 2: Adding Dense layer changes output shape

   The Dense layer with 10 units changes output shape to (None, 10), where None is batch size.

3. Final Answer:

   (None, 10) -> Option C

4. Quick Check:

   Dense units = output shape last dim [OK]

Hint: Dense layer units define output shape's last dimension [OK]

Common Mistakes:

• Confusing input shape with output shape
• Ignoring batch size dimension None
• Assuming output shape keeps base model shape

4. Identify the error in this fine-tuning code snippet:

base_model = tf.keras.applications.ResNet50(weights='imagenet', include_top=False)
base_model.trainable = False

model = tf.keras.Sequential([
    base_model,
    tf.keras.layers.Dense(5, activation='softmax')
])

model.compile(optimizer='adam', loss='sparse_categorical_crossentropy')
model.fit(train_data, epochs=5)

base_model.trainable = True
model.fit(train_data, epochs=5, learning_rate=0.001)

medium

A. Dense layer should be added before base_model

B. base_model.trainable should be True before first fit

C. Loss function is incorrect for classification

D. learning_rate is not a valid argument in model.fit

Solution

1. Step 1: Check model.fit arguments

   The model.fit method does not accept learning_rate as an argument; it belongs to the optimizer.

2. Step 2: Identify correct way to change learning rate

   To change learning rate, create a new optimizer with desired learning rate and recompile the model.

3. Final Answer:

   learning_rate is not a valid argument in model.fit -> Option D

4. Quick Check:

   learning_rate belongs to optimizer, not fit [OK]

Hint: learning_rate must be set in optimizer, not fit() [OK]

Common Mistakes:

• Passing learning_rate to model.fit
• Not recompiling model after changing trainable layers
• Freezing layers after first training instead of before

5. You want to fine-tune a pretrained model on a small dataset. Which approach is best to avoid overfitting?

# Step 1: Freeze base model
base_model.trainable = False
# Step 2: Add new classifier layers
# Step 3: Train new layers
# Step 4: Unfreeze some base layers
# Step 5: Continue training with low learning rate

Which of these is the best practice for Step 5?

hard

A. Use a very low learning rate like 1e-5 to fine-tune base layers

B. Use a high learning rate like 0.1 to speed up training

C. Keep learning rate same as initial training

D. Stop training immediately after unfreezing layers

Solution

1. Step 1: Understand fine-tuning with small data

   Fine-tuning with small data risks overfitting, so training must be gentle.

2. Step 2: Choose learning rate carefully

   A very low learning rate (e.g., 1e-5) helps adjust pretrained weights slowly to avoid overfitting.

3. Final Answer:

   Use a very low learning rate like 1e-5 to fine-tune base layers -> Option A

4. Quick Check:

   Low learning rate prevents overfitting in fine-tuning [OK]

Hint: Use low learning rate when unfreezing pretrained layers [OK]

Common Mistakes:

• Using high learning rate causing weight damage
• Not unfreezing any layers for fine-tuning
• Stopping training too early after unfreezing