Learning rate for fine-tuning in TensorFlow - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Why use a smaller learning rate for fine-tuning?

When fine-tuning a pre-trained neural network, why is it common to use a smaller learning rate compared to training from scratch?

A Because a smaller learning rate helps preserve the learned features and avoids large updates that could destroy useful information.

B Because a smaller learning rate prevents the model from converging too quickly to a good solution.

C Because a smaller learning rate speeds up training by making bigger jumps in the parameter space.

D Because a smaller learning rate increases the chance of overfitting the training data.

💡 Hint

Think about what happens if you change the pre-trained weights too much.

Fine-tuning starts from a model that already learned useful features. Using a smaller learning rate ensures updates are gentle, preserving these features while adapting to new data.

❓ Predict Output

intermediate

Output of learning rate schedule during fine-tuning

Consider this TensorFlow code snippet that sets a learning rate schedule for fine-tuning:

import tensorflow as tf

initial_lr = 0.001
lr_schedule = tf.keras.optimizers.schedules.ExponentialDecay(
    initial_learning_rate=initial_lr,
    decay_steps=1000,
    decay_rate=0.96,
    staircase=True)

optimizer = tf.keras.optimizers.Adam(learning_rate=lr_schedule)

lrs = [optimizer.learning_rate(step).numpy() for step in [0, 1000, 2000, 3000]]
print(lrs)

What is the output printed?

A [0.001, 0.00096, 0.00096, 0.00096]

B [0.001, 0.00096, 0.0009216, 0.0008847360000000001]

C [0.001, 0.00096, 0.0009216, 0.0009216]

D [0.001, 0.00096, 0.0009216, 0.000884736]

💡 Hint

Recall that with staircase=True, the learning rate decays in steps at multiples of decay_steps.

The learning rate decays by multiplying by 0.96 every 1000 steps. At step 0: 0.001, step 1000: 0.001*0.96=0.00096, step 2000: 0.00096*0.96=0.0009216, step 3000: 0.0009216*0.96=0.000884736.

❓ Hyperparameter

advanced

Choosing learning rate for fine-tuning a large pre-trained model

You are fine-tuning a large pre-trained image classification model on a small dataset. Which learning rate choice is most appropriate to avoid overfitting and preserve learned features?

A Use a very high learning rate like 0.1 to quickly adapt to the new data.

B Use a moderate learning rate like 0.01 to balance adaptation and stability.

C Use no learning rate (0) to freeze all layers and not update weights.

D Use a low learning rate like 0.0001 to make small updates and avoid destroying pre-trained weights.

💡 Hint

Think about the size of the dataset and the risk of losing pre-trained knowledge.

A low learning rate allows the model to adjust gently to new data without large changes that could erase useful pre-trained features, especially important with small datasets.

❓ Metrics

advanced

Effect of learning rate on training and validation loss during fine-tuning

During fine-tuning, you observe the following behavior:

• Training loss decreases steadily.
• Validation loss starts increasing after a few epochs.

What does this suggest about the learning rate and model behavior?

A The learning rate is too high, causing the model to overfit and validation loss to increase.

B The learning rate is too low, causing underfitting and poor validation performance.

C The learning rate is appropriate; the model is learning well on both training and validation data.

D The learning rate is irrelevant; this behavior is caused by data imbalance.

💡 Hint

Think about what causes validation loss to increase while training loss decreases.

When validation loss increases but training loss decreases, the model is overfitting. A high learning rate can cause unstable updates that lead to overfitting during fine-tuning.

🔧 Debug

expert

Identifying the cause of unstable training during fine-tuning

You fine-tune a pre-trained model with this optimizer setup:

optimizer = tf.keras.optimizers.Adam(learning_rate=0.01)

After a few batches, the training loss becomes NaN and the model stops learning. What is the most likely cause?

A The learning rate is too low, causing the model to stop updating weights.

B The optimizer Adam does not support learning rates above 0.001.

C The learning rate is too high, causing gradient explosion and NaN loss values.

D The batch size is too large, causing memory overflow.

💡 Hint

Consider what happens when the learning rate is set too high in gradient-based optimization.

A high learning rate like 0.01 can cause very large updates, leading to unstable training and NaN loss values due to exploding gradients.

Practice

1. Why is it recommended to use a smaller learning rate when fine-tuning a pre-trained TensorFlow model?

easy

A. To increase the batch size automatically

B. To speed up training drastically

C. To make gentle updates and avoid destroying learned features

D. To skip the need for validation data

Solution

1. Step 1: Understand fine-tuning purpose

   Fine-tuning adjusts a pre-trained model slightly to new data without losing learned knowledge.

2. Step 2: Reason about learning rate effect

   A smaller learning rate makes small changes, preserving useful features learned before.

3. Final Answer:

   To make gentle updates and avoid destroying learned features -> Option C

4. Quick Check:

   Small learning rate = gentle fine-tuning [OK]

Hint: Small learning rate prevents big changes during fine-tuning [OK]

Common Mistakes:

• Using a large learning rate that overwrites learned features
• Ignoring the need to freeze layers before fine-tuning
• Assuming learning rate affects batch size

2. Which of the following TensorFlow code snippets correctly sets a smaller learning rate for fine-tuning?

easy

A. optimizer = tf.keras.optimizers.Adam(learning_rate=0.0001)

B. optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)

C. optimizer = tf.keras.optimizers.Adam(learning_rate=0.01)

D. optimizer = tf.keras.optimizers.Adam(learning_rate=0.1)

Solution

1. Step 1: Identify typical fine-tuning learning rates

   Fine-tuning usually uses smaller rates like 0.0001 to avoid big weight changes.

2. Step 2: Compare given learning rates

   0.0001 is smaller than 0.001, 0.01, and 0.1, making it suitable for fine-tuning.

3. Final Answer:

   optimizer = tf.keras.optimizers.Adam(learning_rate=0.0001) -> Option A

4. Quick Check:

   Smaller learning rate = 0.0001 [OK]

Hint: Pick the smallest learning rate value for fine-tuning [OK]

Common Mistakes:

• Choosing a learning rate too large for fine-tuning
• Confusing learning rate with batch size
• Not using tf.keras.optimizers.Adam correctly

3. Consider this TensorFlow code for fine-tuning a model:

base_model.trainable = True
optimizer = tf.keras.optimizers.Adam(learning_rate=0.01)
model.compile(optimizer=optimizer, loss='sparse_categorical_crossentropy', metrics=['accuracy'])
history = model.fit(train_data, epochs=3)

What is the likely outcome of using this learning rate during fine-tuning?

medium

A. The model will fine-tune smoothly with stable accuracy improvements

B. The model may have unstable training and lose pre-trained knowledge

C. The training will be very slow but stable

D. The model will skip training and keep old weights

Solution

1. Step 1: Analyze learning rate size

   Learning rate 0.01 is relatively large for fine-tuning, which risks big weight changes.

2. Step 2: Predict training behavior

   Large learning rate can cause unstable training and overwrite useful pre-trained features.

3. Final Answer:

   The model may have unstable training and lose pre-trained knowledge -> Option B

4. Quick Check:

   Large learning rate = unstable fine-tuning [OK]

Hint: Large learning rate can harm fine-tuning stability [OK]

Common Mistakes:

• Assuming higher learning rate always improves training speed
• Ignoring the effect of unfreezing the base model
• Thinking loss function controls learning rate impact

4. You wrote this code to fine-tune a TensorFlow model but notice training loss is not decreasing:

base_model.trainable = True
optimizer = tf.keras.optimizers.Adam(learning_rate=0.1)
model.compile(optimizer=optimizer, loss='categorical_crossentropy')
history = model.fit(train_data, epochs=5)

What is the main issue and how to fix it?

medium

A. Learning rate is too high; reduce it to 0.0001

B. Loss function is wrong; use 'sparse_categorical_crossentropy'

C. Base model should be frozen; set trainable = False

D. Epochs are too few; increase to 50

Solution

1. Step 1: Identify learning rate problem

   Learning rate 0.1 is very high for fine-tuning, causing unstable or no learning.

2. Step 2: Suggest fix

   Lowering learning rate to 0.0001 allows gentle updates and loss decrease.

3. Final Answer:

   Learning rate is too high; reduce it to 0.0001 -> Option A

4. Quick Check:

   High learning rate blocks loss decrease [OK]

Hint: Lower learning rate if loss doesn't improve during fine-tuning [OK]

Common Mistakes:

• Changing loss function unnecessarily
• Freezing base model after unfreezing
• Increasing epochs without fixing learning rate

5. You want to fine-tune a pre-trained TensorFlow model on a small dataset. Which learning rate strategy is best to balance learning speed and stability?

hard

A. Start with a small learning rate (e.g., 0.0001), then gradually increase it

B. Use a large learning rate (e.g., 0.1) from the start for faster training

C. Train all layers from scratch with a medium learning rate (e.g., 0.01)

D. Freeze the base model, train the new layers with a higher learning rate, then unfreeze and fine-tune with a smaller learning rate

Solution

1. Step 1: Understand fine-tuning best practice

   Freeze base model first to train new layers safely with higher learning rate.

2. Step 2: Fine-tune with smaller learning rate after unfreezing

   Unfreeze base model and use smaller learning rate to gently adjust pre-trained weights.

3. Final Answer:

   Freeze the base model, train the new layers with a higher learning rate, then unfreeze and fine-tune with a smaller learning rate -> Option D

4. Quick Check:

   Freeze first, then fine-tune with small learning rate [OK]

Hint: Freeze base, train new layers high LR, then fine-tune low LR [OK]

Common Mistakes:

• Using large learning rate from start causing unstable training
• Training all layers from scratch wasting pre-trained knowledge
• Not freezing base model before fine-tuning