Learning rate for fine-tuning in TensorFlow - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Why use a smaller learning rate for fine-tuning?

When fine-tuning a pre-trained neural network, why is it common to use a smaller learning rate compared to training from scratch?

A Because a smaller learning rate helps preserve the learned features and avoids large updates that could destroy useful information.

B Because a smaller learning rate prevents the model from converging too quickly to a good solution.

C Because a smaller learning rate speeds up training by making bigger jumps in the parameter space.

D Because a smaller learning rate increases the chance of overfitting the training data.

💡 Hint

Think about what happens if you change the pre-trained weights too much.

Fine-tuning starts from a model that already learned useful features. Using a smaller learning rate ensures updates are gentle, preserving these features while adapting to new data.

❓ Predict Output

intermediate

Output of learning rate schedule during fine-tuning

Consider this TensorFlow code snippet that sets a learning rate schedule for fine-tuning:

import tensorflow as tf

initial_lr = 0.001
lr_schedule = tf.keras.optimizers.schedules.ExponentialDecay(
    initial_learning_rate=initial_lr,
    decay_steps=1000,
    decay_rate=0.96,
    staircase=True)

optimizer = tf.keras.optimizers.Adam(learning_rate=lr_schedule)

lrs = [optimizer.learning_rate(step).numpy() for step in [0, 1000, 2000, 3000]]
print(lrs)

What is the output printed?

A [0.001, 0.00096, 0.00096, 0.00096]

B [0.001, 0.00096, 0.0009216, 0.0008847360000000001]

C [0.001, 0.00096, 0.0009216, 0.0009216]

D [0.001, 0.00096, 0.0009216, 0.000884736]

💡 Hint

Recall that with staircase=True, the learning rate decays in steps at multiples of decay_steps.

The learning rate decays by multiplying by 0.96 every 1000 steps. At step 0: 0.001, step 1000: 0.001*0.96=0.00096, step 2000: 0.00096*0.96=0.0009216, step 3000: 0.0009216*0.96=0.000884736.

❓ Hyperparameter

advanced

Choosing learning rate for fine-tuning a large pre-trained model

You are fine-tuning a large pre-trained image classification model on a small dataset. Which learning rate choice is most appropriate to avoid overfitting and preserve learned features?

A Use a very high learning rate like 0.1 to quickly adapt to the new data.

B Use a moderate learning rate like 0.01 to balance adaptation and stability.

C Use no learning rate (0) to freeze all layers and not update weights.

D Use a low learning rate like 0.0001 to make small updates and avoid destroying pre-trained weights.

💡 Hint

Think about the size of the dataset and the risk of losing pre-trained knowledge.

A low learning rate allows the model to adjust gently to new data without large changes that could erase useful pre-trained features, especially important with small datasets.

❓ Metrics

advanced

Effect of learning rate on training and validation loss during fine-tuning

During fine-tuning, you observe the following behavior:

• Training loss decreases steadily.
• Validation loss starts increasing after a few epochs.

What does this suggest about the learning rate and model behavior?

A The learning rate is too high, causing the model to overfit and validation loss to increase.

B The learning rate is too low, causing underfitting and poor validation performance.

C The learning rate is appropriate; the model is learning well on both training and validation data.

D The learning rate is irrelevant; this behavior is caused by data imbalance.

💡 Hint

Think about what causes validation loss to increase while training loss decreases.

When validation loss increases but training loss decreases, the model is overfitting. A high learning rate can cause unstable updates that lead to overfitting during fine-tuning.

🔧 Debug

expert

Identifying the cause of unstable training during fine-tuning

You fine-tune a pre-trained model with this optimizer setup:

optimizer = tf.keras.optimizers.Adam(learning_rate=0.01)

After a few batches, the training loss becomes NaN and the model stops learning. What is the most likely cause?

A The learning rate is too low, causing the model to stop updating weights.

B The optimizer Adam does not support learning rates above 0.001.

C The learning rate is too high, causing gradient explosion and NaN loss values.

D The batch size is too large, causing memory overflow.

💡 Hint

Consider what happens when the learning rate is set too high in gradient-based optimization.

A high learning rate like 0.01 can cause very large updates, leading to unstable training and NaN loss values due to exploding gradients.