Learning rate for fine-tuning in TensorFlow - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the learning rate in machine learning?

The learning rate is a small number that controls how much the model changes its knowledge after seeing new data. It helps the model learn step by step without jumping too far.

beginner

Why do we use a smaller learning rate when fine-tuning a pre-trained model?

We use a smaller learning rate to make small, careful updates. This helps keep the useful knowledge the model already learned and adjusts it gently to the new task.

intermediate

How does a high learning rate affect fine-tuning?

A high learning rate can make the model forget what it learned before and cause unstable training. It might jump over good solutions and perform worse.

beginner

In TensorFlow, how do you set a learning rate for fine-tuning?

You set the learning rate when you create the optimizer, for example:
optimizer = tf.keras.optimizers.Adam(learning_rate=0.0001)
This small rate helps fine-tune the model carefully.

intermediate

What is a common strategy to adjust learning rate during fine-tuning?

A common strategy is to start with a small learning rate and reduce it further if the model stops improving. This is called learning rate scheduling or decay.

Why is a smaller learning rate preferred for fine-tuning?

A To speed up training drastically

B To make small, careful updates preserving learned features

C To ignore previous knowledge completely

D To increase randomness in training

A smaller learning rate helps the model adjust gently without losing what it already learned.

What happens if the learning rate is too high during fine-tuning?

A Model may forget previous knowledge and training becomes unstable

B Model ignores new data

C Model stops training immediately

D Model learns faster and better

A high learning rate can cause the model to forget and jump over good solutions.

In TensorFlow, where do you specify the learning rate?

A When creating the optimizer

B In the model architecture

C In the dataset

D In the loss function

The learning rate is set when you create the optimizer, like Adam or SGD.

What is learning rate scheduling?

A Ignoring learning rate after first epoch

B Fixing learning rate to a constant value

C Changing learning rate during training to improve results

D Randomly changing learning rate every batch

Learning rate scheduling means adjusting the learning rate over time to help training.

Which learning rate is typical for fine-tuning a pre-trained model?

A 1.0

B 0.01

C 0.1

D 0.0001

A small learning rate like 0.0001 is common to fine-tune carefully.

Explain why fine-tuning a model requires a different learning rate than training from scratch.

Think about how the model already knows something useful.

Describe how you would set up learning rate scheduling in TensorFlow for fine-tuning.

Consider how to change learning rate during training.

Practice

1. Why is it recommended to use a smaller learning rate when fine-tuning a pre-trained TensorFlow model?

easy

A. To increase the batch size automatically

B. To speed up training drastically

C. To make gentle updates and avoid destroying learned features

D. To skip the need for validation data

Solution

1. Step 1: Understand fine-tuning purpose

   Fine-tuning adjusts a pre-trained model slightly to new data without losing learned knowledge.

2. Step 2: Reason about learning rate effect

   A smaller learning rate makes small changes, preserving useful features learned before.

3. Final Answer:

   To make gentle updates and avoid destroying learned features -> Option C

4. Quick Check:

   Small learning rate = gentle fine-tuning [OK]

Hint: Small learning rate prevents big changes during fine-tuning [OK]

Common Mistakes:

• Using a large learning rate that overwrites learned features
• Ignoring the need to freeze layers before fine-tuning
• Assuming learning rate affects batch size

2. Which of the following TensorFlow code snippets correctly sets a smaller learning rate for fine-tuning?

easy

A. optimizer = tf.keras.optimizers.Adam(learning_rate=0.0001)

B. optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)

C. optimizer = tf.keras.optimizers.Adam(learning_rate=0.01)

D. optimizer = tf.keras.optimizers.Adam(learning_rate=0.1)

Solution

1. Step 1: Identify typical fine-tuning learning rates

   Fine-tuning usually uses smaller rates like 0.0001 to avoid big weight changes.

2. Step 2: Compare given learning rates

   0.0001 is smaller than 0.001, 0.01, and 0.1, making it suitable for fine-tuning.

3. Final Answer:

   optimizer = tf.keras.optimizers.Adam(learning_rate=0.0001) -> Option A

4. Quick Check:

   Smaller learning rate = 0.0001 [OK]

Hint: Pick the smallest learning rate value for fine-tuning [OK]

Common Mistakes:

• Choosing a learning rate too large for fine-tuning
• Confusing learning rate with batch size
• Not using tf.keras.optimizers.Adam correctly

3. Consider this TensorFlow code for fine-tuning a model:

base_model.trainable = True
optimizer = tf.keras.optimizers.Adam(learning_rate=0.01)
model.compile(optimizer=optimizer, loss='sparse_categorical_crossentropy', metrics=['accuracy'])
history = model.fit(train_data, epochs=3)

What is the likely outcome of using this learning rate during fine-tuning?

medium

A. The model will fine-tune smoothly with stable accuracy improvements

B. The model may have unstable training and lose pre-trained knowledge

C. The training will be very slow but stable

D. The model will skip training and keep old weights

Solution

1. Step 1: Analyze learning rate size

   Learning rate 0.01 is relatively large for fine-tuning, which risks big weight changes.

2. Step 2: Predict training behavior

   Large learning rate can cause unstable training and overwrite useful pre-trained features.

3. Final Answer:

   The model may have unstable training and lose pre-trained knowledge -> Option B

4. Quick Check:

   Large learning rate = unstable fine-tuning [OK]

Hint: Large learning rate can harm fine-tuning stability [OK]

Common Mistakes:

• Assuming higher learning rate always improves training speed
• Ignoring the effect of unfreezing the base model
• Thinking loss function controls learning rate impact

4. You wrote this code to fine-tune a TensorFlow model but notice training loss is not decreasing:

base_model.trainable = True
optimizer = tf.keras.optimizers.Adam(learning_rate=0.1)
model.compile(optimizer=optimizer, loss='categorical_crossentropy')
history = model.fit(train_data, epochs=5)

What is the main issue and how to fix it?

medium

A. Learning rate is too high; reduce it to 0.0001

B. Loss function is wrong; use 'sparse_categorical_crossentropy'

C. Base model should be frozen; set trainable = False

D. Epochs are too few; increase to 50

Solution

1. Step 1: Identify learning rate problem

   Learning rate 0.1 is very high for fine-tuning, causing unstable or no learning.

2. Step 2: Suggest fix

   Lowering learning rate to 0.0001 allows gentle updates and loss decrease.

3. Final Answer:

   Learning rate is too high; reduce it to 0.0001 -> Option A

4. Quick Check:

   High learning rate blocks loss decrease [OK]

Hint: Lower learning rate if loss doesn't improve during fine-tuning [OK]

Common Mistakes:

• Changing loss function unnecessarily
• Freezing base model after unfreezing
• Increasing epochs without fixing learning rate

5. You want to fine-tune a pre-trained TensorFlow model on a small dataset. Which learning rate strategy is best to balance learning speed and stability?

hard

A. Start with a small learning rate (e.g., 0.0001), then gradually increase it

B. Use a large learning rate (e.g., 0.1) from the start for faster training

C. Train all layers from scratch with a medium learning rate (e.g., 0.01)

D. Freeze the base model, train the new layers with a higher learning rate, then unfreeze and fine-tune with a smaller learning rate

Solution

1. Step 1: Understand fine-tuning best practice

   Freeze base model first to train new layers safely with higher learning rate.

2. Step 2: Fine-tune with smaller learning rate after unfreezing

   Unfreeze base model and use smaller learning rate to gently adjust pre-trained weights.

3. Final Answer:

   Freeze the base model, train the new layers with a higher learning rate, then unfreeze and fine-tune with a smaller learning rate -> Option D

4. Quick Check:

   Freeze first, then fine-tune with small learning rate [OK]

Hint: Freeze base, train new layers high LR, then fine-tune low LR [OK]

Common Mistakes:

• Using large learning rate from start causing unstable training
• Training all layers from scratch wasting pre-trained knowledge
• Not freezing base model before fine-tuning