Learning rate for fine-tuning in TensorFlow - Model Metrics & Evaluation

Metrics & Evaluation - Learning rate for fine-tuning

Which metric matters for learning rate fine-tuning and WHY

When adjusting the learning rate for fine-tuning a model, the key metrics to watch are training loss and validation loss. These show how well the model is learning and if it is improving on new data.

A good learning rate helps the loss go down steadily without jumping around or getting stuck. If the learning rate is too high, loss may bounce or get worse. If too low, learning is slow and may stop early.

Also, watch validation accuracy to see if the model is improving on unseen data, which means fine-tuning is effective.

Confusion matrix example during fine-tuning

      Actual \ Predicted | Positive | Negative
      -------------------|----------|---------
      Positive           |    85    |   15
      Negative           |    10    |   90

      Total samples = 200
      TP = 85, FP = 10, TN = 90, FN = 15
    

From this matrix, we can calculate precision and recall to understand model quality after fine-tuning.

Precision vs Recall tradeoff in fine-tuning

Fine-tuning with a proper learning rate balances precision and recall. For example:

• If learning rate is too high, model may overfit noise, increasing false positives (lower precision).
• If learning rate is too low, model may underfit, missing true positives (lower recall).

Choosing the right learning rate helps find a sweet spot where both precision and recall improve, making the model reliable.

Good vs Bad metric values for learning rate fine-tuning

Good: Validation loss steadily decreases, validation accuracy improves, precision and recall both above 80%.

Bad: Validation loss fluctuates or increases, accuracy plateaus or drops, precision or recall below 50%, indicating poor learning rate choice.

Common pitfalls when tuning learning rate

• Too high learning rate: Causes loss to jump or diverge, model fails to learn.
• Too low learning rate: Training is very slow, may get stuck in local minima.
• Ignoring validation metrics: Only watching training loss can hide overfitting.
• Not adjusting learning rate schedule: Fine-tuning often needs smaller learning rates than initial training.

Self-check question

Your fine-tuned model has 98% accuracy but only 12% recall on the positive class. Is it good for production? Why or why not?

Answer: No, it is not good. High accuracy can be misleading if the data is imbalanced. Low recall means the model misses most positive cases, which can be critical depending on the task (e.g., missing fraud or disease). You should adjust the learning rate or other settings to improve recall.

Key Result

Proper learning rate tuning balances training and validation loss to improve precision and recall for effective fine-tuning.

Practice

1. Why is it recommended to use a smaller learning rate when fine-tuning a pre-trained TensorFlow model?

easy

A. To increase the batch size automatically

B. To speed up training drastically

C. To make gentle updates and avoid destroying learned features

D. To skip the need for validation data

Solution

1. Step 1: Understand fine-tuning purpose

   Fine-tuning adjusts a pre-trained model slightly to new data without losing learned knowledge.

2. Step 2: Reason about learning rate effect

   A smaller learning rate makes small changes, preserving useful features learned before.

3. Final Answer:

   To make gentle updates and avoid destroying learned features -> Option C

4. Quick Check:

   Small learning rate = gentle fine-tuning [OK]

Hint: Small learning rate prevents big changes during fine-tuning [OK]

Common Mistakes:

• Using a large learning rate that overwrites learned features
• Ignoring the need to freeze layers before fine-tuning
• Assuming learning rate affects batch size

2. Which of the following TensorFlow code snippets correctly sets a smaller learning rate for fine-tuning?

easy

A. optimizer = tf.keras.optimizers.Adam(learning_rate=0.0001)

B. optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)

C. optimizer = tf.keras.optimizers.Adam(learning_rate=0.01)

D. optimizer = tf.keras.optimizers.Adam(learning_rate=0.1)

Solution

1. Step 1: Identify typical fine-tuning learning rates

   Fine-tuning usually uses smaller rates like 0.0001 to avoid big weight changes.

2. Step 2: Compare given learning rates

   0.0001 is smaller than 0.001, 0.01, and 0.1, making it suitable for fine-tuning.

3. Final Answer:

   optimizer = tf.keras.optimizers.Adam(learning_rate=0.0001) -> Option A

4. Quick Check:

   Smaller learning rate = 0.0001 [OK]

Hint: Pick the smallest learning rate value for fine-tuning [OK]

Common Mistakes:

• Choosing a learning rate too large for fine-tuning
• Confusing learning rate with batch size
• Not using tf.keras.optimizers.Adam correctly

3. Consider this TensorFlow code for fine-tuning a model:

base_model.trainable = True
optimizer = tf.keras.optimizers.Adam(learning_rate=0.01)
model.compile(optimizer=optimizer, loss='sparse_categorical_crossentropy', metrics=['accuracy'])
history = model.fit(train_data, epochs=3)

What is the likely outcome of using this learning rate during fine-tuning?

medium

A. The model will fine-tune smoothly with stable accuracy improvements

B. The model may have unstable training and lose pre-trained knowledge

C. The training will be very slow but stable

D. The model will skip training and keep old weights

Solution

1. Step 1: Analyze learning rate size

   Learning rate 0.01 is relatively large for fine-tuning, which risks big weight changes.

2. Step 2: Predict training behavior

   Large learning rate can cause unstable training and overwrite useful pre-trained features.

3. Final Answer:

   The model may have unstable training and lose pre-trained knowledge -> Option B

4. Quick Check:

   Large learning rate = unstable fine-tuning [OK]

Hint: Large learning rate can harm fine-tuning stability [OK]

Common Mistakes:

• Assuming higher learning rate always improves training speed
• Ignoring the effect of unfreezing the base model
• Thinking loss function controls learning rate impact

4. You wrote this code to fine-tune a TensorFlow model but notice training loss is not decreasing:

base_model.trainable = True
optimizer = tf.keras.optimizers.Adam(learning_rate=0.1)
model.compile(optimizer=optimizer, loss='categorical_crossentropy')
history = model.fit(train_data, epochs=5)

What is the main issue and how to fix it?

medium

A. Learning rate is too high; reduce it to 0.0001

B. Loss function is wrong; use 'sparse_categorical_crossentropy'

C. Base model should be frozen; set trainable = False

D. Epochs are too few; increase to 50

Solution

1. Step 1: Identify learning rate problem

   Learning rate 0.1 is very high for fine-tuning, causing unstable or no learning.

2. Step 2: Suggest fix

   Lowering learning rate to 0.0001 allows gentle updates and loss decrease.

3. Final Answer:

   Learning rate is too high; reduce it to 0.0001 -> Option A

4. Quick Check:

   High learning rate blocks loss decrease [OK]

Hint: Lower learning rate if loss doesn't improve during fine-tuning [OK]

Common Mistakes:

• Changing loss function unnecessarily
• Freezing base model after unfreezing
• Increasing epochs without fixing learning rate

5. You want to fine-tune a pre-trained TensorFlow model on a small dataset. Which learning rate strategy is best to balance learning speed and stability?

hard

A. Start with a small learning rate (e.g., 0.0001), then gradually increase it

B. Use a large learning rate (e.g., 0.1) from the start for faster training

C. Train all layers from scratch with a medium learning rate (e.g., 0.01)

D. Freeze the base model, train the new layers with a higher learning rate, then unfreeze and fine-tune with a smaller learning rate

Solution

1. Step 1: Understand fine-tuning best practice

   Freeze base model first to train new layers safely with higher learning rate.

2. Step 2: Fine-tune with smaller learning rate after unfreezing

   Unfreeze base model and use smaller learning rate to gently adjust pre-trained weights.

3. Final Answer:

   Freeze the base model, train the new layers with a higher learning rate, then unfreeze and fine-tune with a smaller learning rate -> Option D

4. Quick Check:

   Freeze first, then fine-tune with small learning rate [OK]

Hint: Freeze base, train new layers high LR, then fine-tune low LR [OK]

Common Mistakes:

• Using large learning rate from start causing unstable training
• Training all layers from scratch wasting pre-trained knowledge
• Not freezing base model before fine-tuning