Learning rate for fine-tuning in TensorFlow - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the learning rate in machine learning?

The learning rate is a small number that controls how much the model changes its knowledge after seeing new data. It helps the model learn step by step without jumping too far.

beginner

Why do we use a smaller learning rate when fine-tuning a pre-trained model?

We use a smaller learning rate to make small, careful updates. This helps keep the useful knowledge the model already learned and adjusts it gently to the new task.

intermediate

How does a high learning rate affect fine-tuning?

A high learning rate can make the model forget what it learned before and cause unstable training. It might jump over good solutions and perform worse.

beginner

In TensorFlow, how do you set a learning rate for fine-tuning?

You set the learning rate when you create the optimizer, for example: <br>optimizer = tf.keras.optimizers.Adam(learning_rate=0.0001)<br>This small rate helps fine-tune the model carefully.

intermediate

What is a common strategy to adjust learning rate during fine-tuning?

A common strategy is to start with a small learning rate and reduce it further if the model stops improving. This is called learning rate scheduling or decay.

Why is a smaller learning rate preferred for fine-tuning?

A To speed up training drastically

B To make small, careful updates preserving learned features

C To ignore previous knowledge completely

D To increase randomness in training

A smaller learning rate helps the model adjust gently without losing what it already learned.

What happens if the learning rate is too high during fine-tuning?

A Model may forget previous knowledge and training becomes unstable

B Model ignores new data

C Model stops training immediately

D Model learns faster and better

A high learning rate can cause the model to forget and jump over good solutions.

In TensorFlow, where do you specify the learning rate?

A When creating the optimizer

B In the model architecture

C In the dataset

D In the loss function

The learning rate is set when you create the optimizer, like Adam or SGD.

What is learning rate scheduling?

A Ignoring learning rate after first epoch

B Fixing learning rate to a constant value

C Changing learning rate during training to improve results

D Randomly changing learning rate every batch

Learning rate scheduling means adjusting the learning rate over time to help training.

Which learning rate is typical for fine-tuning a pre-trained model?

A 1.0

B 0.01

C 0.1

D 0.0001

A small learning rate like 0.0001 is common to fine-tune carefully.

Explain why fine-tuning a model requires a different learning rate than training from scratch.

Think about how the model already knows something useful.

Describe how you would set up learning rate scheduling in TensorFlow for fine-tuning.

Consider how to change learning rate during training.