Fine-tuning strategy in PyTorch - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is fine-tuning in machine learning?

Fine-tuning is the process of taking a pre-trained model and training it a bit more on a new, often smaller, dataset to adapt it to a specific task.

beginner

Why do we freeze some layers during fine-tuning?

We freeze layers to keep their learned features unchanged, which helps prevent overfitting and reduces training time by only updating certain parts of the model.

intermediate

In PyTorch, how do you freeze layers of a model?

You set the parameter's requires_grad attribute to False, like: <br>for param in model.parameters():<br> param.requires_grad = False

intermediate

What is a common strategy to fine-tune a pre-trained model?

First, freeze most layers and train only the last layers. Then, optionally unfreeze some earlier layers and train with a smaller learning rate.

beginner

How does using a smaller learning rate help during fine-tuning?

A smaller learning rate helps make small adjustments to the pre-trained weights, avoiding large changes that could ruin the learned features.

What does freezing layers in a model mean?

A Stopping updates to those layers during training

B Removing those layers from the model

C Adding more neurons to those layers

D Changing the activation function of those layers

Freezing layers means their weights do not get updated during training.

Why start fine-tuning by training only the last layers?

A Because last layers have fewer parameters

B Because last layers do not affect output

C Because last layers are always frozen

D Because last layers are usually task-specific

Last layers capture task-specific features, so training them first adapts the model quickly.

In PyTorch, which attribute controls if a parameter is trainable?

A grad_enabled

B trainable

C requires_grad

D update_flag

The requires_grad attribute controls if gradients are computed and parameters updated.

What is a benefit of using a pre-trained model for fine-tuning?

A It guarantees perfect accuracy

B It reduces training time and data needed

C It removes the need for validation

D It makes the model smaller

Pre-trained models have learned useful features, so fine-tuning needs less data and time.

What happens if you use a large learning rate during fine-tuning?

A The model might forget learned features

B The model trains faster without issues

C The model becomes smaller

D The model ignores new data

A large learning rate can cause big changes that ruin the pre-trained knowledge.

Explain the step-by-step process of fine-tuning a pre-trained model in PyTorch.

Think about which layers to freeze and how to adjust learning rates.

Describe why fine-tuning is useful compared to training a model from scratch.

Consider the benefits of transfer learning.