Why requires_grad flag in PyTorch? - Purpose & Use Cases

The Big Idea

What if you could tell your model exactly which parts to learn and which to keep fixed, without extra hassle?

The Scenario

Imagine you are training a model by hand, calculating every gradient and update manually for each parameter.

You have a huge network with many layers, and you want to freeze some parts so they don't change during training.

The Problem

Manually tracking which parts to update is slow and confusing.

You might accidentally change parameters that should stay fixed, or waste time recalculating gradients for parts that don't need it.

The Solution

The requires_grad flag in PyTorch tells the system exactly which parameters need gradients.

This way, PyTorch automatically handles gradient calculation only for those parts, saving time and avoiding mistakes.

Before vs After

✗ Before

for param in model.parameters():
    param.grad = None  # manually reset gradients
    if param in freeze_list:
        # manually skip updating
        continue
    # manually compute gradients and update

✓ After

for param in model.parameters():
    if param in freeze_list:
        param.requires_grad = False  # freeze parameters
# PyTorch computes gradients only for requires_grad=True automatically

What It Enables

It enables efficient and error-free training by controlling exactly which parts of the model learn and update.

Real Life Example

When fine-tuning a pre-trained model, you freeze early layers by setting requires_grad=False so only the last layers learn new features.

Key Takeaways

Manually tracking gradients is slow and error-prone.

requires_grad flag automates gradient tracking for selected parameters.

This makes training faster, safer, and easier to control.