Gradient access (.grad) in PyTorch - Model Pipeline Trace

Model Pipeline - Gradient access (.grad)

This pipeline shows how a simple model learns by calculating gradients using PyTorch's .grad attribute. Gradients tell the model how to change its weights to improve predictions.

Data Flow - 5 Stages

1Data Input

4 rows x 2 columns→Provide input features and target labels→4 rows x 2 columns (features), 4 rows x 1 column (labels)

Features: [[1.0, 2.0], [2.0, 3.0], [3.0, 4.0], [4.0, 5.0]], Labels: [[3.0], [5.0], [7.0], [9.0]]

↓

2Model Initialization

2 features→Initialize weights and bias with requires_grad=True→Weights: 2 parameters, Bias: 1 parameter

Weights: tensor([0.5, -0.5], requires_grad=True), Bias: tensor(0.0, requires_grad=True)

↓

3Forward Pass

4 rows x 2 columns→Calculate predictions using linear model: y = Xw + b→4 rows x 1 column

Predictions: tensor([[0.5], [1.5], [2.5], [3.5]], requires_grad=True)

↓

4Loss Calculation

Predictions and labels (4 rows x 1 column each)→Compute mean squared error loss→Scalar loss value

Loss: tensor(10.25, requires_grad=True)

↓

5Backward Pass

Scalar loss→Calculate gradients of loss w.r.t weights and bias using .backward()→Gradients stored in .grad attributes of weights and bias

Weights.grad: tensor([-7.0, -10.0]), Bias.grad: tensor([-3.0])

Training Trace - Epoch by Epoch

Loss
10.25 |*       
 8.00 | *      
 6.00 |  *     
 4.00 |   *    
 2.56 |    *   
 1.00 |     *  
 0.64 |      * 
 0.16 |       *
 0.04 |        *
       ----------------
        1 2 3 4 5 Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	10.25	N/A	Initial loss is high; gradients calculated for first update
2	2.56	N/A	Loss decreased after weight update using gradients
3	0.64	N/A	Loss continues to decrease; gradients guide learning
4	0.16	N/A	Model is learning well; gradients become smaller
5	0.04	N/A	Loss is low; model predictions close to targets

Prediction Trace - 5 Layers

Layer 1: Input features

Layer 2: Linear layer calculation

Layer 3: Loss calculation

Layer 4: Backward pass (.backward())

Layer 5: Access .grad attribute

Model Quiz - 3 Questions

What does the .grad attribute store after calling backward()?

A The gradients of the loss with respect to the tensor

B The updated weights after training

C The input data to the model

D The final prediction values

Key Insight

The .grad attribute in PyTorch stores the gradients of the loss with respect to each parameter. These gradients guide how to update weights to reduce error. Accessing .grad after backward() is key to understanding and controlling learning.