Why automatic differentiation enables training in PyTorch

Introduction

Automatic differentiation helps computers learn by quickly finding how to change numbers to make predictions better.

When training a neural network to recognize images.

When adjusting a model to predict house prices accurately.

When improving a recommendation system based on user feedback.

When tuning parameters in a machine learning model automatically.

Syntax

PyTorch

loss.backward()
optimizer.step()

loss.backward() calculates gradients automatically.

optimizer.step() updates model parameters using these gradients.

Examples

This calculates gradients of the loss with respect to model parameters.

PyTorch

loss = criterion(output, target)
loss.backward()

Clear old gradients, compute new ones, then update parameters.

PyTorch

optimizer.zero_grad()
loss.backward()
optimizer.step()

Sample Model

This program shows how automatic differentiation finds gradients to train a simple model that learns y = 2x + 1.

PyTorch

import torch
import torch.nn as nn
import torch.optim as optim

# Simple model: y = wx + b
class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(1, 1)
    def forward(self, x):
        return self.linear(x)

# Data: x and y (y = 2x + 1)
x = torch.tensor([[1.0], [2.0], [3.0], [4.0]])
y = torch.tensor([[3.0], [5.0], [7.0], [9.0]])

model = SimpleModel()
criterion = nn.MSELoss()
optimizer = optim.SGD(model.parameters(), lr=0.1)

for epoch in range(5):
    optimizer.zero_grad()  # Clear old gradients
    output = model(x)      # Predict
    loss = criterion(output, y)  # Calculate loss
    loss.backward()        # Compute gradients automatically
    optimizer.step()       # Update weights
    print(f"Epoch {epoch+1}, Loss: {loss.item():.4f}")

Important Notes

Automatic differentiation saves time by calculating gradients for all model parameters at once.

Without it, training complex models would be very slow and error-prone.

Summary

Automatic differentiation finds how to change model numbers to reduce errors.

This process is key to training machine learning models efficiently.

Practice

1. Why is automatic differentiation important for training machine learning models in PyTorch?

easy

A. It stores the training data efficiently.

B. It generates random numbers for model initialization.

C. It visualizes the model architecture.

D. It calculates gradients automatically to update model parameters.

Solution

1. Step 1: Understand the role of gradients in training

   Training adjusts model numbers to reduce errors, which requires knowing how errors change with each number.

2. Step 2: Identify what automatic differentiation does

   Automatic differentiation calculates these changes (gradients) automatically, so the model can update correctly.

3. Final Answer:

   It calculates gradients automatically to update model parameters. -> Option D

4. Quick Check:

   Gradients = Automatic differentiation [OK]

Hint: Automatic differentiation = auto gradient calculation [OK]

Common Mistakes:

• Confusing gradient calculation with data storage
• Thinking it generates random numbers
• Assuming it visualizes models

2. Which PyTorch code snippet correctly enables automatic differentiation for a tensor?

easy

A. tensor = torch.tensor([1.0, 2.0], requires_grad=False)

B. tensor = torch.tensor([1.0, 2.0], requires_grad=True)

C. tensor = torch.tensor([1.0, 2.0]).detach()

D. tensor = torch.tensor([1.0, 2.0]).numpy()

Solution

1. Step 1: Check how to enable gradient tracking

   In PyTorch, setting requires_grad=True tells PyTorch to track operations for gradients.

2. Step 2: Identify the correct code snippet

   tensor = torch.tensor([1.0, 2.0], requires_grad=True) sets requires_grad=True, enabling automatic differentiation.

3. Final Answer:

   tensor = torch.tensor([1.0, 2.0], requires_grad=True) -> Option B

4. Quick Check:

   requires_grad=True enables gradients [OK]

Hint: Use requires_grad=True to track gradients [OK]

Common Mistakes:

• Using requires_grad=False disables gradients
• Calling detach() stops gradient tracking
• Converting to numpy removes gradient info

3. What will be the output of this PyTorch code?

import torch
x = torch.tensor(2.0, requires_grad=True)
y = x ** 3
z = y + 2 * x
z.backward()
print(x.grad.item())

medium

A. 8.0

B. 12.0

C. 14.0

D. 10.0

Solution

1. Step 1: Calculate the function and its derivative

   The function is z = x^3 + 2x. Its derivative is dz/dx = 3x^2 + 2.

2. Step 2: Compute the gradient at x=2.0

   Plug in x=2: dz/dx = 3*(2^2) + 2 = 3*4 + 2 = 12 + 2 = 14.

3. Final Answer:

   14.0 -> Option C

4. Quick Check:

   Derivative at 2 = 14 [OK]

Hint: Derivative of x³+2x at 2 is 14 [OK]

Common Mistakes:

• Forgetting to apply chain rule
• Calculating derivative incorrectly
• Using wrong x value

4. Identify the error in this PyTorch code that prevents training:

import torch
x = torch.tensor([1.0, 2.0, 3.0])
y = x.sum()
y.backward()
print(x.grad)

medium

A. x does not have requires_grad=True, so gradients are not tracked.

B. Using sum() prevents gradient calculation.

C. backward() must be called on x, not y.

D. x.grad should be printed before backward() call.

Solution

1. Step 1: Check if gradients are tracked

   By default, tensors have requires_grad=False, so no gradients are stored.

2. Step 2: Identify why backward() fails to compute gradients

   Since x lacks requires_grad=True, x.grad remains None after backward.

3. Final Answer:

   x does not have requires_grad=True, so gradients are not tracked. -> Option A

4. Quick Check:

   requires_grad=True needed for gradients [OK]

Hint: Set requires_grad=True to track gradients [OK]

Common Mistakes:

• Assuming sum() disables gradients
• Calling backward() on wrong tensor
• Printing grad before backward()

5. You want to train a model with parameters w and b using PyTorch. Which sequence correctly uses automatic differentiation to update parameters after computing loss?

hard

A. loss.backward()
with torch.no_grad():
w -= learning_rate * w.grad
b -= learning_rate * b.grad
w.grad.zero_()
b.grad.zero_()

B. loss.backward()
w = w - learning_rate * w.grad
b = b - learning_rate * b.grad
w.grad.zero_()
b.grad.zero_()

C. loss.backward()
w.grad = None
b.grad = None
w -= learning_rate * w.grad
b -= learning_rate * b.grad

D. loss.backward()
w.grad.zero_()
b.grad.zero_()
w -= learning_rate * w.grad
b -= learning_rate * b.grad

Solution

1. Step 1: Understand gradient update with no_grad()

   Parameter updates must be done without tracking gradients, so use torch.no_grad() context.

2. Step 2: Zero gradients after update

   After updating, gradients must be reset with grad.zero_() to avoid accumulation.

3. Final Answer:

   Use loss.backward(), update inside torch.no_grad(), then zero gradients. -> Option A

4. Quick Check:

   Update params inside no_grad, then zero grads [OK]

Hint: Update params inside no_grad, then zero grads [OK]

Common Mistakes:

• Updating params outside no_grad() causes errors
• Not zeroing gradients after update
• Setting grad to None instead of zero_()