Why checkpointing preserves progress in PyTorch

Introduction

Checkpointing saves your model and training state so you don't lose progress if something stops your training.

When training a model takes a long time and you want to pause and continue later.

If your computer might shut down or lose power during training.

When you want to try different training settings but keep the best model so far.

To save intermediate results and avoid starting from scratch after a crash.

Syntax

PyTorch

torch.save({
    'epoch': epoch,
    'model_state_dict': model.state_dict(),
    'optimizer_state_dict': optimizer.state_dict(),
    'loss': loss_value,
}, PATH)

checkpoint = torch.load(PATH)
model.load_state_dict(checkpoint['model_state_dict'])
optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
epoch = checkpoint['epoch']

Use model.state_dict() to save model weights.

Use optimizer.state_dict() to save optimizer state for continuing training.

Examples

Saves only the model weights, useful for inference later.

PyTorch

torch.save(model.state_dict(), 'model.pth')

Saves model, optimizer, and epoch to resume training exactly.

PyTorch

torch.save({
    'model': model.state_dict(),
    'optimizer': optimizer.state_dict(),
    'epoch': epoch
}, 'checkpoint.pth')

Loads checkpoint to continue training from saved state.

PyTorch

checkpoint = torch.load('checkpoint.pth')
model.load_state_dict(checkpoint['model'])
optimizer.load_state_dict(checkpoint['optimizer'])
epoch = checkpoint['epoch']

Sample Model

This program trains a simple model for 3 epochs, saves a checkpoint, then loads it and continues training for 2 more epochs. You see the loss decrease over all 5 epochs without losing progress.

PyTorch

import torch
import torch.nn as nn
import torch.optim as optim

# Simple model
class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(1, 1)
    def forward(self, x):
        return self.linear(x)

model = SimpleModel()
optimizer = optim.SGD(model.parameters(), lr=0.1)
criterion = nn.MSELoss()

# Dummy data
x = torch.tensor([[1.0], [2.0], [3.0], [4.0]])
y = torch.tensor([[2.0], [4.0], [6.0], [8.0]])

# Train for 3 epochs and save checkpoint
for epoch in range(3):
    optimizer.zero_grad()
    outputs = model(x)
    loss = criterion(outputs, y)
    loss.backward()
    optimizer.step()
    print(f"Epoch {epoch+1}, Loss: {loss.item():.4f}")

# Save checkpoint
torch.save({
    'epoch': 3,
    'model_state_dict': model.state_dict(),
    'optimizer_state_dict': optimizer.state_dict(),
    'loss': loss.item()
}, 'checkpoint.pth')

# Load checkpoint and continue training for 2 more epochs
checkpoint = torch.load('checkpoint.pth')
model.load_state_dict(checkpoint['model_state_dict'])
optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
start_epoch = checkpoint['epoch']

for epoch in range(start_epoch, start_epoch + 2):
    optimizer.zero_grad()
    outputs = model(x)
    loss = criterion(outputs, y)
    loss.backward()
    optimizer.step()
    print(f"Epoch {epoch+1}, Loss: {loss.item():.4f}")

Important Notes

Always save both model and optimizer states to resume training properly.

Checkpoints let you avoid losing hours of training if interrupted.

You can also save checkpoints periodically during training to keep backups.

Summary

Checkpointing saves your training progress so you can stop and continue later.

It saves model weights, optimizer state, and other info like epoch number.

This helps avoid losing work and makes training more flexible and safe.

Practice

1. What is the main reason for using checkpointing during PyTorch model training?

easy

A. To save the model's current state so training can resume later without loss

B. To speed up the training by skipping some layers

C. To reduce the size of the training dataset

D. To automatically tune hyperparameters during training

Solution

1. Step 1: Understand checkpointing purpose

   Checkpointing saves the model's current state including weights and optimizer info.

2. Step 2: Connect checkpointing to training progress

   This allows training to stop and resume later without losing progress.

3. Final Answer:

   To save the model's current state so training can resume later without loss -> Option A

4. Quick Check:

   Checkpointing = Save progress [OK]

Hint: Checkpointing means saving progress to continue later [OK]

Common Mistakes:

• Thinking checkpointing speeds up training
• Confusing checkpointing with data reduction
• Assuming checkpointing tunes hyperparameters

2. Which of the following is the correct PyTorch code snippet to save a checkpoint?

easy

A. model.load_state_dict(torch.save('checkpoint.pth'))

B. torch.save(model.state_dict(), 'checkpoint.pth')

C. torch.load('checkpoint.pth')

D. optimizer.save('checkpoint.pth')

Solution

1. Step 1: Identify saving function

   torch.save() is used to save objects like model weights to a file.

2. Step 2: Check correct usage for saving model state

   model.state_dict() returns model weights; saving it with torch.save() is correct.

3. Final Answer:

   torch.save(model.state_dict(), 'checkpoint.pth') -> Option B

4. Quick Check:

   Save model weights = torch.save(state_dict) [OK]

Hint: Use torch.save with model.state_dict() to save checkpoint [OK]

Common Mistakes:

• Using torch.load instead of torch.save to save
• Trying to save optimizer with wrong method
• Confusing load_state_dict with saving

3. Given this code snippet, what will be printed after loading the checkpoint?

model = MyModel()
optimizer = torch.optim.Adam(model.parameters())
checkpoint = torch.load('checkpoint.pth')
model.load_state_dict(checkpoint['model_state'])
optimizer.load_state_dict(checkpoint['optimizer_state'])
epoch = checkpoint['epoch']
print(epoch)

medium

A. An error because checkpoint keys are missing

B. The total number of model parameters

C. The optimizer learning rate

D. The epoch number saved in the checkpoint

Solution

1. Step 1: Understand checkpoint contents

   The checkpoint dictionary contains keys 'model_state', 'optimizer_state', and 'epoch'.

2. Step 2: Identify printed value

   Variable 'epoch' is assigned checkpoint['epoch'], so print(epoch) outputs the saved epoch number.

3. Final Answer:

   The epoch number saved in the checkpoint -> Option D

4. Quick Check:

   Print epoch from checkpoint = epoch number [OK]

Hint: Print shows saved epoch from checkpoint dictionary [OK]

Common Mistakes:

• Thinking print shows model parameters count
• Confusing optimizer state with epoch
• Assuming missing keys cause error here

4. You tried to resume training but got an error: RuntimeError: Error(s) in loading state_dict. What is the most likely cause related to checkpointing?

medium

A. The training data was modified after checkpointing

B. The checkpoint file was saved with torch.load instead of torch.save

C. The model architecture changed after saving the checkpoint

D. The optimizer state was not saved in the checkpoint

Solution

1. Step 1: Understand error meaning

   Loading state_dict errors usually happen if model layers differ from saved checkpoint.

2. Step 2: Connect error to checkpoint cause

   If model architecture changed after saving, weights won't match, causing this error.

3. Final Answer:

   The model architecture changed after saving the checkpoint -> Option C

4. Quick Check:

   State_dict error = architecture mismatch [OK]

Hint: Mismatch model layers cause state_dict loading errors [OK]

Common Mistakes:

• Confusing save/load functions causing error
• Assuming missing optimizer state causes this error
• Blaming training data changes for state_dict error

5. You want to checkpoint your training every 5 epochs to avoid losing progress. Which approach best preserves training progress including optimizer state and epoch count?

hard

A. Save a dictionary with model.state_dict(), optimizer.state_dict(), and current epoch number

B. Save only model.state_dict() every 5 epochs

C. Save optimizer.state_dict() and epoch number but not model weights

D. Save the training data batch every 5 epochs

Solution

1. Step 1: Identify what preserves full training state

   Saving model weights, optimizer state, and epoch number allows full resume.

2. Step 2: Compare options

   Only saving model weights misses optimizer info; saving optimizer and epoch without model is incomplete; saving data batch doesn't preserve progress.

3. Final Answer:

   Save a dictionary with model.state_dict(), optimizer.state_dict(), and current epoch number -> Option A

4. Quick Check:

   Checkpoint = model + optimizer + epoch [OK]

Hint: Checkpoint all: model, optimizer, and epoch for full resume [OK]

Common Mistakes:

• Saving only model weights loses optimizer progress
• Ignoring epoch number causes restart from zero
• Saving training data batch does not preserve model state