Validation loop in PyTorch

Introduction

A validation loop checks how well a trained model works on new data it hasn't seen before. It helps us know if the model is learning correctly or just memorizing.

After training a model for some time to see if it is improving.

To compare different models and pick the best one.

To avoid overfitting by checking performance on unseen data.

When tuning model settings to find the best parameters.

Before using the model in real life to ensure it works well.

Syntax

PyTorch

model.eval()
with torch.no_grad():
    for inputs, labels in validation_loader:
        outputs = model(inputs)
        loss = loss_function(outputs, labels)
        # calculate metrics like accuracy
        # accumulate results for reporting

Use model.eval() to set the model to evaluation mode.

Wrap the loop with torch.no_grad() to save memory and speed up computation.

Examples

Basic validation loop calculating loss on validation data.

PyTorch

model.eval()
with torch.no_grad():
    for inputs, labels in val_loader:
        outputs = model(inputs)
        loss = criterion(outputs, labels)

Validation loop calculating accuracy of the model.

PyTorch

model.eval()
with torch.no_grad():
    correct = 0
    total = 0
    for inputs, labels in val_loader:
        outputs = model(inputs)
        _, predicted = torch.max(outputs, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()
    accuracy = correct / total

Sample Model

This program trains a simple model on random data and then runs a validation loop to check loss and accuracy on validation data after each epoch.

PyTorch

import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader, TensorDataset

# Create simple dataset
x_train = torch.randn(100, 5)
y_train = (x_train.sum(dim=1) > 0).long()
x_val = torch.randn(30, 5)
y_val = (x_val.sum(dim=1) > 0).long()

train_dataset = TensorDataset(x_train, y_train)
val_dataset = TensorDataset(x_val, y_val)

train_loader = DataLoader(train_dataset, batch_size=10)
val_loader = DataLoader(val_dataset, batch_size=10)

# Simple model
class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(5, 2)
    def forward(self, x):
        return self.linear(x)

model = SimpleModel()
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr=0.1)

# Train for 3 epochs
for epoch in range(3):
    model.train()
    for inputs, labels in train_loader:
        optimizer.zero_grad()
        outputs = model(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

    # Validation loop
    model.eval()
    val_loss = 0.0
    correct = 0
    total = 0
    with torch.no_grad():
        for inputs, labels in val_loader:
            outputs = model(inputs)
            loss = criterion(outputs, labels)
            val_loss += loss.item() * inputs.size(0)
            _, predicted = torch.max(outputs, 1)
            total += labels.size(0)
            correct += (predicted == labels).sum().item()
    val_loss /= total
    accuracy = correct / total
    print(f"Epoch {epoch+1}: Validation Loss = {val_loss:.4f}, Accuracy = {accuracy:.4f}")

Important Notes

Always use model.eval() before validation to turn off training-only features like dropout.

Use torch.no_grad() during validation to save memory and speed up calculations.

Calculate metrics like accuracy or loss to understand model performance on validation data.

Summary

A validation loop tests the model on new data to check its performance.

Use model.eval() and torch.no_grad() to run validation properly.

Track metrics like loss and accuracy to see how well the model is doing.

Practice

1. What is the main purpose of a validation loop in PyTorch?

easy

A. To update the model weights during training

B. To check how well the model performs on unseen data

C. To load the training dataset

D. To save the model to disk

Solution

1. Step 1: Understand the role of validation

   The validation loop is used to test the model on new data that it has not seen during training.

2. Step 2: Differentiate from training

   Unlike training, validation does not update model weights but measures performance like accuracy or loss.

3. Final Answer:

   To check how well the model performs on unseen data -> Option B

4. Quick Check:

   Validation loop = test model on new data [OK]

Hint: Validation loop tests model without changing weights [OK]

Common Mistakes:

• Confusing validation with training
• Thinking validation updates model weights
• Mixing validation with data loading

2. Which PyTorch code snippet correctly sets the model for validation?

easy

A. model.eval()

B. model.train()

C. model.validate()

D. model.test()

Solution

1. Step 1: Recall PyTorch model modes

   In PyTorch, model.eval() sets the model to evaluation mode, disabling dropout and batch norm updates.

2. Step 2: Identify correct method for validation

   Validation requires evaluation mode, so model.eval() is correct; others do not exist or are for training.

3. Final Answer:

   model.eval() -> Option A

4. Quick Check:

   Use model.eval() for validation [OK]

Hint: Use model.eval() to disable training behaviors [OK]

Common Mistakes:

• Using model.train() during validation
• Assuming model.validate() exists
• Confusing test() with eval()

3. What will be the output of the following validation loop snippet?

model.eval()
correct = 0
with torch.no_grad():
    for inputs, labels in val_loader:
        outputs = model(inputs)
        predicted = outputs.argmax(dim=1)
        correct += (predicted == labels).sum().item()
accuracy = correct / len(val_loader.dataset)
print(f"Accuracy: {accuracy:.2f}")

medium

A. Prints loss value instead of accuracy

B. Raises an error because of missing optimizer

C. Prints the accuracy of the model on validation data

D. Prints training accuracy instead of validation accuracy

Solution

1. Step 1: Analyze the code logic

   The code sets model to eval mode, disables gradient calculation, and counts correct predictions over validation data.

2. Step 2: Understand the output calculation

   Accuracy is computed as correct predictions divided by total validation samples, then printed.

3. Final Answer:

   Prints the accuracy of the model on validation data -> Option C

4. Quick Check:

   Validation loop code prints accuracy [OK]

Hint: Check if code counts correct predictions for accuracy [OK]

Common Mistakes:

• Expecting loss instead of accuracy
• Thinking optimizer is needed in validation
• Confusing training and validation accuracy

4. Identify the error in this validation loop code:

model.eval()
for inputs, labels in val_loader:
    outputs = model(inputs)
    loss = criterion(outputs, labels)
    loss.backward()
    total_loss += loss.item()

medium

A. Not zeroing gradients before backward

B. Not using model.train() before loop

C. Missing optimizer.step() after loss.backward()

D. Calling loss.backward() during validation

Solution

1. Step 1: Understand validation loop purpose

   Validation should not update model weights or compute gradients, so backward() is not needed.

2. Step 2: Identify the error

   Calling loss.backward() computes gradients, which is incorrect in validation; it wastes resources and may cause errors.

3. Final Answer:

   Calling loss.backward() during validation -> Option D

4. Quick Check:

   No backward() in validation loop [OK]

Hint: No backward() in validation, only in training [OK]

Common Mistakes:

• Running backward() in validation
• Confusing train and eval modes
• Forgetting torch.no_grad() in validation

5. You want to add accuracy tracking in your validation loop. Which code snippet correctly computes and updates accuracy inside the loop?

model.eval()
total_correct = 0
total_samples = 0
with torch.no_grad():
    for inputs, labels in val_loader:
        outputs = model(inputs)
        predicted = outputs.argmax(dim=1)
        # What goes here?
accuracy = total_correct / total_samples
print(f"Validation Accuracy: {accuracy:.2f}")

hard

A. total_correct += (predicted == labels).sum().item()
total_samples += labels.size(0)

B. total_correct += predicted.sum().item()
total_samples += labels.size(0)

C. total_correct += (predicted != labels).sum().item()
total_samples += labels.size(0)

D. total_correct += (predicted == labels).sum()
total_samples += len(val_loader)

Solution

1. Step 1: Calculate correct predictions

   Use (predicted == labels).sum().item() to count correct predictions as a Python number.

2. Step 2: Update total samples count

   Use labels.size(0) to add the batch size to total samples for accurate accuracy calculation.

3. Final Answer:

   total_correct += (predicted == labels).sum().item() total_samples += labels.size(0) -> Option A

4. Quick Check:

   Sum correct matches and batch size for accuracy [OK]

Hint: Sum correct predictions and batch size for accuracy [OK]

Common Mistakes:

• Summing predicted values instead of correct matches
• Using != instead of == for correct count
• Adding batch count incorrectly as len(val_loader)