Training and validation loss tracking in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of training and validation loss tracking code

What will be the printed output after running this PyTorch training loop snippet for 1 epoch?

PyTorch

import torch
import torch.nn as nn

model = nn.Linear(2, 1)
criterion = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)

train_losses = []
val_losses = []

# Dummy data
train_data = torch.tensor([[1.0, 2.0], [3.0, 4.0]])
train_targets = torch.tensor([[1.0], [2.0]])
val_data = torch.tensor([[5.0, 6.0]])
val_targets = torch.tensor([[3.0]])

# Training loop for 1 epoch
model.train()
optimizer.zero_grad()
pred = model(train_data)
loss = criterion(pred, train_targets)
loss.backward()
optimizer.step()
train_losses.append(loss.item())

model.eval()
with torch.no_grad():
    val_pred = model(val_data)
    val_loss = criterion(val_pred, val_targets)
    val_losses.append(val_loss.item())

print(f"Train loss: {train_losses[0]:.4f}")
print(f"Validation loss: {val_losses[0]:.4f}")

A

Train loss: 0.0000
Validation loss: 6.2500

B

Train loss: 1.2500
Validation loss: 0.0000

C

Train loss: 1.2500
Validation loss: 6.2500

D

Train loss: 0.0000
Validation loss: 0.0000

💡 Hint

Remember the model is untrained initially, so losses won't be zero.

The initial random weights cause the model to predict values far from targets, resulting in non-zero losses. The training loss is computed on two samples, and validation loss on one sample, both using mean squared error.

🧠 Conceptual

intermediate

Purpose of tracking validation loss during training

Why do we track validation loss separately from training loss during model training?

A To reduce the size of the training dataset automatically

B To speed up the training process by skipping some training steps

C To increase the model complexity during training

D To check if the model is learning patterns that generalize to new data

💡 Hint

Think about how we know if the model is overfitting or not.

Validation loss helps us see if the model performs well on unseen data, indicating generalization. Training loss alone only shows how well the model fits the training data.

❓ Hyperparameter

advanced

Effect of batch size on training and validation loss tracking

How does increasing the batch size during training typically affect the smoothness of training and validation loss curves?

A Larger batch sizes usually make loss curves smoother but may reduce generalization

B Larger batch sizes always make loss curves more noisy and unstable

C Batch size has no effect on loss curve smoothness

D Smaller batch sizes always produce smoother loss curves

💡 Hint

Think about how averaging over more samples affects loss calculation.

Larger batches average gradients over more samples, reducing noise and making loss curves smoother. However, very large batches can hurt generalization.

❓ Metrics

advanced

Calculating average validation loss over multiple batches

Given validation losses for 3 batches as [0.5, 0.7, 0.6], what is the correct average validation loss to report?

A 0.7

B 0.6

C 0.5

D 1.8

💡 Hint

Add all losses and divide by the number of batches.

Average loss = (0.5 + 0.7 + 0.6) / 3 = 1.8 / 3 = 0.6

🔧 Debug

expert

Identifying the bug in validation loss tracking code

What error will this PyTorch code raise when tracking validation loss, and why? ```python val_losses = [] model.eval() for data, target in val_loader: pred = model(data) loss = criterion(pred, target) val_losses.append(loss) print(f"Validation loss: {sum(val_losses)/len(val_losses):.4f}") ```

A TypeError because loss tensors cannot be summed directly without converting to numbers

B ZeroDivisionError because val_losses list is empty

C SyntaxError due to missing colon after for loop

D No error, prints average validation loss correctly

💡 Hint

Think about the data type of loss and how sum() works.

loss is a tensor, and summing tensors in a list without extracting their scalar values causes a TypeError. We must use loss.item() to get a number.

Practice

1. Why do we track both training and validation loss during model training in PyTorch?

easy

A. To speed up the training process

B. To increase the batch size automatically

C. To monitor if the model is learning and generalizing well to new data

D. To reduce the number of model parameters

Solution

1. Step 1: Understand the purpose of training loss

   Training loss shows how well the model fits the training data during learning.

2. Step 2: Understand the purpose of validation loss

   Validation loss shows how well the model performs on unseen data, indicating generalization.

3. Final Answer:

   To monitor if the model is learning and generalizing well to new data -> Option C

4. Quick Check:

   Training and validation loss track learning and generalization [OK]

Hint: Training loss = fit, validation loss = generalization [OK]

Common Mistakes:

• Thinking validation loss speeds up training
• Confusing loss tracking with batch size changes
• Assuming loss tracking reduces parameters

2. Which PyTorch method should you call before starting the validation phase to ensure correct behavior?

easy

A. model.train()

B. model.eval()

C. model.validate()

D. model.freeze()

Solution

1. Step 1: Recall PyTorch modes

   model.train() sets the model to training mode, enabling dropout and batch norm updates.

2. Step 2: Identify validation mode

   model.eval() sets the model to evaluation mode, disabling dropout and batch norm updates for stable validation.

3. Final Answer:

   model.eval() -> Option B

4. Quick Check:

   Use model.eval() before validation [OK]

Hint: Use model.eval() to disable training-only layers [OK]

Common Mistakes:

• Using model.train() during validation
• Assuming model.validate() exists
• Confusing freeze() with eval()

3. Given the code below, what will be the printed average training loss after one epoch?

train_losses = []
for inputs, targets in train_loader:
    optimizer.zero_grad()
    outputs = model(inputs)
    loss = criterion(outputs, targets)
    loss.backward()
    optimizer.step()
    train_losses.append(loss.item())
avg_train_loss = sum(train_losses) / len(train_losses)
print(f"Average training loss: {avg_train_loss:.4f}")

Assuming losses per batch are [0.8, 0.6, 0.4], what is the output?

medium

A. Average training loss: 0.4000

B. Average training loss: 1.8000

C. Average training loss: 0.0000

D. Average training loss: 0.6000

Solution

1. Step 1: Sum the batch losses

   Sum = 0.8 + 0.6 + 0.4 = 1.8

2. Step 2: Calculate average loss

   Average = 1.8 / 3 = 0.6

3. Final Answer:

   Average training loss: 0.6000 -> Option D

4. Quick Check:

   Sum 1.8 / 3 batches = 0.6 [OK]

Hint: Average = sum of losses ÷ number of batches [OK]

Common Mistakes:

• Printing sum instead of average
• Dividing by wrong number of batches
• Using loss tensor instead of loss.item()

4. What is the bug in this code snippet for calculating average validation loss?

model.eval()
val_losses = []
for inputs, targets in val_loader:
    outputs = model(inputs)
    loss = criterion(outputs, targets)
    val_losses.append(loss)
avg_val_loss = sum(val_losses) / len(val_losses)
print(f"Validation loss: {avg_val_loss:.4f}")

medium

A. Appending loss tensor instead of loss.item()

B. Not calling model.train() before validation

C. Using model.eval() disables loss calculation

D. Forgetting to zero gradients before validation

Solution

1. Step 1: Check what is appended to val_losses

   The code appends loss tensors, not scalar values.

2. Step 2: Understand sum behavior on tensors list

   Summing loss tensors retains computation graphs from each forward pass, preventing memory release and causing high memory usage over multiple epochs.

3. Final Answer:

   Appending loss tensor instead of loss.item() -> Option A

4. Quick Check:

   Use loss.item() to get scalar loss [OK]

Hint: Append loss.item(), not loss tensor, for correct averaging [OK]

Common Mistakes:

• Appending loss tensor directly
• Confusing model.train() and model.eval() roles
• Expecting loss calculation to stop in eval mode

5. You want to track training and validation loss per epoch and plot them to check overfitting. Which approach correctly implements this in PyTorch?

hard

A. Use separate lists for training and validation losses, append average loss each epoch, then plot both lists

B. Append batch losses directly to one list for both training and validation, then plot

C. Only track training loss since validation loss is not useful

D. Call model.train() during validation to get accurate loss

Solution

1. Step 1: Understand loss tracking per epoch

   We need to store average losses per epoch separately for training and validation.

2. Step 2: Use separate lists and append averages

   Appending average loss per epoch to separate lists allows clear plotting and comparison.

3. Final Answer:

   Use separate lists for training and validation losses, append average loss each epoch, then plot both lists -> Option A

4. Quick Check:

   Track average losses per epoch separately [OK]

Hint: Track average losses per epoch separately for clear plots [OK]

Common Mistakes:

• Mixing batch losses in one list
• Ignoring validation loss tracking
• Using model.train() during validation