Epoch-based training in PyTorch - Model Metrics & Evaluation

Metrics & Evaluation - Epoch-based training

Which metric matters for Epoch-based training and WHY

When training a model in epochs, the key metrics to watch are training loss and validation loss. Loss tells us how well the model is learning to predict the right answers. We want the loss to go down as epochs increase. Also, validation accuracy helps us see if the model is improving on new data it hasn't seen before. These metrics show if the model is learning well or just memorizing the training data.

Confusion matrix example after epoch training

Suppose after training for 5 epochs, the model predicts on a test set of 100 samples. The confusion matrix might look like this:

      | Predicted Positive | Predicted Negative |
      |--------------------|--------------------|
      | True Positive (TP): 40 | False Positive (FP): 5 |
      | False Negative (FN): 10 | True Negative (TN): 45 |
    

From this, we can calculate precision, recall, and accuracy to understand model performance after training.

Precision vs Recall tradeoff during epoch training

As epochs increase, the model may become better at finding positive cases (higher recall) but might also make more mistakes (lower precision). For example, in a spam filter, if the model catches almost all spam (high recall) but marks many good emails as spam (low precision), users get annoyed. So, during epoch training, we watch these metrics to find a balance that fits our goal.

What good vs bad metric values look like after epoch training

Good: Training and validation loss both decrease and stabilize. Validation accuracy improves and stays close to training accuracy. Precision and recall are balanced according to the task needs.

Bad: Training loss keeps going down but validation loss goes up (overfitting). Large gap between training and validation accuracy. Precision or recall very low, meaning the model misses many cases or makes many mistakes.

Common pitfalls in epoch-based training metrics

• Accuracy paradox: High accuracy can be misleading if classes are imbalanced.
• Overfitting: Training loss decreases but validation loss increases.
• Data leakage: Validation data accidentally used in training can give false good metrics.
• Early stopping ignored: Not stopping training when validation loss stops improving wastes time and causes overfitting.

Self-check question

Your model has 98% accuracy but only 12% recall on fraud cases after 10 epochs. Is it good for production? Why or why not?

Answer: No, it is not good. The model misses 88% of fraud cases (low recall), which is dangerous. High accuracy is misleading because fraud cases are rare. We need to improve recall to catch more fraud.

Key Result

In epoch-based training, watching loss and validation accuracy over epochs helps detect learning progress and overfitting.

Practice

1. What does one epoch represent in PyTorch training?

easy

A. One full pass through the entire training dataset

B. One batch of data processed by the model

C. One update of the model weights

D. One evaluation of the model on the test set

Solution

1. Step 1: Understand the definition of an epoch

   An epoch means the model sees every example in the training dataset once.

2. Step 2: Differentiate epoch from batch and update

   A batch is a subset of data; multiple batches make an epoch. Weight updates happen after each batch, not the whole dataset.

3. Final Answer:

   One full pass through the entire training dataset -> Option A

4. Quick Check:

   Epoch = full dataset pass [OK]

Hint: Epoch means full dataset seen once by model [OK]

Common Mistakes:

• Confusing batch with epoch
• Thinking epoch is one weight update
• Mixing test evaluation with training epoch

2. Which PyTorch code snippet correctly sets up a loop for 5 epochs?

easy

A. for epoch in range(1, 5):

B. for epoch in range(5, 10):

C. for epoch in range(0, 6):

D. for epoch in range(5):

Solution

1. Step 1: Recall Python range behavior

   range(5) generates 0,1,2,3,4 which is 5 iterations.

2. Step 2: Check other options for iteration count

   range(1,5) gives 4 iterations; range(0,6) gives 6; range(5,10) gives 5 but starts at 5, not zero.

3. Final Answer:

   for epoch in range(5): -> Option D

4. Quick Check:

   range(5) = 5 loops [OK]

Hint: range(5) loops exactly 5 times from 0 to 4 [OK]

Common Mistakes:

• Using range(1,5) which loops 4 times
• Using range(0,6) which loops 6 times
• Starting loop at 5 instead of 0

3. Given this PyTorch training loop snippet, what will be printed after 2 epochs?

for epoch in range(2):
    total_loss = 0
    for batch in dataloader:
        output = model(batch)
        loss = loss_fn(output, batch.labels)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        total_loss += loss.item()
    print(f"Epoch {epoch+1} loss: {total_loss}")

medium

A. Loss values for all batches printed

B. Loss values for epoch 1 and epoch 2 printed

C. Only loss for epoch 2 printed

D. No output printed due to error

Solution

1. Step 1: Analyze the outer loop and print statement

   The outer loop runs 2 times (epochs 0 and 1). After each epoch, it prints total_loss.

2. Step 2: Understand total_loss accumulation and printing

   total_loss sums batch losses per epoch, so print shows loss after each epoch.

3. Final Answer:

   Loss values for epoch 1 and epoch 2 printed -> Option B

4. Quick Check:

   Print inside epoch loop prints each epoch loss [OK]

Hint: Print inside epoch loop shows loss per epoch [OK]

Common Mistakes:

• Thinking only last epoch loss prints
• Confusing batch loss with epoch loss
• Assuming print causes error

4. Identify the error in this epoch-based training code snippet:

for epoch in range(3):
    for batch in dataloader:
        output = model(batch)
        loss = loss_fn(output, batch.labels)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
    print(f"Epoch {epoch+1} done")

medium

A. loss.backward() should be called after optimizer.step()

B. optimizer.step() should be called after optimizer.zero_grad()

C. optimizer.zero_grad() should be called before loss.backward()

D. No error, code is correct

Solution

1. Step 1: Recall correct order of optimizer calls

   Zero gradients before backward pass to avoid accumulating gradients.

2. Step 2: Check code order

   Code calls loss.backward() before optimizer.zero_grad(), which is wrong.

3. Final Answer:

   optimizer.zero_grad() should be called before loss.backward() -> Option C

4. Quick Check:

   Zero grads before backward [OK]

Hint: Always zero gradients before backward pass [OK]

Common Mistakes:

• Zeroing gradients after backward
• Swapping optimizer.step() and zero_grad() order
• Thinking no zero_grad needed

5. You want to train a model for 10 epochs but stop early if validation loss does not improve for 3 epochs. Which approach fits best with epoch-based training?

hard

A. After each epoch, check validation loss and stop if no improvement for 3 epochs

B. Stop training after 3 batches with no improvement

C. Train all 10 epochs regardless of validation loss

D. Check validation loss only after all 10 epochs finish

Solution

1. Step 1: Understand early stopping concept

   Early stopping monitors validation loss after each epoch to decide if training should continue.

2. Step 2: Match early stopping with epoch-based training

   Checking validation loss after each epoch and stopping if no improvement for 3 epochs fits epoch-based training.

3. Final Answer:

   After each epoch, check validation loss and stop if no improvement for 3 epochs -> Option A

4. Quick Check:

   Early stopping checks validation loss per epoch [OK]

Hint: Early stop checks validation loss after each epoch [OK]

Common Mistakes:

• Checking after batches instead of epochs
• Ignoring validation loss during training
• Waiting until all epochs finish to check