Best model saving pattern in PyTorch - Model Metrics & Evaluation

Metrics & Evaluation - Best model saving pattern

Which metric matters for this concept and WHY

When saving the best model during training, the key metric to track is the validation metric that best reflects your goal. For example, if you want a model that predicts correctly, use validation accuracy. If you want to catch rare events, use validation recall. Saving the model with the best value of this metric ensures you keep the most useful version.

Confusion matrix or equivalent visualization (ASCII)

    Confusion Matrix Example:

          Predicted
          P     N
    Actual P  TP    FN
           N  FP    TN

    TP = True Positives
    FP = False Positives
    TN = True Negatives
    FN = False Negatives

    Use this matrix to calculate metrics like accuracy, precision, recall.
    The best model saving pattern depends on which metric you want to maximize.
    

Precision vs Recall tradeoff with concrete examples

Choosing which metric to save your best model on depends on your problem:

• High Precision: Save model with highest precision if you want few false alarms. Example: Spam filter that should not mark good emails as spam.
• High Recall: Save model with highest recall if you want to catch as many positives as possible. Example: Cancer detector that should not miss any cancer cases.
• Balanced (F1 score): Save model with best F1 score if you want a balance between precision and recall.

What "good" vs "bad" metric values look like for this use case

Good model saving pattern means:

• Saving the model checkpoint only when the validation metric improves.
• Not saving models that perform worse or the same as previous best.
• Using early stopping to avoid overfitting.

Bad pattern examples:

• Saving model every epoch regardless of metric.
• Saving based on training metric instead of validation metric.
• Ignoring metric fluctuations and saving worse models.

Metrics pitfalls (accuracy paradox, data leakage, overfitting indicators)

• Accuracy paradox: High accuracy can be misleading if data is imbalanced. Saving best model by accuracy alone may not help.
• Data leakage: If validation data leaks into training, the saved "best" model may not generalize.
• Overfitting: Model saved with best validation metric may still overfit if metric fluctuates or validation set is small.
• Metric choice: Saving based on wrong metric (e.g., training loss) can save a poor model.

Self-check: Your model has 98% accuracy but 12% recall on fraud. Is it good?

No, it is not good for fraud detection. Although accuracy is high, recall is very low. This means the model misses most fraud cases, which is dangerous. You should save and select models based on recall or a metric that values catching fraud cases.

Key Result

Save the model checkpoint that achieves the best validation metric aligned with your goal (e.g., accuracy, recall, or F1) to ensure the best real-world performance.

Practice

1. What is the best practice for saving a PyTorch model during training?

easy

A. Save the model only at the start of training.

B. Save the model only when it improves on validation data.

C. Save the model after every training batch.

D. Save the model only if the training loss increases.

Solution

1. Step 1: Understand model saving timing

   Saving the model only when validation improves ensures you keep the best version, avoiding unnecessary saves.

2. Step 2: Compare other options

   Saving every batch wastes space; saving at start or on loss increase is not useful for best model.

3. Final Answer:

   Save the model only when it improves on validation data. -> Option B

4. Quick Check:

   Save best validation model = C [OK]

Hint: Save model only on validation improvement to keep best [OK]

Common Mistakes:

• Saving model too frequently wastes storage
• Saving only at start misses improvements
• Saving on training loss increase is wrong

2. Which of the following is the correct PyTorch code to save only the model weights?

easy

A. torch.save(model.state_dict(), 'model.pth')

B. torch.save(model, 'model.pth')

C. model.save('model.pth')

D. model.state_dict().save('model.pth')

Solution

1. Step 1: Identify correct saving method

   PyTorch saves weights using torch.save(model.state_dict(), filename).

2. Step 2: Check other options

   Saving the whole model (torch.save(model, 'model.pth')) is possible but less flexible; options C and D are invalid syntax.

3. Final Answer:

   torch.save(model.state_dict(), 'model.pth') -> Option A

4. Quick Check:

   Save weights with state_dict() = A [OK]

Hint: Use torch.save(model.state_dict(), filename) to save weights [OK]

Common Mistakes:

• Trying to save model directly without state_dict
• Using non-existent save methods on model
• Confusing saving weights vs full model

3. Given this code snippet, what will be printed?

import torch
import torch.nn as nn

model = nn.Linear(2, 1)
torch.save(model.state_dict(), 'best.pth')
new_model = nn.Linear(2, 1)
new_model.load_state_dict(torch.load('best.pth'))
print(new_model.weight.shape)

medium

A. torch.Size([1, 2])

B. torch.Size([2, 1])

C. torch.Size([1, 1])

D. Error: shape mismatch

Solution

1. Step 1: Understand model architecture

   nn.Linear(2,1) creates weights of shape [1, 2] (output features, input features).

2. Step 2: Loading weights into new model

   Loading saved weights into identical model keeps weight shape same.

3. Final Answer:

   torch.Size([1, 2]) -> Option A

4. Quick Check:

   Linear(2,1) weight shape = [1, 2] [OK]

Hint: Linear layer weights shape = (out_features, in_features) [OK]

Common Mistakes:

• Confusing input/output dimensions order
• Expecting error when loading identical model
• Misreading weight shape as (2,1)

4. What is wrong with this code snippet for saving the best model?

if val_loss < best_loss:
    best_loss = val_loss
    torch.save(model, 'best_model.pth')

medium

A. There is no condition to check validation loss.

B. It should save model.state_dict() instead of model.

C. It does not update best_loss correctly.

D. It saves the entire model, which is less flexible than saving state_dict.

Solution

1. Step 1: Analyze saving method

   Saving entire model works but is less flexible and may cause issues when loading on different devices or PyTorch versions.

2. Step 2: Compare with best practice

   Best practice is saving model.state_dict() for portability and smaller files.

3. Final Answer:

   It saves the entire model, which is less flexible than saving state_dict. -> Option D

4. Quick Check:

   Save state_dict() preferred over full model [OK]

Hint: Save state_dict() for flexibility, not full model [OK]

Common Mistakes:

• Saving full model without state_dict
• Ignoring portability issues
• Assuming full model save is always best

5. You want to save the best model during training based on validation accuracy. Which code snippet correctly implements this pattern?

best_acc = 0.0
for epoch in range(epochs):
    train()
    val_acc = validate()
    # Save best model here
    ???

hard

A. if val_acc < best_acc:
best_acc = val_acc
torch.save(model.state_dict(), 'best_model.pth')

B. if val_acc == best_acc:
torch.save(model.state_dict(), 'best_model.pth')

C. if val_acc > best_acc:
best_acc = val_acc
torch.save(model.state_dict(), 'best_model.pth')

D. torch.save(model.state_dict(), 'best_model.pth') # save every epoch

Solution

1. Step 1: Identify saving condition

   We save model only if validation accuracy improves (val_acc > best_acc).

2. Step 2: Update best accuracy and save weights

   Update best_acc and save model.state_dict() to keep best weights.

3. Final Answer:

   if val_acc > best_acc: best_acc = val_acc torch.save(model.state_dict(), 'best_model.pth') -> Option C

4. Quick Check:

   Save on val_acc improvement = B [OK]

Hint: Save model only if validation accuracy improves [OK]

Common Mistakes:

• Saving when accuracy decreases
• Saving every epoch wastes space
• Not updating best accuracy variable