Train/val/test split in PyTorch - Model Metrics & Evaluation

Metrics & Evaluation - Train/val/test split

Which metric matters for Train/val/test split and WHY

When we split data into train, validation, and test sets, the main goal is to check how well our model learns and how well it will work on new data.

Training metrics (like loss and accuracy) show how well the model learns from the training data.

Validation metrics help us tune the model and avoid overfitting by checking performance on unseen data during training.

Test metrics give a final unbiased estimate of how the model will perform in the real world.

So, the key metrics are accuracy, loss, precision, recall, or others depending on the task, but measured separately on train, val, and test sets to understand model behavior.

Confusion matrix example for validation set

      | Predicted Positive | Predicted Negative |
      |--------------------|--------------------|
      | True Positive (TP): 50 | False Negative (FN): 10 |
      | False Positive (FP): 5 | True Negative (TN): 35 |

      Total samples in val set = TP + FN + FP + TN = 50 + 10 + 5 + 35 = 100
    

This matrix helps calculate precision, recall, and accuracy on the validation set.

Precision vs Recall tradeoff in validation

Imagine a spam email detector:

• High precision means most emails marked as spam really are spam (few good emails wrongly blocked).
• High recall means most spam emails are caught (few spam emails missed).

During validation, we adjust the model to balance precision and recall depending on what matters more.

Train/val/test split lets us test these tradeoffs on validation data before final testing.

Good vs Bad metric values for train/val/test split

Good:

• Training accuracy: 90%
• Validation accuracy: 88%
• Test accuracy: 87%
• Close values show the model generalizes well.

Bad:

• Training accuracy: 95%
• Validation accuracy: 60%
• Test accuracy: 58%
• Big gap means overfitting; model memorizes training but fails on new data.

Common pitfalls with train/val/test split metrics

• Data leakage: If test data leaks into training, test metrics look too good and mislead.
• Imbalanced splits: Unequal class distribution in splits can give wrong metric impressions.
• Overfitting: High train but low val/test metrics show model memorizes instead of learning.
• Small validation/test sets: Too few samples make metrics unstable and unreliable.

Self-check question

Your model has 98% accuracy on training but only 12% recall on fraud cases in validation. Is it good for production?

Answer: No. The model misses most fraud cases (low recall), which is dangerous. Despite high training accuracy, it does not generalize well. You need to improve recall on validation before trusting the model.

Key Result

Train/val/test split metrics reveal if the model learns well and generalizes; watch for gaps indicating overfitting or data issues.

Practice

1. What is the main purpose of splitting data into train, validation, and test sets in PyTorch?

easy

A. To train the model, tune hyperparameters, and evaluate performance separately

B. To increase the size of the dataset artificially

C. To reduce the number of features in the dataset

D. To convert data into tensors automatically

Solution

1. Step 1: Understand the role of each data split

   The training set is used to teach the model, the validation set helps tune settings, and the test set checks how well the model works on new data.

2. Step 2: Match purpose to options

   Only To train the model, tune hyperparameters, and evaluate performance separately correctly describes these roles. Other options describe unrelated tasks.

3. Final Answer:

   To train the model, tune hyperparameters, and evaluate performance separately -> Option A

4. Quick Check:

   Train/val/test split = separate training, tuning, testing [OK]

Hint: Remember: train teaches, val tunes, test checks [OK]

Common Mistakes:

• Thinking splitting increases dataset size
• Confusing splitting with feature selection
• Assuming splitting converts data types

2. Which of the following is the correct way to split a PyTorch dataset into train and test sets using random_split?

easy

A. train_set, test_set = random_split(dataset, [80%, 20%])

B. train_set, test_set = random_split(dataset, [80, 20])

C. train_set, test_set = random_split(dataset, [0.8, 0.2])

D. train_set, test_set = random_split(dataset, [80])

Solution

1. Step 1: Check the expected input for random_split

   random_split expects the dataset and a list of lengths (integers) for each split, not fractions or percentages.

2. Step 2: Evaluate each option

   train_set, test_set = random_split(dataset, [80, 20]) uses integer lengths [80, 20], which is correct. train_set, test_set = random_split(dataset, [0.8, 0.2]) uses floats, train_set, test_set = random_split(dataset, [80%, 20%]) uses invalid syntax with %, and train_set, test_set = random_split(dataset, [80]) provides only one length.

3. Final Answer:

   train_set, test_set = random_split(dataset, [80, 20]) -> Option B

4. Quick Check:

   random_split needs integer lengths [OK]

Hint: Use integer lengths, not fractions, in random_split [OK]

Common Mistakes:

• Passing fractions or percentages instead of integers
• Providing only one length instead of two
• Using invalid syntax like % in list

3. Given the code below, what will be the length of val_set?

from torch.utils.data import random_split

full_dataset = list(range(100))
train_size = 70
val_size = 20
test_size = 10
train_set, val_set, test_set = random_split(full_dataset, [train_size, val_size, test_size])
print(len(val_set))

medium

A. 10

B. 70

C. 20

D. 100

Solution

1. Step 1: Understand random_split lengths

   The list [70, 20, 10] tells random_split to split the dataset into parts of those exact lengths.

2. Step 2: Identify val_set length

   val_set corresponds to the second split with length 20, so len(val_set) will be 20.

3. Final Answer:

   20 -> Option C

4. Quick Check:

   val_size = 20 means val_set length = 20 [OK]

Hint: Lengths in random_split match output sizes exactly [OK]

Common Mistakes:

• Confusing sizes with percentages
• Assuming val_set length is sum of others
• Forgetting order of splits in output

4. You wrote this code to split your dataset but get a ValueError: Sum of input lengths does not equal the length of the input dataset!. What is the error?

from torch.utils.data import random_split

dataset = list(range(50))
train_set, test_set = random_split(dataset, [30, 25])

medium

A. The sum of lengths 30 + 25 is greater than dataset length 50

B. random_split requires fractions, not lengths

C. dataset must be a PyTorch tensor, not a list

D. random_split cannot split into two parts

Solution

1. Step 1: Check dataset length and split lengths

   Dataset length is 50. The sum of split lengths is 30 + 25 = 55, which is more than 50.

2. Step 2: Understand random_split requirements

   random_split requires the sum of split lengths to exactly match dataset length.

3. Final Answer:

   The sum of lengths 30 + 25 is greater than dataset length 50 -> Option A

4. Quick Check:

   Sum of split lengths must equal dataset length [OK]

Hint: Sum of split lengths must equal dataset size exactly [OK]

Common Mistakes:

• Passing split lengths that don't sum to dataset size
• Thinking random_split accepts fractions
• Assuming dataset must be tensor, not list

5. You want to split a PyTorch dataset of 1000 samples into 70% train, 15% validation, and 15% test sets. Which code correctly performs this split using random_split?

hard

A. train_set, val_set, test_set = random_split(dataset, [0.7, 0.15, 0.15])

B. train_set, val_set, test_set = random_split(dataset, [700, 150, 100])

C. train_set, val_set, test_set = random_split(dataset, [700, 150])

D. train_set, val_set, test_set = random_split(dataset, [700, 150, 150])

Solution

1. Step 1: Calculate integer split sizes

   70% of 1000 is 700, 15% is 150, so splits are [700, 150, 150].

2. Step 2: Check options for correctness

   train_set, val_set, test_set = random_split(dataset, [700, 150, 150]) uses correct integer lengths summing to 1000. train_set, val_set, test_set = random_split(dataset, [0.7, 0.15, 0.15]) uses floats (invalid), train_set, val_set, test_set = random_split(dataset, [700, 150]) misses one split, train_set, val_set, test_set = random_split(dataset, [700, 150, 100]) sums to 950 (incorrect).

3. Final Answer:

   train_set, val_set, test_set = random_split(dataset, [700, 150, 150]) -> Option D

4. Quick Check:

   Use integer lengths summing to dataset size [OK]

Hint: Convert percentages to integers summing to dataset size [OK]

Common Mistakes:

• Passing fractions instead of integers
• Not summing split sizes to dataset length
• Missing one of the splits