Why regularization controls overfitting in PyTorch - Model Pipeline Impact

Model Pipeline - Why regularization controls overfitting

This pipeline shows how adding regularization helps a model avoid overfitting by keeping it simple and improving its ability to generalize to new data.

Data Flow - 6 Stages

1Data in

1000 rows x 10 columns→Raw dataset with features and labels→1000 rows x 10 columns

[[5.1, 3.5, ..., 1.4], label=0]

↓

2Preprocessing

1000 rows x 10 columns→Normalize features to zero mean and unit variance→1000 rows x 10 columns

[[0.1, -0.3, ..., 0.5], label=0]

↓

3Feature Engineering

1000 rows x 10 columns→No change, use all features→1000 rows x 10 columns

[[0.1, -0.3, ..., 0.5], label=0]

↓

4Model Trains

1000 rows x 10 columns→Train neural network with L2 regularization (weight decay)→Model weights updated

Weights updated with penalty on large values

↓

5Metrics Improve

Validation set: 200 rows x 10 columns→Evaluate loss and accuracy on validation data→Loss and accuracy values

Loss=0.25, Accuracy=85%

↓

6Prediction

New sample: 1 row x 10 columns→Model predicts class probabilities→1 row x 3 classes

[0.1, 0.7, 0.2]

Training Trace - Epoch by Epoch

Loss
1.2 |*       
0.9 | **     
0.6 |  ***   
0.3 |    ****
     --------
     Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	1.2	40%	High loss and low accuracy, model just started learning
5	0.6	70%	Loss decreased, accuracy improved, model learning well
10	0.4	80%	Loss continues to decrease, accuracy rises
15	0.35	83%	Loss stabilizes, accuracy improves slowly
20	0.33	85%	Model converged with good generalization due to regularization

Prediction Trace - 3 Layers

Layer 1: Input Layer

Layer 2: Hidden Layer with ReLU

Layer 3: Output Layer with Softmax

Model Quiz - 3 Questions

What effect does L2 regularization have during training?

A It removes features from the dataset

B It increases the model complexity to fit training data better

C It penalizes large weights to keep the model simple

D It speeds up training by skipping some data

Key Insight

Regularization like L2 adds a penalty for large weights, which keeps the model simpler. This prevents the model from memorizing noise in training data, helping it perform better on new data by reducing overfitting.