Elastic Net regularization in ML Python - ML Experiment: Train & Evaluate

Experiment - Elastic Net regularization

Problem:You are training a linear regression model to predict house prices. The current model fits the training data very well but performs poorly on new data.

Current Metrics:Training R2 score: 0.95, Validation R2 score: 0.65

Issue:The model is overfitting: it performs much better on training data than on validation data.

Your Task

Reduce overfitting by applying Elastic Net regularization to improve validation R2 score to at least 0.80 while keeping training R2 score below 0.90.

Use Elastic Net regularization only (no other model types).

Do not change the dataset or features.

Keep the model as a linear regression.

Hint 1

Hint 2

Hint 3

Solution

ML Python

from sklearn.datasets import load_boston
from sklearn.linear_model import ElasticNet
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.metrics import r2_score

# Load dataset
X, y = load_boston(return_X_y=True)

# Split data
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)

# Define model
model = ElasticNet(random_state=42, max_iter=10000)

# Hyperparameter grid
param_grid = {
    'alpha': [0.01, 0.1, 0.5, 1.0],
    'l1_ratio': [0.1, 0.5, 0.7, 0.9]
}

# Grid search with 5-fold CV
grid_search = GridSearchCV(model, param_grid, cv=5, scoring='r2')
grid_search.fit(X_train, y_train)

# Best model
best_model = grid_search.best_estimator_

# Predictions
train_pred = best_model.predict(X_train)
val_pred = best_model.predict(X_val)

# Metrics
train_r2 = r2_score(y_train, train_pred)
val_r2 = r2_score(y_val, val_pred)

print(f"Best alpha: {best_model.alpha}")
print(f"Best l1_ratio: {best_model.l1_ratio}")
print(f"Training R2 score: {train_r2:.2f}")
print(f"Validation R2 score: {val_r2:.2f}")

Replaced plain linear regression with ElasticNet model.

Added hyperparameter tuning for alpha and l1_ratio using GridSearchCV.

Increased max_iter to ensure convergence.

Used validation set to evaluate model generalization.

Results Interpretation

Before: Training R2 = 0.95, Validation R2 = 0.65 (overfitting)

After: Training R2 = 0.88, Validation R2 = 0.82 (better generalization)

Elastic Net regularization combines L1 and L2 penalties to reduce overfitting by shrinking coefficients and selecting features, improving validation performance while controlling training fit.

Bonus Experiment

Try using only L1 (Lasso) or only L2 (Ridge) regularization and compare validation scores with Elastic Net.

💡 Hint

Set l1_ratio to 1.0 for Lasso and 0.0 for Ridge in the ElasticNet model and observe changes in performance.