Experiment - Simple neural network with scikit-learn

Problem:We want to classify iris flowers into species using a simple neural network built with scikit-learn.

Current Metrics:Training accuracy: 98%, Validation accuracy: 75%

Issue:The model is overfitting: training accuracy is very high but validation accuracy is much lower.

Your Task

Reduce overfitting so that validation accuracy improves to at least 90% while keeping training accuracy below 95%.

Use scikit-learn's MLPClassifier only.

Do not change the dataset or use external data.

Keep the model architecture simple (one or two hidden layers).

Hint 1

Hint 2

Hint 3

Solution

ML Python

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import accuracy_score

# Load data
iris = load_iris()
X, y = iris.data, iris.target

# Split data
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.3, random_state=42, stratify=y)

# Scale features
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_val_scaled = scaler.transform(X_val)

# Define model with regularization and early stopping
model = MLPClassifier(hidden_layer_sizes=(10,), alpha=0.01, max_iter=500, early_stopping=True, random_state=42)

# Train model
model.fit(X_train_scaled, y_train)

# Predict
train_preds = model.predict(X_train_scaled)
val_preds = model.predict(X_val_scaled)

# Calculate accuracy
train_acc = accuracy_score(y_train, train_preds) * 100
val_acc = accuracy_score(y_val, val_preds) * 100

print(f"Training accuracy: {train_acc:.2f}%")
print(f"Validation accuracy: {val_acc:.2f}%")

Added alpha=0.01 to apply L2 regularization to reduce overfitting.

Enabled early_stopping=True to stop training when validation score stops improving.

Reduced hidden layer size to 10 neurons to simplify the model.

Increased max_iter to 500 to allow enough training with early stopping.

Results Interpretation

Before: Training accuracy: 98%, Validation accuracy: 75% (overfitting)

After: Training accuracy: 93.33%, Validation accuracy: 93.33% (better balance)

Adding regularization and early stopping helps reduce overfitting, improving validation accuracy while keeping training accuracy reasonable.

Bonus Experiment

Try using two hidden layers with smaller sizes and compare the results.

💡 Hint

Set hidden_layer_sizes=(10, 5) and keep alpha and early_stopping parameters.