Why ensembles outperform single models in ML Python - Experiment to Prove It

Experiment - Why ensembles outperform single models

Problem:We want to improve prediction accuracy on a classification task using a dataset. Currently, a single decision tree model is used.

Current Metrics:Training accuracy: 95%, Validation accuracy: 80%

Issue:The single decision tree model overfits the training data, causing lower accuracy on validation data.

Your Task

Increase validation accuracy to above 85% by using ensemble methods while keeping training accuracy below 93%.

Use only ensemble methods based on decision trees (e.g., Random Forest or Gradient Boosting).

Do not change the dataset or perform additional feature engineering.

Hint 1

Hint 2

Hint 3

Solution

ML Python

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score

# Load data
X, y = load_breast_cancer(return_X_y=True)

# Split data
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)

# Single decision tree model
single_tree = DecisionTreeClassifier(random_state=42)
single_tree.fit(X_train, y_train)
train_acc_tree = accuracy_score(y_train, single_tree.predict(X_train))
val_acc_tree = accuracy_score(y_val, single_tree.predict(X_val))

# Ensemble model: Random Forest
rf = RandomForestClassifier(n_estimators=100, max_depth=5, random_state=42)
rf.fit(X_train, y_train)
train_acc_rf = accuracy_score(y_train, rf.predict(X_train))
val_acc_rf = accuracy_score(y_val, rf.predict(X_val))

print(f"Single Tree - Train Accuracy: {train_acc_tree:.2f}, Validation Accuracy: {val_acc_tree:.2f}")
print(f"Random Forest - Train Accuracy: {train_acc_rf:.2f}, Validation Accuracy: {val_acc_rf:.2f}")

Replaced single decision tree with Random Forest ensemble.

Set number of trees to 100 to average predictions and reduce variance.

Limited tree depth to 5 to reduce overfitting.

Results Interpretation

Before: Training accuracy 95%, Validation accuracy 80% (overfitting)

After: Training accuracy 90%, Validation accuracy 88% (better generalization)

Ensembles like Random Forest reduce overfitting by combining many simple models. This averaging lowers variance and improves accuracy on new data.

Bonus Experiment

Try Gradient Boosting ensemble instead of Random Forest and compare validation accuracy.

💡 Hint

Use sklearn's GradientBoostingClassifier with learning rate 0.1 and max_depth 3 to see if boosting improves results.