Gradient Boosting (GBM) in ML Python

Introduction

Gradient Boosting helps us build strong prediction models by combining many simple models step-by-step to fix mistakes and improve accuracy.

When you want to predict house prices based on features like size and location.

When you need to classify emails as spam or not spam with high accuracy.

When you want to improve a weak model by correcting its errors gradually.

When you have tabular data and want a powerful model without deep neural networks.

Syntax

ML Python

from sklearn.ensemble import GradientBoostingClassifier
model = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1, max_depth=3, random_state=42)
model.fit(X_train, y_train)
predictions = model.predict(X_test)

n_estimators controls how many simple models (trees) are combined.

learning_rate controls how much each new tree corrects the previous ones.

Examples

Using fewer trees but a higher learning rate to train faster.

ML Python

from sklearn.ensemble import GradientBoostingClassifier
model = GradientBoostingClassifier(n_estimators=50, learning_rate=0.2, random_state=42)
model.fit(X_train, y_train)

Using Gradient Boosting for regression tasks with deeper trees.

ML Python

from sklearn.ensemble import GradientBoostingRegressor
model = GradientBoostingRegressor(n_estimators=200, max_depth=4, random_state=42)
model.fit(X_train, y_train)

Sample Model

This example trains a Gradient Boosting model on the Iris flower dataset to classify flower types and prints the accuracy on test data.

ML Python

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import accuracy_score

# Load data
iris = load_iris()
X, y = iris.data, iris.target

# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Create model
model = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1, max_depth=3, random_state=42)

# Train model
model.fit(X_train, y_train)

# Predict
predictions = model.predict(X_test)

# Check accuracy
accuracy = accuracy_score(y_test, predictions)
print(f"Accuracy: {accuracy:.2f}")

Important Notes

Gradient Boosting builds trees one after another, each fixing errors from the last.

Too many trees or too high learning rate can cause overfitting (model learns noise).

Always split data into training and testing to check real performance.

Summary

Gradient Boosting combines many simple models to make a strong one.

It works well for both classification and regression problems.

Adjust n_estimators and learning_rate to balance speed and accuracy.