Gradient Boosting for regression in ML Python

Introduction

Gradient Boosting helps us make better predictions by combining many simple models step-by-step. It focuses on fixing mistakes from earlier tries to improve accuracy.

When you want to predict a number, like house prices or temperatures.

When simple models don't give good enough results and you want better accuracy.

When you have data with many features and want a model that learns complex patterns.

When you want a model that can handle different types of data without much preparation.

Syntax

ML Python

from sklearn.ensemble import GradientBoostingRegressor
model = GradientBoostingRegressor(n_estimators=100, learning_rate=0.1, max_depth=3)
model.fit(X_train, y_train)
predictions = model.predict(X_test)

n_estimators controls how many simple models are combined.

learning_rate controls how much each new model fixes the errors.

Examples

A smaller learning rate and fewer trees for slower, careful learning.

ML Python

model = GradientBoostingRegressor(n_estimators=50, learning_rate=0.05, max_depth=2)

More trees and deeper trees for a stronger, faster model but risk of overfitting.

ML Python

model = GradientBoostingRegressor(n_estimators=200, learning_rate=0.2, max_depth=4)

Sample Model

This program creates fake data for regression, trains a Gradient Boosting model, and shows how well it predicts new data by printing the error and some predictions.

ML Python

from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.metrics import mean_squared_error

# Create sample data
X, y = make_regression(n_samples=200, n_features=5, noise=10, random_state=42)

# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Create model
model = GradientBoostingRegressor(n_estimators=100, learning_rate=0.1, max_depth=3, random_state=42)

# Train model
model.fit(X_train, y_train)

# Predict
predictions = model.predict(X_test)

# Calculate error
mse = mean_squared_error(y_test, predictions)

print(f"Mean Squared Error: {mse:.2f}")
print(f"First 5 predictions: {predictions[:5]}")

Important Notes

Gradient Boosting can take longer to train than simple models because it builds many trees one after another.

Choosing the right learning_rate and n_estimators is important to avoid overfitting or underfitting.

It works well with default settings but tuning can improve results for your specific data.

Summary

Gradient Boosting builds a strong prediction model by combining many simple models step-by-step.

It is useful for predicting numbers when you want better accuracy than simple models.

Adjusting parameters like n_estimators and learning_rate helps control learning speed and accuracy.