Gradient Boosting for regression in ML Python - ML Experiment: Train & Evaluate

Experiment - Gradient Boosting for regression

Problem:Predict house prices using a gradient boosting regression model.

Current Metrics:Training R2 score: 0.95, Validation R2 score: 0.70

Issue:The model is overfitting: training score is very high but validation score is much lower.

Your Task

Reduce overfitting so that validation R2 score improves to at least 0.85 while keeping training R2 below 0.90.

Do not change the dataset or features.

Only adjust gradient boosting hyperparameters.

Hint 1

Hint 2

Hint 3

Solution

ML Python

from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.metrics import r2_score

# Load data
X, y = fetch_california_housing(return_X_y=True)

# Split data
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)

# Define model with adjusted hyperparameters
model = GradientBoostingRegressor(
    n_estimators=200,      # more trees
    learning_rate=0.05,    # smaller step size
    max_depth=3,           # limit tree depth
    subsample=0.8,         # use 80% of data per tree
    random_state=42
)

# Train model
model.fit(X_train, y_train)

# Predict
train_preds = model.predict(X_train)
val_preds = model.predict(X_val)

# Calculate R2 scores
train_r2 = r2_score(y_train, train_preds)
val_r2 = r2_score(y_val, val_preds)

print(f"Training R2 score: {train_r2:.2f}")
print(f"Validation R2 score: {val_r2:.2f}")

Reduced learning rate from default 0.1 to 0.05 to slow learning and improve generalization.

Increased number of estimators from default 100 to 200 to compensate for smaller learning rate.

Limited max_depth to 3 to prevent overly complex trees.

Added subsample=0.8 to use random 80% of data per tree, adding randomness to reduce overfitting.

Results Interpretation

Before: Training R2 = 0.95, Validation R2 = 0.70 (overfitting)

After: Training R2 = 0.88, Validation R2 = 0.86 (better generalization)

Reducing learning rate, limiting tree depth, and adding subsampling helps reduce overfitting in gradient boosting, improving validation performance.

Bonus Experiment

Try adding early stopping to stop training when validation score stops improving.

💡 Hint

Use the 'validation_fraction' and 'n_iter_no_change' parameters in GradientBoostingRegressor.

Practice

1. What is the main idea behind Gradient Boosting for regression?

easy

A. Combining many simple models step-by-step to improve predictions

B. Using a single complex model to predict values

C. Randomly guessing values and selecting the best guess

D. Using only one decision tree without updates

Solution

1. Step 1: Understand Gradient Boosting concept

   Gradient Boosting builds a strong model by adding simple models one after another, each fixing errors of the previous.

2. Step 2: Compare options with this idea

   Only Combining many simple models step-by-step to improve predictions describes combining many simple models step-by-step to improve predictions.

3. Final Answer:

   Combining many simple models step-by-step to improve predictions -> Option A

4. Quick Check:

   Gradient Boosting = Combining simple models [OK]

Hint: Remember: Gradient Boosting adds models one by one [OK]

Common Mistakes:

• Thinking it uses only one model
• Confusing with random guessing
• Assuming it uses a single complex model

2. Which of the following is the correct way to create a Gradient Boosting Regressor in Python using scikit-learn?

easy

A. import GradientBoostingRegressor
model = GradientBoostingRegressor()

B. from sklearn.linear_model import GradientBoostingRegressor
model = GradientBoostingRegressor(learning_rate=0.1)

C. from sklearn.ensemble import GradientBoostingClassifier
model = GradientBoostingClassifier(n_estimators=100)

D. from sklearn.ensemble import GradientBoostingRegressor
model = GradientBoostingRegressor(n_estimators=100, learning_rate=0.1)

Solution

1. Step 1: Identify correct import and class for regression

   GradientBoostingRegressor is in sklearn.ensemble, not sklearn.linear_model or a classifier.

2. Step 2: Check syntax correctness

   from sklearn.ensemble import GradientBoostingRegressor model = GradientBoostingRegressor(n_estimators=100, learning_rate=0.1) correctly imports and creates the model with parameters n_estimators and learning_rate.

3. Final Answer:

   Correct import and model creation with sklearn.ensemble.GradientBoostingRegressor -> Option D

4. Quick Check:

   Correct import and class = from sklearn.ensemble import GradientBoostingRegressor model = GradientBoostingRegressor(n_estimators=100, learning_rate=0.1) [OK]

Hint: Use sklearn.ensemble for GradientBoostingRegressor [OK]

Common Mistakes:

• Importing from wrong module
• Using classifier instead of regressor
• Missing parameters or wrong syntax

3. What will be the output of the following code snippet?

from sklearn.ensemble import GradientBoostingRegressor
import numpy as np

X = np.array([[1], [2], [3], [4], [5]])
y = np.array([1.5, 3.5, 5.5, 7.5, 9.5])
model = GradientBoostingRegressor(n_estimators=10, learning_rate=0.5)
model.fit(X, y)
pred = model.predict(np.array([[6]]))
print(round(pred[0], 1))

medium

A. 10.0

B. 11.5

C. 12.0

D. 9.5

Solution

1. Step 1: Understand training data pattern

   y roughly equals 2*x - 0.5 (1.5, 3.5, 5.5, 7.5, 9.5). So for x=6, expected y ~ 11.5.

2. Step 2: Predict with Gradient Boosting model

   Model with 10 estimators and learning rate 0.5 fits this pattern well, predicting close to 11.5 for input 6.

3. Final Answer:

   11.5 -> Option B

4. Quick Check:

   Prediction for 6 ≈ 11.5 [OK]

Hint: Check pattern in y to guess prediction quickly [OK]

Common Mistakes:

• Ignoring the linear pattern in data
• Confusing classifier with regressor output
• Rounding errors or wrong rounding

4. Identify the error in this Gradient Boosting regression code and fix it:

from sklearn.ensemble import GradientBoostingRegressor
X = [[1], [2], [3]]
y = [2, 4, 6]
model = GradientBoostingRegressor(n_estimators=50)
model.fit(X, y)
print(model.predict([4]))

medium

A. Import GradientBoostingClassifier instead

B. Change n_estimators to 1

C. Change predict input to [[4]] instead of [4]

D. Change y to a numpy array

Solution

1. Step 1: Check input shape for predict method

   Model expects 2D array for predict, but [4] is 1D. It should be [[4]] to match training input shape.

2. Step 2: Fix predict input shape

   Changing predict input to [[4]] fixes the error and allows prediction.

3. Final Answer:

   Change predict input to [[4]] instead of [4] -> Option C

4. Quick Check:

   Predict input shape must match training input [OK]

Hint: Always use 2D array for predict input in scikit-learn [OK]

Common Mistakes:

• Passing 1D array to predict
• Changing unrelated parameters
• Using classifier instead of regressor

5. You want to improve your Gradient Boosting regression model's accuracy on a dataset but notice it overfits. Which combination of parameter changes is best to reduce overfitting?

hard

A. Decrease n_estimators and decrease learning_rate

B. Decrease n_estimators and increase learning_rate

C. Increase n_estimators and decrease learning_rate

D. Increase n_estimators and increase learning_rate

Solution

1. Step 1: Understand overfitting in Gradient Boosting

   Overfitting means model fits training data too closely, losing generalization.

2. Step 2: Adjust parameters to reduce overfitting

   Decreasing n_estimators reduces model complexity; decreasing learning_rate slows learning, both help reduce overfitting.

3. Final Answer:

   Decrease n_estimators and decrease learning_rate -> Option A

4. Quick Check:

   Lower complexity and slower learning reduce overfitting [OK]

Hint: Lower n_estimators and learning_rate to fight overfitting [OK]

Common Mistakes:

• Increasing both parameters causing more overfitting
• Increasing learning_rate alone
• Ignoring parameter effects on overfitting