Gradient Boosting for regression in ML Python - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

How does Gradient Boosting improve regression predictions?

Imagine you want to predict house prices. Gradient Boosting builds many small models one after another. What is the main way it improves predictions?

A It randomly selects features to reduce overfitting.

B Each model predicts independently and their results are averaged.

C It uses a single large model trained once on all data.

D Each new model tries to fix the errors made by all previous models.

💡 Hint

Think about how mistakes from earlier models guide the next ones.

Gradient Boosting builds models sequentially. Each new model learns to predict the errors (residuals) of the combined previous models, improving overall accuracy.

❓ Predict Output

intermediate

Output of Gradient Boosting regression training metrics

What will be the printed training loss after 3 iterations of this Gradient Boosting Regressor?

ML Python

from sklearn.datasets import make_regression
from sklearn.ensemble import GradientBoostingRegressor

X, y = make_regression(n_samples=100, n_features=1, noise=5, random_state=42)
model = GradientBoostingRegressor(n_estimators=3, learning_rate=1.0, max_depth=1, random_state=42)
model.fit(X, y)
print(round(model.train_score_[-1], 2))

A 0.00

B 0.25

C 0.12

D 0.50

💡 Hint

Look at how train_score_ stores deviance (loss) after each iteration.

The train_score_ attribute stores the loss after each boosting iteration. After 3 iterations, the loss is approximately 0.12 for this setup.

❓ Model Choice

advanced

Choosing the best base learner for Gradient Boosting regression

You want to use Gradient Boosting to predict a continuous target with complex nonlinear relationships. Which base learner is best to use?

A Linear regression models as base learners

B Decision trees with shallow depth (e.g., max_depth=3)

C Single-layer perceptrons (simple neural networks)

D K-nearest neighbors as base learners

💡 Hint

Consider which model type can capture nonlinear patterns well and is commonly used in Gradient Boosting.

Gradient Boosting typically uses shallow decision trees as base learners because they can capture nonlinearities and interactions efficiently while being fast to train.

❓ Hyperparameter

advanced

Effect of learning rate in Gradient Boosting regression

What happens if you set the learning rate too high in a Gradient Boosting regression model?

A The model may overfit and fail to generalize well.

B The model will train slower but be more accurate.

C The model will ignore the residuals and not improve.

D The model will always underfit the training data.

💡 Hint

Think about how a large step size affects the model updates.

A high learning rate causes the model to make large updates, which can lead to overfitting and unstable training, reducing generalization.

🔧 Debug

expert

Why does this Gradient Boosting regression model raise a ValueError?

Consider this code snippet:

from sklearn.ensemble import GradientBoostingRegressor
model = GradientBoostingRegressor(n_estimators=100, loss='log_loss')
model.fit(X_train, y_train)

Why does it raise a ValueError?

A Because 'log_loss' is not a valid loss function for regression tasks.

B Because n_estimators must be less than 50 for Gradient Boosting.

C Because GradientBoostingRegressor requires y_train to be categorical.

D Because the input data X_train is missing.

💡 Hint

Check the allowed loss functions for regression in Gradient Boosting.

'log_loss' is a loss function used for classification, not regression. GradientBoostingRegressor only accepts regression losses like 'squared_error' or 'absolute_error'.