Gradient Boosting (GBM) in ML Python - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

How does Gradient Boosting improve model predictions?

Gradient Boosting builds a strong model by combining many weak models. What is the main way it improves predictions at each step?

A It fits a new model to the residual errors of the previous model to reduce overall error.

B It randomly selects features to build each new model independently.

C It averages the predictions of all previous models without adjustment.

D It increases the depth of the decision trees in each iteration.

💡 Hint

Think about how the model learns from mistakes made before.

Gradient Boosting fits each new model to the errors (residuals) of the combined previous models, improving predictions step-by-step.

❓ Predict Output

intermediate

Output of training loss during Gradient Boosting

Consider this Python snippet training a Gradient Boosting Regressor on a simple dataset. What is the printed training loss after 3 iterations?

ML Python

from sklearn.ensemble import GradientBoostingRegressor
from sklearn.metrics import mean_squared_error
import numpy as np

X = np.array([[1], [2], [3], [4], [5]])
y = np.array([1.5, 3.5, 3.0, 5.0, 7.5])

model = GradientBoostingRegressor(n_estimators=3, learning_rate=1.0, max_depth=1, random_state=42)
model.fit(X, y)
pred = model.predict(X)
loss = mean_squared_error(y, pred)
print(round(loss, 2))

A 0.50

B 0.00

C 0.12

D 1.25

💡 Hint

Check how well the model fits the small dataset after 3 boosting steps.

The model reduces error progressively; after 3 estimators, the mean squared error is about 0.12.

❓ Model Choice

advanced

Choosing the best base learner for Gradient Boosting

You want to use Gradient Boosting for a classification task with many categorical features. Which base learner is most suitable?

A Linear regression models

B Decision trees with limited depth

C K-nearest neighbors

D Support vector machines

💡 Hint

Think about what base learners are commonly used in Gradient Boosting and handle categorical data well.

Decision trees are the standard base learners in Gradient Boosting and handle categorical features effectively.

❓ Hyperparameter

advanced

Effect of learning rate in Gradient Boosting

What happens if you set the learning rate too high in a Gradient Boosting model?

A The model ignores residuals and stops learning.

B The model automatically prunes trees to prevent overfitting.

C The model will train slower but generalize better.

D The model may overfit quickly and have unstable training.

💡 Hint

Consider how a large step size affects the model updates.

A high learning rate causes large updates that can lead to overfitting and unstable training.

❓ Metrics

expert

Interpreting feature importance in Gradient Boosting

After training a Gradient Boosting model, you get these feature importances: {'age': 0.6, 'income': 0.3, 'gender': 0.1}. What does this mean?

A The 'age' feature contributes most to reducing prediction error in the model.

B All features contribute equally to the model's predictions.

C The 'gender' feature is the most important for predictions.

D Feature importance values indicate the correlation between features.

💡 Hint

Think about what feature importance measures in Gradient Boosting.

Feature importance shows how much each feature helps reduce error; higher means more contribution.

Practice

1. What is the main idea behind Gradient Boosting (GBM)?

easy

A. Using a single deep neural network for prediction

B. Combining many weak models to create a strong model

C. Clustering data points into groups

D. Reducing data dimensions using PCA

Solution

1. Step 1: Understand the concept of boosting

   Boosting means combining many simple models (weak learners) to improve overall prediction.

2. Step 2: Identify Gradient Boosting's approach

   Gradient Boosting builds models sequentially, each correcting errors of the previous one, making a strong model.

3. Final Answer:

   Combining many weak models to create a strong model -> Option B

4. Quick Check:

   Boosting = Combining weak models [OK]

Hint: Boosting means many weak models combined [OK]

Common Mistakes:

• Confusing boosting with deep learning
• Thinking GBM clusters data
• Mixing boosting with dimensionality reduction

2. Which of the following is the correct way to import GradientBoostingClassifier from scikit-learn?

easy

A. import GradientBoostingClassifier from sklearn

B. from sklearn import GradientBoostingClassifier

C. from sklearn.ensemble import GradientBoostingClassifier

D. import GradientBoostingClassifier from sklearn.ensemble

Solution

1. Step 1: Recall correct import syntax in Python

   Python imports classes or functions using 'from module import class' syntax.

2. Step 2: Identify the correct module for GradientBoostingClassifier

   GradientBoostingClassifier is in sklearn.ensemble, so correct import is from sklearn.ensemble import GradientBoostingClassifier.

3. Final Answer:

   from sklearn.ensemble import GradientBoostingClassifier -> Option C

4. Quick Check:

   Correct import syntax = from sklearn.ensemble import GradientBoostingClassifier [OK]

Hint: Use 'from sklearn.ensemble import GradientBoostingClassifier' [OK]

Common Mistakes:

• Using 'import' instead of 'from ... import ...'
• Importing from wrong module
• Wrong order of import statement

3. What will be the output of the following code snippet?

from sklearn.ensemble import GradientBoostingRegressor
X = [[1], [2], [3], [4]]
y = [2, 4, 6, 8]
gbm = GradientBoostingRegressor(n_estimators=100, learning_rate=0.1)
gbm.fit(X, y)
pred = gbm.predict([[5]])
print(round(pred[0], 1))

medium

A. 9.0

B. 10.0

C. 8.0

D. 6.0

Solution

1. Step 1: Understand the training data and model

   X and y show a linear relation y = 2 * x. The model is GradientBoostingRegressor with 100 trees and learning rate 0.1.

2. Step 2: Predict for input 5

   Gradient Boosting can extrapolate somewhat beyond training data, especially with many estimators and moderate learning rate, so prediction is close to 10.0.

3. Final Answer:

   9.0 -> Option A

4. Quick Check:

   Prediction near linear extrapolation = 9.0 [OK]

Hint: Tree boosting can approximate linear extrapolation with enough estimators [OK]

Common Mistakes:

• Expecting exact linear output
• Ignoring learning rate effect
• Confusing classification with regression output

4. Identify the error in this Gradient Boosting code snippet:

from sklearn.ensemble import GradientBoostingClassifier
X = [[0], [1], [2]]
y = [0, 1, 0]
gbm = GradientBoostingClassifier(n_estimators='100')
gbm.fit(X, y)

medium

A. n_estimators should be an integer, not a string

B. X should be a numpy array, not a list

C. GradientBoostingClassifier cannot handle binary targets

D. Missing learning_rate parameter

Solution

1. Step 1: Check parameter types

   n_estimators expects an integer number of trees, but '100' is a string, causing a type error.

2. Step 2: Validate other parts

   X as list is acceptable, binary targets are valid, learning_rate is optional with default 0.1.

3. Final Answer:

   n_estimators should be an integer, not a string -> Option A

4. Quick Check:

   Parameter types must match expected types [OK]

Hint: Check parameter types carefully [OK]

Common Mistakes:

• Passing numbers as strings
• Assuming lists are invalid input
• Thinking learning_rate is mandatory

5. You want to improve a Gradient Boosting model's accuracy but training is very slow. Which combination of hyperparameters is best to try first?

hard

A. Increase n_estimators and decrease learning_rate

B. Increase both n_estimators and learning_rate

C. Set n_estimators to 1 and learning_rate to 0.01

D. Decrease n_estimators and increase learning_rate

Solution

1. Step 1: Understand hyperparameter effects

   More n_estimators means more trees and slower training; higher learning_rate speeds learning but risks overfitting.

2. Step 2: Balance speed and accuracy

   Decreasing n_estimators reduces training time; increasing learning_rate compensates to keep accuracy.

3. Final Answer:

   Decrease n_estimators and increase learning_rate -> Option D

4. Quick Check:

   Fewer trees + higher learning rate = faster training [OK]

Hint: Fewer trees + higher learning rate speeds training [OK]

Common Mistakes:

• Increasing both slows training
• Too low n_estimators hurts accuracy
• Too low learning_rate slows learning