Boosting concept in ML Python - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to create a simple AdaBoost classifier using scikit-learn.

ML Python

from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier

model = AdaBoostClassifier(base_estimator=[1], n_estimators=50, random_state=42)
model.fit(X_train, y_train)

Drag options to blanks, or click blank then click option'

AKNeighborsClassifier()

BRandomForestClassifier()

CDecisionTreeClassifier(max_depth=1)

DSVC()

Attempts:

3 left

💡 Hint

Common Mistakes

Using complex models like RandomForest or SVC as base estimators.

Not specifying max_depth=1 for the decision tree.

AdaBoost commonly uses a simple decision tree with max_depth=1 as its base estimator, called a stump.

2fill in blank

medium

Complete the code to calculate the weighted error of a weak learner in boosting.

ML Python

weighted_error = sum(sample_weights * (predictions != y_true)) / [1]

Drag options to blanks, or click blank then click option'

Asum(sample_weights)

Blen(predictions)

Csum(y_true)

Dlen(sample_weights)

Attempts:

3 left

💡 Hint

Common Mistakes

Dividing by the number of samples instead of sum of weights.

Using sum(y_true) which is unrelated.

The weighted error divides the weighted sum of wrong predictions by the total sum of weights.

3fill in blank

hard

Fix the error in updating sample weights after a boosting iteration.

ML Python

sample_weights = sample_weights * np.exp([1] * (predictions != y_true))
sample_weights /= sample_weights.sum()

Drag options to blanks, or click blank then click option'

A1

B-alpha

Calpha

D-1

Attempts:

3 left

💡 Hint

Common Mistakes

Using positive alpha which decreases weights for errors.

Using constants 1 or -1 which ignore alpha.

Weights increase for misclassified samples, so the exponent must be negative alpha times the error indicator.

4fill in blank

hard

Fill both blanks to compute the alpha (learner weight) in AdaBoost.

ML Python

alpha = 0.5 * np.log((1 - [1]) / [2])

Drag options to blanks, or click blank then click option'

Aweighted_error

B1 - weighted_error

D1 + weighted_error

Attempts:

3 left

💡 Hint

Common Mistakes

Swapping numerator and denominator.

Using incorrect expressions like 1 + weighted_error.

Alpha is calculated as 0.5 * log((1 - error) / error) where error is the weighted error.

5fill in blank

hard

Fill all three blanks to create a dictionary comprehension that maps each weak learner to its alpha weight if alpha is positive.

ML Python

learner_weights = [1]: [2] for learner, [3] in zip(learners, alphas) if [2] > 0}

Drag options to blanks, or click blank then click option'

Alearner

Balpha

Dweight

Attempts:

3 left

💡 Hint

Common Mistakes

Using wrong variable names like 'weight' or inconsistent names.

Not filtering by positive alpha.

The dictionary maps each learner to its alpha weight, filtering only positive alphas.

Practice

1. What is the main idea behind boosting in machine learning?

easy

A. Randomly selecting features for training

B. Using a single complex model to fit data

C. Reducing the size of the dataset

D. Combining many weak models to create a strong model

Solution

1. Step 1: Understand boosting concept

   Boosting builds a strong model by combining many simple (weak) models.

2. Step 2: Compare options with definition

   Only Combining many weak models to create a strong model correctly describes this idea; others describe different techniques.

3. Final Answer:

   Combining many weak models to create a strong model -> Option D

4. Quick Check:

   Boosting = Combining weak models [OK]

Hint: Boosting = many weak models combined [OK]

Common Mistakes:

• Thinking boosting uses one complex model
• Confusing boosting with feature selection
• Believing boosting reduces dataset size

2. Which of the following is the correct syntax to create an AdaBoost classifier in Python using scikit-learn?

easy

A. from sklearn.ensemble import AdaBoostClassifier
model = AdaBoostClassifier()

B. from sklearn.ensemble import AdaBoost
model = AdaBoost()

C. from sklearn.boost import AdaBoostClassifier
model = AdaBoostClassifier()

D. import AdaBoost from sklearn.ensemble
model = AdaBoost()

Solution

1. Step 1: Recall correct import path

   In scikit-learn, AdaBoostClassifier is in sklearn.ensemble module.

2. Step 2: Check syntax correctness

   from sklearn.ensemble import AdaBoostClassifier model = AdaBoostClassifier() uses correct import and class name; others have wrong module or syntax.

3. Final Answer:

   from sklearn.ensemble import AdaBoostClassifier\nmodel = AdaBoostClassifier() -> Option A

4. Quick Check:

   Correct import and class name = from sklearn.ensemble import AdaBoostClassifier model = AdaBoostClassifier() [OK]

Hint: AdaBoostClassifier is in sklearn.ensemble [OK]

Common Mistakes:

• Using wrong module like sklearn.boost
• Incorrect import syntax
• Wrong class name without 'Classifier'

3. Consider this Python code using AdaBoost:

from sklearn.datasets import load_iris
from sklearn.ensemble import AdaBoostClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target, random_state=42)
model = AdaBoostClassifier(n_estimators=10, random_state=42)
model.fit(X_train, y_train)
preds = model.predict(X_test)
print(round(accuracy_score(y_test, preds), 2))

What is the printed output?

medium

A. 0.85

B. 0.75

C. 0.97

D. 0.60

Solution

1. Step 1: Understand the dataset and model

   Iris dataset is simple; AdaBoost with 10 estimators usually achieves accuracy around 0.85 on this split.

2. Step 2: Check typical AdaBoost accuracy on iris

   Common results show accuracy near 85% on test split with random_state=42 and 10 estimators.

3. Final Answer:

   0.85 -> Option A

4. Quick Check:

   Typical AdaBoost iris accuracy = 0.85 [OK]

Hint: AdaBoost on iris usually scores ~0.85 accuracy [OK]

Common Mistakes:

• Assuming low accuracy for simple dataset
• Confusing accuracy with training score
• Ignoring random_state effect

4. The following code tries to train an AdaBoost model but raises an error:

from sklearn.ensemble import AdaBoostClassifier
model = AdaBoostClassifier(n_estimators='ten')
model.fit(X_train, y_train)

What is the cause of the error?

medium

A. Model cannot be trained without specifying 'learning_rate'

B. Missing import for 'X_train' and 'y_train'

C. 'n_estimators' must be an integer, not a string

D. AdaBoostClassifier does not have 'n_estimators' parameter

Solution

1. Step 1: Check parameter types

   n_estimators expects an integer number of weak learners, not a string.

2. Step 2: Identify error cause

   Passing 'ten' as string causes a type error; other options are incorrect because imports or learning_rate are not mandatory.

3. Final Answer:

   'n_estimators' must be an integer, not a string -> Option C

4. Quick Check:

   n_estimators type error = 'n_estimators' must be an integer, not a string [OK]

Hint: n_estimators must be int, not string [OK]

Common Mistakes:

• Thinking learning_rate is required
• Ignoring parameter type requirements
• Assuming missing imports cause this error

5. You want to improve a weak decision tree model using boosting. Which approach best fits this goal?

hard

A. Increase the depth of a single decision tree

B. Use Gradient Boosting to sequentially correct errors of weak trees

C. Use random forests to average many deep trees

D. Apply PCA to reduce features before training the tree

Solution

1. Step 1: Understand boosting application

   Boosting improves weak models by sequentially correcting their errors.

2. Step 2: Match approach to boosting

   Gradient Boosting fits this by building trees one after another to fix mistakes.

3. Final Answer:

   Use Gradient Boosting to sequentially correct errors of weak trees -> Option B

4. Quick Check:

   Boosting = sequential error correction [OK]

Hint: Boosting fixes errors step-by-step [OK]

Common Mistakes:

• Confusing boosting with random forests
• Trying to fix with one big tree
• Using PCA unrelated to boosting