Stacking and blending in ML Python - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Difference between stacking and blending

Which statement correctly describes the main difference between stacking and blending in ensemble learning?

A Stacking combines models by averaging predictions, blending uses majority voting.

B Blending requires no separate data for meta-model training, stacking does.

C Blending trains base models sequentially, stacking trains them in parallel.

D Stacking uses cross-validation to train the meta-model, while blending uses a holdout validation set.

💡 Hint

Think about how the meta-model is trained in each method.

Stacking typically uses cross-validation to generate predictions for training the meta-model, ensuring no data leakage. Blending uses a separate holdout set for this purpose.

❓ Predict Output

intermediate

Output of stacking predictions code

What is the output of the following Python code snippet that performs stacking predictions?

ML Python

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=100, n_features=4, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Base models
model1 = LogisticRegression(random_state=42)
model2 = DecisionTreeClassifier(random_state=42)

# Train base models
model1.fit(X_train, y_train)
model2.fit(X_train, y_train)

# Generate base predictions for test set
preds1 = model1.predict_proba(X_test)[:, 1]
preds2 = model2.predict_proba(X_test)[:, 1]

# Stack predictions as features
stacked_features = np.column_stack((preds1, preds2))

# Meta-model
meta_model = LogisticRegression(random_state=42)
meta_model.fit(stacked_features, y_test)

# Final predictions
final_preds = meta_model.predict(stacked_features)

print(sum(final_preds))

A 18

B 20

C 12

D 16

💡 Hint

Count how many final predictions are positive (1) in the test set.

The meta-model is trained on the stacked base model predictions for the test set labels. The sum of final_preds counts how many samples are predicted as class 1.

❓ Hyperparameter

advanced

Choosing meta-model for stacking

Which meta-model choice is generally best when stacking base models with diverse prediction scales and distributions?

A A linear regression model without regularization

B A logistic regression model with L2 regularization

C A decision tree with max_depth=1

D A k-nearest neighbors model with k=1

💡 Hint

Consider a model that handles different scales and avoids overfitting.

Logistic regression with L2 regularization balances fitting and generalization, handling diverse base model outputs well. Linear regression is for continuous targets, decision trees with depth=1 are too simple, and kNN with k=1 overfits easily.

❓ Metrics

advanced

Evaluating blending ensemble performance

You trained a blending ensemble with three base classifiers and a meta-model on a holdout set. Which metric best reflects the meta-model's ability to improve over base models?

A Training loss of base models

B Mean squared error of base models on training data

C F1-score of the meta-model on the holdout set

D Accuracy of the meta-model on the holdout set

💡 Hint

Think about a metric that balances precision and recall for classification.

F1-score balances precision and recall, showing how well the meta-model improves classification performance on the holdout set. Accuracy alone may be misleading if classes are imbalanced.

🔧 Debug

expert

Debugging stacking code with data leakage

Consider this stacking code snippet. What is the main issue causing data leakage?

from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=200, n_features=5, random_state=0)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)

base_model = LogisticRegression(random_state=0)
base_model.fit(X_train, y_train)

# Using base model predictions on training data to train meta-model
train_preds = base_model.predict_proba(X_train)[:, 1]
meta_model = LogisticRegression(random_state=0)
meta_model.fit(train_preds.reshape(-1, 1), y_train)

# Predict on test data
test_preds = base_model.predict_proba(X_test)[:, 1]
final_preds = meta_model.predict(test_preds.reshape(-1, 1))

print(sum(final_preds))

A The meta-model is trained on base model predictions from the same training data, causing overfitting.

B The base model is not trained before generating predictions.

C The test data is used to train the meta-model.

D The base model predictions are not reshaped correctly before meta-model training.

💡 Hint

Think about how the meta-model training data is generated.

Training the meta-model on base model predictions from the same training data causes data leakage and overfitting. Proper stacking uses out-of-fold predictions or separate data for meta-model training.