Bagging vs Boosting vs Stacking in Python: Key Differences & Code
bagging builds multiple independent models on random data samples and averages their results, boosting builds models sequentially to correct previous errors, and stacking combines different model predictions using a meta-model for better accuracy. All three are ensemble methods but differ in how they train and combine base models.Quick Comparison
Here is a quick table comparing bagging, boosting, and stacking on key factors.
| Factor | Bagging | Boosting | Stacking |
|---|---|---|---|
| Training Style | Parallel, independent models | Sequential, models learn from errors | Parallel base models + meta-model |
| Goal | Reduce variance | Reduce bias and variance | Combine strengths of different models |
| Model Dependency | Models are independent | Models depend on previous ones | Base models independent, meta-model dependent |
| Common Algorithms | Random Forest | AdaBoost, Gradient Boosting | Any classifiers/regressors |
| Risk of Overfitting | Lower risk | Higher risk if too many rounds | Depends on base and meta models |
| Use Case | When data is noisy | When model needs improvement on errors | When combining diverse models |
Key Differences
Bagging (Bootstrap Aggregating) trains multiple copies of the same model on different random subsets of data. Each model votes or averages predictions, which reduces variance and helps with noisy data. Models are trained independently and in parallel.
Boosting trains models one after another. Each new model focuses on the mistakes of the previous ones, improving overall accuracy by reducing bias and variance. This sequential dependency can lead to better performance but risks overfitting if not controlled.
Stacking combines predictions from multiple different models (base learners) by training a new model (meta-learner) on their outputs. This approach leverages the strengths of diverse models and can improve accuracy beyond bagging or boosting alone. It requires careful training to avoid data leakage.
Code Comparison: Bagging Example
This example shows how to use BaggingClassifier with a decision tree on the Iris dataset.
from sklearn.datasets import load_iris from sklearn.model_selection import train_test_split from sklearn.tree import DecisionTreeClassifier from sklearn.ensemble import BaggingClassifier from sklearn.metrics import accuracy_score # Load data iris = load_iris() X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target, random_state=42) # Base model base_clf = DecisionTreeClassifier(random_state=42) # Bagging ensemble bagging_clf = BaggingClassifier(base_estimator=base_clf, n_estimators=10, random_state=42) bagging_clf.fit(X_train, y_train) # Predict and evaluate preds = bagging_clf.predict(X_test) accuracy = accuracy_score(y_test, preds) print(f"Bagging accuracy: {accuracy:.3f}")
Boosting Equivalent
This example uses AdaBoostClassifier with a decision tree on the same Iris dataset.
from sklearn.ensemble import AdaBoostClassifier # Base model base_clf = DecisionTreeClassifier(max_depth=1, random_state=42) # Boosting ensemble boosting_clf = AdaBoostClassifier(estimator=base_clf, n_estimators=50, random_state=42) boosting_clf.fit(X_train, y_train) # Predict and evaluate preds = boosting_clf.predict(X_test) accuracy = accuracy_score(y_test, preds) print(f"Boosting accuracy: {accuracy:.3f}")
When to Use Which
Choose bagging when your main goal is to reduce variance and handle noisy data, especially with unstable models like decision trees.
Choose boosting when you want to reduce bias and improve model accuracy by focusing on hard-to-predict samples, but watch out for overfitting.
Choose stacking when you want to combine different types of models to leverage their strengths and achieve better overall performance, especially in complex problems.