# Ensemble Learning ## Overview - **Ensemble Methods in Machine Learning** - Reviews voting, bagging, boosting, and Bayesian averaging; explains how statistical, computational, and representational differences between learners can make their combination useful. [Dietterich 2000]( "Ensemble Methods in Machine Learning; Multiple Classifier Systems"). ## Bagging and randomized trees - **Bagging Predictors** - Fits predictors on bootstrap samples and aggregates their outputs. Particularly useful for unstable learners whose predictions change substantially when the training data change. [Breiman 1996]( "Bagging predictors; Mach. Learn."). - **Random Subspaces** - Builds decision trees on randomly selected feature subsets and combines their predictions. Feature sampling occurs per tree; Breiman's random forest also samples candidate features at individual splits. [Tin Kam Ho 1998]( "The random subspace method for constructing decision forests; TPAMI"). - **Random Forests** - Combines randomized trees; the widely used construction couples bootstrap sampling with random feature subsets at each split. Individual tree strength and error correlation both affect ensemble performance. [Breiman 2001]( "Random Forests; Mach. Learn."). - **Bias in Random Forest Variable Importance** - Shows selection and importance biases with mixed predictor types or category counts; investigates conditional inference trees and subsampling as remedies. [Strobl et al. 2007]( "Bias in random forest variable importance measures: Illustrations, sources and a solution; BMC Bioinformatics"). - **Variable Selection Using Random Forests** - Uses importance rankings and nested models to select variables; distinguishes identifying relevant variables from choosing a smaller set for prediction. [Genuer et al. 2010]( "Variable selection using random forests; Pattern Recognit. Lett."). - **Random Forest Methodology and Practical Guidance** - Reviews tuning, implementations, applications, and pitfalls in bioinformatics, including biases in variable-importance measures. [Boulesteix et al. 2012]( "Overview of random forest methodology and practical guidance with emphasis on computational biology and bioinformatics; WIREs Data Min & Knowl"). - **Extremely Randomized Trees** - Randomizes candidate features and split thresholds, then selects among those candidates. The original Extra-Trees construction uses the full training sample rather than bootstrap replicates. [Geurts et al. 2006]( "Extremely randomized trees; Mach. Learn."). ## Boosting foundations - **The Strength of Weak Learnability** - Proves equivalence of weak and strong learnability in the distribution-free PAC model. The result assumes a weak learner for the required distributions, not merely one classifier above chance on one dataset. [Schapire 1990]( "The strength of weak learnability; Mach. Learn."). - **AdaBoost** - Sequentially reweights training examples and combines weak classifiers with error-dependent weights. Misclassified examples gain weight, directing later learners toward current errors. [Freund and Schapire 1997]( "A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting; JCSS"). - **Arcing the Edge** - Formulates adaptive reweighting as optimization of functions of the edge; develops an alternative perspective on the behavior of voting classifiers. [Breiman 1997]( "Arcing the Edge; technical report"). - **Boosting the Margin** - Relates voting-classifier generalization bounds to the margin distribution, helping explain continued improvement after training error reaches zero. [Schapire et al. 1998]( "Boosting the margin: a new explanation for the effectiveness of voting methods; Ann. Statist."). - **A Short Introduction to Boosting** - Compact overview of AdaBoost, training-error bounds, margins, and connections with support-vector machines. [Freund and Schapire 1999]( "A Short Introduction to Boosting; J. Japanese Soc. AI"). ## Gradient boosting - **Boosting Algorithms as Gradient Descent** - Treats boosting as functional gradient descent on prediction or margin losses; introduces the DOOM II algorithm. [Mason et al. 1999]( "Boosting Algorithms as Gradient Descent; NeurIPS"). - **Gradient Boosting Machine** - Builds an additive model by fitting base learners to negative loss gradients. The loss determines the target: residuals for squared error, with other objectives supporting classification and robust regression. [Friedman 2001]( "Greedy function approximation: A gradient boosting machine.; Ann. Statist."). - **Stochastic Gradient Boosting** - Fits each boosting update on a random subsample drawn without replacement. Adds row sampling to gradient boosting; distinct from stochastic gradient descent on model parameters. [Friedman 2002]( "Stochastic gradient boosting; Comput. Stat. Data Anal."). - **XGBoost** - Combines regularized tree objectives, first- and second-order loss information, sparsity-aware split finding, and weighted quantile sketches with systems optimizations for large datasets. [Chen and Guestrin 2016]( "XGBoost: A Scalable Tree Boosting System; KDD"). - **Out-of-Core GPU Gradient Boosting** - Combines external-memory data access with gradient-based sampling in XGBoost to train on data exceeding GPU memory capacity while limiting the working set. [Ou 2020]( "Out-of-Core GPU Gradient Boosting; arXiv"). - **LightGBM** - Introduces gradient-based one-side sampling, retaining large-gradient rows while sampling others, and exclusive feature bundling, combining sparse features that rarely take nonzero values together. [Ke et al. 2017]( "LightGBM: A Highly Efficient Gradient Boosting Decision Tree; NeurIPS"). - **CatBoost** - Uses ordered boosting and permutation-based target statistics for categorical features to reduce prediction shift caused by reusing training targets. [Prokhorenkova et al. 2018]( "CatBoost: unbiased boosting with categorical features; NeurIPS"). ## Learning how to combine predictors - **Voting and Averaging** - Hard voting combines class labels; soft voting averages predicted probabilities. Regression ensembles commonly average numerical predictions. Weights can be fixed or estimated on separate validation data. [docs](). - **Stacked Generalization** - Trains a second-level model on base-model predictions for observations excluded from their fitting data. Learns how to combine predictors with different errors and biases. [Wolpert 1992]( "Stacked generalization; Neural Networks"). - **Ensemble Selection** - Greedily selects models from a fitted library to optimize ensemble performance on validation data. Selection with replacement permits unequal model weights. [Caruana et al. 2004]( "Ensemble selection from libraries of models; ICML"). - **Super Learner** - Uses cross-validated risk to learn a combination of candidate prediction algorithms; theoretical guarantees depend on the loss and regularity conditions. [van der Laan et al. 2007]( "Super Learner; Stat. Appl. Genet. Mol. Biol."). - **Stacking predictive distributions** - Chooses mixture weights using held-out predictive scores, often via leave-one-out inference. Yao et al. target prediction when all candidate models may be wrong; weights are not posterior probabilities that models are true. [Yao et al. 2018]( "Using stacking to average Bayesian predictive distributions; Bayesian Anal."). ## Distributional prediction and statistical estimation - **Quantile Regression Forests** - Uses forest-derived observation weights to estimate the conditional response distribution and its quantiles, rather than only the conditional mean. [Meinshausen 2006]( "Quantile Regression Forests; JMLR"). - **Generalized Random Forests** - Learns adaptive neighborhoods for local moment equations, supporting quantiles and heterogeneous effects. Its inference guarantees require the paper's sampling and regularity assumptions. [Athey et al. 2019]( "Generalized Random Forests; The Annals of Statistics"). - **SWAG** - Fits a Gaussian approximation to SGD iterates using their mean and a low-rank-plus-diagonal covariance; sampled weights yield averaged predictions and approximate Bayesian uncertainty. [Maddox et al. 2019]( "A Simple Baseline for Bayesian Uncertainty in Deep Learning; NeurIPS"). - **NGBoost** - Boosts parameters of a predictive distribution using natural gradients and a proper scoring rule. Produces distributional predictions instead of interpreting variation among ordinary trees as an interval. [Duan et al. 2020]( "NGBoost: Natural Gradient Boosting for Probabilistic Prediction; ICML"). - **Jackknife+** - Combines leave-one-out residuals with test-point predictions to form intervals. Under exchangeability and symmetric fitting, the general coverage bound is 1−2α for nominal 1−α; near-nominal results need further stability assumptions. [Barber et al. 2021]( "Predictive inference with the jackknife+; The Annals of Statistics"). ## Neural ensembles - **Deep Ensembles** - Averages predictive distributions from separately initialized and trained networks. The method uses proper scoring rules and studies predictive uncertainty in classification and regression. [Lakshminarayanan et al. 2017]( "Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles; NeurIPS"). - **Snapshot Ensembles** - Uses cyclic learning rates to collect several model checkpoints along one training trajectory, then averages their predictions. Multiple checkpoints still require evaluation at inference time. [Huang et al. 2017]( "Snapshot Ensembles: Train 1, get M for free; ICLR"). - **Fast Geometric Ensembling** - Uses short learning-rate cycles late in training to collect nearby predictors for an ensemble, motivated by low-loss paths connecting neural-network solutions. [Garipov et al. 2018]( "Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs; NeurIPS"). - **BatchEnsemble** - Shares weight matrices across ensemble members and modulates them with member-specific rank-one factors. Reduces parameter storage while retaining separate predictions; members are not independently parameterized networks. [Wen et al. 2020]( "BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning; ICLR"). - **TabM** - Adapts BatchEnsemble to tabular multilayer perceptrons, sharing most weights across predictors and averaging their outputs. [Gorishniy et al. 2025]( "TabM: Advancing tabular deep learning with parameter-efficient ensembling; ICLR"); [code](). ## Model merging and compression - **Knowledge Distillation** - Trains a student to match temperature-scaled teacher probabilities, including those from an ensemble. Retains information about relative class probabilities while reducing the models needed at inference. [Hinton et al. 2014]( "Distilling the Knowledge in a Neural Network; NeurIPS DL Workshop"). - **Model Soups** - Averages weights of compatible models fine-tuned from a shared pretrained initialization. Produces one model for inference; weight averaging is a different operation from averaging predictions. [Wortsman et al. 2022]( "Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time; ICML"). - **TIES-Merging** - Merges fine-tuned models by trimming small parameter changes, resolving conflicting update signs and combining sign-aligned changes; targets interference between task-specific updates. [Yadav et al. 2023]( "TIES-Merging: Resolving Interference When Merging Models; NeurIPS"). ## Evaluation - **Out-of-Bag Evaluation** - Estimates performance using predictions from bootstrap members that excluded each observation. Reusing these scores for extensive model selection still calls for a separate final evaluation. [Breiman 2001]( "Random Forests; Mach. Learn."); [docs](). - **Uncertainty Under Dataset Shift** - Compares uncertainty methods, including ensembles, across shifted classification datasets. Calibration on the original distribution does not establish calibration after a shift. [Ovadia et al. 2019]( "Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift; NeurIPS"). - **Stacking Leakage** - Train the combiner on held-out or out-of-fold base predictions. The documented `prefit` mode risks overfitting when base models and the combiner use the same training observations. [docs](). ## Software - **scikit-learn** - Bagging, random forests, Extra-Trees, AdaBoost, gradient boosting, voting, and stacking with estimator APIs. Its guide distinguishes histogram-based and conventional gradient boosting. [docs](). - **XGBoost** - Tree boosting for classification, regression, and ranking, with CPU/GPU and external-memory workflows. [code](), [docs](). - **LightGBM** - Histogram-based tree boosting with leaf-wise growth, categorical splits, and distributed training. Algorithm and device settings determine which optimizations are used. [docs](). - **CatBoost** - Gradient boosting with categorical-feature processing, CPU/GPU training, and prediction tools. [docs](), [docs](). - **ranger** - C++ random forests with an R interface, including classification, regression, probability, and survival forests. [code](). - **randomForest** - R classification and regression forests. Classification and Regression by randomForest explains fitting, out-of-bag estimates, and importance measures. [Liaw and Wiener 2002]( "Classification and Regression by randomForest; R News"); [code](). - **grf** - R/C++ generalized forests for regression, quantiles, causal effects, and other statistical targets. [code](), [code](). - **SuperLearner** - R implementation of cross-validated ensemble prediction, with wrappers for candidate learners and methods for estimating their combination. [code](). ## Related topics - [Models](https://mlpapers.org/models/) - [Neural Networks](https://mlpapers.org/neural-nets/) - [Feature Selection](https://mlpapers.org/feature-selection/) - [Optimization](https://mlpapers.org/optimization/)