Ensemble Learning

Introduction

An ensemble model combines several predictive models to produce one final prediction. The central idea is that a group of moderately accurate, diverse learners can outperform a single learner. Ensemble learning is especially important in data science because real-world datasets often contain noise, nonlinear relationships, missing information, and interacting variables. No individual algorithm is best for every dataset, but an ensemble can reduce the weaknesses of its components.

Ensemble methods are widely used for credit-risk assessment, fraud detection, customer churn prediction, medical classification, demand forecasting, recommendation systems, and image recognition. This chapter focuses on five major approaches: bagging, random forests, boosting, voting, and stacking. It also compares predictive performance, computational cost, and interpretability.

Learning Objectives

By the end of this chapter, a learner should be able to:

• explain why ensembles can improve generalisation;
• distinguish bagging from boosting;
• apply the logic of random forests and gradient-boosted trees;
• use voting and stacking for classification and regression;
• compare ensembles with individual models using suitable evaluation measures;
• assess computational cost, interpretability, overfitting risk, and operational constraints;
• design a reliable ensemble-learning workflow.

1. Foundations of Ensemble Learning

1.1 The basic idea

Suppose a dataset contains input variables X and a target variable y. An individual learner produces a prediction f_j(x). An ensemble combines predictions from M learners, where M is the number of component models.

For regression, a simple average is:

Here, ŷ(x) is the ensemble prediction and f_j(x) is the prediction of model j. The formula assumes that all component predictions are on the same numerical scale and that equal weighting is appropriate.

For classification, each model may vote for a class. The predicted class is the class receiving the greatest number of votes:

The symbol I(·) equals 1 when its condition is true and 0 otherwise; k represents a possible class. This hard-voting rule is suitable when class labels, rather than calibrated probabilities, are available.

1.2 Why ensembles work

Three ideas explain much of the benefit:

1. Error reduction: averaging can reduce random errors.
2. Variance reduction: models that react differently to the training sample can be combined to make a more stable predictor.
3. Bias correction: sequential methods can focus on errors made by earlier models and reduce systematic underfitting.

Diversity is essential. If every model makes exactly the same error, combining them provides little benefit. Diversity can be created by changing the training sample, the input variables, the algorithm, the hyperparameters, or the representation of the data.

For an idealised regression ensemble with unbiased component errors, equal error variance σ², and pairwise error correlation ρ, the variance of the average is:

This relationship assumes equal variances and equal pairwise correlations. If ρ is low, increasing M substantially reduces variance. If ρ is close to 1, the models behave similarly and the improvement is small. The equation does not imply that adding unlimited models always improves performance; computational cost, bias, noise, and overfitting still matter.

2. Bagging

2.1 Definition and procedure

Bagging, short for bootstrap aggregating, trains multiple models independently on different bootstrap samples of the training data. A bootstrap sample is created by sampling observations with replacement until the sample has the same size as the original training set. Some records appear more than once, while others are omitted.

A bagging procedure is:

1. Draw M bootstrap samples from the training data.
2. Train one base learner on each sample.
3. Combine predictions by averaging for regression or majority voting for classification.
4. Evaluate on a validation set or on observations not used in a particular bootstrap sample.

Bagging is particularly effective for high-variance learners, such as fully grown decision trees. A single deep tree may change greatly when the training data changes slightly. Averaging many such trees makes the final prediction more stable.

2.2 Out-of-bag estimation

Because a bootstrap sample omits some original observations, each observation can be predicted by models whose bootstrap samples did not contain it. These are called out-of-bag (OOB) predictions. The proportion of observations omitted from one bootstrap sample approaches:

Here, n is the number of training observations and e is the base of natural logarithms. Thus, approximately 36.8% of observations are out-of-bag for a particular bootstrap sample when n is sufficiently large. OOB predictions provide an internal validation estimate, although a separate test set remains valuable for final unbiased assessment.

2.3 Strengths and limitations

Bagging usually reduces variance without greatly increasing bias. It is relatively easy to parallelise because component models are trained independently. However, bagging may be computationally expensive when many large trees are trained. It also provides less direct interpretability than a single decision tree, and it may not improve a model that is already low variance.

3. Random Forests

3.1 From bagging to random forests

A random forest is an ensemble of decision trees that introduces randomness in two ways:

• each tree is trained on a bootstrap sample or a related subsample of observations;
• at each split, the tree considers only a random subset of input features.

The second mechanism reduces correlation between trees. Without feature subsampling, strong predictors may repeatedly create the same first split, causing trees to behave similarly. Lower correlation increases the value of averaging.

For classification, the forest commonly uses majority voting. For regression, it commonly averages tree outputs. Important hyperparameters include:

• number of trees;
• maximum tree depth;
• minimum observations required for a split or leaf;
• number of features considered at each split;
• bootstrap or subsampling strategy;
• class weights for imbalanced classification.

Increasing the number of trees usually stabilises predictions but increases training time and memory use. Beyond a point, the accuracy gain is small.

3.2 Interpretation of random forests

A random forest is less transparent than one tree, but several tools improve interpretation:

• Permutation importance: randomly shuffle one feature and measure the resulting performance decline. A larger decline indicates greater predictive usefulness, although correlated features can distort the result.
• Impurity-based importance: aggregate split-quality improvements attributed to a feature. This can favour continuous variables or variables with many possible split points.
• Partial dependence: estimate the average prediction as one feature changes while other features are averaged over the dataset. It can hide interactions and should be interpreted cautiously.
• Individual conditional expectation: display prediction curves for individual observations, revealing heterogeneous effects.
• Local explanation methods: approximate a model’s behaviour near one observation, but explanations are approximations rather than causal proof.

Feature importance is not the same as causal importance. A feature can be predictive because it is a proxy for another variable, because of data leakage, or because of a historical process.

4. Boosting

4.1 General principle

Boosting builds an ensemble sequentially. Each new learner attempts to improve the current ensemble, often by concentrating on observations that previous learners predicted poorly. In contrast, bagging trains component models independently.

A simplified additive model is:

Here, F_M(x) is the final model, F₀(x) is the initial prediction, h_m(x) is the mth weak learner, and η is the learning rate. The learning rate is positive and usually less than or equal to 1. A smaller learning rate often requires more learners but can improve generalisation when properly regularised.

4.2 AdaBoost

AdaBoost assigns larger weights to observations misclassified by earlier learners. The next learner therefore focuses more strongly on difficult cases. For binary classification with labels coded as −1 and +1, a common learner-weight expression is:

where ε_m is the weighted error rate of learner m and ln is the natural logarithm. This expression assumes 0 < ε_m < 0.5; a learner with error below random guessing receives a positive weight. AdaBoost can be sensitive to noisy labels and extreme outliers because it may repeatedly emphasise them.

4.3 Gradient boosting

Gradient boosting fits each new learner to the negative gradient of a chosen loss function. For squared-error regression, the negative gradient is the residual:

where y_i is the observed target for observation i and F_{m−1}(x_i) is the current prediction. The next tree approximates these residuals. For other losses, such as logistic loss, the training target is a pseudo-residual derived from the loss gradient rather than simply y_i minus the prediction.

A general optimisation view is:

where h_m approximates the direction that reduces the loss. Gradient boosting can model nonlinear relationships and interactions, but it requires careful control of depth, learning rate, number of trees, subsampling, and regularisation.

5. Gradient-Boosted Trees in Practice

Gradient-boosted trees are among the strongest general-purpose methods for structured, tabular data. Implementations such as XGBoost, LightGBM, and CatBoost add engineering improvements, including efficient split finding, regularisation, handling of categorical variables, and parallel computation.

Common controls include:

A practical tuning strategy is to choose a validation scheme, set a relatively small learning rate, use early stopping to select the effective number of trees, and then refine depth and regularisation. Random search or Bayesian optimisation can be more efficient than exhaustive grid search, but the validation design must remain sound.

Boosting often achieves excellent accuracy, but it can overfit noisy data, especially with deep trees, too many iterations, or weak validation discipline. Training is sequential, so it is less naturally parallel than bagging, although modern implementations parallelise parts of split computation.

6. Voting Ensembles

Voting combines predictions from different model types, such as logistic regression, a decision tree, and a nearest-neighbour model.

6.1 Hard voting

Hard voting selects the class with the greatest number of predicted labels. It is simple and does not require calibrated probabilities. However, it treats a highly confident and a barely confident prediction as equivalent.

6.2 Soft voting

Soft voting averages predicted class probabilities:

Here, p_{jk}(x) is model j’s probability for class k and w_j is its non-negative weight. The final class is the class with the greatest combined probability. Soft voting is useful when probabilities are reasonably calibrated and component models make complementary errors. Poorly calibrated probabilities can make soft voting misleading; calibration methods such as Platt scaling or isotonic regression may help, but calibration must be fitted without contaminating the test set.

7. Stacking

Stacking, or stacked generalisation, trains a meta-model to combine predictions from base models. For example, a random forest, logistic regression model, and gradient-boosted tree may generate predictions that become inputs to a logistic regression meta-model.

The critical requirement is that meta-model training uses out-of-fold predictions. If base predictions are generated on the same observations used to fit the base models, they may be unrealistically accurate. The meta-model then learns from leakage and performs poorly on new data.

A sound procedure is:

1. Divide the training data into K folds.
2. For each fold, train each base model on the other K−1 folds.
3. Predict the held-out fold and store these predictions.
4. Combine all held-out predictions to form a leakage-resistant meta-training set.
5. Train the meta-model on these predictions and the target.
6. Retrain each base model on all available training data.
7. For new data, obtain predictions from all retrained base models and pass them to the meta-model.

The number of folds K affects computation and variance. Stratified folds are generally appropriate for classification, while time-ordered splits are required when future observations must not influence past predictions.

8. Comparing Ensemble Methods with Individual Models

Accuracy should not be the only selection criterion. A model is useful when its predictive benefit justifies its cost, complexity, and risk.

Useful evaluation measures include accuracy, balanced accuracy, precision, recall, F1 score, ROC-AUC, PR-AUC, log loss, and calibration error for classification. For regression, common measures include mean absolute error (MAE), root mean squared error (RMSE), and R².

For n observations, actual values y_i, and predictions ŷ_i:

MAE has the same units as the target and gives linear penalty to errors. RMSE also has the target’s units but penalises large errors more strongly. These metrics assume numerical predictions and should be interpreted in the context of the decision problem.

Use cross-validation or a carefully designed validation set for model selection, and reserve a final test set for one-time performance estimation. In time-dependent settings, use temporal validation rather than random shuffling. Compare models using the same data splits and report uncertainty where possible.

9. Applied Case: Predicting Customer Churn

Background

The following is an illustrative case. A telecommunications provider wants to identify customers likely to cancel service within the next month. The dataset includes contract type, monthly charge, tenure, service usage, complaint history, and customer-support contacts.

Problem

The company wants high recall because failing to identify a likely churner is costly. However, retention campaigns are expensive, so precision and probability calibration also matter. Data leakage is a major risk: a post-cancellation administrative code must not be used as a predictor.

Analysis and approach

The data science team first defines the prediction time and removes variables unavailable at that time. It creates a stratified training-validation-test split. A regularised logistic regression model provides a transparent baseline. A single decision tree is added to capture nonlinear rules.

A random forest is then trained using class weights and permutation importance. A gradient-boosted tree is tuned with shallow trees, a small learning rate, and early stopping. Finally, a soft-voting model combines the logistic, forest, and boosted-tree probabilities after calibration is assessed.

Performance is evaluated using recall, precision, PR-AUC, log loss, and calibration plots. The team does not select the model solely by PR-AUC: a model with slightly lower PR-AUC but better calibration may support more reliable campaign budgeting.

Outcome

Suppose the boosted model provides the best validation discrimination, while the voting model provides better calibration and nearly equal recall. The company selects the voting model because campaign decisions use predicted probabilities. The model is monitored monthly for drift in customer behaviour and quarterly for subgroup performance.

Learning points

The case illustrates that ensemble selection depends on the business loss, not merely a leaderboard score. It also demonstrates the importance of leakage prevention, probability calibration, transparent baselines, consistent validation, and post-deployment monitoring.

10. Challenges, Governance, and Best Practices

Overfitting and leakage

Ensembles can overfit through excessive depth, excessive iterations, repeated tuning against the test set, or leakage in feature construction. Use pipelines, nested validation when appropriate, early stopping, regularisation, and an untouched test set.

Imbalanced data

Accuracy can be misleading when the positive class is rare. Consider class weights, threshold adjustment, stratified evaluation, PR-AUC, recall, precision, and cost-sensitive learning. Resampling must occur inside the training folds rather than before cross-validation.

Missing and categorical data

Preprocessing must be fitted only on training data. Some boosted-tree implementations handle missing values and categorical variables directly, but their behaviour should be documented and tested. Consistent feature definitions are essential in production.

Interpretability and accountability

A complex ensemble may be unsuitable where decisions require clear explanations, such as lending, employment, insurance, or clinical support. Explanations should describe model associations and local reasoning, not claim causality. Organisations should document data provenance, feature definitions, model versions, validation results, limitations, and human review procedures.

Fairness and privacy

Compare error rates, calibration, selection rates, and other relevant measures across protected or vulnerable groups, while recognising that fairness criteria can conflict. Avoid unnecessary personal data, protect training records, control access, and assess whether proxy variables reproduce discriminatory patterns. Legal requirements vary by jurisdiction, so technical evaluation must be combined with organisational and legal review.

Production deployment

Record the exact preprocessing and model pipeline. Monitor input drift, prediction drift, outcome performance, latency, resource consumption, and subgroup behaviour. Establish retraining triggers and rollback procedures. Reproducibility requires fixed data snapshots or documented extraction logic, versioned code, and recorded hyperparameters.

11. Contemporary Developments

Modern ensemble systems increasingly combine heterogeneous learners, automated hyperparameter optimisation, calibrated prediction intervals, and distributed training. Explainability tools are becoming standard components of model governance. For large-scale tabular problems, highly optimised gradient-boosting libraries remain competitive, while hybrid systems may combine neural representations with tree ensembles. These developments do not remove the need for sound problem definition, leakage control, validation, fairness assessment, and human oversight.

Chapter Summary

Ensemble learning combines multiple predictive models to improve generalisation. Bagging trains learners independently on bootstrap samples and mainly reduces variance. Random forests add random feature selection to reduce correlation among trees. Boosting builds learners sequentially to correct earlier errors; AdaBoost reweights difficult observations, while gradient boosting follows the loss gradient and is especially effective with decision trees. Voting combines predictions directly, whereas stacking uses a meta-model and requires out-of-fold predictions to avoid leakage.

Ensembles often improve predictive performance, but they require additional computation and are generally less interpretable than individual models. Model selection should consider the decision objective, validation design, calibration, fairness, operational cost, and monitoring requirements. A reliable ensemble workflow uses a transparent baseline, leakage-resistant preprocessing, appropriate metrics, careful tuning, independent testing, documentation, and post-deployment governance.

Key Terms

Ensemble learning: Combining multiple models to form a single predictor.

Bagging: Bootstrap aggregating through independent models trained on resampled data.

Bootstrap sample: A sample drawn with replacement from a dataset.

Out-of-bag prediction: A prediction made by models whose bootstrap samples omitted the observation.

Random forest: A bagged tree ensemble with random feature selection at splits.

Boosting: Sequential construction of learners that improve an existing ensemble.

AdaBoost: A boosting method that increases emphasis on previously misclassified observations.

Gradient boosting: Boosting that fits learners to directions suggested by the loss gradient.

Learning rate: A parameter controlling each learner’s contribution.

Weak learner: A model that performs better than a simple baseline but may be insufficient alone.

Hard voting: Combining predicted class labels by majority vote.

Soft voting: Combining predicted class probabilities.

Stacking: Training a meta-model on predictions from base models.

Out-of-fold prediction: A prediction for a fold produced by a model that did not train on that fold.

Variance: Sensitivity of a model to changes in the training sample.

Bias: Systematic error caused by restrictive modelling assumptions.

Calibration: Agreement between predicted probabilities and observed frequencies.

Permutation importance: Performance change after randomly permuting a feature.

Data leakage: Unintended use of information unavailable at prediction time.

Concept drift: Change in the relationship between inputs and the target over time.

Review Questions

Short Answer Questions

1. What is the main purpose of combining models in an ensemble?
2. How does bagging differ from boosting?
3. Why does a random forest select only a subset of features at each split?
4. Distinguish hard voting from soft voting.
5. Why are out-of-fold predictions necessary in stacking?

Long Answer Questions

1. Explain the procedure, assumptions, strengths, and limitations of bagging.
2. Describe how gradient-boosted trees are trained and identify important hyperparameters.
3. Compare random forests and gradient-boosted trees in terms of accuracy, computation, and interpretability.
4. Explain how model evaluation should be designed for an imbalanced classification problem.
5. Discuss ethical, governance, and monitoring requirements for deploying an ensemble in a high-impact domain.

Analytical / Application Questions

1. A fraud dataset has 2% fraudulent transactions. Which evaluation measures and validation strategy would you use, and why?
2. A stacking model performs extremely well on validation data but poorly on the test set. Diagnose possible causes and propose corrective actions.
3. A company must choose between a slightly more accurate boosted model and a better-calibrated voting model for allocating retention offers. What additional analysis should guide the decision?

Multiple Choice Questions with Answers

Question 1

What is the primary purpose of bagging?

A. To reduce variance by averaging models trained on resampled data

B. To remove all features except the most important one

C. To train one model sequentially on residual errors

D. To guarantee causal interpretation of predictions

Correct Answer: A. To reduce variance by averaging models trained on resampled data

Question 2

Which additional source of randomness distinguishes a random forest from basic bagging of decision trees?

A. Random feature subsets considered at tree splits

B. Random target labels assigned to observations

C. Random evaluation metrics selected after testing

D. Random removal of the validation set

Correct Answer: A. Random feature subsets considered at tree splits

Question 3

In boosting, what is the usual role of the learning rate?

A. It controls the contribution of each new learner

B. It determines the number of classes in the target

C. It measures the percentage of missing values

D. It converts regression into classification

Correct Answer: A. It controls the contribution of each new learner

Question 4

For squared-error gradient boosting, what do the residuals commonly represent?

A. The observed target minus the current prediction

B. The number of features in the dataset

C. The average class probability

D. The bootstrap sampling fraction only

Correct Answer: A. The observed target minus the current prediction

Question 5

Which statement best describes soft voting?

A. It combines predicted class probabilities, often using weighted averages

B. It selects the deepest tree in the ensemble

C. It averages only the training errors

D. It uses no information from component models

Correct Answer: A. It combines predicted class probabilities, often using weighted averages

Question 6

What is the main danger of training a stacking meta-model on predictions from base models fitted on the same observations?

A. Data leakage can make the meta-model learn from overly optimistic predictions

B. The base models become impossible to train

C. The target variable becomes continuous

D. The number of classes is automatically doubled

Correct Answer: A. Data leakage can make the meta-model learn from overly optimistic predictions

Question 7

Which metric penalises large regression errors more strongly than MAE?

A. RMSE

B. Median

C. Recall

D. Accuracy

Correct Answer: A. RMSE

Question 8

Approximately what proportion of observations is omitted from one large bootstrap sample?

A. 36.8%

B. 2.0%

C. 50.0%

D. 90.0%

Correct Answer: A. 36.8%

Question 9

Why can permutation importance be misleading when features are strongly correlated?

A. Related features may substitute for one another, reducing the apparent importance of one feature

B. It always assigns identical importance to every feature

C. It changes the target labels permanently

D. It can only be used for unsupervised learning

Correct Answer: A. Related features may substitute for one another, reducing the apparent importance of one feature

Question 10

Which practice is most appropriate for a time-dependent prediction problem?

A. Use time-ordered validation that prevents future information entering training

B. Randomly shuffle all observations before every split

C. Use the test set repeatedly for hyperparameter tuning

D. Include variables recorded after the prediction time

Correct Answer: A. Use time-ordered validation that prevents future information entering training