Ensemble Learning

Introduction

Ensemble learning combines multiple predictive models to produce a model that is often more accurate, stable, and robust than any single constituent model. The central idea is not simply to create many models, but to combine models whose errors are sufficiently different. In business analytics, ensembles are widely used for credit-risk assessment, fraud detection, customer churn, demand forecasting, recommendation, medical diagnosis, and operational decision support.

A single decision tree may be easy to explain but unstable: a small change in training data can produce a substantially different tree. An ensemble can reduce this instability, improve predictive performance, or exploit complementary modelling assumptions. However, ensembles also introduce costs: increased computation, more complicated interpretation, calibration problems, data leakage risks, and governance challenges.

This chapter examines bagging, random forests, boosting, gradient boosting, voting, and stacking. It also develops the ideas of bias, variance, diversity, calibration, feature importance, computational trade-offs, and responsible deployment.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain why combining models can improve generalisation.
2. Distinguish bagging, random forests, boosting, voting, and stacking.
3. Apply ensemble methods to classification and regression problems.
4. Analyse the roles of diversity, bias, variance, and correlation.
5. Evaluate calibration, feature importance, computational cost, and overfitting.
6. Design a leakage-resistant ensemble modelling workflow.
7. Interpret ensemble outputs for managerial and regulatory decisions.

1. Foundations of Ensemble Learning

1.1 Bias, variance, and error decomposition

Suppose a learning algorithm estimates an unknown function f(x) using training data D, producing prediction f̂_D(x). For a regression problem with response y = f(x) + ε, where ε is irreducible random noise with expected value 0 and variance σ²_ε, the expected squared prediction error at x can be decomposed as:

The three terms are, respectively, irreducible noise, squared bias, and variance. The expectation E_D is over possible training samples and E_D,ε is over both training samples and noise. The decomposition applies to squared-error loss and helps explain why ensembles work; it is not a universal decomposition for every loss function.

High-bias models, such as very shallow trees, systematically underfit. High-variance models, such as fully grown trees, fit training peculiarities and may generalise poorly. Ensemble methods can reduce variance, reduce bias, or achieve both through sequential correction and model diversity.

1.2 What is diversity?

Diversity means that constituent models do not make identical errors. Diversity can arise from:

• different bootstrap samples;
• different subsets of predictors;
• different algorithms or hyperparameters;
• different feature representations;
• different random seeds; and
• sequentially emphasising previously mispredicted observations.

Accuracy alone is insufficient. Combining ten highly accurate but identical models may provide almost no benefit. Conversely, a somewhat weaker model can improve an ensemble if its errors complement those of stronger models.

For binary classifiers, a practical diversity analysis compares error indicators e_i(x), where e_i(x) = 1 if model i misclassifies x and 0 otherwise. Useful diagnostics include pairwise disagreement, correlation of prediction residuals, and the proportion of observations on which models make different decisions. No single diversity measure is universally best; diversity must be considered alongside individual model quality.

1.3 Averaging and the role of correlation

Assume m unbiased regression estimators, each with variance σ² and equal pairwise error correlation ρ. Their simple average has variance:

Here m is the number of models, σ² is the variance of each estimator, and ρ is the pairwise correlation of their errors. The formula assumes equal variances and equal pairwise correlations. When ρ = 1, averaging provides no variance reduction. When ρ = 0, variance decreases to σ²/m. As m becomes large, the variance approaches σ²ρ rather than necessarily reaching zero. This explains why randomisation and diversity are important in bagging and random forests.

2. Bagging

2.1 Principle

Bagging, or bootstrap aggregating, trains the same base algorithm on multiple bootstrap samples and combines the predictions. A bootstrap sample of size n is formed by drawing n observations with replacement from the original training set of size n. Approximately 1 − e⁻¹ ≈ 63.2% of distinct observations appear in a bootstrap sample on average, leaving approximately 36.8% out-of-bag observations.

For regression, the bagged prediction is usually the mean:

For classification, it may be the majority vote:

B is the number of base models, f̂_b is the prediction of model b, and h_b is its class prediction. Averaging class probabilities is often preferable when probability quality matters.

2.2 Why bagging works

Bagging is particularly effective for unstable learners, especially decision trees. Each tree sees a different sample and therefore responds differently to sampling variation. Aggregation reduces variance while preserving much of the base learner’s flexibility. Bagging usually has less effect on stable, high-bias learners.

Bagging does not automatically reduce bias. If every base learner systematically ignores an important nonlinear relationship, averaging them will not correct that omission. It can also be ineffective when base-model errors are strongly correlated.

2.3 Out-of-bag estimation

An observation is out of bag for a particular tree if it was not selected in that tree’s bootstrap sample. Predictions can be aggregated only over trees for which the observation was out of bag. This produces an out-of-bag estimate of generalisation performance without requiring a separate validation set.

Out-of-bag evaluation is useful, but it is not a substitute for careful time-based validation in temporal problems, nor does it eliminate all forms of leakage. If preprocessing, target encoding, or feature construction uses the complete dataset before resampling, out-of-bag estimates can still be optimistic.

3. Random Forests

3.1 Definition and algorithm

A random forest is an ensemble of decision trees that combines bootstrap sampling with random feature selection at each split. A typical algorithm is:

1. For b = 1,…,B, draw a bootstrap sample.
2. Grow a decision tree, often deeply and without pruning.
3. At each split, randomly select m_try predictors from the available p predictors.
4. Choose the best split among only those m_try predictors.
5. Aggregate tree predictions by averaging or voting.

Random feature selection reduces correlation between trees. A dominant predictor that would otherwise appear at the root of nearly every tree is not always available, allowing alternative predictors and structures to contribute.

3.2 Hyperparameters

Important hyperparameters include:

For classification, the default m_try is often related to √p, while regression defaults are commonly related to p/3, but these are heuristics rather than universal rules. Hyperparameters should be selected using validation appropriate to the data-generating process.

3.3 Feature importance

Random forests commonly report impurity-based importance. For a tree node t, a split’s weighted impurity decrease can be represented as:

I(t) is the impurity at node t, L and R are its child nodes, and w terms represent the proportions or weights of samples reaching the nodes. Total importance for a feature is the sum of its decreases over splits using that feature, often averaged over trees.

Impurity importance can be biased toward continuous variables or categorical variables with many levels and toward variables correlated with other predictors. Permutation importance is an alternative: randomly permute one feature in a validation set and measure the performance deterioration. If the baseline loss is L and the permuted loss is L_perm, permutation importance is often expressed as:

The feature index is j. Importance should be computed on held-out data, and correlated features should be interpreted as groups rather than isolated causal drivers. Neither measure proves that a feature causes the outcome.

4. Boosting

4.1 Sequential learning

Bagging trains models independently and reduces variance through averaging. Boosting trains models sequentially, with later models focusing on errors made by earlier models. The base learners are usually weak learners, such as shallow trees.

A general additive model has the form:

F₀ is the initial prediction, h_m is the mth weak learner, η is the learning rate, and M is the number of boosting iterations. Smaller η usually requires more iterations and can improve generalisation when paired with suitable regularisation.

4.2 AdaBoost

For binary classification with labels y_i ∈ {−1,+1}, AdaBoost assigns weights to observations. Initially, all observations have equal weight. At iteration m, a weak classifier h_m is trained and its weighted error is:

The classifier weight is:

The observation weights are then updated approximately according to:

Weights are normalised after updating. These equations assume binary labels coded as −1 and +1, a weak learner error below 0.5, and the exponential-loss formulation. Correctly classified cases receive reduced weight, while misclassified cases receive increased weight. AdaBoost can be sensitive to outliers and mislabeled observations because it repeatedly emphasises difficult cases.

4.3 Gradient boosting

Gradient boosting generalises the sequential correction idea by fitting each new learner to the negative gradient of a chosen loss function. Let the empirical objective be:

L is the loss, Ω is a regularisation term, and F is the current prediction function. At iteration m, pseudo-residuals are:

A weak learner is fitted to r_im, and the model is updated:

For squared-error regression, L(y,F) = 1/2(y − F)², so the negative gradient is y_i − F(x_i), the ordinary residual. For other losses, the pseudo-residual is not necessarily the ordinary residual.

Gradient boosting can model nonlinearities and interactions without manually specifying them. It is powerful but can overfit when trees are too deep, the learning rate is too large, the number of iterations is excessive, or validation is inadequate.

4.4 Regularisation and modern implementations

Regularisation strategies include:

• shrinkage through a small learning rate;
• early stopping based on validation performance;
• shallow trees or limited tree depth;
• row subsampling, sometimes called stochastic gradient boosting;
• column subsampling;
• minimum leaf constraints; and
• explicit penalties on model complexity.

Modern gradient-boosting systems, including histogram-based and highly engineered implementations, improve speed by discretising continuous features, exploiting parallelism, handling missing values, and optimising memory access. Their implementation details differ, so claims about missing-value handling, categorical variables, and regularisation must be checked for the specific software and version.

5. Voting and Stacking

5.1 Voting ensembles

Voting combines predictions from heterogeneous models. Hard voting selects the most frequent predicted class. Soft voting averages predicted class probabilities:

Here p̂_j is model j’s probability for class k and α_j is its nonnegative weight. Soft voting is meaningful only when probabilities are on comparable scales; calibration may therefore be necessary. In regression, voting is generally weighted averaging of predictions.

5.2 Stacking

Stacking trains a meta-model to combine base-model predictions. The crucial requirement is that the meta-model must be trained on out-of-fold predictions rather than predictions generated by base models on the same observations used to fit them.

A leakage-resistant procedure is:

1. Split the training data into K folds.
2. For each fold, fit every base model on the other K − 1 folds.
3. Predict the held-out fold and store those predictions.
4. Combine all held-out predictions to form a new meta-feature matrix.
5. Fit the meta-model on these out-of-fold predictions.
6. Refit base models on all training data and predict the test set.
7. Apply the fitted meta-model to the test-set base predictions.

If a base model predicts its own training observations, it may appear unrealistically accurate. The meta-model then learns to exploit overfit signals and test performance is overstated.

Stacking is most useful when base models have complementary inductive biases—for example, a linear model, a tree ensemble, and a nearest-neighbour model. A simple, regularised meta-model is often preferable because the meta-feature dimension is small and overfitting remains possible.

6. Calibration and Decision Quality

A classifier can rank cases correctly while producing poor probabilities. A model that assigns probability 0.9 to events that occur only 0.7 of the time is overconfident. Calibration means that among cases assigned probability p, the event occurs approximately p of the time.

For binary outcomes, the Brier score is:

p_i is the predicted probability and y_i ∈ {0,1} is the observed outcome. Lower is better. The Brier score is sensitive to both calibration and discrimination. Reliability diagrams compare predicted probability bins with observed event frequencies. Log loss is another proper scoring rule that strongly penalises confident incorrect predictions.

Tree ensembles, particularly boosted models, can be miscalibrated. Calibration methods include Platt scaling, isotonic regression, and beta calibration. Calibration must be fitted using data separate from the model-fitting data, commonly through cross-validation. In rare-event settings, calibration should reflect the deployment prevalence; a change in base rates can make previously calibrated probabilities unsuitable.

Business decisions require more than a probability threshold of 0.5. If the benefit of a true positive is B and the cost of a false positive is C, a simplified threshold under equal and known decision consequences may be derived from expected utility. In practice, organisations should include false-negative costs, operational capacity, fairness constraints, and intervention effects rather than treating a statistical threshold as a universal rule.

7. Worked Example: A Small Bagging Calculation

Consider three regression trees predicting demand for a product. Their predictions for one store-week are 100, 120, and 110 units. The bagged prediction is:

If the actual demand is 116 units, the absolute error is |110 − 116| = 6 units. This simple average does not show whether bagging improves performance; improvement must be assessed across unseen observations and compared with a baseline. If the three trees make highly correlated errors, adding more trees may produce little additional benefit.

Now suppose two calibrated classifiers estimate default probabilities of 0.20 and 0.50, with weights 0.7 and 0.3. The soft-voting probability is:

The result is 0.29, provided that the component probabilities are calibrated or at least comparable and the weights were selected without test-set leakage.

8. Applied Case: Illustrative Loan-Default Analytics

Background

A fictional retail bank wants to identify applicants who may default within twelve months. The data include income, debt-to-income ratio, repayment history, account age, product usage, and recent application behaviour. The outcome is imbalanced: defaults are uncommon relative to non-defaults.

Problem

The bank wants strong ranking performance for prioritising manual review, reliable probabilities for pricing and provisioning, and explanations suitable for customers, auditors, and risk committees. The data also contain time dependence: economic conditions and lending policies change over time.

Analysis

A logistic regression provides a transparent baseline. A single decision tree is easy to communicate but unstable. A random forest can capture nonlinearities and interactions while reducing tree variance. Gradient boosting may produce stronger discrimination but requires careful tuning and calibration. A stacked model could combine the logistic model, random forest, and boosted trees, but only if out-of-fold predictions are used.

Performance should be assessed using a time-based holdout: earlier applications for training and later applications for testing. Accuracy is inappropriate as the primary metric because a model that predicts every applicant as non-default may appear accurate. Suitable measures include area under the precision–recall curve, recall at a fixed review capacity, expected financial loss, calibration error, and Brier score.

Approach and solution

The bank first defines the prediction point and ensures that every feature was available at that time. It performs preprocessing within each training fold. It tunes a random forest and gradient-boosting model using rolling or time-aware validation. Class weights are selected based on the business objective, not merely to maximise a generic score. Probabilities are calibrated on a separate validation period.

The final policy uses the ensemble probability together with operational constraints. Applicants above a review threshold are referred for additional assessment, while automated adverse decisions require a documented reason-code process. Feature importance is supplemented by local explanations, stability analysis, and monotonicity checks where domain knowledge requires that risk should not decrease as debt burden increases.

Outcome

In this illustrative case, the ensemble improves ranking and reduces variance relative to the single tree. However, the strongest model is not automatically the best policy: calibration, fairness, stability under economic shift, interpretability, and legal defensibility determine whether it is deployable.

Learning points

1. Time-aware validation is essential when applicant behaviour and macroeconomic conditions change.
2. Ranking performance and probability calibration are different objectives.
3. Class imbalance should be addressed with metrics and decision costs, not only resampling.
4. Ensemble feature importance is evidence of predictive association, not causation.
5. Human review and governance remain necessary in high-impact decisions.

9. Comparing Ensemble Methods

Bagging and random forests can train trees in parallel. Boosting is more sequential because later learners depend on earlier ones, although individual computations can still be parallelised. Random forests often require more memory for many trees; boosting may require more tuning and can be more sensitive to data quality. Model selection should consider latency, retraining frequency, hardware, monitoring, and the cost of errors—not only cross-validation score.

10. Challenges, Governance, and Best Practice

Data leakage

Leakage occurs when information unavailable at prediction time enters training. Common sources include post-outcome variables, global target encoding, imputation performed before splitting, and stacking predictions generated in-sample. Leakage is especially dangerous because ensembles can exploit subtle leakage very effectively.

Distribution shift

An ensemble assumes that the relationship between predictors and outcomes remains sufficiently stable. Population shift, concept drift, policy changes, and economic shocks can degrade performance. Monitor feature distributions, prediction distributions, calibration, outcome rates, and subgroup metrics.

Fairness and accountability

A model can be accurate overall yet produce unequal error rates across groups. Protected attributes should not be removed automatically as a solution; proxy variables may preserve their information, and removing them may prevent fairness measurement. Organisations should assess relevant fairness criteria, document trade-offs, provide contestability, and comply with applicable data-protection and sector-specific regulations.

Interpretability

Global feature importance describes aggregate predictive contribution, not individual causality. Partial dependence and accumulated local effects can reveal average relationships, but correlated features complicate interpretation. Local explanations should be stress-tested for stability and should not be presented as guaranteed causal reasons.

Practical workflow

A robust workflow is:

1. Define the decision, prediction horizon, loss, and cost of errors.
2. Establish a simple baseline.
3. Create leakage-resistant train, validation, and test splits.
4. Select an ensemble appropriate to the data and operational constraints.
5. Tune hyperparameters using nested or carefully separated validation.
6. Evaluate discrimination, calibration, subgroup performance, and stability.
7. Inspect feature importance and individual explanations.
8. Calibrate probabilities when decisions require risk estimates.
9. Document data, versions, assumptions, limitations, and approval controls.
10. Monitor after deployment and establish retraining or rollback criteria.

Contemporary Developments

Current ensemble practice increasingly combines gradient boosting with automated hyperparameter optimisation, distributed training, categorical-feature handling, and early stopping. Explainable boosting models and constrained boosting permit more interpretable or monotonic relationships in selected applications. Neural and tree models may be combined in multimodal systems, although such hybrids increase governance complexity. Research continues on uncertainty estimation, conformal prediction, fairness-aware ensembles, robust learning under distribution shift, and energy-efficient model training.

These developments do not invalidate classical methods. A well-validated random forest or carefully calibrated gradient-boosting model may be preferable to a more complex system when the data volume, latency, auditability, or organisational capability is limited.

Chapter Summary

Ensemble learning combines multiple models to improve predictive performance through variance reduction, bias correction, or exploitation of complementary inductive biases. Bagging uses bootstrap samples and aggregation; random forests add random feature selection to reduce tree correlation. AdaBoost sequentially emphasises difficult observations, while gradient boosting fits successive learners to loss-function gradients. Voting combines model outputs directly, whereas stacking learns a meta-model from leakage-resistant out-of-fold predictions.

The effectiveness of an ensemble depends on both constituent accuracy and diversity. Correlated errors limit the benefit of averaging. Feature importance must be interpreted cautiously, especially with correlated predictors and high-cardinality variables. Calibration is distinct from discrimination and is essential when probabilities support pricing, triage, credit, or resource allocation. Professional deployment requires time-aware validation where appropriate, leakage prevention, monitoring, fairness assessment, governance, and consideration of computational and operational trade-offs.

Key Terms

Ensemble learning: Combining multiple predictive models into a single predictive system.

Bagging: Bootstrap aggregating, in which models trained on resampled data are averaged or voted.

Bootstrap sample: A sample formed by drawing observations with replacement.

Out-of-bag observation: An observation not selected for a particular bootstrap sample.

Random forest: A bagged tree ensemble using random feature subsets at splits.

Diversity: The extent to which constituent models make different errors.

Bias: Systematic prediction error caused by restrictive assumptions or underfitting.

Variance: Sensitivity of predictions to the particular training sample.

Boosting: Sequential construction of learners that correct or emphasise previous errors.

AdaBoost: A boosting method that updates observation weights according to classification errors.

Gradient boosting: Sequential fitting of learners to negative loss gradients.

Learning rate: The shrinkage factor applied to each boosting update.

Early stopping: Ending training when validation performance ceases to improve.

Voting: Direct combination of predictions from multiple models.

Stacking: Learning a meta-model that combines predictions from base models.

Calibration: Agreement between predicted probabilities and observed frequencies.

Permutation importance: Performance deterioration after permuting a feature.

Data leakage: Use of information in training that would not be available at prediction time.

Concept drift: Change over time in the relationship between predictors and outcomes.

Review Questions

Short Answer Questions

1. Why does reducing correlation among ensemble members improve averaging performance?
2. Distinguish bagging from boosting.
3. What is an out-of-bag estimate?
4. Why must stacking use out-of-fold predictions?
5. How does calibration differ from discrimination?

Long Answer Questions

1. Explain the bias–variance rationale for bagging and random forests, including the role of error correlation.
2. Derive and interpret the gradient-boosting update for squared-error regression.
3. Compare random forests and gradient boosting in terms of accuracy, tuning, computation, and interpretability.
4. Discuss the strengths and limitations of impurity-based and permutation feature importance.
5. Design a complete governance and monitoring framework for an ensemble used in credit decisions.

Analytical / Application Questions

1. A business has a highly nonlinear response, 200 predictors, and noisy labels. Recommend an ensemble strategy and justify choices of validation, regularisation, and evaluation metrics.
2. Three calibrated models produce probabilities 0.10, 0.30, and 0.60 with weights 0.5, 0.3, and 0.2. Calculate the weighted soft-voting probability and explain what assumptions are needed for interpretation.
3. A stacked model performs exceptionally well in cross-validation but poorly on a future-period test set. Diagnose at least four possible causes and propose corrective actions.

Multiple Choice Questions with Answers

Question 1

What is the primary purpose of bootstrap aggregation (bagging)?

A. To reduce variance by averaging models trained on resampled data

B. To remove all irrelevant features before training

C. To guarantee calibrated probabilities

D. To convert a regression problem into classification

Correct Answer: A. To reduce variance by averaging models trained on resampled data

Question 2

Why does a random forest select a random subset of features at each split?

A. To ensure every tree has identical structure

B. To reduce correlation among trees

C. To eliminate the need for validation data

D. To increase the bias of every tree deliberately

Correct Answer: B. To reduce correlation among trees

Question 3

In gradient boosting, what does a new learner generally approximate?

A. The original feature covariance matrix

B. The negative gradient of the loss with respect to current predictions

C. The number of training observations

D. The impurity of the root node only

Correct Answer: B. The negative gradient of the loss with respect to current predictions

Question 4

Which procedure is essential to avoid leakage when training a stacking meta-model?

A. Using in-sample predictions from every base model

B. Using out-of-fold predictions from the base models

C. Removing all base models with different algorithms

D. Training the meta-model on the test labels

Correct Answer: B. Using out-of-fold predictions from the base models

Question 5

A model assigns probabilities 0.2 and 0.5 with voting weights 0.7 and 0.3. What is the weighted probability?

A. 0.21

B. 0.26

C. 0.29

D. 0.35

Correct Answer: C. 0.29

Question 6

Which statement about calibration is correct?

A. A calibrated model must have perfect accuracy

B. Calibration concerns agreement between predicted probabilities and observed frequencies

C. Calibration is relevant only for regression

D. Calibration automatically removes protected-group disparities

Correct Answer: B. Calibration concerns agreement between predicted probabilities and observed frequencies

Question 7

What is a common limitation of impurity-based feature importance in tree ensembles?

A. It cannot be computed for trees

B. It is always causal

C. It can favour continuous or high-cardinality variables

D. It requires calibrated probabilities

Correct Answer: C. It can favour continuous or high-cardinality variables

Question 8

For AdaBoost with binary labels coded −1 and +1, what happens to the weight of a correctly classified observation in the usual update?

A. It is generally reduced

B. It is always set to zero

C. It is increased more than a misclassified observation

D. It is replaced by the class prior

Correct Answer: A. It is generally reduced

Question 9

Which validation design is generally most appropriate when predicting future outcomes from historical business data?

A. Randomly mixing all future and past observations without restriction

B. Time-aware validation that respects the prediction chronology

C. Validation using only training-set predictions

D. No validation if the ensemble has many trees

Correct Answer: B. Time-aware validation that respects the prediction chronology

Question 10

If ensemble members have pairwise error correlation close to one, what is the likely effect of adding more members?

A. Large variance reduction is guaranteed

B. Little additional benefit from averaging is expected

C. Bias becomes exactly zero

D. Calibration becomes perfect

Correct Answer: B. Little additional benefit from averaging is expected