Machine Learning and Ensemble Methods

Introduction

Predictive analytics has progressed from relatively simple statistical models to machine-learning systems capable of representing nonlinear relationships, interactions and complex decision boundaries. This development is valuable in business analytics because customer behaviour, operational performance, risk and demand are rarely governed by one linear relationship. At the same time, greater predictive power can reduce transparency, increase data and computing requirements, and make governance more difficult.

This chapter examines four important families of predictive methods: random forests, gradient boosting, support vector machines (SVMs) and introductory neural networks. It also compares these methods with interpretable statistical models such as linear and logistic regression. The central professional question is not simply “Which model is most accurate?” but “Which model is appropriate for this decision, data environment and risk context?”

Learning Objectives

By the end of this chapter, you should be able to:

• explain the principles of tree ensembles, support vector machines and neural networks;
• distinguish bagging from boosting;
• describe the role of hyperparameters, validation and feature scaling;
• apply suitable evaluation metrics to classification and regression problems;
• compare interpretable models with more complex machine-learning methods;
• identify overfitting, leakage, bias and governance risks; and
• communicate model results in a way that supports responsible business decisions.

1. From Statistical Models to Machine Learning

A supervised-learning problem contains observations with known outcomes. Let xᵢ represent the features for observation i and yᵢ represent its observed target. A model estimates a function f such that:

Here, εᵢ represents unexplained error. In regression, yᵢ is usually numerical; in classification, yᵢ is a category or class probability. The model is trained using a dataset and then evaluated on observations not used for fitting.

A linear regression model assumes an additive linear relationship:

where ŷᵢ is the predicted response, β₀ is the intercept, βⱼ are coefficients, and xᵢⱼ are feature values. This model is attractive because each coefficient has a direct interpretation, subject to the usual assumptions concerning specification, error behaviour and data quality.

For binary classification, logistic regression models the probability of class 1:

The coefficients are additive on the log-odds scale. Logistic regression is often a strong baseline because it is fast, comparatively transparent and effective when the relationship is approximately linear in the log-odds.

Machine-learning models relax some of these structural assumptions. They can discover nonlinear effects and interactions automatically. However, flexibility increases the possibility of fitting random variation rather than general patterns. The fundamental objective remains generalisation: performing well on future data, not merely reproducing the training sample.

Bias, variance and complexity

Prediction error can be understood conceptually as a combination of bias, variance and irreducible noise. A highly constrained model may have high bias because it cannot represent the true structure. A very flexible model may have high variance because its predictions change substantially when the training sample changes. Cross-validation, regularisation, early stopping and careful feature selection are tools for managing this trade-off.

The standard workflow is:

1. define the business decision and target;
2. split data into training, validation and test sets;
3. prepare features using training data only;
4. fit candidate models;
5. tune hyperparameters using cross-validation or a validation set;
6. select a model using performance, interpretability and operational criteria; and
7. evaluate once on untouched test data before deployment.

For time-dependent data, random splitting may be inappropriate. A temporal split, in which earlier observations predict later observations, better represents deployment conditions.

2. Decision Trees and Random Forests

Decision-tree foundations

A decision tree recursively divides observations according to feature thresholds. For example, a tree might first ask whether customer tenure is below six months and then ask whether recent complaints exceed a specified value. Each terminal node, or leaf, produces a prediction.

For classification, a common impurity measure is the Gini impurity:

where t is a node, K is the number of classes, and pₖ|ₜ is the proportion of observations in node t belonging to class k. Gini impurity is zero when a node contains only one class. A split is useful when it reduces weighted impurity. Entropy is another possible criterion:

with the convention that 0 log₂(0) equals 0. For regression, a tree commonly minimises within-node squared error or variance.

Trees are easy to explain locally: a prediction follows a sequence of if–then rules. A single deep tree, however, can be unstable and overfit. Small changes in the data may produce a different tree.

Bagging and random forests

Bagging, or bootstrap aggregating, fits many models to bootstrap samples drawn with replacement from the training data and averages their predictions. For regression, an ensemble prediction may be:

where B is the number of trees and Tᵦ(x) is the prediction from tree b. For classification, predictions may be combined by majority vote or by averaging class probabilities.

A random forest adds feature randomness: at each split, only a randomly selected subset of features is considered. This reduces correlation between trees. Averaging many less-correlated high-variance trees can lower ensemble variance without requiring a linear relationship.

Important random-forest hyperparameters include:

Random forests generally require little feature scaling because tree splits depend on ordering rather than distance. They can handle nonlinear effects and interactions, and they provide useful variable-importance diagnostics. However, raw importance measures can be biased when features have different cardinalities or are strongly correlated. Permutation importance and partial-dependence analysis are often more informative, although they too require careful interpretation.

Out-of-bag estimation

Because each bootstrap sample omits some training observations on average, the omitted observations can provide an out-of-bag (OOB) performance estimate. OOB evaluation is useful for internal diagnostics, but it does not replace a genuinely independent test set when a high-stakes model is being validated.

3. Gradient Boosting

Gradient boosting builds an additive model sequentially. Each new weak learner, often a shallow decision tree, focuses on reducing the loss left by the existing ensemble. A general formulation is:

where Fₘ is the model after iteration m, hₘ is the newly fitted learner, and η is the learning rate. The learner is selected to approximate the negative gradient of the loss with respect to the current predictions.

For squared-error regression, the negative gradient is the residual:

Thus, a new tree is fitted to current residuals. For other losses, such as logistic loss, the pseudo-residual is derived from the relevant gradient rather than being a simple observed-minus-predicted difference.

A small learning rate combined with more trees often produces a strong model, but training takes longer. Tree depth controls interaction complexity. Subsampling rows or features at each iteration can reduce variance and improve generalisation; this is commonly called stochastic gradient boosting.

Key hyperparameters include the number of boosting iterations, learning rate, tree depth, minimum leaf size, subsampling fraction and regularisation parameters. Early stopping halts training when validation performance stops improving. Without appropriate regularisation, boosting may overfit, especially when trees are deep or the number of iterations is excessive.

Gradient boosting is frequently highly competitive on structured tabular data. It can represent nonlinearities and interactions and often performs well with mixed feature effects. Its disadvantages include greater sensitivity to tuning, longer training and more complicated explanations than a single tree or regression model. Modern implementations such as gradient-boosted decision-tree libraries may incorporate efficient computation, missing-value strategies and explicit regularisation, but the underlying modelling principles remain the same.

Random forests versus gradient boosting

Random forests construct many trees largely independently and average them. Gradient boosting constructs trees sequentially, with each tree correcting weaknesses in the current model. Random forests are often robust and relatively easy to tune. Boosting can achieve higher predictive accuracy on many tabular problems but may be more sensitive to noise, leakage and hyperparameter choices. Both should be compared against a transparent baseline rather than assumed to be superior.

4. Support Vector Machines

Classification principle

A linear support vector classifier seeks a separating hyperplane:

where w is a vector of feature weights and b is an intercept. The signed margin for observation i is yᵢ(wᵀxᵢ + b), where yᵢ is coded as −1 or +1. A soft-margin SVM solves an optimisation problem of the form:

subject to:

The variables ξᵢ are slack variables that permit violations, and C controls the trade-off between a wide margin and training violations. A large C penalises violations more strongly; a smaller C permits more violations in exchange for a wider margin. The optimisation formulation assumes labelled observations and is most directly applicable to binary classification, although multiclass extensions exist.

The observations that lie closest to the boundary, or violate it, are support vectors. They determine the fitted boundary more directly than observations far away.

Kernels and nonlinear boundaries

The kernel method computes inner products in an implicit transformed feature space. A common radial basis function (RBF) kernel is:

where γ controls how rapidly similarity decreases with distance. A large γ creates highly local influence and can overfit; a small γ creates smoother, broader influence. The RBF kernel is useful when the boundary is nonlinear, but interpretation becomes less direct.

SVMs are sensitive to feature scale because distances and inner products determine the margin and kernel values. Standardisation is usually required. For a feature x with training mean μ and standard deviation σ, a common transformation is:

The values μ and σ must be calculated on the training data only and then applied unchanged to validation, test and production data. If σ = 0, the feature has no variation in the training sample and should normally be removed or handled separately.

SVMs can perform well in high-dimensional spaces and in problems with moderate sample sizes. They may become computationally expensive for very large datasets, and probability estimates often require an additional calibration procedure. Missing values, categorical variables and class imbalance also require explicit treatment.

5. Introductory Neural Networks

A basic feed-forward neural network consists of layers of interconnected units. A neuron first computes a weighted sum and bias:

It then applies an activation function aⱼ = g(zⱼ). Common activations include the rectified linear unit (ReLU), g(z) = max(0,z), and the sigmoid function, g(z) = 1/[1 + exp(−z)]. Hidden layers allow the network to represent increasingly complex transformations.

For binary classification, a final sigmoid unit can produce an estimated probability. For K mutually exclusive classes, a softmax output is commonly used:

The probabilities sum to 1 across the K classes. A common training loss for a single multiclass observation with one-hot target values yₖ is cross-entropy:

where pₖ is the predicted probability and yₖ is 1 for the observed class and 0 otherwise.

Training uses backpropagation to calculate gradients of the loss with respect to weights, followed by an optimisation method such as stochastic gradient descent or Adam. In simplified form, a parameter θ is updated as:

where η is the learning rate and ∇θL is the gradient of the loss. This equation describes an update rule, not a guarantee that every update improves test performance.

Neural networks are powerful when there is sufficient data, appropriate representation and computational capacity. They are particularly important for images, text, audio and other unstructured data. For ordinary business tables with limited observations, boosted trees may be more effective and easier to manage. Neural networks can overfit, be sensitive to architecture and scaling, and require careful monitoring of training and validation performance.

6. Evaluation and Model Selection

A model should be evaluated using metrics aligned with the decision. For binary classification, accuracy is:

Precision and recall are:

Here TP, TN, FP and FN denote true positives, true negatives, false positives and false negatives. Accuracy may be misleading under class imbalance. For example, identifying rare fraud requires attention to false negatives and investigation costs, not merely the percentage of correct predictions.

The F1 score is the harmonic mean of precision and recall:

For regression, mean absolute error and root mean squared error are common:

MAE has the same units as the target and weights all absolute errors equally. RMSE also has the target’s units but penalises large errors more strongly. Neither metric alone establishes that a model creates business value.

For probabilistic classification, calibration matters. A model is calibrated when, among cases assigned probability approximately 0.7, about 70% actually belong to the positive class. A well-ranked model may still be poorly calibrated. Threshold selection should reflect the costs and benefits of decisions rather than defaulting to 0.5.

Interpretable and complex models

Interpretability is not a binary property. A model may be globally understandable but locally difficult to explain, or locally explainable while its overall behaviour remains complex. Feature-importance rankings do not prove causation. Explanation methods such as permutation importance, partial dependence and local attribution should be treated as diagnostic aids, not as substitutes for domain knowledge or causal analysis.

7. Worked Example: Churn Prediction

Consider a subscription business with 1,000 customers. In a hold-out test set, a classifier produces the following results:

• TP = 72;
• TN = 760;
• FP = 48; and
• FN = 120.

The total is 1,000. Accuracy is:

Precision is:

Recall is:

The model is correct for most customers, but it identifies only 37.5% of actual churners. If the company’s intervention is inexpensive and missing a likely churner is costly, the decision-maker may lower the probability threshold to increase recall. This will usually increase the number of false positives. The appropriate threshold depends on intervention capacity, customer experience and estimated costs—not on a metric considered in isolation.

A sensible modelling comparison would begin with logistic regression as a transparent baseline, followed by a random forest and gradient-boosted trees. The models should use identical time-aware splits and identical leakage controls. If boosting improves recall only marginally but is much harder to govern, the simpler model may be preferable.

8. Applied Case: Illustrative Retail Loan-Retention System

Background

A fictional retail bank wants to identify personal-loan customers who may become inactive during the next 60 days. The bank has transaction summaries, account tenure, repayment history, service contacts and product holdings. The retention team can contact only 10% of customers each month.

Problem

The bank initially proposes a complex neural network because it expects maximum predictive accuracy. However, the system will influence customer communications and may be reviewed by compliance staff. The bank therefore needs to compare predictive performance, operational constraints and explainability.

Analysis

The analytics team defines the target using only information available at the prediction date. It creates a time-based training period and a later test period. A major leakage risk is discovered: a “closure request” field was recorded after customers had already begun the exit process. Including it would produce excellent test performance but would not represent a valid early-warning system.

The team fits logistic regression, a random forest and gradient boosting. Numerical features for logistic regression are standardised where appropriate; tree models do not require the same scaling. Class weights and threshold selection are considered because inactive customers are a minority. The team evaluates recall among the top 10% of ranked customers, expected retention value and calibration.

Approach and solution

The boosted-tree model gives the best ranking performance, while logistic regression is easier to explain and produces more stable probability estimates after calibration. The bank chooses the boosted model for prioritisation but adds governance controls:

• a documented feature catalogue and data lineage;
• exclusion of post-outcome information;
• fairness checks across legally and operationally relevant groups;
• a reviewable reason code for each contact recommendation;
• a human approval step before communication; and
• monthly drift and calibration monitoring.

The model is not permitted to deny services or alter interest rates automatically. It only prioritises customers for a retention offer, and customers receive standard privacy and communication protections.

Outcome

The pilot improves retention-team efficiency because staff focus on higher-risk customers. However, the bank finds that one group receives too many offers because of a proxy variable related to branch service access. The feature is removed, and the model is retrained. This slightly reduces predictive performance but improves fairness and operational acceptability.

Learning points

The case demonstrates that model selection is multidimensional. A small performance improvement does not automatically justify greater complexity. Data leakage can make a model appear successful while making it useless in production. Human oversight, feature governance, calibration and monitoring are components of the analytical solution, not administrative additions.

9. Implementation Best Practices and Risks

Data preparation and validation

Preprocessing must be fitted within each training fold. Applying imputation, scaling or feature selection to the complete dataset before cross-validation can leak information from validation observations. Pipelines should combine preprocessing and modelling so that the same operations are reproducible in training and deployment.

Categorical variables require an appropriate encoding strategy. High-cardinality categories can create sparse or unstable representations. Missingness may itself carry information, but treating missing values as meaningful should be justified rather than assumed.

Imbalance and thresholding

Oversampling, undersampling, class weights and threshold adjustment address different problems. Resampling changes the training distribution; thresholding changes the decision rule after probabilities are estimated. If oversampling is used, it must occur inside the training folds, never before the split. Evaluation should use the natural deployment prevalence unless there is a clear reason otherwise.

Explainability and governance

Organisations should document the purpose, population, target definition, training period, features, exclusions, hyperparameters, validation design, performance, limitations and approval status. Explanations should be accurate and understandable to the intended audience. A feature associated with an outcome is not necessarily a cause of that outcome.

Potential ethical risks include discriminatory effects, privacy invasion, inappropriate use of sensitive or proxy variables, automation bias and unequal error rates. Relevant privacy, consumer-protection, sectoral and data-protection requirements must be considered. High-impact decisions require stronger evidence, controls and human review than low-risk recommendations.

Deployment and monitoring

A model can degrade through data drift, concept drift, process changes or altered customer behaviour. Monitoring should include input distributions, missingness, prediction distributions, calibration, error rates, subgroup performance and outcome delay. Performance metrics must be interpreted carefully when labels arrive late. A rollback plan and a defined retraining policy are essential.

10. Contemporary Developments

Modern machine-learning practice increasingly combines automated hyperparameter search, explainable-AI tools, model registries, feature stores and continuous monitoring. Gradient-boosted trees remain prominent for structured business data, while neural networks dominate many unstructured-data applications. Transfer learning and pretrained representations can reduce the data required for specialised tasks, but they introduce additional concerns concerning provenance, bias, licensing and security.

Automated machine learning can accelerate model comparison, but it does not remove the need for target definition, leakage prevention, ethical review or business interpretation. Generative AI may assist with coding and documentation, but generated code and explanations require testing, validation and human accountability.

Chapter Summary

Machine-learning models extend traditional statistical approaches by representing nonlinear relationships, interactions and complex boundaries. Random forests use bootstrap samples and random feature selection to reduce the variance of decision trees. Gradient boosting builds trees sequentially to reduce a chosen loss and can be highly effective on structured data. Support vector machines seek a margin that separates classes and can use kernels for nonlinear patterns; scaling is usually essential. Neural networks transform inputs through weighted layers and activation functions and are trained using gradient-based optimisation.

Model quality must be assessed using metrics aligned with the decision, including precision, recall, F1, MAE, RMSE and calibration. Cross-validation, time-aware splitting, leakage prevention, regularisation and threshold selection are central to reliable practice. Complex models may improve predictive performance, but they bring greater explanation, monitoring and governance requirements. A transparent baseline, rigorous validation and responsible deployment are more valuable than complexity for its own sake.

Key Terms

Bias: Systematic error caused by restrictive assumptions or an unrepresentative process.

Boosting: Sequentially combining weak learners so later learners address earlier errors.

Calibration: Agreement between predicted probabilities and observed event frequencies.

Decision tree: A model that recursively partitions observations using feature-based rules.

Early stopping: Ending iterative training when validation performance ceases to improve.

Ensemble: A combination of multiple predictive models.

Feature scaling: Transforming variables to comparable numerical scales.

Gradient boosting: An additive ensemble method that follows gradients of a loss function.

Hyperparameter: A setting selected before or during training rather than directly estimated as a model parameter.

Kernel: A function that computes similarity in an implicit feature space.

Leakage: Use of information unavailable at the time a real prediction would be made.

Margin: Distance between a separating boundary and the closest observations in an SVM.

Neural network: A layered model of weighted transformations and activation functions.

Random forest: An ensemble of randomly diversified decision trees.

Recall: The proportion of actual positive cases correctly identified.

Regularisation: A technique that discourages excessive model complexity.

Support vector: An observation that strongly influences an SVM boundary.

Variance: Sensitivity of a fitted model to the particular training sample.

Review Questions

5 Short Answer Questions

1. What is the principal difference between bagging and boosting?
2. Why are random forests generally less sensitive to feature scaling than SVMs?
3. What does the SVM parameter C control?
4. Define model calibration.
5. Why can accuracy be misleading for an imbalanced classification problem?

5 Long Answer Questions

1. Explain how a random forest reduces the instability of an individual decision tree.
2. Describe gradient boosting, including the roles of the learning rate, tree depth and early stopping.
3. Compare logistic regression, random forests and neural networks for a tabular business prediction problem.
4. Explain the importance of leakage prevention and time-aware validation in predictive analytics.
5. Discuss the ethical and governance requirements for a model that prioritises customers for intervention.

3 Analytical / Application Questions

1. A fraud model has high accuracy but very low recall. Which additional metrics and operational information should be examined before deployment, and why?
2. A company has 5,000 observations, 40 mostly numerical features and a highly nonlinear target. Design a model-comparison plan involving a transparent baseline, random forest and gradient boosting.
3. A neural network performs better on a test set than logistic regression, but its predictions are poorly calibrated and difficult to explain. Recommend whether and how it should be deployed in a high-impact decision process.

Multiple Choice Questions with Answers

Question 1

Which characteristic most directly distinguishes gradient boosting from a random forest?

A. Gradient boosting builds trees sequentially to correct errors from earlier trees.

B. Gradient boosting cannot represent nonlinear relationships.

C. A random forest always uses only one decision tree.

D. A random forest requires all features to be standardised.

Correct Answer: A. Gradient boosting builds trees sequentially to correct errors from earlier trees.

Question 2

What is the main purpose of feature subsampling in a random forest?

A. To ensure every tree has identical predictions

B. To reduce correlation among trees

C. To convert regression into classification

D. To eliminate the need for a test set

Correct Answer: B. To reduce correlation among trees

Question 3

In a soft-margin SVM, a larger value of C generally places greater emphasis on:

A. Allowing classification violations

B. Reducing the number of features to zero

C. Penalising margin violations

D. Standardising the target variable

Correct Answer: C. Penalising margin violations

Question 4

Why is feature scaling usually important for an RBF-kernel SVM?

A. The kernel uses distances between observations.

B. Scaling changes categorical labels into targets.

C. Scaling guarantees perfect classification.

D. The SVM cannot calculate an intercept without scaling.

Correct Answer: A. The kernel uses distances between observations.

Question 5

A classifier has TP = 80 and FN = 20. What is its recall?

A. 20%

B. 40%

C. 80%

D. 100%

Correct Answer: C. 80%

Question 6

Which measure is especially appropriate when large regression errors should be penalised more strongly?

A. MAE

B. RMSE

C. Recall

D. Precision

Correct Answer: B. RMSE

Question 7

What does early stopping attempt to prevent?

A. Feature scaling

B. Overfitting during iterative training

C. The use of validation data

D. The calculation of gradients

Correct Answer: B. Overfitting during iterative training

Question 8

Which action is an example of data leakage?

A. Using an earlier transaction total to predict future churn

B. Calculating scaling parameters from the training fold

C. Using a post-closure field to predict whether closure will occur

D. Evaluating a final model on an untouched test set

Correct Answer: C. Using a post-closure field to predict whether closure will occur

Question 9

What is a key advantage of logistic regression compared with many high-complexity models?

A. It always gives the highest accuracy.

B. Its coefficients can provide comparatively direct explanations.

C. It never requires a target variable.

D. It automatically solves class imbalance.

Correct Answer: B. Its coefficients can provide comparatively direct explanations.

Question 10

What does a calibrated probability of 0.7 approximately mean?

A. The model is correct exactly 70% of the time for every individual.

B. About 70% of cases assigned that probability are expected to be positive.

C. The decision threshold must be 0.7.

D. The model has a recall of 70%.

Correct Answer: B. About 70% of cases assigned that probability are expected to be positive.