Machine Learning Models and Ensembles

Introduction

Predictive analytics converts historical observations into estimates of future or unknown outcomes. In business, these outcomes may include customer churn, loan default, product demand, claim cost, delivery delay, or conversion probability. Classical time-series models remain valuable when temporal structure is dominant, but machine-learning models are useful when many explanatory variables interact in nonlinear ways.

This chapter examines three practical model families: k-nearest neighbours (k-NN), random forests, and gradient boosting. It also explains hyperparameter tuning, model complexity, interpretability, ensemble learning, and business-oriented model selection. The central principle is that the best predictive model is not necessarily the most complex or the model with the lowest validation error. It must also be reliable, explainable enough for its context, operationally feasible, and aligned with the consequences of incorrect predictions.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain how k-NN, random forests, and gradient-boosting models make predictions.
2. Distinguish parameters learned from data from hyperparameters selected by the analyst.
3. Tune models using validation procedures without allowing information leakage.
4. Explain the relationship between model complexity, generalisation, and interpretability.
5. Compare models using appropriate predictive and business metrics.
6. Select a model appropriate to a stated business requirement.
7. Identify governance, ethical, and operational risks in machine-learning deployment.

1. Predictive Modelling as a Business Process

A predictive project begins by defining the decision, not by selecting an algorithm. A retailer may need to decide which customers receive a retention offer. A bank may need to decide whether an application is referred for manual review. A manufacturer may need to schedule maintenance before a machine fails.

The target variable determines the modelling task:

For regression, mean absolute error is

where y_i is the observed value, ŷ_i is the prediction, and n is the number of observations. MAE is expressed in the same units as the target and gives each error a linear weight.

Root mean squared error is

RMSE has the same units as the target but penalises large errors more heavily. It is useful when unusually large errors have disproportionate business consequences.

For binary classification, predicted probabilities should be assessed as well as class labels. Log loss for observations with binary outcomes is

where y_i is 0 or 1 and p_i is the predicted probability of class 1. Probabilities must lie strictly between 0 and 1 for the logarithms to be finite. Lower log loss is better, and confident incorrect predictions are penalised strongly.

A modelling dataset should be divided into training, validation, and test data, or evaluated through cross-validation plus a final untouched test set. When observations have a time order, random splitting can leak future information into the past. A time-based split is usually more appropriate: earlier observations train the model, a later period validates it, and the most recent period provides final testing.

2. k-Nearest Neighbours

2.1 Core idea

The k-nearest neighbours method predicts an observation from similar observations in the training data. Similarity is usually measured by distance in feature space. The method is called instance-based or lazy learning because it does not construct a simple global equation during training; most computation occurs when a prediction is requested.

For two observations with feature vectors x and z, Euclidean distance is

where p is the number of features. This formula is appropriate when numeric features are meaningfully comparable after suitable scaling.

For classification, the predicted class is commonly the majority class among the k closest training observations. A distance-weighted alternative gives closer neighbours greater influence, for example with weight

where d_i is the distance to neighbour i and ε is a small positive constant preventing division by zero. For regression, the prediction may be the mean of the neighbours’ target values:

These rules assume that nearby observations tend to have similar outcomes.

2.2 Scaling and feature design

Distance-based methods are highly sensitive to measurement scale. Suppose annual income is measured in dollars and account age in years. Without scaling, income may dominate the distance merely because its numerical values are larger. Standardisation transforms a feature using

where μ_j and σ_j are calculated from the training data only. The resulting variable has training-set mean approximately zero and standard deviation approximately one. Scaling parameters must not be estimated from validation or test observations.

Categorical variables require careful encoding. One-hot encoding can create many dimensions, while arbitrary numeric codes can incorrectly imply order. Missing values must also be handled before distance calculations.

2.3 Choosing k

A small k produces flexible predictions and can be strongly affected by noise. A large k smooths predictions but may overlook local structure. The best value is selected through validation. Odd values can reduce ties in binary classification, but this is not a substitute for empirical evaluation.

The number of features also matters. In high-dimensional spaces, observations may become similarly distant, a phenomenon often called the curse of dimensionality. Feature selection, domain-informed variables, or dimensionality reduction may therefore improve k-NN.

2.4 Worked example

Assume a regression problem has three nearest customers with observed annual spending of £80, £100, and £120. With k = 3, the unweighted prediction is

Thus, the predicted spending is £100. If the nearest customer has distance 1, the second distance 2, and the third distance 4, inverse-distance weights are 1, 0.5, and 0.25. The weighted prediction is

The closer customer has greater influence. This calculation is meaningful only if the feature representation and distance measure reflect business similarity.

3. Random Forests

3.1 Decision trees as building blocks

A decision tree recursively divides observations using rules such as “account age ≤ 12 months”. Each terminal node produces a prediction. A classification node commonly predicts the most frequent class, while a regression node predicts the mean target in that node.

Tree growth is controlled by rules such as maximum depth, minimum observations required for a split, and minimum observations in a leaf. Deep trees can represent complex interactions but may overfit. Pruning or explicit limits can reduce this risk.

3.2 Bagging and random feature selection

A random forest combines many decision trees. Each tree is trained on a bootstrap sample: a sample of training observations drawn with replacement. At each split, only a random subset of features is considered. These two sources of randomness reduce correlation among trees.

For regression with B trees, the forest prediction is the mean:

where T_beta(x) is the prediction of tree β. For classification, the forest may use majority vote or average class probabilities. Averaging generally reduces variance when individual trees are imperfect but not perfectly correlated.

A useful variance relationship for an idealised average of B equally variable predictors is

where σ² is the variance of an individual predictor and ρ is the average pairwise correlation between predictors. As B increases, the second term decreases; reducing correlation through random feature selection is therefore important. This relationship is a conceptual approximation, not a guarantee for every forest.

3.3 Important hyperparameters

Random forests usually require less feature scaling than k-NN because tree splits depend on ordering rather than distance. They can handle nonlinearities and interactions well, but extrapolate poorly in regression: predictions generally remain within the range represented by terminal-node training outcomes.

3.4 Out-of-bag evaluation

Because bootstrap samples omit some training observations for each tree, the omitted observations can provide an out-of-bag estimate of performance. This can be useful during development, although a final independent test period remains preferable for honest reporting.

4. Gradient Boosting

4.1 Sequential error correction

Gradient boosting builds an additive model sequentially. Each new weak learner attempts to improve the current model, often by fitting the remaining errors or, more generally, the negative gradient of a chosen loss function.

For squared-error regression, begin with a constant prediction, often the training mean. At iteration m, a tree is fitted to residuals

and the model is updated as

where h_m(x) is the new tree and η is the learning rate, with 0 < η ≤ 1 in common implementations. More generally, the fitted target is related to the negative derivative of the loss with respect to the current prediction. The exact update depends on the loss function.

The final model is an additive ensemble:

Here M is the number of boosting iterations. Small trees and a small learning rate can produce strong performance, but usually require more iterations.

4.2 Main hyperparameters

Important controls include the number of boosting iterations, learning rate, tree depth, minimum leaf size, row subsampling, and column subsampling. Learning rate and number of iterations interact: reducing the learning rate often requires more trees. Excessive iterations can overfit, so early stopping based on a validation set is common.

Gradient boosting often achieves excellent accuracy on structured business data. However, it is sensitive to noisy data, leakage, poor hyperparameter choices, and uncalibrated probabilities. Modern implementations such as XGBoost, LightGBM, and CatBoost add engineering improvements, but the underlying principles remain sequential additive modelling and loss minimisation.

5. Ensembles and Model Comparison

An ensemble combines predictions from multiple models. Bagging, as used in random forests, primarily reduces variance by averaging diverse models. Boosting constructs models sequentially to reduce bias and improve the current fit. A simple heterogeneous ensemble may average probabilities from k-NN, a forest, and a boosting model.

For a weighted probability ensemble,

subject to w_j ≥ 0 and ∑w_j = 1. The weights may be selected using validation data. The validation predictions must be generated without training on the corresponding validation observations; otherwise the ensemble evaluation is optimistic.

Model comparison should consider:

• predictive performance on unseen data;
• calibration, when probabilities guide decisions;
• stability across time periods or customer segments;
• computational cost and latency;
• ease of explanation and audit;
• data requirements and missing-value behaviour;
• consequences of false positives and false negatives;
• fairness and regulatory obligations.

Accuracy alone can be misleading. If only 2% of transactions are fraudulent, a classifier that predicts “not fraud” for every case has 98% accuracy but no practical value. A confusion matrix separates true positives, true negatives, false positives, and false negatives. Precision is

and recall is

where TP, FP, and FN denote true positives, false positives, and false negatives. The appropriate balance depends on the cost of each error.

6. Hyperparameter Tuning Without Leakage

A parameter is estimated from training data; a hyperparameter controls the learning procedure and is selected by the analyst. Examples include k in k-NN, tree depth, number of trees, learning rate, and number of boosting iterations.

A practical tuning workflow is:

1. Define the target, prediction horizon, and business loss.
2. Split data according to operational timing.
3. Build a preprocessing pipeline so transformations are fitted only on training folds.
4. Choose a small, defensible search space.
5. Use grid search, random search, or Bayesian optimisation within training data.
6. Select using cross-validation or a validation period.
7. Refit the selected configuration on permitted development data.
8. Evaluate once on the untouched test set.
9. Examine subgroup performance, calibration, stability, and operational impact.

Random search can be more efficient than an exhaustive grid when only a few hyperparameters strongly affect performance. Nested cross-validation is useful when an unbiased estimate is required after extensive tuning, although time-based validation is generally preferable for forecasting applications.

7. Complexity, Interpretability, and Explainability

Model complexity concerns the flexibility of a model, not simply the number of lines of code. A large forest may be computationally complex but relatively robust, while a shallow tree is structurally simple. k-NN has little explicit training but can have high prediction-time cost because it searches the training set.

Interpretability is the extent to which a person can understand the model’s structure and reasoning. Explainability includes methods used to describe predictions from complex models. These concepts are related but not identical.

Useful tools include:

• feature importance based on split improvements;
• permutation importance, which measures performance loss after shuffling a feature;
• partial dependence, showing average prediction changes as a feature varies;
• local explanations for an individual case;
• surrogate models, such as a small decision tree approximating a complex model.

These tools have limitations. Correlated features can share or obscure importance. Partial-dependence plots can be misleading in unrealistic regions of feature space. Local explanations are approximations rather than proof of causality. A feature that is predictive is not necessarily a cause of the outcome.

For high-stakes uses, documentation should state the data sources, target definition, training period, validation design, features, hyperparameters, known limitations, intended use, prohibited use, and monitoring plan.

8. Illustrative Case: Telecom Customer Churn

Background

A telecommunications provider wants to identify customers likely to cancel within the next 30 days. The business can contact only 10% of its customer base, so ranking customers is more useful than producing an unqualified list.

Problem

The dataset contains tenure, monthly charges, contract type, service usage, complaints, payment history, and customer-support contacts. The target is whether cancellation occurs in the following 30 days. A major risk is leakage: a cancellation request, recorded after the prediction date, must not be used as a predictor.

Analysis and approach

The analyst uses the first 18 months for development and the next three months as a time-based test period. Numeric variables are standardised for k-NN. The candidate models are:

• k-NN with several values of k and distance weighting;
• a random forest with tuned depth, leaf size, and class weighting;
• gradient boosting with shallow trees, learning-rate tuning, and early stopping.

Because the retention team needs a ranked list, the models are assessed using precision among the top 10% of customers, recall, PR-AUC, calibration, and expected campaign profit. Suppose the contact cost is £2, the expected gross margin retained from a successfully persuaded customer is £80, and the estimated persuasion probability differs by customer. The final decision should use expected incremental value, not merely churn probability:

The probability in this expression must represent incremental success caused by the intervention, not merely the probability of churn. A predictive churn model alone does not establish that a contact will change behaviour. A controlled experiment is needed to estimate treatment effect.

Outcome

Assume gradient boosting gives the strongest test-period ranking, while the random forest is slightly easier to operate and more stable across customer segments. The firm selects gradient boosting for ranking, but applies probability calibration and a conservative decision threshold. It retains the forest as a monitoring benchmark. A quarterly review checks drift in contract mix, missingness, precision at the campaign capacity, and performance by age group and region.

Learning points

1. The prediction date defines which variables are legitimate.
2. Time-based evaluation is essential when future decisions are involved.
3. The business objective may be ranking or profit rather than accuracy.
4. Prediction of churn is not the same as prediction of treatment response.
5. A slightly less accurate model may be preferable if it is more stable, explainable, or operationally reliable.

9. Limitations, Governance, and Responsible Use

Machine-learning models learn associations in historical data. If historical decisions contain discrimination, the model may reproduce it. Sensitive attributes may be excluded from inputs but still inferred through proxies such as location, language, or purchasing patterns. Fairness assessment should therefore examine relevant group-level error rates, calibration, access to beneficial decisions, and the legal context.

Data protection principles commonly require a clear purpose, data minimisation, appropriate security, retention limits, and transparency. Organisations should document lawful use, access controls, and the consequences of automated decisions. Where individuals may be materially affected, meaningful human review and an appeal route may be necessary.

Operational risks include concept drift, data drift, broken pipelines, changing definitions, adversarial behaviour, and silent changes in upstream systems. Monitoring should compare current feature distributions with development data, track missingness, measure delayed outcomes, and trigger review when performance or fairness deteriorates.

Artificial intelligence tools do not remove the need for professional judgement. Automated hyperparameter search can optimise the wrong metric, and generative systems can produce unsupported explanations. Every model should have an accountable owner, versioned data and code, reproducible evaluation, and a defined retirement process.

10. Selecting a Model for Business Requirements

A practical selection framework is:

The recommendation should state the trade-off explicitly. For example: “Gradient boosting is selected because it improves top-decile recall by 8 percentage points on the time-based test period, while its explanations, calibration, and monitoring plan satisfy the business governance requirement.” Such a statement is stronger than claiming that one algorithm is universally best.

Best-Practice Checklist

• Define the decision, target, horizon, and cost of errors before modelling.
• Prevent temporal and preprocessing leakage.
• Scale distance-based features using training data only.
• Tune hyperparameters inside the development process.
• Preserve an untouched, realistic test period.
• Compare calibration and subgroup performance, not only aggregate accuracy.
• Record model versions, data definitions, and assumptions.
• Use explanations as evidence about model behaviour, not as causal proof.
• Monitor drift, data quality, fairness, and business outcomes after deployment.
• Review whether the model remains appropriate as the business changes.

Chapter Summary

k-nearest neighbours predicts from nearby observations and therefore depends strongly on feature scaling, distance choice, and the value of k. Random forests average diverse decision trees, reducing variance through bootstrap sampling and random feature selection. Gradient boosting builds an additive sequence of weak learners that corrects previous errors and can provide excellent performance on structured data.

Hyperparameter tuning must be separated from final testing and designed to prevent leakage. Model comparison should include suitable predictive metrics, calibration, stability, computational requirements, interpretability, fairness, and business costs. Complex ensembles may improve accuracy but can require stronger governance and explanation procedures. A responsible selection process matches model properties to the decision, data, operational environment, and consequences of error.

Key Terms

k-nearest neighbours: A method that predicts from the outcomes of the closest training observations.

Distance metric: A rule for measuring similarity or separation between observations.

Standardisation: Transformation of a feature by subtracting its training mean and dividing by its training standard deviation.

Decision tree: A recursive set of rules that produces predictions in terminal nodes.

Random forest: An ensemble of decision trees trained with bootstrap samples and random feature subsets.

Bootstrap sample: A sample drawn with replacement from a dataset.

Bagging: Variance-reduction strategy that averages models trained on resampled data.

Gradient boosting: Sequential additive modelling in which new learners reduce the current loss.

Learning rate: A multiplier controlling the contribution of each boosting learner.

Hyperparameter: A modelling control selected by the analyst rather than directly learned as a model parameter.

Cross-validation: A procedure that repeatedly trains and validates a model on different data partitions.

Data leakage: Unintended use of information unavailable at the prediction time.

Overfitting: Learning noise or peculiarities of development data rather than general patterns.

Calibration: Agreement between predicted probabilities and observed frequencies.

Precision: The proportion of predicted positives that are actually positive.

Recall: The proportion of actual positives that are correctly identified.

Feature importance: An estimate of how much a feature contributes to model predictions under a specified method.

Concept drift: Change over time in the relationship between predictors and the target.

Model governance: Policies and controls covering model development, use, monitoring, documentation, and retirement.

Review Questions

5 Short Answer Questions

1. Why must features usually be scaled before applying Euclidean-distance k-NN?
2. What two sources of randomness are commonly used in a random forest?
3. What is the role of the learning rate in gradient boosting?
4. Why can accuracy be misleading for a highly imbalanced classification problem?
5. What is data leakage, and why is it dangerous?

5 Long Answer Questions

1. Compare k-NN, random forests, and gradient boosting in terms of assumptions, preprocessing, complexity, and interpretability.
2. Explain how time-based validation differs from random cross-validation and when it should be used.
3. Discuss how bagging and boosting address prediction error differently.
4. Design a hyperparameter-tuning workflow for a business classification problem, including safeguards against leakage.
5. Explain why predictive importance does not establish that a variable causes the outcome.

3 Analytical / Application Questions

1. A fraud team can investigate only the top 5% of transactions. Which metrics and validation design would you use to compare a forest and a boosting model, and why?
2. A k-NN model performs poorly after new regional variables are added. Diagnose at least three possible causes and propose corrective actions.
3. A gradient-boosting model has the best test score but produces unstable results for a legally protected group. Recommend a deployment decision and a governance plan.

Multiple Choice Questions with Answers

Question 1

Which model is most directly affected by the numerical scale of its input features?

A. k-nearest neighbours

B. Random forest

C. Gradient boosting with tree learners

D. A majority-class rule

Correct Answer: A. k-nearest neighbours

Question 2

In a random forest, bootstrap sampling and random feature selection primarily help to do what?

A. Increase the correlation among trees

B. Reduce ensemble variance by creating diverse trees

C. Guarantee unbiased predictions

D. Eliminate the need for a test set

Correct Answer: B. Reduce ensemble variance by creating diverse trees

Question 3

For squared-error regression, the residual at boosting iteration m is defined as:

A. yᵢ + Fₘ₋₁(xᵢ)

B. Fₘ₋₁(xᵢ) − yᵢ for every loss function

C. yᵢ − Fₘ₋₁(xᵢ)

D. yᵢ/Fₘ₋₁(xᵢ)

Correct Answer: C. yᵢ − Fₘ₋₁(xᵢ)

Question 4

If a binary classifier identifies 40 true positives and 10 false positives, its precision is:

A. 20%

B. 40%

C. 75%

D. 80%

Correct Answer: D. 80%

Question 5

Which evaluation design is generally most appropriate when predicting future outcomes from time-ordered data?

A. Randomly mixing all observations before splitting

B. Training on later observations and testing on earlier observations

C. Training on earlier observations and testing on a later period

D. Evaluating only on the training data

Correct Answer: C. Training on earlier observations and testing on a later period

Question 6

What is the main danger of selecting hyperparameters using the final test set?

A. The model becomes unable to calculate probabilities

B. The reported test performance becomes optimistically biased

C. Feature scaling becomes mathematically impossible

D. The target variable becomes continuous

Correct Answer: B. The reported test performance becomes optimistically biased

Question 7

A k-NN regression model uses three neighbours with target values 80, 100, and 120 and equal weights. What is its prediction?

A. 90

B. 100

C. 110

D. 120

Correct Answer: B. 100

Question 8

Which statement best distinguishes prediction from causal inference?

A. A predictive feature necessarily causes the target

B. Prediction uses no historical data

C. A variable can improve prediction without being a cause

D. Causal inference never requires assumptions

Correct Answer: C. A variable can improve prediction without being a cause

Question 9

Which hyperparameter interaction is characteristic of gradient boosting?

A. A smaller learning rate often requires more boosting iterations

B. A smaller learning rate always requires fewer trees

C. Learning rate affects only feature scaling

D. The number of iterations has no relationship to overfitting

Correct Answer: A. A smaller learning rate often requires more boosting iterations

Question 10

Why might a business select a slightly less accurate model?

A. It has no need for monitoring

B. It may offer better stability, explainability, or operational feasibility

C. It guarantees causal conclusions

D. It eliminates all fairness risks

Correct Answer: B. It may offer better stability, explainability, or operational feasibility