Statistical Learning and Model Evaluation

Introduction

Machine learning becomes useful in business only when a model performs reliably on future cases and improves a decision. A model that explains historical data perfectly may fail when customer behaviour, prices, competitors, or economic conditions change. Statistical learning provides the concepts needed to understand this problem: uncertainty, sampling, inference, generalisation, bias, variance, and the disciplined separation of training, validation, and test data.

Model evaluation is not merely a technical exercise. A classifier may achieve high accuracy while missing nearly every fraudulent transaction. A demand model may have a low average error but systematically underforecast products whose stockouts are costly. Consequently, analysts must distinguish statistical performance from economic value. This chapter develops a framework for doing so and applies it to model selection, regularisation, cross-validation, hyperparameter tuning, and business decisions.

Learning objectives

By the end of this chapter, you should be able to:

• explain the statistical learning problem and the role of probability and inference;
• distinguish empirical performance from expected performance on new observations;
• diagnose underfitting, overfitting, bias, and variance;
• design appropriate training, validation, and test procedures;
• apply cross-validation and hyperparameter tuning without contaminating evaluation;
• explain how regularisation controls model complexity;
• select metrics that reflect business consequences;
• interpret confidence, uncertainty, fairness, and data-governance issues; and
• distinguish predictive performance from economic value.

1. The statistical learning problem

Let X denote a vector of input variables, such as customer tenure, transaction amount, or advertising exposure, and let Y denote an outcome, such as churn, sales, or default. A statistical learning model estimates a relationship between them:

Here, f is the unknown systematic relationship and ε is random noise or unmeasured influence. The equation is a conceptual data-generating model rather than a claim that all relationships are linear. It assumes that the observed outcome contains both explainable structure and irreducible uncertainty.

A fitted model, written f̂, is constructed from a sample of observations. For a new input X₀, the prediction is f̂(X₀). The central objective is good performance on observations not used to fit the model. This property is called generalisation.

Supervised and unsupervised learning

In supervised learning, the training data contain both inputs and known outcomes. Regression predicts a numerical outcome, while classification predicts a category or probability. In unsupervised learning, outcomes are not supplied; the algorithm seeks structure such as clusters or lower-dimensional representations. This chapter concentrates on supervised learning because its evaluation requires comparison between predictions and observed outcomes.

Probability and conditional prediction

Business data are usually uncertain. A customer with a given profile does not have a guaranteed outcome. Classification models often estimate a conditional probability:

This is the probability that the event Y = 1 occurs given inputs X. A decision threshold converts probability into an action. For example, a company might contact customers when p(X) ≥ 0.40, rather than using the conventional threshold of 0.50. The appropriate threshold depends on the costs and benefits of decisions.

For a continuous outcome, a common loss function is squared error:

where y is the observed value and ŷ is the prediction. The expected prediction risk is:

The expectation is over the population of future observations. It is generally unknown, so analysts estimate it using a representative evaluation sample.

2. Inference, estimation, and generalisation

Statistical inference uses sample information to reason about a broader population. A sample mean, for instance, estimates a population mean. Likewise, an evaluation metric calculated on a test set estimates the model's future performance. The estimate is itself uncertain because a different sample would produce a different result.

For independent test observations, the sample mean loss is:

Here, n_test is the number of test observations. This estimate is most informative when the test observations are independent of model development and are drawn from the same target population as future cases. If observations are time-dependent, geographically clustered, duplicated, or selected through a biased process, the estimate may be misleading.

A confidence interval expresses sampling uncertainty around an estimate. Under suitable conditions, an approximate 95% interval for a sample mean is:

where SE(mean) is the standard error. This approximation relies on sufficiently independent observations and an appropriate sampling distribution; it is not automatically valid for clustered or heavily dependent data. For classification metrics such as recall, bootstrap intervals or specialised methods may be preferable.

Statistical significance and practical importance are different. With a very large dataset, a tiny improvement may be statistically detectable but economically irrelevant. Conversely, a meaningful improvement may be difficult to detect with a small sample. Model comparison should therefore report effect size, uncertainty, operational constraints, and financial consequences.

3. Bias, variance, and the generalisation problem

A model can fail because it is too simple, too sensitive to the training sample, or affected by unavoidable noise. The bias–variance framework formalises this trade-off for squared-error prediction.

Suppose repeated training samples are drawn from the same population. At a fixed input x₀, let E[f̂(x₀)] denote the average prediction across these fitted models. The expected squared prediction error can be decomposed as:

The terms are:

• σ²: irreducible noise variance, which cannot be eliminated using the available inputs;
• [E(f̂(x₀)) − f(x₀)]²: squared bias, representing systematic error from restrictive assumptions; and
• Var(f̂(x₀)): variance, representing sensitivity to the particular training sample.

The decomposition assumes squared-error loss and a fixed input. It is a conceptual guide, not a complete explanation of every metric or algorithm.

A highly constrained linear model may have high bias and low variance. A very deep decision tree may have low training error but high variance. Regularisation, more data, feature engineering, early stopping, and ensemble methods can alter this balance.

Underfitting and overfitting

Underfitting occurs when a model cannot capture important structure. Training and validation performance are both poor. Overfitting occurs when a model captures random peculiarities in training data. Training performance is strong, but validation or test performance is weak.

A useful diagnostic is to plot learning curves: training and validation error against training-set size or model complexity. If both errors remain high and close together, the model may have high bias. If training error is low while validation error is much higher, variance or leakage may be present.

4. Data partitioning and leakage control

Training, validation, and test sets

The training set estimates model parameters. The validation set supports choices such as algorithm, feature set, regularisation strength, or classification threshold. The test set provides a final, approximately unbiased estimate after development decisions are complete.

A typical workflow is:

1. define the target population, outcome horizon, and business decision;
2. split data using a design appropriate to time, groups, and classes;
3. fit candidate models using training data;
4. use validation data or cross-validation to select a model and hyperparameters;
5. freeze the procedure;
6. evaluate once on the untouched test set; and
7. assess economic value and operational risks.

The exact proportions are context-dependent. A random 70:15:15 split is not a universal rule. Small datasets may require cross-validation, while large datasets can support separate holdout sets. Time-dependent problems should usually use earlier observations for training and later observations for validation and testing.

Data leakage

Leakage occurs when information unavailable at prediction time influences model training or evaluation. Examples include using a post-sale cancellation field to predict whether a sale will occur, scaling all data before splitting, or allowing records from the same household to appear in both training and test sets.

All learned preprocessing—imputation values, scaling parameters, feature-selection decisions, and encoding dictionaries—must be estimated within the training portion of each split. In cross-validation, preprocessing belongs inside the pipeline so that each validation fold remains unseen during fitting.

Stratification and grouped splitting

For imbalanced classification, stratified splitting preserves approximately the same class proportions across partitions. When several rows belong to one customer, patient, account, or store, grouped splitting prevents information from the same entity appearing in both development and evaluation data. These methods address different problems and may need to be combined.

5. Cross-validation and hyperparameter tuning

In k-fold cross-validation, the training data are divided into k approximately equal folds. Each fold is used once as validation data while the remaining k − 1 folds are used for fitting. The cross-validation estimate is:

where L_j is the loss on fold j. Common choices include 5-fold or 10-fold cross-validation. Larger k uses more data for each fit but can be computationally expensive and may produce more correlated validation estimates.

Cross-validation is especially useful when the development sample is not large enough to sacrifice a substantial validation set. It does not remove bias from a poor data-generating process, leakage, or a mismatch between historical and future populations.

Nested cross-validation

If the same cross-validation results are used both to select hyperparameters and to report final performance, the reported estimate can be optimistic. Nested cross-validation separates these tasks. An outer loop estimates performance, while an inner loop selects hyperparameters. It is useful for honest comparative studies, though operational teams often use a final untouched test set instead.

Hyperparameters

Parameters are learned from data, such as regression coefficients. Hyperparameters are choices specified before or around training, such as tree depth, number of neighbours, learning rate, or regularisation strength. A tuning procedure evaluates candidate values using validation data or inner cross-validation.

The selection criterion must reflect the task. For example, tuning for accuracy while the deployment objective is expected profit can select the wrong model. Tuning and threshold selection should be performed without using the final test set.

6. Regularisation and model complexity

Regularisation adds a penalty for complexity to the training objective. In linear regression, ridge regression minimises:

Here, n is the number of training observations, p is the number of predictors, β₀ is the intercept, βⱼ are coefficients, and λ ≥ 0 controls the penalty. Predictors are commonly standardised before penalisation so that coefficient shrinkage is comparable across units. The intercept is usually not penalised.

When λ = 0, the objective reduces to ordinary least squares. As λ increases, coefficients are shrunk toward zero, often reducing variance at the cost of some bias.

Lasso regression uses an absolute-value penalty:

The lasso can set some coefficients exactly to zero, creating a sparse model. With strongly correlated predictors, its selected variables can be unstable. Elastic net combines both penalties and may be preferable when predictors are numerous and correlated.

Regularisation does not guarantee better results. Its strength must be selected using validation or cross-validation, and the penalty must be appropriate to the model and objective.

7. Metrics for regression and classification

No metric is universally best. The metric should represent the decision, the costs of errors, and the distribution of outcomes.

Regression metrics

Mean absolute error is:

It is expressed in the units of the target and is less sensitive to extreme errors than squared-error metrics.

Mean squared error is:

Root mean squared error is RMSE = √MSE. It has the target's units but penalises large errors more strongly. Mean absolute percentage error can be problematic when actual values are zero or close to zero. For intermittent demand, scaled errors or business-specific stockout and overstock costs may be more meaningful.

Classification metrics

For binary classification, define true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN). Accuracy is:

Recall or sensitivity is TP/(TP + FN), while precision is TP/(TP + FP). Specificity is TN/(TN + FP). The F1 score is the harmonic mean of precision and recall:

These measures depend on a classification threshold. A confusion matrix should therefore accompany threshold-sensitive metrics.

The receiver operating characteristic curve examines true-positive rate against false-positive rate across thresholds; its area, ROC-AUC, measures ranking ability. In highly imbalanced problems, precision–recall curves may be more informative. A model can have a high ROC-AUC yet produce too few useful positive predictions at the operating range that matters.

Calibration asks whether predicted probabilities correspond to observed frequencies. Among cases assigned a probability near 0.70, approximately 70% should experience the event if the model is well calibrated. Calibration plots, reliability measures, and recalibration may be important in credit, health, and risk applications.

Worked example: threshold choice

Suppose 100 customers are evaluated. At a chosen threshold, TP = 18, FN = 2, FP = 20, and TN = 60. Then:

• accuracy = (18 + 60)/100 = 0.78, or 78%;
• recall = 18/(18 + 2) = 0.90, or 90%;
• precision = 18/(18 + 20) ≈ 0.474, or 47.4%; and
• specificity = 60/(60 + 20) = 0.75, or 75%.

The model identifies most positive cases, but more than half of its alerts are false positives. Whether this is acceptable depends on the cost of contacting an uninterested customer and the value of retaining a true churn-risk customer.

8. From statistical performance to economic value

A predictive model creates value only through a decision. Let an action produce benefit B when a true positive is acted upon, cost C_FP for a false positive, cost C_FN for a missed positive, and cost C_TN for a true-negative action or non-action, depending on the setting. Expected value can be calculated from the confusion matrix:

The terms must be defined consistently; some applications have no cost for TN, while others have opportunity costs. This formula is useful for comparing thresholds and policies, but it assumes the estimated costs and benefits are credible and that actions do not change future outcomes in complicated ways.

A probability threshold should reflect expected utility, not a default convention. If intervention cost is C and successful intervention yields net benefit V, a simplified rule is to act when the estimated probability exceeds C/(V + C), assuming calibrated probabilities, constant costs, and no capacity constraint. Real businesses often face limited staff, treatment effects, delayed outcomes, and heterogeneous customer value, so direct profit simulation is preferable.

Incremental evaluation is particularly important. A churn model may identify customers likely to leave, but contacting customers who would have stayed anyway may waste money. The relevant question may be which customers are persuadable by an intervention, requiring experiments or uplift modelling rather than ordinary risk prediction.

9. Illustrative case: subscription churn intervention

Background

The following is a fictional illustrative case. A subscription company wants to reduce monthly cancellations. It has 80,000 historical customer-month records containing tenure, service usage, payment history, support contacts, and a cancellation outcome in the following 30 days.

Problem

Management initially proposes maximising accuracy. However, only 8% of customer-months result in cancellation. A model that predicts “no cancellation” for everyone would achieve 92% accuracy and deliver no useful retention targeting.

Analysis

The analytics team defines the prediction time clearly: only information available at the start of the month may be used. It assigns customers, rather than rows, to development and test groups to avoid repeated-customer leakage. Because the business will deploy the model next quarter, the most recent three months are reserved as a temporal test set. Earlier data are split into training and validation components.

The team compares logistic regression with ridge regularisation, a random forest, and gradient boosting. Imputation, scaling, and categorical encoding are fitted inside each cross-validation pipeline. Hyperparameters are selected using stratified five-fold cross-validation, with area under the precision–recall curve as one criterion and simulated contribution margin as another.

The final model is evaluated on the untouched temporal test set. At the selected operating threshold, it identifies 45% of cancellers, contacts 12% of customers, and has 31% precision. These statistics are not sufficient to approve deployment. The firm estimates that a retention offer costs £8, a retained customer contributes £45 in incremental margin, and only some contacted customers are persuadable.

Approach and solution

The company runs a randomised holdout experiment among eligible high-risk customers: some receive the offer and others do not. It measures incremental retention and net contribution rather than attributing every retained customer to the model. The model is deployed only where expected value is positive and contact capacity permits. A monitoring dashboard tracks calibration, recall, precision, customer segments, offer costs, and realised margin.

Outcome

The predictive model improves targeting, but the experiment reveals that customers with the highest predicted risk are not necessarily the most responsive to an offer. The company therefore revises its objective from “predict cancellation” to “prioritise profitable intervention,” combining risk, expected treatment effect, and customer value.

Learning points

First, class imbalance makes accuracy inadequate. Second, temporal and customer-level splitting can be more credible than a random row split. Third, cross-validation supports model selection but cannot substitute for a deployment experiment. Finally, economic value depends on actions, costs, capacity, and causal effects, not only on predictive scores.

10. Governance, ethics, and operational practice

Evaluation data can encode historical discrimination. A model may show strong overall performance while producing materially lower recall or calibration for a protected group. Analysts should report subgroup confusion matrices, calibration, sample sizes, and uncertainty where legally and ethically appropriate. Fairness criteria can conflict; improving one measure may worsen another, especially when base rates differ. The chosen criterion must be justified by the application and governed transparently.

Privacy requires data minimisation, lawful processing, access controls, retention limits, and secure handling of identifiers. Sensitive variables may be excluded from prediction but still needed for auditing; this distinction should be governed carefully. Explainability is not a promise that every complex model has a simple causal explanation. Feature importance describes association with predictions, not necessarily causal influence.

Production evaluation should monitor data drift, concept drift, missingness, latency, calibration, subgroup performance, and business outcomes. A model's test score can decay when prices, policies, customer composition, or behaviour change. Retraining should be controlled, documented, and validated rather than triggered blindly.

Best-practice evaluation workflow

A robust workflow has the following characteristics:

1. specify the decision, prediction horizon, population, and costs before modelling;
2. prevent leakage through time-aware, group-aware, and pipeline-based design;
3. select metrics that reflect both statistical quality and business consequences;
4. use validation or nested procedures for model and hyperparameter selection;
5. preserve an untouched test set or outer evaluation loop;
6. report uncertainty, baselines, subgroup results, and practical effect sizes;
7. calibrate probabilities when decisions rely on risk estimates;
8. evaluate thresholds under capacity and cost constraints;
9. test interventions when the model changes customer or employee treatment; and
10. monitor the deployed system and establish conditions for review or withdrawal.

Chapter Summary

Statistical learning estimates relationships between uncertain inputs and outcomes, with the central goal of generalising to future observations. Expected risk is approximated from data, but its estimate depends on representative sampling, independence, and leakage-free evaluation. The bias–variance framework explains why overly simple models underfit and overly flexible models overfit.

Training data fit parameters, validation procedures guide choices, and an untouched test set provides final evidence. Cross-validation is valuable for limited data, while nested cross-validation or a final holdout protects against optimistic selection. Regularisation controls complexity by shrinking coefficients or otherwise restricting models.

Regression and classification require different metrics, and threshold-dependent measures must be interpreted through confusion matrices and operating conditions. Accuracy is often unsuitable for imbalanced classes. Calibration assesses probability reliability, while economic analysis connects predictions to costs, benefits, capacity, and causal intervention effects. Ethical evaluation requires attention to subgroup performance, privacy, transparency, drift, and governance.

Key Terms

Statistical learning: Estimating relationships between inputs and outcomes from data.

Generalisation: Performance on new observations from the target population.

Loss function: A rule assigning a cost to prediction error.

Risk: Expected loss over future observations.

Bias: Systematic prediction error caused by restrictive assumptions or model structure.

Variance: Sensitivity of a fitted model to its training sample.

Overfitting: Strong training performance accompanied by weak performance on new data.

Underfitting: Failure to capture important structure in the data.

Data leakage: Use of information in development that would not be available at prediction time.

Training set: Data used to estimate model parameters.

Validation set: Data used to select models, hyperparameters, or thresholds.

Test set: Held-out data used for final performance estimation.

Cross-validation: Repeated fitting and validation across data folds.

Hyperparameter: A modelling choice selected outside the parameter-fitting process.

Regularisation: A complexity-control method that penalises model flexibility.

Precision: The proportion of predicted positives that are true positives.

Recall: The proportion of actual positives that are identified.

Calibration: Agreement between predicted probabilities and observed frequencies.

Concept drift: Change in the relationship between inputs and outcomes over time.

Economic value: Net benefit created by decisions informed by a model.

Review Questions

Short Answer Questions

1. What is the difference between a training set, validation set, and test set?

1. Define bias and variance in the context of predictive modelling.

1. Why can accuracy be misleading for an imbalanced classification problem?

1. What is data leakage? Give one example.

1. What does calibration mean for a probabilistic classifier?

Long Answer Questions

1. Explain the bias–variance decomposition under squared-error loss and discuss how regularisation affects it.

1. Compare MAE, RMSE, precision, recall, F1, ROC-AUC, and calibration as evaluation tools. Identify situations in which each is useful.

1. Describe a leakage-resistant workflow for selecting and evaluating a machine-learning model.

1. Explain how cross-validation supports hyperparameter tuning and why nested cross-validation may be necessary.

1. Discuss why predictive performance and economic value may diverge, using a business example.

Analytical / Application Questions

1. A fraud model produces TP = 90, FN = 10, FP = 140, and TN = 760. Calculate accuracy, precision, recall, and specificity. Which metric would you prioritise if missed fraud costs substantially more than investigating a false alert?

1. A retailer's demand model has an MAE of 12 units and an RMSE of 28 units. What does the difference suggest about its errors, and what additional business information would you request before approving deployment?

1. Design an evaluation plan for a churn model in which each customer appears in multiple monthly records and the company will deploy the model next quarter. Specify the split strategy, leakage controls, validation method, metrics, and economic test.

Multiple Choice Questions with Answers

Question 1

Which dataset should be used for the final, approximately unbiased estimate after model and hyperparameter decisions are complete?

A. Training set

B. Validation set

C. Test set

D. Synthetic set

Correct Answer: C. Test set

Question 2

In the bias–variance decomposition for squared-error prediction, which component represents unpredictable noise?

A. Squared bias

B. Model variance

C. Irreducible noise variance

D. Validation error

Correct Answer: C. Irreducible noise variance

Question 3

A classifier has TP = 30, FN = 10, FP = 20, and TN = 40. What is its recall?

A. 0.50

B. 0.60

C. 0.75

D. 0.80

Correct Answer: C. 0.75

Question 4

What is the primary purpose of regularisation?

A. To increase data leakage

B. To control model complexity and reduce overfitting

C. To guarantee causal conclusions

D. To remove the need for testing

Correct Answer: B. To control model complexity and reduce overfitting

Question 5

Which practice best prevents preprocessing leakage during cross-validation?

A. Fit preprocessing on the complete dataset before splitting

B. Fit preprocessing separately within each training fold

C. Use the test set to calculate missing-value replacements

D. Delete all categorical variables

Correct Answer: B. Fit preprocessing separately within each training fold

Question 6

Why may precision–recall analysis be more informative than accuracy for rare-event classification?

A. It ignores false positives and false negatives

B. It focuses on positive predictions and their recovery of actual positives

C. It always produces higher scores

D. It does not depend on a threshold

Correct Answer: B. It focuses on positive predictions and their recovery of actual positives

Question 7

A well-calibrated group of cases assigned predicted probability 0.70 should, approximately, experience the event at what frequency?

A. 7%

B. 30%

C. 70%

D. 100%

Correct Answer: C. 70%

Question 8

Which splitting strategy is most appropriate when future deployment predicts later periods from earlier data?

A. Randomly mix all dates

B. Use a time-aware split with later observations held out

C. Duplicate recent observations in training

D. Use only positive cases for testing

Correct Answer: B. Use a time-aware split with later observations held out

Question 9

If a false positive costs £8 and a true positive produces £45 of net benefit, under the simplified threshold rule in the chapter, what probability threshold is implied?

A. 0.08

B. 0.15

C. 0.36

D. 0.85

Correct Answer: B. 0.15

Question 10

Why might a high-risk churn customer not be the best customer to target with a retention offer?

A. Risk prediction always has zero value

B. High risk does not necessarily imply responsiveness to the intervention

C. Precision and recall are identical

D. Test sets cannot contain churners

Correct Answer: B. High risk does not necessarily imply responsiveness to the intervention