14. Statistical Learning and Model Evaluation

Introduction and Context

Statistical learning provides the bridge between statistical inference and machine learning. Its central concern is not merely whether a model describes observed data, but whether it can make reliable predictions or decisions for new cases. This distinction changes how models are constructed, assessed, interpreted, and governed.

A model fitted to historical data may appear highly accurate while failing in practice because it has memorised noise, learned from information that would not be available at prediction time, or been evaluated on data that are too similar to the training records. Conversely, a slightly less complex model may perform more reliably because it captures stable relationships rather than accidental patterns. Statistical learning therefore requires a disciplined treatment of loss, uncertainty, validation, generalisation, and the consequences of errors.

For data scientists, model evaluation is also a decision problem. The appropriate metric depends on the cost of false positives and false negatives, the prevalence of the outcome, the quality of probability estimates, and the intended operational use. In high-stakes settings, evaluation must additionally consider subgroup performance, fairness, explainability, robustness, privacy, and human oversight.

Learning Objectives

By the end of this chapter, students should be able to:

1. Explain empirical risk, population risk, loss functions, and generalisation error.

2. Interpret the bias–variance trade-off and relate it to model complexity.

3. Distinguish validation, test, cross-validation, bootstrap, and time-aware evaluation.

4. Select suitable regression and classification metrics for a stated decision context.

5. Explain regularisation, feature selection, calibration, and model interpretability.

6. Identify leakage, overfitting, dataset shift, and other evaluation failures.

7. Design an uncertainty-aware and ethically responsible evaluation framework.

8. Apply these ideas to a realistic data-science case.

From Inference to Prediction: Risk and Loss

In classical inference, an analyst may ask whether a coefficient differs from zero, estimate a population parameter, or construct a confidence interval. Statistical learning often asks a related but distinct question: how well will a fitted function perform on future observations drawn from a target population?

Let an observation be represented by predictors X and response Y. A prediction rule is written as f(X). A loss function L(Y, f(X)) quantifies the consequence of a prediction. The population risk is

R(f) = E[L(Y, f(X))],

where the expectation is taken over the data-generating distribution. Because this distribution is unknown, it is estimated using a sample. The empirical risk is

R_n(f) = (1/n) Σ L(y_i, f(x_i)).

Empirical risk minimisation chooses a model that performs well on observed data. However, minimising empirical risk without controlling complexity can produce a model that fits random variation. The practical objective is low population risk, often called good generalisation.

The loss function embodies a decision context. Squared-error loss, (y − ŷ)^2, heavily penalises large errors and is common in regression. Absolute-error loss, |y − ŷ|, is less sensitive to extreme observations. In binary classification, zero–one loss records whether a class is correctly predicted, while log loss evaluates the quality of predicted probabilities:

−[y log(p) + (1 − y) log(1 − p)].

A model assigning probability 0.99 to an event that does not occur receives a much larger penalty than a model assigning probability 0.60. Thus, accuracy and probability quality are not interchangeable.

Important terminology

A parameter is a quantity describing a model or population that is estimated from data. A hyperparameter is a setting chosen during model development, such as a tree depth or regularisation strength. Training data are used to estimate model parameters; validation data support model or hyperparameter selection; test data provide a final, ideally untouched, estimate of performance. A prediction target is the outcome the model is intended to estimate. A decision threshold converts a predicted probability or score into an action or class.

The Bias–Variance Trade-off

Prediction error can be understood through the balance between bias and variance. Bias is systematic error caused by an overly restrictive model. For example, a linear model may have high bias when the true relationship is strongly nonlinear. Variance is sensitivity to the particular sample used for training. A highly flexible model may change substantially when the training sample changes.

For squared-error prediction, expected test error can be expressed conceptually as:

Expected error = irreducible noise + bias² + variance.

Irreducible noise arises from randomness or unmeasured influences and cannot be eliminated merely by choosing another algorithm. Increasing model complexity generally reduces training error and may reduce bias, but often increases variance. At first, this improves test performance; beyond an appropriate level, overfitting causes test error to rise.

Underfitting occurs when a model is too simple to represent important structure. Overfitting occurs when it captures noise, outliers, or sample-specific regularities. The best model is therefore not necessarily the one with the lowest training error or the largest number of parameters. It is the model with an appropriate expected performance on future data, subject to computational, interpretive, and operational constraints.

The trade-off is not a fixed property of an algorithm. It depends on sample size, noise, feature quality, model class, regularisation, and the similarity between training and deployment populations. More data can reduce variance, but additional data from the wrong population may not improve deployment performance.

Controlling Complexity: Regularisation and Feature Selection

Regularisation modifies the fitting objective by adding a penalty for model complexity. In linear regression, ridge regression minimises

Σ(y_i − ŷ_i)^2 + λ Σ β_j²,

where λ controls the strength of the penalty. Ridge shrinks coefficients toward zero and is especially useful when predictors are numerous or correlated. Lasso regression uses an absolute-value penalty:

Σ(y_i − ŷ_i)^2 + λ Σ |β_j|.

Because the absolute-value penalty can set some coefficients exactly to zero, lasso can perform embedded feature selection. Elastic net combines ridge and lasso penalties and can be useful when correlated predictors occur in groups.

The penalty parameter must be selected using training and validation procedures, not by inspecting test performance repeatedly. Predictors should generally be standardised before penalised regression when their scales differ; otherwise, the penalty affects variables unevenly.

Feature selection may be filter-based, wrapper-based, or embedded. Filter methods rank variables using criteria such as correlation or mutual information before model fitting. Wrapper methods evaluate subsets according to model performance. Embedded methods, such as lasso or tree-based selection, perform selection during training. Feature selection can improve efficiency and interpretability, but selecting variables using the full dataset before cross-validation causes leakage and produces optimistic estimates. Selection must occur within each training fold.

Regularisation is not a substitute for sound data design. A leaked variable, a biased sample, or an incorrectly defined target cannot be repaired by tuning λ.

Validation as a Design Problem

A random train–test split is appropriate only when observations are reasonably independent and identically distributed and when future cases resemble randomly selected current cases. In many applications, these assumptions are false.

A basic holdout procedure divides data into training and test sets. When data are limited, k-fold cross-validation divides the training data into k folds, fits the model on k−1 folds, and evaluates it on the remaining fold. The process is repeated so that each observation is used for validation once. The mean score estimates expected performance, while variation across folds indicates instability.

Stratified cross-validation preserves the approximate class proportion in each fold and is useful for imbalanced classification. Grouped cross-validation keeps records from the same person, household, organisation, or experiment in one fold, preventing related observations from appearing in both training and validation sets. Time-series evaluation must respect temporal order: models are trained on earlier periods and tested on later periods. Randomly mixing future observations into training can create an unrealistic and invalid estimate.

Nested cross-validation separates hyperparameter selection from performance estimation. An inner loop selects the model or hyperparameters; an outer loop estimates performance. This is valuable when many alternatives are considered and data are limited. Bootstrapping, which repeatedly samples observations with replacement, can estimate variability, confidence intervals, or optimism, although its assumptions must be considered carefully.

A robust evaluation pipeline includes preprocessing inside the resampling procedure. Imputation, scaling, encoding, feature selection, and dimensionality reduction must be learned from the training portion of each fold and then applied to its validation portion. This prevents information from the validation data entering model construction.

Choosing Metrics That Match the Decision

Regression

Mean squared error (MSE) is the average squared prediction error, while root mean squared error (RMSE) returns the measure to the response scale. RMSE gives substantial influence to large errors. Mean absolute error (MAE) is easier to interpret and more robust to outliers. Mean absolute percentage error can be misleading when actual values are near zero. R-squared measures the proportion of sample variation explained relative to a baseline, but it is not a universal measure of predictive accuracy and can be negative on test data.

Metrics should be reported with uncertainty, such as confidence intervals or fold-to-fold variation. A small difference in average RMSE may not justify choosing a more complex or less interpretable model.

Classification

A confusion matrix contains true positives, true negatives, false positives, and false negatives. Accuracy is the proportion of correct classifications, but it can be deceptive under class imbalance. Precision is TP/(TP+FP), and recall or sensitivity is TP/(TP+FN). Specificity is TN/(TN+FP). The F1 score is the harmonic mean of precision and recall and is useful when both matter, although it ignores true negatives and probability calibration.

The receiver operating characteristic (ROC) curve displays sensitivity against false-positive rate across thresholds; its area under the curve (ROC-AUC) measures ranking ability across thresholds. Precision–recall curves are often more informative when the positive class is rare. Neither AUC directly specifies the operational threshold or the costs of errors.

A cost-sensitive evaluation may define expected cost as

C = c_FP P(FP) + c_FN P(FN) + c_TP P(TP) + c_TN P(TN),

with costs assigned according to the application. Threshold selection should follow the decision objective, not an automatic preference for 0.5.

Ranking, probability, and operational metrics

When a model prioritises cases, metrics such as precision at k, recall at k, lift, or gains may be appropriate. When probabilities support resource allocation, risk communication, or triage, calibration is essential. A model is calibrated if, among cases assigned probability p, approximately p proportion experience the event. Calibration can be assessed using reliability diagrams, calibration intercept and slope, and proper scoring rules such as log loss or the Brier score.

Discrimination and calibration answer different questions. A model may rank high-risk cases correctly but systematically overestimate their absolute risk. Conversely, a calibrated model may provide useful probabilities even with modest discrimination. Recalibration methods include Platt scaling and isotonic regression, but they require separate validation data or careful cross-validation.

Uncertainty, Stability, and Statistical Comparison

A single test score conceals uncertainty. Sources include finite sample size, sampling variation, random initialisation, hyperparameter choices, and changes in the deployment population. Report means with intervals or distributions across resamples where feasible. When comparing models, evaluate paired differences on the same folds or cases rather than comparing unrelated averages.

Prediction uncertainty can be separated conceptually into aleatoric uncertainty, arising from inherent randomness, and epistemic uncertainty, arising from limited knowledge or data. Prediction intervals in regression describe uncertainty around a future observation, whereas confidence intervals describe uncertainty about an estimated parameter or mean response. They should not be confused.

For probabilistic machine learning, uncertainty estimates should be tested for calibration and sharpness. A narrow interval that frequently misses the true value is not useful. In high-risk applications, abstention or referral to a human may be preferable when uncertainty is high or the input is outside the training distribution.

Explainability and Responsible Evaluation

Explainability has several meanings. A transparent model, such as a small decision tree or a regularised linear model, can be understood directly. Post-hoc methods explain predictions from complex models. Global explanations describe average model behaviour; local explanations describe an individual prediction. Feature importance, partial dependence, accumulated local effects, and Shapley-based methods can be informative, but each has assumptions and possible failure modes.

An explanation is not necessarily a causal explanation. A high feature importance may reflect correlation, proxy information, or data collection practice rather than a variable’s causal influence. Explanations can also be unstable when correlated features compete for importance. They should therefore be checked for consistency, plausibility, and sensitivity to perturbations.

Responsible evaluation includes subgroup analysis. Overall accuracy can conceal poor performance for a minority group. Analysts should compare relevant error rates, calibration, coverage, and threshold effects across groups, while recognising that fairness criteria may conflict mathematically and that group labels may be incomplete or sensitive. The choice of fairness criterion must be connected to the social and institutional context.

Privacy, data minimisation, access control, documentation, and secure handling of sensitive attributes are part of professional model evaluation. In regulated settings, records should support traceability of data sources, model versions, decisions, validation results, and changes after deployment. An evaluation that ignores foreseeable harm is statistically incomplete, even if its numerical score is high.

A Practical Evaluation Workflow

A sound workflow begins by defining the prediction target, prediction time, eligible population, and intended action. The analyst then specifies the loss and metrics before examining results. Data are audited for missingness, duplicates, leakage, sampling bias, temporal structure, and shifts in measurement.

Next, a simple baseline is established. Examples include predicting the training mean in regression, the majority class in classification, or a well-calibrated historical rate. Candidate models are then fitted through a reproducible pipeline. Resampling reflects the anticipated deployment process, and all preprocessing is nested within training folds.

The analyst reports primary and secondary metrics, uncertainty, subgroup results, calibration, computational cost, and operational constraints. Thresholds are selected using validation data and documented. The untouched test set is used once, or as sparingly as possible, for final confirmation. Before deployment, stress tests examine missing inputs, distribution shift, adversarial or unusual cases, and plausible changes in costs.

After deployment, evaluation continues through monitoring of data drift, prediction drift, calibration, performance where labels become available, subgroup outcomes, and human override patterns. A model is not finished when it is deployed; it becomes part of a socio-technical system that must be reviewed over time.

Applied Case: Predicting Hospital Readmission

Background

The following is an illustrative case. A hospital wishes to identify adult patients who are likely to be readmitted within 30 days after discharge. The intended use is to offer additional follow-up support, not to deny treatment.

Situation and problem

An initial team trains a gradient-boosting model on five years of electronic health-record data. The reported accuracy is 92%, and the team recommends deployment. However, only 8% of patients are readmitted. A model that predicts “no readmission” for everyone would achieve 92% accuracy, so the reported result provides little evidence of usefulness.

Further inspection identifies a serious leakage variable: a discharge-planning code entered after clinicians had already arranged follow-up for patients considered high risk. This code would not be available at the intended prediction time for all patients. The original random split also places multiple admissions from the same patient in both training and test sets.

Analysis and revised strategy

The team first defines the prediction time as the point at which discharge planning begins and removes variables recorded afterward. It uses a grouped, temporal split so that patient records do not cross partitions and later periods are evaluated as future data. A logistic regression with regularisation, a calibrated gradient-boosting model, and a historical-risk baseline are compared.

Because the intervention team can contact only 15% of discharged patients, recall at the top 15% of ranked cases becomes an operational metric. Precision at 15%, calibration, subgroup recall, and expected intervention cost are also reported. Cross-validation is performed within the development period, with imputation and feature selection fitted separately within each fold. The final test period remains untouched until the design is fixed.

The model’s probabilities are recalibrated on a validation period. A threshold is selected according to the relative costs of missed high-risk patients, unnecessary follow-up, and staff capacity. Cases with unusual patterns or low-confidence predictions are referred for manual review rather than treated as automatically high or low risk.

Outcome

The revised evaluation produces lower overall accuracy than the original report but gives a more credible estimate of deployment performance. It reveals that performance varies across age and language-access groups, prompting additional data-quality review and targeted monitoring. The hospital decides to run a prospective pilot with human oversight rather than immediately automate decisions.

Learning points

The case illustrates that an apparently strong metric can be invalid because of imbalance, leakage, dependence between observations, or mismatch between evaluation and deployment. It also shows why ranking, calibration, capacity constraints, subgroup analysis, and prospective monitoring are often more relevant than accuracy alone.

Common Failure Modes and Best Practices

Frequent errors include repeatedly tuning against the test set, selecting features before cross-validation, using random splits for temporal data, reporting only accuracy, ignoring the baseline, and interpreting association as causation. Other failures include treating AUC as a complete evaluation, presenting unstable feature explanations as facts, and deploying probability estimates without calibration.

Best practice is to pre-specify the target and evaluation plan, use leakage-resistant pipelines, preserve an honest test set, compare against simple baselines, report uncertainty, and align metrics with decisions. Evaluation should include calibration, subgroup analysis, robustness checks, and documentation. Simpler models should be preferred when they provide comparable performance with lower operational and interpretive risk. Finally, deployment should include monitoring, version control, clear ownership, and a process for model withdrawal or revision.

Current Developments

Contemporary evaluation increasingly addresses distribution shift, foundation models, generative AI, and automated machine-learning systems. In shifting environments, static test sets may become obsolete; rolling-origin evaluation, drift detection, and continual recalibration become important. For generative systems, evaluation must consider factuality, task success, abstention, toxicity, privacy leakage, and robustness to prompts, rather than relying on a single similarity score.

Automated model-selection platforms can accelerate experimentation but also increase the risk of hidden multiple comparisons and test-set overuse. Reproducibility therefore requires recording data versions, search spaces, random seeds where relevant, preprocessing, and selection criteria. Uncertainty-aware AI is also gaining importance: systems should communicate confidence, recognise out-of-distribution inputs, and support escalation when automated predictions are unreliable.

Chapter Summary

Statistical learning connects prediction with statistical reasoning through loss functions, risk, generalisation, and uncertainty. The bias–variance trade-off explains why excessive flexibility produces overfitting, while regularisation and carefully designed feature selection control complexity. Valid evaluation depends on separating training, validation, and test roles; using cross-validation appropriately; preventing leakage; and respecting groups and time.

No metric is universally best. Regression, classification, ranking, probability, and operational decisions require different measures. Calibration complements discrimination, while uncertainty intervals and stability analysis qualify reported scores. Explainability, subgroup analysis, privacy, fairness, documentation, and monitoring extend evaluation beyond numerical performance. A responsible data scientist evaluates not only whether a model predicts, but whether it remains reliable, interpretable, appropriate, and safe in its intended context.

Key Terms

Empirical risk: Average loss measured on observed training data.

Population risk: Expected loss over the target data-generating population.

Loss function: Rule that quantifies the consequence of a prediction error.

Generalisation: Ability of a model to perform well on new relevant observations.

Bias: Systematic prediction error caused by restrictive assumptions or model structure.

Variance: Sensitivity of a fitted model to changes in the training sample.

Overfitting: Learning sample-specific noise that harms performance on new data.

Regularisation: Penalisation of model complexity during fitting.

Hyperparameter: Model-development setting selected rather than directly estimated from data.

Data leakage: Inappropriate information flow from validation, test, or future data into training.

Cross-validation: Resampling procedure for estimating performance or selecting models.

Calibration: Agreement between predicted probabilities and observed event frequencies.

Discrimination: Ability to rank cases according to their relative outcome risk.

Precision: Proportion of predicted positive cases that are actually positive.

Recall: Proportion of actual positive cases correctly identified.

Dataset shift: Change in the distribution of data between development and deployment.

Aleatoric uncertainty: Uncertainty arising from inherent randomness in outcomes.

Epistemic uncertainty: Uncertainty arising from limited knowledge or data.

Explainability: Ability to communicate or analyse how a model produces predictions.

Baseline model: Simple reference model used for meaningful performance comparison.

Review Questions

Short Answer Questions

1. Distinguish empirical risk from population risk.

2. What is the purpose of a loss function?

3. Explain why accuracy can be misleading for an imbalanced classification problem.

4. What is calibration, and how does it differ from discrimination?

5. Why must preprocessing be performed within cross-validation folds?

Long Answer Questions

1. Explain the bias–variance trade-off and discuss how regularisation influences it.

2. Compare holdout validation, k-fold cross-validation, nested cross-validation, and bootstrap evaluation.

3. Discuss the strengths and limitations of RMSE, MAE, accuracy, F1 score, ROC-AUC, precision–recall measures, and log loss.

4. Explain how temporal structure, repeated individuals, and dataset shift affect evaluation design.

5. Develop a responsible evaluation framework for a high-stakes predictive model, including uncertainty, fairness, explainability, and monitoring.

Analytical / Application Questions

1. A fraud model has 99.5% accuracy but detects only 20% of fraudulent transactions. Which additional metrics and evaluation design would you use, and why?

2. A hospital model is trained using random cross-validation on five years of records, with several admissions per patient. Identify the likely sources of optimistic bias and propose a revised validation strategy.

3. A model ranks customers accurately but assigns probabilities that are systematically too high. Explain the operational risks and recommend methods for assessing and improving calibration.

22. Multiple Choice Questions with Answers

Question 1

Which quantity represents the expected loss of a prediction rule over the target population?

A. Training error

B. Population risk

C. Validation accuracy

D. Regularisation strength

Correct Answer: B. Population risk

Question 2

A model has very low training error but much higher test error. Which explanation is most appropriate?

A. It is underfitting because its variance is too low

B. It is overfitting sample-specific variation

C. Its irreducible error has disappeared

D. Its test set must be larger than its training set

Correct Answer: B. It is overfitting sample-specific variation

Question 3

Which method can set some linear-model coefficients exactly to zero?

A. Ridge regression

B. Ordinary least squares

C. Lasso regression

D. Principal component scoring

Correct Answer: C. Lasso regression

Question 4

Why is random cross-validation inappropriate for many time-series problems?

A. It always produces too few observations

B. It prevents the use of numerical predictors

C. It can allow future information to influence evaluation of earlier periods

D. It cannot calculate classification metrics

Correct Answer: C. It can allow future information to influence evaluation of earlier periods

Question 5

A classifier is used to select the 10% of cases with the highest predicted risk. Which metric is especially relevant?

A. Precision at 10%

B. Training loss only

C. R-squared

D. Mean absolute error

Correct Answer: A. Precision at 10%

Question 6

A model assigns a probability of 0.8 to a group of cases. What does good calibration imply?

A. The model classifies exactly 80% of those cases correctly

B. Approximately 80% of those cases experience the event

C. The model has an ROC-AUC of 0.8

D. The model has no false positives

Correct Answer: B. Approximately 80% of those cases experience the event

Question 7

Which action is an example of data leakage?

A. Scaling predictors using parameters learned from each training fold

B. Using a discharge code entered after the prediction time as a predictor

C. Comparing a model with a majority-class baseline

D. Reporting performance separately by subgroup

Correct Answer: B. Using a discharge code entered after the prediction time as a predictor

Question 8

Why should patient records be kept within the same fold in grouped cross-validation?

A. To increase the number of model parameters

B. To ensure every fold has identical sample sizes

C. To prevent highly related records from appearing in both training and validation data

D. To guarantee perfect calibration

Correct Answer: C. To prevent highly related records from appearing in both training and validation data

Question 9

Which statement best distinguishes a confidence interval from a prediction interval?

A. A prediction interval concerns a future observation and generally includes outcome noise

B. A confidence interval always has greater width

C. A prediction interval estimates only a model coefficient

D. They are identical under all regression models

Correct Answer: A. A prediction interval concerns a future observation and generally includes outcome noise

Question 10

Why should a model’s explanation not automatically be interpreted as a causal explanation?

A. Explanations cannot be computed for any model

B. Feature importance may reflect correlation or proxy information rather than causal influence

C. Causal relationships are always unrelated to prediction

D. Only accuracy metrics can reveal causation

Correct Answer: B. Feature importance may reflect correlation or proxy information rather than causal influence