Model Evaluation, Validation and Explainability

Introduction

A predictive model is useful only when its reported performance reflects how it will behave on future, unseen data. A model may achieve excellent training accuracy yet fail in deployment because it has memorised noise, benefited from information that would not be available at prediction time, or been evaluated with an inappropriate metric. Model evaluation is therefore not a final administrative step; it is a scientific process for estimating generalisation, comparing alternatives, identifying risks and communicating evidence.

This chapter develops a practical framework for evaluating classification, regression and ranking models. It explains holdout validation, cross-validation and bootstrap methods; discusses leakage, overfitting and statistical significance; and introduces calibration, robustness, fairness and model explainability. The emphasis is on designing evaluations that are valid for the data-generating process and the intended decision.

Learning Objectives

By the end of this chapter, you should be able to:

• design training, validation and test splits for realistic deployment settings;
• select and interpret classification, regression and ranking metrics;
• apply cross-validation and bootstrap estimation appropriately;
• detect leakage, overfitting and sources of optimistic bias;
• compare models using uncertainty intervals and significance tests;
• assess calibration, robustness and group fairness;
• distinguish global from local explanations and identify their limitations; and
• prepare an evaluation report suitable for technical and governance review.

1. Generalisation and the Evaluation Problem

Let D denote the distribution of future examples and let a model f produce a prediction from input x. The expected risk under loss function L is:

Here, y is the true outcome, and the expectation is over the population or future-data distribution. Since D is unknown, practitioners estimate risk using a finite sample. The central assumption is that evaluation data are sufficiently representative of future use. If this assumption fails—for example, because a hospital changes its patient population or a fraudster changes behaviour—ordinary test performance may be misleading.

A useful evaluation distinguishes three roles:

The test set must be treated as a sealed assessment. Repeatedly inspecting it and modifying the model gradually turns it into a validation set, producing selection bias.

Sampling units and dependence

The independent unit may be a person, transaction, patient, household, device or time period. Rows are not automatically independent. Multiple records from one patient, or repeated measurements from one machine, can cause information from the same entity to appear in both training and test sets. Grouped or time-aware splitting is then required.

For a temporal deployment, training must precede testing chronologically. Random splitting can allow future patterns to influence the past and creates an unrealistic estimate. For grouped data, all observations belonging to one group should normally remain in one partition.

2. Holdout Evaluation and Data Splitting

A holdout design partitions data once. A common arrangement is 60–80% for training, 10–20% for validation and 10–20% for testing, although the appropriate proportions depend on sample size, model complexity and the cost of uncertainty.

For classification, stratification preserves approximately the class proportions in each partition. Stratification is useful for imbalanced outcomes, but it does not solve group dependence or temporal drift. A minority class with very few examples may require repeated cross-validation or domain-specific sampling rather than an arbitrary split.

A sound workflow is:

1. define the prediction time and information available at that time;
2. identify groups, time ordering and sampling constraints;
3. create a final test set using the deployment-like split;
4. perform feature engineering and model selection using only the remaining development data;
5. freeze the pipeline and evaluate once on the test set; and
6. report uncertainty, subgroup results and the evaluation protocol.

Leakage

Data leakage occurs when information unavailable at prediction time, or information influenced by the outcome, enters model training or evaluation. Examples include:

• normalising the entire dataset before splitting;
• using a diagnosis entered after a clinical event to predict that event;
• allowing duplicate customers across training and test sets;
• selecting features using correlations computed from the test set; and
• imputing missing values using statistics calculated from all observations.

Preprocessing must be fitted inside each training fold and then applied to its validation fold. A pipeline that combines preprocessing and the estimator is safer than manually transforming data before cross-validation.

3. Cross-Validation and Bootstrap Methods

k-fold cross-validation

In k-fold cross-validation, the development data are divided into k approximately equal folds. Each fold is used once as validation data while the remaining k − 1 folds are used for training. If the validation loss in fold j is L_j, the cross-validation estimate is:

The folds should be formed according to the data structure: stratified folds for classification, grouped folds for clustered observations and rolling or expanding folds for time series. Common choices are k = 5 or k = 10. Larger k uses more training data per fit but is computationally more expensive and can produce more correlated validation estimates. Leave-one-out cross-validation uses k equal to the number of observations, but its estimate can have high variance and may be computationally costly.

Cross-validation is primarily a model-selection tool. Once the design is selected, a final model is commonly retrained on all development data and evaluated on the untouched test set. If hyperparameters are tuned extensively, nested cross-validation provides a less biased estimate: an outer loop estimates performance and an inner loop performs model selection.

Bootstrap evaluation

Bootstrap methods repeatedly sample n observations with replacement from a dataset of size n. Each resample is used to fit or evaluate a model. Approximately 1 − e⁻¹, or about 63.2%, of distinct observations appear in a typical bootstrap sample; the remaining observations are out-of-bag cases.

For an estimated metric θ̂, B bootstrap replicates θ̂₁, …, θ̂_B can be used to estimate uncertainty. The bootstrap standard error is:

where θ̄ is the mean of the B bootstrap estimates. Percentile intervals use selected quantiles of the bootstrap distribution. Bootstrap procedures are useful for confidence intervals and comparing metrics, especially when analytical standard errors are difficult. They are less reliable when the sample is very small, observations are strongly dependent, or the resampling scheme does not preserve groups or time order. Cluster bootstrap and block bootstrap are alternatives for grouped and temporal data.

4. Classification Metrics

Suppose a binary classifier predicts positive or negative outcomes. A confusion matrix contains true positives (TP), false positives (FP), true negatives (TN) and false negatives (FN).

Accuracy is:

It is suitable only when class and error costs are reasonably balanced. Precision and recall are:

Precision answers, “Among predicted positives, how many are correct?” Recall answers, “Among actual positives, how many were found?” The F1 score is their harmonic mean:

The F1 score is defined when precision + recall is non-zero. It weights precision and recall equally and therefore is not appropriate when false positives and false negatives have very different consequences.

Specificity, or true-negative rate, is:

Balanced accuracy is the mean of recall and specificity:

Threshold-independent metrics

A probabilistic classifier can be evaluated over all thresholds using the receiver operating characteristic (ROC) curve, which plots true-positive rate against false-positive rate. The area under this curve, ROC-AUC, can be interpreted as the probability that a randomly selected positive receives a higher score than a randomly selected negative, subject to ties and sampling assumptions.

Precision–recall curves are often more informative when the positive class is rare because they focus on positive predictive performance. Neither ROC-AUC nor PR-AUC directly specifies the operational threshold. A deployment report should therefore include threshold-specific confusion matrices and the consequences of errors.

Multiclass and multilabel settings

For multiclass classification, metrics may be macro-averaged, giving each class equal weight, or micro-averaged, aggregating decisions across all instances. Weighted averaging accounts for class frequencies. The averaging method must be reported because it can materially change conclusions.

5. Regression Metrics

For observed values yᵢ and predictions ŷᵢ, the mean absolute error is:

MAE is expressed in the units of the target and is relatively less sensitive to extreme errors than squared-error measures. Mean squared error is:

Root mean squared error is RMSE = √MSE, also in target units. Squaring makes MSE and RMSE more sensitive to large errors. The coefficient of determination is:

where ȳ is the sample mean of observed targets. R² compares the model with a constant-mean baseline under squared-error loss. It can be negative on test data when the model performs worse than that baseline; it is not a percentage accuracy measure.

Mean absolute percentage error, MAPE = (100/n)∑| (yᵢ − ŷᵢ)/yᵢ |, is undefined when yᵢ = 0 and unstable near zero. It should therefore be used cautiously. Always compare against a meaningful baseline, such as a seasonal-naïve forecast, mean predictor or last-value predictor.

6. Ranking and Retrieval Metrics

Search, recommendation and information-retrieval systems often produce an ordered list rather than a single label. Precision at k is the fraction of the first k results that are relevant:

Recall at k is the fraction of all relevant items retrieved in the first k positions:

Mean reciprocal rank (MRR) averages the reciprocal rank of the first relevant result across queries. Normalised discounted cumulative gain (NDCG) gives larger weight to relevant items near the top. With graded relevance rᵢ at rank i:

NDCG@k = DCG@k/IDCG@k, where IDCG is the DCG of the ideal ordering. Ranking metrics must reflect user value: a recommendation system may need diversity, novelty, coverage or business constraints in addition to relevance.

7. Overfitting, Model Selection and Statistical Comparison

Overfitting occurs when a model captures idiosyncrasies of the development sample that do not generalise. Increasing model complexity usually lowers training error but may increase validation error. Regularisation, early stopping, simpler features, more data and appropriate cross-validation can reduce this risk.

A reported point estimate is incomplete without uncertainty. For a mean metric over n independent cases, a rough normal-theory interval is estimate ± 1.96 × SE, where 1.96 corresponds approximately to a 95% interval under suitable assumptions. This approximation may be poor for bounded, skewed or dependent metrics; bootstrap intervals or paired methods are often preferable.

When comparing two models on the same cases, paired analysis is more powerful than treating scores as independent. For classification predictions, McNemar’s test examines the discordant prediction counts. For paired continuous losses, a paired t-test may be reasonable if differences are approximately independent and near-normal; otherwise, a permutation test or bootstrap confidence interval for the paired difference is preferable.

Statistical significance does not equal practical importance. A tiny improvement may be statistically detectable in a very large dataset but operationally irrelevant. Conversely, a meaningful improvement may fail to reach significance with limited data. Report the effect size, uncertainty, decision threshold and practical consequences.

8. Calibration and Robustness

A classifier is calibrated if among cases assigned probability p, approximately a proportion p are positive. For example, among 1,000 cases assigned risk 0.20, about 200 should be positive over a sufficiently large and comparable population. Discrimination and calibration are different: a model can rank cases well but produce probabilities that are systematically too high or too low.

Calibration can be assessed using reliability diagrams, calibration intercept and slope, and proper scoring rules. The Brier score for binary outcomes is:

where pᵢ is the predicted probability and yᵢ ∈ {0,1}. Lower values are better. Calibration methods such as Platt scaling and isotonic regression must themselves be fitted on data separate from the final test assessment.

Robustness asks whether performance remains acceptable under plausible changes: missing features, noise, distribution shift, adversarial inputs, changed prevalence or subgroup variation. Sensitivity analysis should vary important assumptions and evaluate realistic perturbations. Out-of-distribution detection may warn about unfamiliar inputs, but it does not guarantee safe predictions.

9. Fairness and Responsible Evaluation

Evaluation should examine performance across relevant groups, such as age, sex, disability, geography or socioeconomic category, subject to legal and ethical requirements. Useful quantities include group-specific error rates, selection rates, calibration and predictive values.

Common fairness criteria can conflict. Equalised opportunity seeks similar true-positive rates across groups; equalised odds seeks similar true-positive and false-positive rates. Demographic parity seeks similar selection rates. When base rates differ, it is generally impossible to satisfy all these criteria simultaneously except in special cases. Therefore, fairness is not a single metric-selection exercise: stakeholders must decide which harms matter, whether the labels are appropriate, and what interventions are acceptable.

Sensitive attributes may be needed for auditing even when they are excluded from prediction. Governance should address consent, privacy, access control, documentation, contestability and monitoring. Removing a sensitive variable does not guarantee fairness because correlated variables can encode it indirectly.

10. Explainability: From Model Behaviour to Human Reasons

Explainability concerns how model behaviour is described to users, developers and auditors. A global explanation describes overall patterns, whereas a local explanation describes one prediction.

Global techniques

• Feature importance summarises how much features contribute to predictive performance or model decisions.
• Partial dependence plots estimate the average predicted response as one feature varies, but can be misleading when features are strongly correlated.
• Accumulated local effects are often safer with correlated features because they aggregate local changes.
• Surrogate models approximate a complex model with an interpretable model, but fidelity must be measured rather than assumed.
• Decision trees and rule lists can be intrinsically interpretable, although a large tree may still be difficult to understand.

Local techniques

Local surrogate methods such as LIME fit a simple model near one instance. SHAP-style explanations allocate a prediction difference among features using a cooperative-game framework. These methods can be useful for diagnosis, but explanations are not automatically causal. A feature may be associated with an outcome without causing it, and an explanation can be unstable under small changes in data or background distribution.

Counterfactual explanations state how an input would need to change to obtain a different prediction. A useful counterfactual should be plausible, actionable, sufficiently close to the original case and compliant with immutable-feature constraints. For example, “income must increase” may be neither actionable nor ethically acceptable if the model does not identify a realistic path.

Explanations should be evaluated for fidelity, stability, comprehensibility and fairness. They should not expose personal data or provide a misleading sense of certainty. In high-impact decisions, explanation should complement—not replace—performance, subgroup analysis, human review and recourse procedures.

11. Integrated Evaluation Workflow

A professional evaluation can follow this sequence:

1. Specify the decision. Define the prediction target, prediction horizon, users, costs and acceptable failure modes.
2. Audit the data. Identify missingness, duplicates, groups, temporal order, label quality and possible leakage.
3. Choose the split. Use random, stratified, grouped or temporal partitioning according to deployment.
4. Build a reproducible pipeline. Fit all preprocessing only within training data.
5. Establish baselines. Compare with simple, transparent and operationally meaningful alternatives.
6. Tune using development data. Use cross-validation or nested cross-validation where appropriate.
7. Select metrics in advance. Include primary, secondary, threshold-specific and subgroup metrics.
8. Assess uncertainty. Use paired bootstrap, confidence intervals or suitable statistical tests.
9. Check calibration and robustness. Test probability quality, shifts, missingness and plausible perturbations.
10. Evaluate fairness and explanations. Report group results and explanation limitations.
11. Freeze and test. Run the final pipeline once on the untouched test set.
12. Monitor after deployment. Track drift, performance, calibration, subgroup outcomes and incident reports.

12. Illustrative Case: Hospital Readmission Risk

Background and problem

A hospital develops a model to estimate whether a patient will be readmitted within 30 days of discharge. The model is intended to support, not replace, care-coordination decisions. A major risk is leakage: discharge summaries and post-discharge billing codes may contain information unavailable when the intervention decision is made.

Analysis and approach

The team defines the prediction time as 24 hours before discharge. Patients, rather than individual records, are treated as sampling units. Data from earlier months are used for development and a later month is reserved for temporal testing. Preprocessing, imputation and feature selection occur within cross-validation folds.

The team compares logistic regression, a regularised tree ensemble and a simple baseline based on the historical readmission rate. Because readmission is relatively uncommon, accuracy is not the primary metric. The report includes PR-AUC, recall at a fixed intervention capacity, precision, calibration and Brier score. A cost analysis recognises that unnecessary outreach consumes staff time while missed high-risk patients may face preventable harm.

Outcome

Suppose the tree ensemble has higher PR-AUC but is poorly calibrated, systematically overestimating risk. Calibration is fitted using development data and then assessed on the temporal test set. Group-specific recall and false-positive rates reveal lower recall for a rural subgroup, prompting investigation of data availability and access barriers rather than simply changing the threshold.

Global explanations identify recent emergency visits and medication complexity as influential features. Local explanations are shown to clinicians with a warning that they describe associations, not causes. The deployment plan includes monthly drift checks, quarterly subgroup audits and a process for clinicians to challenge incorrect predictions.

Learning points

The case demonstrates that a model is not adequately evaluated by one score. Valid temporal design, leakage prevention, operational thresholds, calibration, subgroup analysis, explanation quality and monitoring are all part of evidence for responsible use.

Challenges, Limitations and Contemporary Practice

No evaluation completely predicts future behaviour. Labels may be noisy or reflect historical decisions rather than the underlying construct. Distribution shift can invalidate a previously representative test set. Metrics can reward undesirable behaviour when the objective is incomplete. Explainability tools can be unstable, and fairness metrics may conflict with one another.

Modern machine-learning practice increasingly uses data and model cards, reproducible experiment tracking, privacy-preserving evaluation, subgroup stress testing and continuous monitoring. Automated machine-learning systems must still preserve a leakage-safe validation design. Generative and foundation models require additional evaluation for hallucination, prompt sensitivity, memorisation, toxicity, retrieval quality and domain-specific reliability. Human oversight remains necessary where errors have material consequences.

Chapter Summary

Model evaluation estimates how well a system will generalise to future data. Holdout sets are simple but must reflect deployment; cross-validation supports model selection; bootstrap methods quantify uncertainty when their sampling assumptions are appropriate. Classification evaluation should combine discrimination, threshold-specific errors and calibration. Regression metrics differ in their sensitivity to large errors and target scale, while ranking metrics evaluate ordered retrieval.

Leakage, dependence, temporal structure and repeated test-set use are major sources of optimistic bias. Statistical comparison should report paired effect sizes and uncertainty, not only p-values. Robust evaluation includes calibration, stress testing, distribution-shift analysis and subgroup fairness assessment. Global and local explanations can improve understanding and debugging, but they are not causal proof and require fidelity and stability checks. A credible evaluation is therefore a documented lifecycle involving data design, metrics, uncertainty, governance, explanation and post-deployment monitoring.

Key Terms

Generalisation: Performance on new data drawn from the intended use population.

Holdout set: Data partition reserved for validation or final testing.

Cross-validation: Repeated partitioning in which different folds serve as validation data.

Nested cross-validation: Cross-validation with an inner selection loop and outer performance-estimation loop.

Bootstrap: Resampling observations with replacement to estimate sampling variability.

Data leakage: Use of information that would not legitimately be available at prediction time.

Overfitting: Learning sample-specific noise or patterns that do not generalise.

Precision: Proportion of predicted positives that are actual positives.

Recall: Proportion of actual positives that are detected.

Calibration: Agreement between predicted probabilities and observed frequencies.

Brier score: Mean squared error of probabilistic binary predictions.

ROC-AUC: Area under the receiver operating characteristic curve.

PR-AUC: Area under the precision–recall curve.

MAE: Mean absolute error for regression.

RMSE: Square root of mean squared error.

NDCG: A ranking metric that discounts lower-ranked results and supports graded relevance.

Robustness: Ability to maintain acceptable behaviour under plausible perturbations or shifts.

Fairness: Evaluation of whether outcomes and errors meet relevant ethical or policy criteria across groups.

Global explanation: Description of overall model behaviour.

Local explanation: Description of the factors associated with one prediction.

Review Questions

Short Answer Questions

1. Why must a final test set remain untouched during model selection?
2. Distinguish precision from recall.
3. What is data leakage? Give one example.
4. Why can a model have strong discrimination but poor calibration?
5. What is the difference between a global and a local explanation?

Long Answer Questions

1. Explain how to design a leakage-resistant evaluation for grouped, time-dependent data.
2. Compare holdout validation, k-fold cross-validation and bootstrap evaluation, including their assumptions and limitations.
3. Discuss why accuracy may be inappropriate for an imbalanced classification problem and propose a metric set.
4. Explain calibration, the Brier score and the operational importance of reliable probabilities.
5. Discuss the ethical and technical limitations of fairness metrics and explainability methods.

Analytical / Application Questions

1. A fraud detector has 95% accuracy but detects only 20% of fraudulent transactions. Analyse why accuracy is misleading and recommend a revised evaluation plan.
2. Two ranking models have similar NDCG@10, but one places relevant results consistently at ranks 1–2 while the other places them at ranks 8–10. Which additional metric or analysis would help distinguish their practical value? Justify your answer.
3. A clinical model performs well in development but its calibration deteriorates after deployment in a new region. Design an investigation covering data shift, recalibration, subgroup performance and monitoring.

Multiple Choice Questions with Answers

Question 1

Which dataset should be used only after model selection and pipeline development are complete?

A. Training set

B. Validation set

C. Final test set

D. Bootstrap sample

Correct Answer: C. Final test set

Question 2

Which practice most directly prevents preprocessing leakage during cross-validation?

A. Using a larger test set

B. Fitting preprocessing separately inside each training fold

C. Removing all missing values before splitting

D. Increasing the number of model features

Correct Answer: B. Fitting preprocessing separately inside each training fold

Question 3

If TP = 80 and FN = 20, what is recall?

A. 20%

B. 50%

C. 80%

D. 100%

Correct Answer: C. 80%

Question 4

Which metric is generally most sensitive to large regression errors?

A. MAE

B. MSE

C. Median absolute error

D. Recall

Correct Answer: B. MSE

Question 5

What does ROC-AUC approximately measure?

A. The probability calibration of predictions

B. The fraction of all cases classified correctly at one threshold

C. The probability that a randomly selected positive is ranked above a randomly selected negative

D. The average absolute prediction error

Correct Answer: C. The probability that a randomly selected positive is ranked above a randomly selected negative

Question 6

Which metric is especially useful for evaluating ordered search results with graded relevance?

A. NDCG

B. Specificity

C. RMSE

D. Brier score

Correct Answer: A. NDCG

Question 7

A calibrated group of predictions assigned probability 0.30 should, over many comparable cases, contain approximately what positive fraction?

A. 3%

B. 30%

C. 70%

D. 100%

Correct Answer: B. 30%

Question 8

Why is paired comparison useful when evaluating two models on the same test cases?

A. It eliminates all sampling uncertainty

B. It accounts for the correspondence between the models’ outcomes on each case

C. It guarantees statistical significance

D. It makes calibration unnecessary

Correct Answer: B. It accounts for the correspondence between the models’ outcomes on each case

Question 9

Which statement about fairness criteria is correct?

A. All fairness criteria can always be satisfied simultaneously

B. Removing sensitive attributes guarantees fairness

C. Different fairness criteria may conflict, especially when group base rates differ

D. Fairness is measured only by overall accuracy

Correct Answer: C. Different fairness criteria may conflict, especially when group base rates differ

Question 10

What is a key limitation of a local feature-attribution explanation?

A. It always proves that the features are causal

B. It may describe associations without establishing causation

C. It can only be used with linear models

D. It cannot be produced for a single prediction

Correct Answer: B. It may describe associations without establishing causation