11. Model Evaluation, Validation and Explainability

Introduction and Context

Building a predictive model is only one part of data science. A model becomes useful when its performance is measured appropriately, when its reported results are likely to generalise beyond the data used for development, and when its decisions can be interpreted responsibly. Model evaluation, validation and explainability therefore connect statistical modelling with professional decision-making.

A model may achieve high accuracy and still be unsuitable for deployment. It may have learned accidental patterns, benefited from information that would not be available at prediction time, or perform poorly for an important subgroup. Similarly, a model with slightly lower predictive performance may be preferable if its errors are easier to detect, its decisions are explainable, or its outputs support regulatory and organisational requirements.

This chapter examines the principles and practices used to assess machine-learning models. It covers the selection of evaluation metrics, the design of training, validation and test strategies, cross-validation, hyperparameter tuning, overfitting diagnosis, model comparison, and interpretable and explainable artificial intelligence. The central theme is that evaluation is not a single numerical exercise; it is a structured argument about whether a model is reliable for a particular purpose.

Learning Objectives

After completing this chapter, the learner should be able to:

1. Distinguish training, validation and test data and design suitable data-splitting strategies.

2. Select evaluation metrics according to the prediction task, business objective and error costs.

3. Apply cross-validation and hyperparameter-tuning methods without causing data leakage.

4. Diagnose overfitting, underfitting, instability and distribution shift.

5. Compare models using statistical, operational and ethical criteria.

6. Explain individual predictions and global model behaviour using suitable methods.

7. Recognise the limitations and risks of explainability techniques.

8. Develop an evaluation plan for a real-world data-science project.

Evaluation as a Decision Problem

Model evaluation begins with a question: what does it mean for the model to be good? The answer depends on the task and the consequences of errors. In a medical screening system, missing a genuine case may be more harmful than incorrectly referring a healthy person. In fraud detection, investigators may tolerate some false alarms if undetected fraud is costly. In demand forecasting, a small average error may be less important than systematically underestimating demand during peak periods.

A metric is therefore not merely a technical summary. It expresses a preference about errors. Evaluation should identify the target population, prediction horizon, available information, acceptable error types, operational constraints and decision threshold. Only then can a numerical score be interpreted meaningfully.

A useful evaluation plan usually contains four elements:

• a clearly defined prediction target;
• a data-partitioning and validation design;
• one or more primary and secondary metrics; and
• criteria for deployment, monitoring and reassessment.

The distinction between predictive performance and decision performance is important. A probability model may be well calibrated, but the organisation may still use an inappropriate threshold. Conversely, a model with modest ranking performance may improve decisions if it is integrated effectively into a workflow.

Choosing Metrics for the Task

Classification metrics

For binary classification, the confusion matrix records four outcomes. A true positive is a positive instance correctly identified; a true negative is a negative instance correctly rejected. A false positive is a negative instance incorrectly classified as positive, while a false negative is a positive instance missed by the model.

Accuracy is defined as:

Accuracy = (TP + TN) / (TP + TN + FP + FN)

It is intuitive but can be misleading when classes are imbalanced. If only 2 per cent of transactions are fraudulent, a model that labels every transaction as legitimate obtains 98 per cent accuracy while detecting no fraud.

Precision measures the proportion of predicted positives that are correct:

Precision = TP / (TP + FP)

Recall, also called sensitivity or true-positive rate, measures the proportion of actual positives detected:

Recall = TP / (TP + FN)

Specificity, or true-negative rate, is:

Specificity = TN / (TN + FP)

The F1 score is the harmonic mean of precision and recall:

F1 = 2(Precision × Recall) / (Precision + Recall)

The F1 score is useful when both precision and recall matter, but it ignores true negatives and assumes that the two error types deserve comparable emphasis. In applications with asymmetric costs, a cost-sensitive measure or an explicit expected-loss calculation may be preferable.

The receiver operating characteristic (ROC) curve plots recall against the false-positive rate across classification thresholds. The area under this curve, ROC-AUC, measures ranking ability across thresholds. For highly imbalanced data, the precision-recall curve often provides a more informative view because it focuses on performance for the positive class. Neither AUC measure by itself identifies the best operational threshold.

Probability quality and calibration

A classifier may output probabilities rather than labels. Calibration concerns whether predicted probabilities correspond to observed frequencies. Among cases assigned a probability of 0.8, approximately 80 per cent should be positive if the model is well calibrated. Reliability diagrams and the Brier score can be used for assessment.

Discrimination and calibration are different. A model can rank cases correctly but produce probabilities that are systematically too high. Conversely, a model can be well calibrated but have weak ability to distinguish high-risk from low-risk cases. In credit, insurance and clinical settings, both properties may matter.

Regression metrics

For continuous outcomes, mean absolute error (MAE) is:

MAE = (1/n) Σ |yi − ŷi|

It expresses the average absolute deviation in the original units and is relatively robust to extreme errors. Mean squared error (MSE) is:

MSE = (1/n) Σ (yi − ŷi)²

Because errors are squared, MSE and its square root, RMSE, penalise large errors more strongly. This is appropriate when large mistakes are disproportionately costly, but it can make the evaluation sensitive to outliers.

The coefficient of determination, R², compares the model with a baseline that predicts the mean of the target. It indicates the proportion of variance explained relative to that baseline, although it should not be interpreted as a universal measure of practical usefulness. Mean absolute percentage error can be intuitive for business reporting, but it is unstable or undefined when actual values are zero or close to zero.

Metrics should be calculated on the relevant scale. If a target is log-transformed during modelling, errors may need to be examined after transforming predictions back to the original scale. Segment-level and time-period-level errors may reveal patterns hidden by an overall average.

Ranking, retrieval and specialised metrics

Recommendation and search systems frequently use ranking metrics such as precision at k, recall at k, mean reciprocal rank and normalized discounted cumulative gain. These metrics recognise that users often inspect only the first few results. Clustering requires different evaluation approaches, such as silhouette analysis, stability assessment and domain validation, because there may be no known labels. For probabilistic forecasting, prediction-interval coverage and proper scoring rules are relevant.

Designing Training, Validation and Test Strategies

The training set is used to estimate model parameters. The validation set supports choices such as algorithm selection, feature engineering, regularisation and hyperparameter tuning. The test set is reserved for final, relatively unbiased assessment. The test set must not repeatedly influence development decisions; otherwise it becomes another validation set and its estimate becomes optimistic.

A common workflow is to divide data into training, validation and test portions, for example 60:20:20 or 70:15:15. The exact proportions depend on sample size and problem complexity. With limited data, cross-validation can replace a single validation split while retaining a final test set.

Random splitting is not always appropriate. In time-dependent forecasting, training must precede validation and test periods chronologically. Randomly placing future observations in the training set can produce look-ahead bias. In grouped data, such as multiple records from the same patient, customer or machine, all observations belonging to a group should generally remain in one partition. Otherwise the model may recognise the individual rather than learn a general pattern.

Stratified splitting preserves approximate class proportions and is useful for imbalanced classification. Spatial, temporal and group-based splits are forms of distribution-aware validation. The correct question is not “Which split is conventional?” but “What future or unseen cases will this model encounter?”

Data leakage

Data leakage occurs when information unavailable at prediction time, or information derived from the evaluation data, enters model development. Examples include scaling the entire dataset before splitting, imputing missing values using test-set statistics, selecting features using all labels, and using a post-outcome variable as a predictor. Leakage can produce impressive validation scores and disappointing deployment performance.

A robust practice is to place preprocessing, feature selection and modelling inside a single pipeline. Each operation is fitted only on the training portion of each fold. Leakage is especially common in medical records, financial transactions and temporal data, where the order in which information becomes available must be respected.

Cross-Validation and Hyperparameter Tuning

In k-fold cross-validation, the training data are divided into k folds. The model is trained on k−1 folds and evaluated on the remaining fold; the process is repeated until every fold has served as validation data. The average score estimates expected performance, while the variation across folds indicates stability.

Stratified k-fold cross-validation maintains class proportions. Grouped cross-validation keeps related observations together. Time-series cross-validation uses forward or expanding windows so that validation observations occur after training observations. Repeated cross-validation can provide a more stable estimate, although it increases computational cost.

Cross-validation is also used to select hyperparameters, such as tree depth, regularisation strength, number of neighbours or learning rate. Grid search evaluates a predefined set of combinations, while random search samples combinations from specified distributions and may explore high-impact parameters more efficiently. Bayesian optimisation treats the search as a sequential decision problem, using previous results to select promising configurations.

A serious concern is repeated experimentation. If thousands of configurations are evaluated against the same validation folds, the chosen configuration may fit noise in those folds. Nested cross-validation addresses this problem. An inner loop selects hyperparameters, and an outer loop estimates generalisation performance. The outer test fold remains uninvolved in model selection.

Hyperparameter tuning should be constrained by operational and governance considerations. The most accurate model may be too slow, expensive, opaque or difficult to maintain. Search spaces should reflect plausible deployment conditions rather than encourage arbitrary optimisation of a single score.

Diagnosing Generalisation and Model Failure

Underfitting occurs when a model is too simple to capture important structure. It generally produces high training and validation error. Overfitting occurs when a model captures noise or idiosyncrasies in the training data. It often produces low training error but substantially worse validation error.

Learning curves, which plot training and validation performance against training-set size, help distinguish these conditions. A persistent gap suggests high variance and possible overfitting; similarly high errors suggest high bias or inadequate features. Validation-score variability across folds may indicate small sample size, influential observations or heterogeneous subpopulations.

Regularisation, simpler models, feature reduction, early stopping, more data and data augmentation can reduce overfitting. However, regularisation should not be applied mechanically. A model may appear overfit because the validation distribution differs from the training distribution. In that case, the underlying issue is distribution shift rather than ordinary variance.

Performance should be examined by subgroup, time period, geography, product category and other relevant dimensions. Aggregate performance can hide systematic failures. Confidence intervals or bootstrap estimates can communicate uncertainty around metrics. When comparing two models, paired evaluation on the same cases is usually more informative than comparing scores obtained on different samples. For classification, threshold-dependent comparisons should use the same threshold policy or compare complete curves where appropriate.

Explainability and Interpretability

Interpretability refers broadly to how understandable a model or its behaviour is to humans. Explainability methods attempt to provide reasons, summaries or evidence for model outputs. A simple linear model may be intrinsically interpretable, whereas a deep neural network may require post hoc explanation. Interpretability is not identical to accuracy, and an explanation is not proof that the model uses a genuine causal mechanism.

Global understanding

Global methods describe overall model behaviour. In a linear model, coefficients indicate the expected change in the prediction associated with a one-unit feature change, holding other variables constant under the model assumptions. Coefficients can be misleading when features are strongly correlated or measured on different scales.

Decision trees provide path-based rules, although large trees become difficult to inspect. Feature importance in ensembles may be calculated through impurity reduction or permutation importance. Permutation importance measures the performance decrease after a feature is disrupted, but correlated features can share or conceal importance.

Partial dependence plots show the average predicted response as a feature varies, averaging over other features. Individual conditional expectation plots display this relationship for individual observations and can reveal heterogeneity hidden by an average curve. These plots may be misleading when they generate combinations of feature values that are unrealistic in the data.

Local explanations

Local explanation methods focus on one prediction. LIME approximates a complex model near a selected observation with a simpler surrogate model. SHAP methods allocate a prediction among features using concepts related to Shapley values from cooperative game theory. Such explanations can identify which features pushed a prediction upward or downward relative to a baseline.

Counterfactual explanations answer a different question: what minimal change in the input would produce a different outcome? They can be useful in lending, customer support and clinical decision support, but suggested changes must be actionable, feasible and ethically acceptable. Changing an immutable attribute or proposing a harmful intervention is not a meaningful counterfactual.

Explanations should be evaluated for fidelity, stability, comprehensibility and usefulness. A visually attractive explanation may not faithfully represent the model. Local explanations can change substantially when the input or sampling procedure changes. Feature attribution also does not establish causation. If a model uses a proxy for a protected characteristic, explaining the proxy does not remove the fairness concern.

An Illustrative Case: Predicting Loan Delinquency

Background

A regional lender develops a model to identify customers who may become seriously delinquent within the next six months. The purpose is not automatic rejection but prioritisation of early financial counselling.

Situation and problem

The data contain repayment history, account activity, income information, application details and records of customer contact. The initial modelling team randomly divides all records and reports a high ROC-AUC. However, the model performs poorly after deployment, particularly for recently acquired customers.

Analysis

Investigation identifies three problems. First, several records describing collection activity occurred after the prediction date and therefore leaked outcome-related information. Second, customers with multiple monthly records were split across training and test sets, enabling the model to recognise customer-specific patterns. Third, evaluation focused on ROC-AUC even though the counselling team could contact only a small number of customers each week.

Approach and solution

The team defines a fixed prediction date for each observation and removes variables unavailable at that date. It uses group-based splitting so that a customer appears in only one partition, and it performs chronological validation to simulate future deployment. Because intervention capacity is limited, precision at the top k customers and recall among high-risk customers become primary metrics. Calibration is also assessed because staff need meaningful risk probabilities.

The team compares a regularised logistic regression with a gradient-boosted tree model. The tree model ranks customers somewhat better, but the logistic model is easier to audit and produces more stable subgroup performance. SHAP-based inspection of the tree model reveals that recent contact attempts are influential, but these attempts are not consistently available at prediction time. The final decision is to deploy the calibrated logistic model initially, accompanied by monitoring and periodic review.

Outcome

The revised evaluation produces lower but more credible performance estimates. The organisation gains a model aligned with its intervention capacity and a documented rationale for its variables and threshold. Performance is monitored by customer segment and over time rather than by one aggregate score alone.

Learning points

The case demonstrates that leakage control can reduce apparent performance while increasing credibility. It also shows that model choice depends on operational constraints, that ranking metrics must reflect workflow capacity, and that explainability can uncover problematic predictors rather than merely justify a preferred model.

Challenges, Risks and Professional Responsibilities

Evaluation results are affected by sampling bias, measurement error, missing data, label noise and changes in the data-generating process. A test set is not permanently representative; its relevance declines if the population, policy or behaviour changes. Small samples can produce unstable estimates, and multiple subgroup comparisons can generate apparently alarming results by chance.

Fairness assessment should examine whether errors, calibration or access differ across relevant groups. No single fairness criterion is universally correct, and some criteria cannot generally be satisfied simultaneously when base rates differ. Fairness analysis should involve domain experts and affected stakeholders, not only technical staff.

Privacy must be protected when creating validation datasets and explanations. Explanations may reveal sensitive information or enable users to infer protected attributes. Security teams should consider whether explanation interfaces expose model weaknesses. Documentation should record the intended use, data sources, evaluation design, known limitations, threshold policy and monitoring plan.

Human oversight is essential in high-impact contexts. A model score should not be treated as an unquestionable fact, particularly when labels reflect historical decisions or institutional inequalities. Automated systems require mechanisms for appeal, correction and responsible retirement.

Emerging Directions and Best Practices

Modern evaluation increasingly includes uncertainty estimation, conformal prediction, distribution-shift detection and stress testing. Conformal methods can produce prediction sets or intervals with specified coverage assumptions. Robustness testing examines performance under missing features, noisy inputs and plausible changes in population. For generative and foundation models, evaluation may combine factuality, safety, instruction following, calibration, human preference and task-specific utility.

Best practice is to:

1. Define the deployment decision and error costs before selecting metrics.

2. Replicate the expected production setting in the data split.

3. Keep preprocessing and feature selection inside validation pipelines.

4. Preserve a final test set and avoid repeated informal test-set inspection.

5. Report uncertainty, subgroup performance and threshold-dependent results.

6. Compare models on predictive, operational, ethical and maintenance criteria.

7. Use explanations to investigate behaviour, not to claim causality automatically.

8. Monitor performance, calibration, data quality and fairness after deployment.

9. Maintain documentation, version control and reproducible evaluation code.

10. Revalidate the system whenever data, policy or intended use changes.

Chapter Summary

Model evaluation is a structured process for determining whether a model is reliable and useful for a defined purpose. Metric selection must reflect the target, class balance, error costs and operational workflow. Training, validation and test sets serve different roles, while time, group and stratified splits help reproduce realistic deployment conditions. Leakage is a major source of falsely optimistic results.

Cross-validation estimates performance and supports hyperparameter tuning, but repeated experimentation can overfit the validation process; nested cross-validation provides stronger protection. Overfitting, underfitting, instability and distribution shift require different diagnoses and remedies. Models should be compared using uncertainty, subgroup behaviour, operational constraints and governance requirements, not merely one score.

Interpretability and explainability methods include coefficients, feature importance, partial dependence, local surrogate explanations, SHAP and counterfactuals. These tools support understanding and auditing but do not automatically establish causation, fairness or truth. Responsible evaluation continues after deployment through monitoring, documentation, human oversight and periodic revalidation.

Key Terms

Accuracy: The proportion of all predictions that are correct.

Calibration: Agreement between predicted probabilities and observed outcome frequencies.

Confusion matrix: A table of true-positive, true-negative, false-positive and false-negative outcomes.

Cross-validation: A procedure that repeatedly trains and evaluates a model on different partitions of available data.

Data leakage: Use of information during development that would be unavailable at prediction time or improperly derived from evaluation data.

Discrimination: A model’s ability to distinguish between different outcome classes or levels of risk.

Explainability: The provision of information intended to clarify how a model produces or supports a prediction.

F1 score: The harmonic mean of precision and recall.

False negative: A positive case incorrectly classified as negative.

False positive: A negative case incorrectly classified as positive.

Generalisation: The ability of a model to perform well on relevant unseen data.

Hyperparameter: A model-development setting selected rather than directly learned from the training observations.

Overfitting: Learning noise or sample-specific patterns that reduce performance on unseen data.

Precision: The proportion of predicted positive cases that are truly positive.

Recall: The proportion of actual positive cases correctly detected.

Regularisation: A technique that constrains model complexity to improve generalisation.

Test set: Data reserved for final assessment after model-development decisions are complete.

Validation set: Data used to select models, features, thresholds or hyperparameters.

Interpretability: The extent to which humans can understand a model or its behaviour.

Distribution shift: A change between the data used for development and the data encountered in deployment.

Review Questions

Short Answer Questions

1. Why can accuracy be misleading for an imbalanced classification problem?

2. Distinguish precision from recall and give one context in which each might be prioritised.

3. What is the purpose of a final test set?

4. Explain the difference between calibration and discrimination.

5. What is data leakage? Provide one example.

Long Answer Questions

1. Explain how training, validation and test sets should be designed for a time-dependent prediction problem.

2. Compare MAE, MSE, RMSE and R², including situations in which each may be useful or misleading.

3. Describe k-fold cross-validation, stratified cross-validation, grouped cross-validation and nested cross-validation.

4. Discuss the causes, diagnosis and remedies of overfitting.

5. Compare global and local explainability methods and discuss their limitations.

Analytical / Application Questions

1. A disease-screening model is used where false negatives are much more harmful than false positives. Which metrics and threshold-analysis methods would you use, and why?

2. A customer dataset contains several transactions per customer. Design a validation strategy that avoids overly optimistic performance estimates.

3. A model has excellent overall AUC but poor calibration and substantially lower recall for a minority subgroup. Should it be deployed? Develop a reasoned evaluation and remediation plan.

22. Multiple Choice Questions with Answers

Question 1

A fraud detector is evaluated on data in which only 1% of transactions are fraudulent. Which metric is most likely to be misleading if used alone?

A. Accuracy

B. Recall

C. Precision

D. PR-AUC

Correct Answer: A. Accuracy

Question 2

A dataset contains repeated observations from the same patient. Which validation strategy best prevents patient-specific information from leaking across partitions?

A. Random splitting of all observations

B. Grouped cross-validation by patient

C. Increasing the number of features

D. Using training accuracy

Correct Answer: B. Grouped cross-validation by patient

Question 3

What is the main purpose of keeping a final test set untouched during model development?

A. To increase the training sample size

B. To select hyperparameters repeatedly

C. To obtain a less biased estimate of final generalisation performance

D. To replace preprocessing

Correct Answer: C. To obtain a less biased estimate of final generalisation performance

Question 4

A model has very low training error but substantially higher validation error. Which diagnosis is most consistent with this pattern?

A. Underfitting

B. Overfitting

C. Perfect calibration

D. Class balancing

Correct Answer: B. Overfitting

Question 5

In a medical screening system, which metric directly measures the proportion of actual positive cases detected?

A. Specificity

B. Precision

C. Recall

D. Accuracy

Correct Answer: C. Recall

Question 6

Why is chronological validation preferred for many forecasting problems?

A. It guarantees zero prediction error

B. It prevents future information from being used to predict the past

C. It always produces higher AUC

D. It removes the need for a test set

Correct Answer: B. It prevents future information from being used to predict the past

Question 7

Which statement best distinguishes calibration from discrimination?

A. Calibration concerns probability reliability, whereas discrimination concerns ranking or separation

B. Calibration applies only to regression, whereas discrimination applies only to clustering

C. Calibration measures training error, whereas discrimination measures test size

D. They are two names for the same property

Correct Answer: A. Calibration concerns probability reliability, whereas discrimination concerns ranking or separation

Question 8

What is a key limitation of permutation feature importance when predictors are strongly correlated?

A. It cannot be computed for any model

B. Importance may be divided among correlated variables or appear misleading

C. It always proves causation

D. It requires the target to be continuous

Correct Answer: B. Importance may be divided among correlated variables or appear misleading

Question 9

A counterfactual explanation for a loan decision suggests changing a customer’s immutable age. What is the principal concern?

A. The explanation is necessarily globally accurate

B. The suggested change is not actionable and may be ethically inappropriate

C. Counterfactuals cannot be used with classifiers

D. Age is never a model feature

Correct Answer: B. The suggested change is not actionable and may be ethically inappropriate

Question 10

A team repeatedly selects models using the same validation folds and reports the best validation score as final performance. What risk has increased?

A. Validation-process overfitting

B. Under-sampling of the test set

C. Improved causal identification

D. Perfect subgroup fairness

Correct Answer: A. Validation-process overfitting