Model Evaluation, Explainability and Robustness

Introduction

A deep-learning model is not successful merely because its training loss decreases. A deployable model must perform well on unseen data, communicate reliable confidence, behave fairly across relevant populations, provide explanations appropriate to its users, and remain dependable when data or operating conditions change. These requirements are especially important in high-impact applications such as medical diagnosis, credit assessment, autonomous systems, education and public services.

Model evaluation is therefore a multidimensional engineering and scientific activity. It includes measurement of predictive performance, validation of generalisation, analysis of uncertainty and calibration, assessment of fairness and interpretability, testing against adversarial manipulation, and monitoring for distribution shift and failure. No single metric captures all these properties. Evaluation must instead connect technical evidence to the risks and decisions associated with the application.

Learning Objectives

By the end of this chapter, students should be able to:

1. Select suitable performance metrics for classification, regression and imbalanced-data problems.
2. Design sound validation and test procedures that avoid leakage.
3. Explain aleatoric and epistemic uncertainty and assess calibration.
4. Compare major explainability methods and recognise their limitations.
5. Evaluate fairness, robustness, adversarial vulnerability and distribution shift.
6. Identify model failure modes and construct a practical evaluation protocol.

1. Evaluation as a System of Evidence

Evaluation should answer several distinct questions:

• Predictive validity: Does the model make accurate predictions on relevant unseen cases?
• Generalisability: Does performance hold across time, locations, devices and populations?
• Reliability: Do confidence scores correspond to actual frequencies of correctness?
• Safety: What happens under noise, missing values, unusual inputs or deliberate attacks?
• Equity: Are errors and benefits distributed acceptably across groups?
• Interpretability: Can users understand the basis and limitations of predictions?
• Operational fitness: Does the model meet latency, cost, privacy and human-workflow requirements?

A model may have high accuracy but poor calibration, or excellent average performance but unacceptable results for a minority group. Evaluation must consequently report disaggregated results, uncertainty intervals where appropriate, and the conditions under which the results were obtained.

2. Performance Metrics

2.1 Confusion-matrix measures

For binary classification, predictions can be described using four counts:

The principal metrics are:

Accuracy is suitable when classes and error costs are reasonably balanced. It can be misleading under class imbalance.

Precision answers: among predicted positives, how many are positive? It is important when false alarms are costly.

Recall, also called sensitivity or true-positive rate, answers: among actual positives, how many are detected? It is important when missed positives are costly.

Specificity is the true-negative rate and is important when false positives impose substantial costs.

F1 is the harmonic mean of precision and recall. It is defined when both precision and recall are meaningful and nonzero; it gives equal importance to them and does not directly include true negatives.

For a classifier that outputs a score, the receiver operating characteristic (ROC) curve plots recall against false-positive rate as the decision threshold varies. The area under this curve, AUROC, measures ranking ability across thresholds. The precision–recall curve is often more informative when positives are rare because it focuses on positive predictions. Neither AUROC nor AUPRC alone determines the threshold appropriate for deployment.

2.2 Thresholds and decision costs

A probability estimate must usually be converted into an action. If the cost of a false negative is C_FN and the cost of a false positive is C_FP, a simplified threshold under calibrated probabilities and equal action benefits is:

where p̂ is the predicted probability of the positive class. This expression assumes binary decisions, known and constant costs, calibrated probabilities, and no additional constraints such as capacity limits. In practice, organisations should use an explicit cost matrix and assess consequences with domain experts rather than selecting a threshold solely from accuracy.

2.3 Regression metrics

For observed values y_i and predictions ŷ_i, with n observations:

MAE is expressed in the units of the target and gives linear penalty to errors.

MSE and its square root, RMSE, penalise large errors more strongly. RMSE is in the target units:

Mean absolute percentage error can be useful for positive targets but becomes unstable or undefined when y_i is zero or close to zero. R² is a relative measure comparing residual variation with variation around the sample mean; it may be negative on test data and should not be interpreted as a universal measure of accuracy.

2.4 Structured and probabilistic outputs

For segmentation, intersection over union (IoU) is commonly used:

For probabilistic classification, log loss evaluates the entire probability distribution. For K classes and one-hot labels y_ik:

Here p_ik is the predicted probability for class k, and probabilities must be valid and positive. Log loss strongly penalises confident incorrect predictions, making it useful for evaluating probabilistic forecasts.

3. Validation Design and Generalisation

A typical workflow separates data into training, validation and test sets. The training set estimates model parameters; the validation set selects architectures, hyperparameters and thresholds; the test set is used once, or very sparingly, for final unbiased estimation. Repeatedly inspecting the test set turns it into an informal validation set and produces optimistic results.

Random splitting is inappropriate when observations are related. Patient records should be split by patient rather than by individual scan; frames from one video should not appear in both training and test sets; time-dependent forecasting should use a temporal split. For grouped observations, group-aware cross-validation prevents information from the same entity appearing in multiple folds. For time series, walk-forward validation preserves temporal order.

In k-fold cross-validation, data are divided into k folds. Each fold is used once as validation data while the other k−1 folds are used for training. The mean score estimates expected performance, while variation across folds indicates sensitivity to the sample. Confidence intervals can be estimated with suitable resampling methods, but they should not be interpreted as guaranteeing future performance under distribution shift.

Data leakage occurs when information unavailable at prediction time influences training or evaluation. Examples include normalising the full dataset before splitting, using a post-outcome variable as a feature, duplicate records across partitions, and selecting features using test labels. Every learned preprocessing step must be fitted on training data only and then applied to validation and test data.

A strong evaluation report specifies the data-generating context, inclusion criteria, missing-data treatment, split strategy, random seeds, hyperparameter search procedure, metrics, confidence intervals and exclusions. External validation on a different site or later period is often more informative than another random split.

4. Uncertainty and Calibration

A prediction is incomplete without an indication of how much it should be trusted. Aleatoric uncertainty arises from inherent noise or ambiguity in the data, such as sensor noise or two visually indistinguishable classes. Epistemic uncertainty reflects limited knowledge, such as insufficient training examples or unfamiliar inputs. More data may reduce epistemic uncertainty but cannot necessarily remove aleatoric uncertainty.

For classification, calibration means that predictions with probability p correspond, approximately, to a proportion p of correct outcomes. A model that assigns probability 0.8 to 100 cases should be correct about 80 times if it is well calibrated in that population.

A reliability diagram groups predictions into probability bins and compares mean confidence with observed accuracy. The expected calibration error (ECE) is commonly estimated as:

B_m is bin m, M is the number of bins, and |B_m| is the number of observations in that bin. ECE depends on binning choices and is not a complete calibration assessment.

Calibration can be improved using temperature scaling, which changes logits by a learned positive temperature on a validation set, or methods such as Platt scaling and isotonic regression. Calibration must be evaluated on data representative of deployment; a model calibrated in one hospital or demographic group may be miscalibrated in another.

Neural networks can estimate uncertainty using deep ensembles, stochastic dropout procedures, Bayesian neural networks or distributional prediction. These methods differ in computational cost and assumptions. Prediction intervals should be tested for coverage and sharpness: an interval that always covers the target but is extremely wide is not practically useful.

5. Explainability and Interpretability

Interpretability concerns how understandable a model or its decisions are. Intrinsic interpretability is provided by models whose structure is directly inspectable, such as small decision trees or sparse linear models. Post-hoc explainability analyses a trained model after the fact.

Global explanations describe overall behaviour, such as feature importance, partial-dependence plots or concepts learned by a vision model. Local explanations describe one prediction, for example which pixels or features influenced a classification. A local explanation is not automatically a causal explanation: highlighting a feature does not prove that changing it would change the outcome.

Common methods include:

• Permutation importance: measures performance degradation after a feature is randomly permuted. Correlated features can make the result unstable.
• Partial dependence: estimates average prediction as a feature varies, but may generate unrealistic combinations of correlated features.
• LIME: fits a simple local surrogate around an instance; results can depend on sampling and the chosen locality.
• SHAP: assigns feature contributions using Shapley-value ideas. Exact computation is often expensive, and assumptions about feature dependence affect interpretation.
• Saliency and integrated gradients: identify input regions influential to neural predictions. Visual prominence does not establish human meaningfulness or causal relevance.
• Counterfactual explanations: describe changes that could alter a prediction. A valid counterfactual should be plausible, actionable, sufficiently close, and respectful of immutable attributes.

Explanations should be evaluated for fidelity, stability, comprehensibility and usefulness. Fidelity asks whether the explanation accurately represents the model. Stability asks whether small irrelevant changes produce similar explanations. Users should be told whether an explanation is a model rationale, a correlation-based attribution, or a causal account.

6. Fairness and Responsible Evaluation

Fairness is application-dependent and cannot be reduced to one universal metric. Let A denote a protected or monitored group attribute, Ŷ the prediction and Y the true outcome. Demographic parity requires similar positive-prediction rates across groups, while equal opportunity requires similar true-positive rates. Equalised odds requires both true-positive and false-positive rates to be similar. Predictive parity requires similar positive predictive values.

These criteria can conflict, particularly when base rates differ between groups. Consequently, fairness assessment requires a normative decision about which errors matter, consultation with affected communities, and careful consideration of legal requirements. Performance should be disaggregated by relevant groups and intersections, while protecting privacy and avoiding unreliable conclusions from very small samples.

Fairness problems can enter through historical labels, under-representation, measurement differences, proxy variables, selective missingness and unequal access to recourse. Removing a protected attribute does not guarantee fairness because correlated variables may encode it. Mitigation may occur before modelling through data improvement, during modelling through constrained optimisation or reweighting, or after modelling through threshold and policy adjustments. Every intervention involves trade-offs that should be documented.

7. Robustness, Adversarial Vulnerability and Distribution Shift

Robustness is the ability to maintain acceptable behaviour under relevant perturbations. Ordinary robustness tests may include sensor noise, blur, compression, missing values, spelling variation, lighting changes and corrupted records. The perturbation set should reflect realistic failure conditions rather than arbitrary transformations.

An adversarial example is an input intentionally modified to induce an incorrect output, often while remaining visually or semantically similar to the original. Attacks may be white-box, where the attacker knows model details, or black-box, where the attacker queries the system. Threat modelling should specify attacker capability, access, goals and acceptable perturbation size.

Defences include adversarial training, input validation, robust feature design, detection mechanisms, ensembles and certified robustness methods. No defence is universal: improving resistance to one attack can reduce clean accuracy or expose vulnerability to another. Security evaluation should be performed independently from training and should include adaptive attackers.

Distribution shift occurs when the deployment distribution differs from the training distribution. Covariate shift changes P(X), label shift changes P(Y), and concept shift changes P(Y|X). A model may fail because of new equipment, changing user behaviour, seasonal effects, policy changes or population movement. Monitoring may use feature statistics, prediction distributions, reference sets, drift detectors and outcome-based performance checks. Drift detection is not equivalent to detecting performance degradation: a changed input distribution may be harmless, while a subtle change in the relationship between inputs and outcomes may be damaging.

8. Model Failure Modes and Monitoring

A failure-mode analysis should ask where, when and why the model can fail. Typical modes include:

• spurious correlations, such as learning hospital identifiers instead of pathology;
• shortcut learning, where easy but irrelevant cues dominate;
• rare subgroup failure hidden by aggregate metrics;
• out-of-distribution inputs receiving overconfident predictions;
• label noise and inconsistent annotation;
• feedback loops caused by decisions changing future training data;
• automation bias, where users accept incorrect model outputs;
• infrastructure failures, including preprocessing version mismatch and unavailable features.

A production monitoring plan should track input quality, missingness, latency, prediction confidence, subgroup outcomes, drift indicators, calibration and confirmed errors. It should define alert thresholds, owners, escalation routes, rollback procedures and retraining criteria. Logging must respect privacy, retention and access-control requirements.

9. An Integrated Evaluation Protocol

A practical protocol can proceed as follows:

1. Define the decision, users, affected groups, costs and safety constraints.
2. Audit labels, sampling, duplicates, missingness and representation.
3. Choose a split that reflects deployment, including groups or time where necessary.
4. Establish simple baselines and a meaningful acceptance threshold.
5. Train models using leakage-free preprocessing and validation.
6. Report task-appropriate metrics with uncertainty and disaggregated results.
7. Evaluate calibration, threshold decisions and selective prediction or abstention.
8. Test interpretability methods for fidelity and stability.
9. Conduct fairness, robustness, adversarial and shift analyses.
10. Perform external or prospective validation before deployment.
11. Establish monitoring, human oversight, incident response and model retirement.

A model card or evaluation dossier should record intended use, prohibited use, training data, performance by subgroup, limitations, ethical risks, version history and monitoring arrangements.

10. Illustrative Case: AI-Assisted Chest-Radiograph Triage

Background

A hospital develops a neural network to prioritise chest radiographs that may show a serious abnormality. The system does not make a final diagnosis; it assigns a risk score to support radiologist workflow.

Problem

Initial testing reports 96% accuracy. However, serious abnormalities occur in only 4% of examinations. The hospital must determine whether the system is safe and useful across scanners, sites and patient groups.

Analysis

Because of prevalence imbalance, accuracy is inadequate. The team reports sensitivity, specificity, precision, negative predictive value, AUROC and area under the precision–recall curve. Images from the same patient are grouped in one partition, and a later-month external test set is added. Calibration plots reveal that predictions are overconfident on a scanner not represented in training. Subgroup analysis shows lower sensitivity for a less-represented age group.

Review of saliency maps suggests that the model sometimes uses image borders and device markers. This indicates shortcut learning rather than reliable anatomical reasoning. Robustness tests with altered contrast and cropping produce substantial degradation. The team therefore adds scanner diversity, removes identifying artefacts where appropriate, improves labels through multi-reader adjudication and introduces an abstention policy for low-quality or unfamiliar images.

Outcome

The revised system is evaluated prospectively. It is used only for prioritisation, with radiologist review retained. Thresholds are selected using workflow costs rather than accuracy alone. Monitoring tracks calibration, scanner-specific performance, subgroup sensitivity and override rates. Deployment is paused if safety thresholds are breached.

Learning Points

The case demonstrates that high aggregate accuracy can conceal prevalence effects, calibration failure, shortcut learning and subgroup risk. Evaluation must include realistic splits, external validation, technical robustness, human factors and governance.

Challenges and Limitations

Evaluation metrics are estimates, not permanent properties of a model. Small or biased test sets produce uncertain conclusions. Fairness metrics may conflict, and subgroup labels may be incomplete or sensitive. Explanations can create false reassurance, especially when users mistake associations for causes. Robustness testing can never enumerate every future perturbation. Calibration and performance can deteriorate after deployment, while monitoring itself may be affected by delayed or selectively observed labels.

A further limitation is organisational: technically strong evaluation may have little impact if decision rights, accountability, procurement requirements and incident procedures are unclear. Responsible deployment therefore requires collaboration among machine-learning engineers, domain professionals, affected users, legal specialists, security teams and governance bodies.

Contemporary Developments

Current practice is moving toward foundation-model evaluation, red-teaming, uncertainty-aware abstention, conformal prediction, data-centric testing and continuous post-deployment evaluation. Conformal methods can construct prediction sets or intervals with coverage guarantees under exchangeability assumptions, but guarantees may weaken under distribution shift. Synthetic data and generative models can expand testing, although synthetic examples may reproduce or hide real-world biases. Automated evaluation agents can improve coverage but require human-designed threat models and validation.

The central trend is a shift from a one-time test score toward lifecycle assurance: documenting intended use, evaluating before release, monitoring during operation and revising or retiring systems when evidence changes.

Chapter Summary

Reliable deep-learning evaluation is multidimensional. Classification metrics such as precision, recall, specificity and F1 must be selected according to prevalence and error costs; regression requires metrics whose penalty structure and units match the application. Leakage-free validation, group-aware splitting, temporal evaluation and external testing are essential for credible generalisation claims. Calibration assesses whether confidence reflects observed correctness, while uncertainty analysis distinguishes irreducible data noise from limited model knowledge.

Explainability methods provide global or local attributions, but they must be assessed for fidelity, stability and appropriate interpretation. Fairness requires subgroup analysis and an explicit choice among potentially conflicting criteria. Robustness evaluation includes realistic perturbations, adversarial threat modelling and distribution-shift analysis. Finally, deployment requires failure-mode analysis, monitoring, human oversight, documentation and clear intervention procedures. A trustworthy model is not simply accurate; it is suitably accurate, calibrated, understandable, equitable, secure and governable for its intended use.

Key Terms

Accuracy: Proportion of all predictions that are correct.

Calibration: Agreement between predicted probabilities and observed outcome frequencies.

Concept shift: Change in the relationship P(Y|X) between inputs and outcomes.

Data leakage: Use of information during training or evaluation that would not be available at prediction time.

Demographic parity: Similar positive-prediction rates across specified groups.

Distribution shift: Difference between training and deployment data distributions.

Epistemic uncertainty: Uncertainty caused by limited knowledge or data.

Aleatoric uncertainty: Irreducible uncertainty arising from inherent variability or noise.

Equalised odds: Similar true-positive and false-positive rates across groups.

Explainability: Methods for describing or attributing a model’s predictions.

External validation: Evaluation on data from a different site, period or population.

F1 score: Harmonic mean of precision and recall.

False positive: Negative case incorrectly predicted as positive.

Calibration error: Quantified discrepancy between confidence and observed accuracy.

Robustness: Ability to maintain acceptable performance under relevant perturbations.

Shortcut learning: Reliance on easy but irrelevant predictive cues.

Test set: Held-out data used for final performance estimation.

Threat model: Specification of an attacker’s access, capability and objective.

Uncertainty: Lack of complete confidence in a prediction or outcome.

Validation set: Data used to select models, hyperparameters or operating thresholds.

Review Questions

Short Answer Questions

1. Why can accuracy be misleading for a highly imbalanced classification problem?
2. Distinguish aleatoric uncertainty from epistemic uncertainty.
3. What is the purpose of a reliability diagram?
4. Give two examples of data leakage.
5. Why is external validation valuable?

Long Answer Questions

1. Explain how precision, recall, specificity and F1 should be selected according to application costs.
2. Compare intrinsic interpretability with post-hoc explainability, including limitations of each.
3. Explain how fairness criteria can conflict when base rates differ between groups.
4. Describe a rigorous evaluation protocol for a neural model before deployment.
5. Discuss how distribution shift, monitoring and model retirement should be managed across the model lifecycle.

Analytical / Application Questions

1. A fraud detector is used where fraud occurs in 1% of transactions. Explain why accuracy alone is inappropriate and recommend a metric and threshold-selection process.
2. A medical model has excellent internal AUROC but poor performance at a new hospital. Diagnose possible causes and design an investigation.
3. A vision model highlights a hospital logo when classifying disease. Identify the failure mode and propose data, modelling and evaluation interventions.

Multiple Choice Questions with Answers

Question 1

Which metric answers the question: among cases predicted positive, what proportion is truly positive?

A. Recall

B. Precision

C. Specificity

D. Accuracy

Correct Answer: B. Precision

Question 2

Why should patient-level grouping be used when several images belong to the same patient?

A. To increase the number of training examples

B. To guarantee perfect calibration

C. To prevent related records appearing across data partitions

D. To eliminate class imbalance

Correct Answer: C. To prevent related records appearing across data partitions

Question 3

A well-calibrated model assigning probability 0.8 to many comparable cases should be correct approximately what fraction of the time?

A. 8%

B. 20%

C. 50%

D. 80%

Correct Answer: D. 80%

Question 4

Which uncertainty is most directly associated with limited training data or unfamiliar inputs?

A. Epistemic uncertainty

B. Aleatoric uncertainty

C. Measurement certainty

D. Deterministic uncertainty

Correct Answer: A. Epistemic uncertainty

Question 5

What is a major limitation of a saliency map?

A. It cannot be computed for any neural network

B. It automatically proves a causal relationship

C. Visual prominence may not establish causal or human-meaningful reasoning

D. It can only be used for regression

Correct Answer: C. Visual prominence may not establish causal or human-meaningful reasoning

Question 6

Which validation strategy is most appropriate for forecasting future demand?

A. Randomly shuffle all observations before splitting

B. Use a temporal or walk-forward split

C. Use only the training loss

D. Place all recent observations in training and old observations in testing

Correct Answer: B. Use a temporal or walk-forward split

Question 7

What does equal opportunity primarily require across groups?

A. Equal positive-prediction rates

B. Equal true-positive rates

C. Equal sample sizes

D. Equal feature distributions

Correct Answer: B. Equal true-positive rates

Question 8

Which situation is an example of shortcut learning?

A. The model uses a medically relevant lesion

B. The model uses a hospital logo correlated with the label

C. The model is evaluated on an external dataset

D. The model reports a confidence interval

Correct Answer: B. The model uses a hospital logo correlated with the label

Question 9

What is distribution shift?

A. A change in model parameters during backpropagation

B. A difference between training and deployment data conditions

C. A method for balancing classes

D. A type of activation function

Correct Answer: B. A difference between training and deployment data conditions

Question 10

Why is a test set repeatedly used during hyperparameter tuning problematic?

A. It reduces the number of features

B. It causes the model to overfit evidence from the test set

C. It guarantees fairness

D. It converts classification into regression

Correct Answer: B. It causes the model to overfit evidence from the test set