Model Evaluation, Validation and Explainability

Introduction

Predictive modelling is not complete when an algorithm produces predictions. A model is useful only when its performance is measured fairly, its results generalise to new observations, and its outputs can be understood and appropriately governed. Model evaluation addresses the question, “How accurate and useful are the predictions?” Validation addresses, “Will the model perform on future or unseen data?” Explainability addresses, “Why did the model produce this result, and can users trust it?”

These questions are especially important in business analytics. A forecasting model may guide inventory purchases, a classification model may identify customers at risk of leaving, and a credit model may influence lending decisions. In each case, a high training score alone is insufficient. The analyst must control overfitting, prevent data leakage, select suitable metrics, examine errors across groups and time, and communicate model behaviour to decision-makers.

Learning objectives

By the end of this chapter, you should be able to:

• distinguish training, validation and test data;
• select suitable holdout and cross-validation strategies;
• calculate and interpret MAE, RMSE, R-squared, precision, recall, F1, ROC-AUC and calibration;
• evaluate regression, classification and time-series models;
• explain predictions using feature importance and local explanation methods;
• identify leakage, bias, instability and other evaluation risks; and
• recommend a defensible model for a business application.

1. Generalisation and the evaluation problem

A model learns a relationship from a sample. Its practical value depends on its expected performance on observations that were not used to fit it. This property is called generalisation. A model that memorises accidental patterns in the training data has low training error but may have high future error; this is overfitting. A model that is too simple to capture important structure is underfitting.

The usual workflow separates data into:

1. Training set: used to estimate model parameters.
2. Validation set: used to select algorithms, features and hyperparameters.
3. Test set: used once, at the end, to estimate performance on unseen data.

For a random independent sample, a common allocation is 60–80% training, 10–20% validation and 10–20% test. The proportions are guidelines rather than rules. The test set must represent the intended deployment population. For time-dependent data, observations must normally be ordered chronologically: earlier observations are used for training and later observations for validation or testing.

A data leak occurs when information unavailable at prediction time enters the training process. Examples include using a customer’s eventual cancellation status to predict cancellation, scaling the complete dataset before splitting it, or randomly splitting a time series so that future values appear in training. Leakage can create impressive but false performance.

The complete modelling pipeline—not just the estimator—must be fitted inside the training process. Imputation, scaling, feature selection and encoding should be learned from training data only. The same transformations are then applied to validation and test data.

2. Cross-validation

Cross-validation uses repeated training and validation subsets to obtain a more stable estimate of model performance. In k-fold cross-validation, the training data are divided into k approximately equal folds. Each fold serves once as validation data while the other k − 1 folds are used for training. If m_i is the metric obtained on fold i, the mean cross-validation score is:

Here, k is the number of folds and m_i is calculated using only predictions for fold i. The formula applies when the folds are constructed appropriately and the metric is averaged consistently. For error metrics, lower values are usually better; for measures such as R-squared or ROC-AUC, higher values are generally better.

A standard choice is k = 5 or k = 10. Larger k uses more data for each fitted model but requires more computation and may produce more correlated estimates. Cross-validation does not remove sampling uncertainty; it estimates performance under the chosen splitting design.

Stratified k-fold cross-validation preserves the approximate class proportions in each fold and is useful for classification, especially when the positive class is uncommon. Group cross-validation keeps observations from the same customer, household, patient or organisation in one fold, preventing identity leakage. Time-series cross-validation respects temporal order. In a rolling-origin design, a model is trained on earlier data and evaluated on a subsequent time block, then the training window advances.

Cross-validation should not repeatedly be used to make decisions about a final test set. Hyperparameters are selected using training data and cross-validation; the untouched test set is used for final confirmation. When many models are compared, even the test set can become indirectly overused, so a nested cross-validation design may be appropriate for rigorous studies.

3. Regression evaluation

Regression predicts a numerical quantity such as sales, delivery time or energy consumption. Evaluation should use a scale and loss function aligned with the business decision.

Mean absolute error

For n observations, actual values y_i and predictions ŷ_i, mean absolute error is:

MAE measures the average magnitude of error in the same units as the target. It treats an error of 10 as twice as serious as an error of 5 and is less affected by extreme errors than RMSE. It does not indicate the direction of error because positive and negative errors are converted to absolute values.

Root mean squared error

RMSE is also expressed in target units. Squaring gives greater influence to large errors; consequently, RMSE is useful when serious forecasting failures are particularly costly. MAE and RMSE are comparable across models only when they are calculated on the same target scale and test observations.

For example, if actual demand is [100, 120, 80] and predictions are [90, 130, 85], the errors are [10, −10, −5]. Thus MAE = (10 + 10 + 5)/3 = 8.33 units, while RMSE = √[(100 + 100 + 25)/3] ≈ 8.66 units. The slightly larger RMSE reflects the penalty for squared errors.

R-squared

R-squared compares model error with variation around the mean of the observed target:

Here, ȳ is the sample mean of actual values. The denominator is the total variation in the observed outcomes. R² indicates the proportional reduction in squared error relative to predicting the sample mean. It can be negative on test data when the model performs worse than that baseline. R² is not a percentage of predictions that are correct, and a high value does not guarantee low business cost or unbiased errors.

Regression evaluation should also examine residuals e_i = y_i − ŷ_i. Important diagnostics include average error (bias), error by time period, error by customer or product segment, and error against predicted value. A residual pattern may reveal non-linearity, changing variance, seasonality or missing variables.

4. Classification evaluation

Classification predicts categories such as fraud/not fraud or churn/no churn. For a binary classifier, predictions can be represented by a confusion matrix:

The classification threshold converts a predicted probability into a class. Changing the threshold changes TP, FP, FN and TN. Therefore, a model and its threshold should be evaluated together.

Precision, recall and F1

Precision is the proportion of predicted positives that are truly positive. It matters when false alarms are costly, such as sending expensive retention offers to customers who would not leave.

Recall, also called sensitivity or true-positive rate, is the proportion of actual positives detected. It matters when missing a positive case is costly, such as failing to identify a fraudulent transaction.

F1 is the harmonic mean of precision and recall. It is defined when both precision and recall are meaningful and their sum is non-zero. It gives equal weight to the two measures and is not generally appropriate when false positives and false negatives have very different financial consequences. In imbalanced problems, accuracy can be misleading: predicting the majority class may achieve high accuracy while detecting no rare cases.

ROC-AUC

The receiver operating characteristic curve plots recall (true-positive rate) against the false-positive rate across classification thresholds. ROC-AUC is the area under this curve. Conceptually, it is the probability that a randomly selected positive case receives a higher score than a randomly selected negative case, subject to handling of ties. A value of 0.5 represents random ranking, while 1.0 represents perfect ranking. ROC-AUC evaluates ranking rather than the quality of a particular probability threshold. With severe class imbalance, precision–recall curves may provide a more informative view of positive-case performance.

Calibration

A classifier is calibrated when, among cases assigned a probability of 0.70, approximately 70% are positive over a sufficiently large and relevant group. Discrimination and calibration are different: a model can rank cases well (high ROC-AUC) but produce probabilities that are systematically too high or too low.

Calibration can be assessed using reliability diagrams, calibration curves and a proper scoring rule such as the Brier score:

Here, p_i is the predicted probability of the positive class and y_i is 1 for a positive outcome and 0 otherwise. Lower Brier score is better, although it reflects both calibration and refinement. Calibration methods such as Platt scaling or isotonic regression must themselves be fitted using validation data, not the final test outcomes.

5. Evaluation for time-series forecasting

Time-series evaluation has an additional requirement: the future must not influence the past. Random k-fold validation is generally inappropriate when observations are temporally dependent. Instead, use chronological holdouts or rolling-origin validation.

A simple baseline is essential. Examples include a naïve forecast equal to the last observed value, a seasonal-naïve forecast equal to the value from the previous season, or a historical mean. A sophisticated model should outperform a sensible baseline on the relevant horizon and cost measure.

Forecast accuracy should be reported separately by horizon, such as one day ahead, seven days ahead and thirty days ahead. Errors may also be summarised by season, product, region and demand level. Percentage metrics can be unstable when actual values are zero or close to zero; MAE, RMSE or scale-free alternatives may be safer. Prediction intervals should be evaluated for coverage: a nominal 90% interval should contain approximately 90% of future observations over repeated comparable cases, although finite samples and distribution shifts create uncertainty.

6. Feature importance and explainability

Explainability includes methods for understanding a model globally and explaining an individual prediction locally.

Permutation importance measures the increase in validation error after randomly shuffling one feature while leaving other features unchanged. If shuffling feature j increases error substantially, the model relied on information associated with that feature. It should be calculated on held-out data and repeated to assess variation. Correlated features can share importance, so shuffling one may understate the importance of a group or produce unstable rankings.

Tree-based importance measures such as impurity reduction are fast but can favour continuous or high-cardinality features and may not reflect causal importance. Feature importance means contribution to model prediction, not proof that the feature causes the outcome.

Partial dependence shows the average predicted response as a feature varies over a selected range, averaging over other observations. It can be misleading when feature combinations in that range are unrealistic or when features are strongly correlated.

Individual conditional expectation displays prediction curves for individual observations, revealing heterogeneous effects that an average partial-dependence plot can hide.

Local surrogate methods, such as LIME, approximate a complex model near one observation with an interpretable simpler model. Their results depend on how neighbourhoods are generated and which variables are perturbed.

SHAP-style explanations allocate a prediction among features using contribution values based on cooperative-game ideas. A common additive form is:

Here, f(x) is the model prediction for observation x, φ₀ is a baseline prediction, p is the number of features and φ_j is the contribution attributed to feature j under a specified background and feature-dependence assumption. The explanation is locally additive, but it is not automatically causal. Different background datasets, link functions and treatments of correlated features can produce different explanations.

Explainability should combine statistical evidence with domain knowledge. A useful explanation states the model version, input values, reference population, uncertainty and known limitations. It should not disclose sensitive information unnecessarily or create a false impression of certainty.

7. An integrated evaluation framework

A professional evaluation process can be organised as follows:

1. Define the decision, prediction time, target horizon and cost of errors.
2. Establish a simple business and statistical baseline.
3. Split data according to independence, groups and time.
4. Build preprocessing and modelling steps into a leakage-resistant pipeline.
5. Use cross-validation appropriate to the data-generating process.
6. Select metrics that reflect business priorities; report several complementary metrics.
7. Tune the decision threshold separately from probability estimation.
8. Examine calibration, subgroup performance and residual or error patterns.
9. Interpret the selected model using global and local methods.
10. Confirm final performance once on an untouched test set.
11. Document assumptions, versions, data dates, exclusions and limitations.
12. Monitor performance and calibration after deployment.

No single metric determines the best model. Model choice is a multi-objective decision involving accuracy, stability, latency, cost, fairness, interpretability and operational risk.

8. Illustrative case: predicting customer churn

Background and problem

A subscription company wants to identify customers likely to cancel during the next 30 days so that retention teams can contact them. The positive class—customers who cancel—is 12% of the historical population. Contacting every customer is too costly, while failing to identify a likely cancellation loses future revenue.

Analysis and approach

The analyst defines the prediction timestamp as the end of each customer’s latest completed billing period. Features are restricted to information available at that timestamp: usage frequency, tenure, service incidents, payment history and recent support contacts. The outcome is cancellation within the following 30 days. Records are split chronologically, with the final two months retained as the test period. Customers appearing in multiple records are kept within the appropriate temporal and group structure to reduce leakage.

A logistic regression and a gradient-boosted tree model are compared against a majority-class baseline. Stratified, time-aware validation is used within the development period. The analyst reports precision, recall, F1 and ROC-AUC, but also examines precision at the number of customers the retention team can contact each week. Because the company needs reliable risk probabilities for budget planning, calibration curves and the Brier score are included.

Suppose the selected threshold produces TP = 180, FP = 120 and FN = 60. Then precision = 180/(180 + 120) = 0.60 and recall = 180/(180 + 60) = 0.75. The F1 score is 2 × (0.60 × 0.75)/(0.60 + 0.75) ≈ 0.667. Management may accept this threshold if the expected value of preventing 180 cancellations exceeds the cost of contacting 300 customers. A lower threshold could increase recall but also increase contact cost.

Explainability and governance

Permutation importance identifies recent usage decline and unresolved service incidents as influential variables. Local explanations are reviewed for a sample of correct and incorrect predictions. The team checks whether explanations depend on protected attributes or their proxies, whether performance differs across customer segments, and whether the model uses post-cancellation information. Customer-facing communications describe the available support rather than claiming that the model knows a customer’s intentions.

Outcome and learning points

The model is approved for a monitored pilot rather than immediate full deployment. Its performance is tracked monthly, with alerts for changes in class prevalence, recall, calibration and subgroup error rates. The case demonstrates that evaluation is not merely a competition for the highest ROC-AUC. It connects data design, metrics, threshold decisions, explainability, operational capacity and responsible use.

9. Limitations, ethics and governance

Evaluation estimates future behaviour; it does not guarantee it. Distribution shift may occur when prices, competitors, regulations or customer behaviour change. Small test sets produce uncertain estimates, and repeated experimentation can lead to optimistic conclusions. Confidence intervals, bootstrap summaries or repeated validation can communicate uncertainty, although their assumptions must be checked.

Fairness should be assessed where decisions affect people. Compare relevant error rates, calibration and selection rates across legally and ethically meaningful groups, while recognising that some fairness criteria cannot all be satisfied simultaneously when base rates differ. Avoid using sensitive attributes merely to improve prediction without a lawful and ethical purpose. They may, however, be needed for auditing disparity.

Explainability does not establish causation. A feature may be predictive because it is a proxy, because it records a consequence rather than a cause, or because of historical discrimination. High-performing models should therefore undergo data protection review, documentation, access control, human oversight and an appeal or correction process where appropriate. Model cards, data sheets and versioned evaluation reports support accountability.

Contemporary practice increasingly includes automated monitoring, drift detection, conformal prediction for distribution-free prediction sets or intervals under stated assumptions, and responsible AI review. These tools support—not replace—sound data design and professional judgement.

Chapter Summary

Reliable predictive analytics requires evaluation on data that represent future use and were not used to fit or tune the model. Training, validation and test sets serve different purposes, while cross-validation improves development estimates when its splitting design matches the data. MAE and RMSE measure regression error in target units; R² compares squared error with a mean baseline. Precision, recall and F1 describe threshold-dependent classification performance, whereas ROC-AUC evaluates ranking across thresholds. Calibration assesses whether predicted probabilities correspond to observed frequencies.

Time-series evaluation must preserve chronology and should compare models with naïve and seasonal baselines. Explainability methods—including permutation importance, partial dependence, local surrogates and SHAP-style additive explanations—help users understand model behaviour but do not prove causation. A defensible evaluation also examines leakage, subgroup performance, uncertainty, drift, operational costs and governance requirements. The best model is therefore not necessarily the most complex or the one with the highest single score; it is the model whose validated performance and explanations support the intended decision responsibly.

Key Terms

Generalisation: Performance on new observations beyond the training data.

Overfitting: Learning sample-specific noise that harms performance on new data.

Underfitting: Using a model too simple to capture important structure.

Training set: Data used to estimate model parameters.

Validation set: Data used to select models, features or hyperparameters.

Test set: Held-out data used for final performance estimation.

Data leakage: Unavailable or inappropriate information entering model development.

Cross-validation: Repeated fitting and validation across data folds.

MAE: Average absolute prediction error.

RMSE: Square root of average squared prediction error.

R-squared: Relative reduction in squared error compared with a mean baseline.

Precision: Proportion of predicted positives that are truly positive.

Recall: Proportion of actual positives detected.

F1 score: Harmonic mean of precision and recall.

ROC-AUC: Threshold-independent measure of ranking discrimination.

Calibration: Agreement between predicted probabilities and observed frequencies.

Permutation importance: Change in held-out performance after shuffling a feature.

Local explanation: Explanation of one individual prediction.

Distribution shift: Change in the data-generating process between development and deployment.

Feature importance: A measure of how much model predictions depend on features, not necessarily causal influence.

Review Questions

Short Answer Questions

1. What is the distinct purpose of a validation set and a test set?
2. Why can accuracy be misleading for an imbalanced classification problem?
3. State one important difference between MAE and RMSE.
4. What does calibration mean for a predicted probability of 0.70?
5. Why is random cross-validation usually unsuitable for time-series data?

Long Answer Questions

1. Explain how a complete modelling pipeline can suffer from data leakage and describe controls that prevent it.
2. Compare precision, recall, F1 and ROC-AUC, including circumstances in which each is useful.
3. Explain chronological holdout and rolling-origin validation for forecasting.
4. Compare permutation importance, partial dependence and local explanation methods.
5. Discuss why a model with the highest test score may not be the best model for deployment.

Analytical / Application Questions

1. A fraud model has TP = 240, FP = 60 and FN = 100. Calculate precision, recall and F1, and explain whether the threshold should be changed if missed fraud is much more costly than investigation.
2. A retailer randomly splits three years of daily sales into training and test records. Identify two evaluation risks and propose a better validation design.
3. A churn model has high ROC-AUC but poor calibration and substantially lower recall for one customer group. Recommend an evaluation and governance plan before deployment.

Multiple Choice Questions with Answers

Question 1

Which dataset should normally be used only once for final performance estimation?

A. Training set

B. Validation set

C. Test set

D. Feature engineering set

Correct Answer: C. Test set

Question 2

If false negatives are especially costly, which metric is usually most directly relevant?

A. Recall

B. Precision

C. R-squared

D. Mean absolute error

Correct Answer: A. Recall

Question 3

For actual values [10, 20] and predictions [8, 23], what is the MAE?

A. 2.0

B. 2.5

C. 3.0

D. 5.0

Correct Answer: B. 2.5

Question 4

What is a defining property of time-series cross-validation?

A. It randomly shuffles all observations

B. It preserves temporal order

C. It uses only the test set

D. It removes seasonal effects automatically

Correct Answer: B. It preserves temporal order

Question 5

A calibrated group assigned predicted probability 0.80 should, over many comparable cases, have approximately what positive rate?

A. 20%

B. 50%

C. 80%

D. 100%

Correct Answer: C. 80%

Question 6

ROC-AUC primarily measures a classifier’s ability to do what?

A. Produce unbiased residuals

B. Rank positive cases above negative cases

C. Choose the cheapest threshold

D. Guarantee calibrated probabilities

Correct Answer: B. Rank positive cases above negative cases

Question 7

Which method estimates feature importance by shuffling a feature and measuring performance change?

A. Permutation importance

B. Seasonal decomposition

C. One-hot encoding

D. Rolling-origin validation

Correct Answer: A. Permutation importance

Question 8

Why can a majority-class classifier have misleadingly high accuracy?

A. It always has perfect calibration

B. It may ignore a rare but important positive class

C. Accuracy uses squared errors

D. It necessarily has high ROC-AUC

Correct Answer: B. It may ignore a rare but important positive class

Question 9

If TP = 30 and FP = 10, precision is:

A. 0.25

B. 0.50

C. 0.75

D. 0.80

Correct Answer: C. 0.75

Question 10

What is the main caution when interpreting feature importance?

A. It always proves causation

B. It may be unstable with correlated features and is not causal evidence

C. It can only be calculated for linear models

D. It is identical to calibration

Correct Answer: B. It may be unstable with correlated features and is not causal evidence