Model Selection, Evaluation, and Explainability

Introduction

Building a machine-learning model is not complete when an algorithm produces predictions. A professional practitioner must determine whether the model generalises to unseen data, whether its performance is useful for the intended decision, whether a simpler model would be preferable, and whether its predictions can be explained responsibly. Model selection, evaluation, and explainability connect technical modelling with reliable application.

A model may obtain excellent accuracy on its training data and still fail in deployment because it has memorised noise, because the test data do not represent future cases, or because the chosen metric ignores costly errors. This chapter presents a practical framework for selecting and evaluating models. It covers data splitting, cross-validation, baselines, metrics, hyperparameter tuning, bias and variance, regularisation, feature importance, and introductory local and global explainability.

Learning Objectives

By the end of this chapter, you should be able to:

• distinguish training, validation, and test data;
• select suitable evaluation metrics for classification and regression;
• use cross-validation without contaminating evaluation data;
• compare models with meaningful baselines;
• explain the roles of hyperparameters and regularisation;
• diagnose underfitting, overfitting, bias, and variance;
• interpret feature importance and introductory explainability methods;
• recognise limitations, fairness risks, and governance responsibilities in model evaluation.

1. The Model Selection Problem

Suppose a dataset contains observations (x_i, y_i), where x_i is a vector of input features and y_i is the target. A learning algorithm uses training data to estimate a predictive function f̂(x). Model selection involves choosing among algorithms, feature representations, and hyperparameter settings so that the selected model performs well on future observations.

A model's performance on a sample is an estimate of its generalisation performance: the expected performance on cases drawn from the same target population. This assumption is called the independent and identically distributed, or i.i.d., assumption. It can be inappropriate for time series, changing populations, repeated measurements, or data collected through a biased process.

Model selection must therefore answer three questions:

1. How well does the model fit data used for learning?
2. How well does it perform on data not used to fit or tune it?
3. Is its performance adequate, understandable, fair, and operationally useful?

2. Training, Validation, and Test Sets

A dataset is commonly divided into three parts.

Model parameters are learned from data, such as the coefficients in linear regression or the split rules in a decision tree. Hyperparameters are chosen before or around training, such as tree depth, learning rate, or regularisation strength.

A typical random split might allocate 60–80% to training, 10–20% to validation, and 10–20% to testing. The exact proportions depend on the dataset size and the cost of holding out data. For classification, stratified splitting is often appropriate: each partition approximately preserves the class proportions. For time-dependent data, random splitting may cause future information to enter the past; chronological splitting is then safer.

Data leakage occurs when information unavailable at prediction time influences training or evaluation. Examples include calculating a normalisation mean using the entire dataset, using a post-outcome variable as a predictor, and selecting features after examining the test results. Leakage usually produces an optimistically biased estimate.

A sound workflow is:

1. define the prediction task and evaluation metric;
2. separate the test set before extensive experimentation;
3. fit preprocessing steps only on training data;
4. tune models using training data and validation or cross-validation;
5. choose a final model;
6. evaluate it once on the untouched test set;
7. document the data, procedure, metric, and uncertainty.

3. Cross-Validation

When data are limited, a single validation split may give an unstable estimate. In k-fold cross-validation, the training portion is divided into k approximately equal folds. For each fold, the model is trained on the other k − 1 folds and evaluated on the held-out fold. The cross-validation score is commonly the mean of the k scores:

Here, k is the number of folds and s_j is the score obtained on fold j. The formula assumes that all fold scores use the same metric and that a larger score consistently means better performance. For a loss where smaller is better, the mean loss is reported instead.

The sample standard deviation of the fold scores indicates variability:

This is not a guarantee of future uncertainty, but it reveals sensitivity to the particular partitions. Common choices are k = 5 or k = 10. Larger k uses more training data in each fit but requires more computation and may produce more correlated fold estimates.

In stratified k-fold cross-validation, folds preserve class proportions. In grouped cross-validation, all observations from one group—for example, one patient or customer—remain in the same fold. In time-series cross-validation, training data precede validation data. These designs prevent dependence between training and validation observations.

Preprocessing must be inside the cross-validation procedure. For example, each fold should calculate its own scaling parameters from its training subset. A pipeline is a useful implementation pattern because it applies transformations and the estimator in the correct order.

4. Baselines and Evaluation Metrics

A baseline is a simple reference method against which a learned model is compared. For classification, a majority-class classifier may always predict the most common class. For regression, a baseline may always predict the training mean or median. A complex model that does not outperform a sensible baseline may not justify its cost or risk.

4.1 Confusion-Matrix Metrics

For binary classification, predictions can be represented by:

Accuracy is the proportion of correct predictions:

It can be misleading under class imbalance. Precision measures the proportion of predicted positives that are correct:

Recall, also called sensitivity or true-positive rate, measures the proportion of actual positives detected:

The F1 score is the harmonic mean of precision and recall:

These formulas require non-zero denominators. The choice between precision and recall depends on consequences. In a screening system, missing a genuine case may be more harmful than reviewing a false alarm, so recall may be prioritised. In an expensive intervention, precision may be more important.

Specificity measures the proportion of actual negatives correctly identified:

A receiver operating characteristic curve displays true-positive rate against false-positive rate at different thresholds. The area under this curve, ROC-AUC, measures ranking ability across thresholds, but can appear optimistic when negative cases greatly outnumber positive cases. Precision–recall curves may be more informative for rare positive events.

4.2 Regression Metrics

For actual values y_i and predictions ŷ_i, with n observations, mean absolute error is:

MAE has the same units as the target and gives linear importance to errors. Mean squared error is:

Its square root, root mean squared error, is:

RMSE also uses target units but penalises large errors more heavily. Mean absolute percentage error can be problematic when actual values are zero or close to zero. R², the coefficient of determination, is commonly defined as:

Here, ȳ is the mean of the actual target values. R² compares the model with predicting the sample mean; it may be negative when the model performs worse than that baseline. It is not automatically a measure of practical usefulness.

4.3 Choosing Metrics Professionally

Metric selection should reflect the decision. Before modelling, specify the primary metric, acceptable minimum performance, important subgroup checks, and the relative costs of errors. Report confidence intervals or variation across folds when possible. A single rounded score hides uncertainty and may encourage unjustified claims.

5. Hyperparameter Tuning

Hyperparameters control the learning process or model complexity. Examples include the number of neighbours in k-nearest neighbours, maximum tree depth, number of trees in a forest, learning rate, and regularisation strength.

Grid search evaluates a predefined set of combinations. Random search samples combinations from specified distributions and can be more efficient when only a few hyperparameters strongly influence performance. Successive-halving methods allocate more resources to promising configurations. Whatever method is used, the test set must not guide the search.

A robust procedure uses nested cross-validation when an unbiased estimate is especially important. The outer loop estimates generalisation. Within each outer training portion, an inner loop selects hyperparameters. The outer validation fold is never used for tuning. Nested validation is computationally expensive but avoids reporting a score influenced by repeated model selection.

Hyperparameter tuning should include practical constraints. A model with a slightly better score may be unacceptable if it is too slow, difficult to monitor, expensive to retrain, or impossible to explain to affected users.

6. Bias, Variance, and Regularisation

Prediction error can be understood through bias and variance. Bias is systematic error caused by an overly restrictive model or unsuitable assumptions. Variance is sensitivity to the particular training sample. High bias commonly produces underfitting: both training and validation performance are poor. High variance commonly produces overfitting: training performance is strong but validation performance is substantially worse.

For squared-error prediction, a simplified decomposition at a fixed input is:

Noise variance represents irreducible randomness in the target. The decomposition assumes a specified data-generating process and squared loss; it is a conceptual framework rather than a direct calculation from one dataset.

Regularisation discourages overly complex solutions. Ridge regression minimises squared error plus an L2 penalty:

Here, β₀ is the intercept, β_j are feature coefficients, x_ij is feature j for observation i, p is the number of features, and λ ≥ 0 controls penalty strength. The intercept is usually not penalised. Larger λ generally shrinks coefficients more strongly, reducing variance but potentially increasing bias.

Lasso regression uses an L1 penalty:

Lasso can set some coefficients exactly to zero, providing a form of feature selection. Elastic net combines L1 and L2 penalties. Because penalty effects depend on feature scale, numerical features should generally be standardised before regularised linear modelling. Regularisation strength must be selected using validation or cross-validation.

7. Feature Importance and Explainability

Explainability concerns how a model's behaviour can be understood by people. Interpretability is not identical to accuracy, and an explanation is not proof that a model is fair or causally correct.

7.1 Global Explanations

Permutation importance measures how much a score deteriorates when one feature's values are randomly shuffled while other features remain unchanged. If shuffling feature j reduces performance substantially, the model relied on information associated with that feature. The procedure should be repeated to assess variability and performed on validation or test data, not only training data.

Permutation importance can be misleading when features are correlated: shuffling one feature may not hurt because a correlated substitute remains available. Built-in tree impurity importance can also favour high-cardinality or continuous variables. Therefore, importance rankings should be treated as model-dependent evidence, not as universal rankings of causal importance.

Partial dependence describes the average predicted response as a selected feature varies, averaging over other observations. It can be misleading when the plotted feature combinations are unrealistic or rare. Accumulated local effects are often preferable when correlated features make partial-dependence extrapolation problematic.

7.2 Local Explanations

A local explanation describes one prediction. For a linear model, the contribution of a feature can often be related directly to its coefficient and input value, provided scaling and reference values are understood. For complex models, methods such as LIME approximate the model near one case with a simpler surrogate. SHAP methods allocate a prediction difference among features using Shapley-value ideas from cooperative game theory. They require careful choices of background data and can be computationally demanding.

An explanation should state whether it is global or local, which model it explains, what reference or baseline is used, and what limitations apply. Explanations should not expose sensitive information or enable harmful manipulation.

8. Worked Example: Selecting a Customer-Churn Model

Assume a telecommunications provider wants to identify customers likely to leave within 30 days. The positive class is churn. Missing a likely churner may lead to lost revenue, while contacting a customer unnecessarily incurs a smaller campaign cost.

The data are divided chronologically: earlier customers and observations form the development data, while the most recent period is reserved as the final test set. This reflects the intended deployment direction. Preprocessing, including missing-value imputation and scaling, is placed inside a pipeline.

Three models are compared using five-fold stratified cross-validation on the development data:

The random forest has higher recall but slightly lower F1. Because the business prioritises identifying potential churners, the team investigates thresholds rather than automatically selecting the highest accuracy. A validation analysis shows that lowering the probability threshold increases recall but also increases campaign volume. The decision threshold is therefore chosen using the intervention budget and an explicitly documented cost trade-off.

Feature analysis shows that recent service complaints, contract type, and months since the last support interaction are important predictors. However, support interaction may partly reflect previous retention interventions, and contract type may correlate with socioeconomic characteristics. The team checks performance by customer segment, documents limitations, and avoids claiming that these features cause churn.

Finally, the selected pipeline is refitted on all development data and evaluated once on the recent test period. The report includes the test confusion matrix, recall, precision, calibration assessment, subgroup results, confidence or variability information, and a monitoring plan. If performance declines because customer behaviour changes, retraining or threshold review may be required.

This case illustrates that model selection is not simply choosing the algorithm with the highest score. It combines data design, metric choice, threshold policy, explainability, fairness, operational constraints, and monitoring.

9. Limitations, Ethics, and Governance

Evaluation results are conditional on the data and procedure used. Distribution shift occurs when deployment data differ from development data. Concept drift occurs when the relationship between features and target changes. Sampling bias can produce apparently strong performance for one population and poor performance for another.

In high-impact applications, practitioners should assess disparate error rates and calibration across relevant groups, while respecting privacy and legal requirements. A feature may be legally permissible yet act as a proxy for a protected characteristic. Removing the feature does not necessarily remove the underlying bias because other variables may encode similar information.

Explanations must be communicated in a way appropriate to the audience. A technical feature-importance chart is not sufficient notice for a person affected by an automated decision. Organisations should maintain documentation of data provenance, model version, evaluation design, intended use, known limitations, human oversight, and procedures for appeal or correction.

Security is also relevant. Publicly exposing detailed explanations can help an attacker reverse-engineer decision boundaries or manipulate inputs. Explainability should therefore be balanced with privacy and robustness.

10. Best-Practice Workflow

A practical workflow is as follows:

1. Define the target, prediction time, population, and decision consequences.
2. Establish a simple baseline.
3. Choose a split strategy appropriate to the data-generating process.
4. Prevent leakage by fitting every learned transformation within training folds.
5. Select metrics that reflect error costs and class prevalence.
6. Compare several models, including a simple interpretable model.
7. Tune hyperparameters using cross-validation, keeping the test set untouched.
8. Examine training and validation performance for bias–variance symptoms.
9. Evaluate calibration, subgroup performance, and operational constraints.
10. Use global and local explanations cautiously, checking correlated features and unrealistic scenarios.
11. Report uncertainty, limitations, and the final test result honestly.
12. Monitor performance, data quality, drift, fairness, and explanation stability after deployment.

Chapter Summary

Reliable machine learning requires more than fitting an algorithm. Training, validation, and test data have distinct roles, and leakage must be prevented. Cross-validation provides a more stable comparison when data are limited, while specialised folds are needed for grouped, imbalanced, or time-dependent data. Baselines reveal whether complexity adds value.

Classification metrics such as precision, recall, F1, specificity, ROC-AUC, and precision–recall analysis answer different questions. MAE, MSE, RMSE, and R² provide complementary views of regression error. Hyperparameter tuning must be separated from final testing. Bias and variance explain underfitting and overfitting, while regularisation controls complexity.

Feature importance, permutation methods, partial dependence, LIME, and SHAP can support interpretation, but explanations are model-dependent and do not establish causation or fairness. Professional evaluation also considers data shift, subgroup performance, privacy, security, governance, operational cost, and post-deployment monitoring.

Key Terms

Model selection: Choosing a model, representation, or hyperparameter configuration for a task.

Generalisation: Performance on new data from the target population.

Training set: Data used to estimate model parameters.

Validation set: Data used to compare models or tune hyperparameters.

Test set: Held-out data used for final evaluation.

Data leakage: Unintended use of information unavailable at prediction time.

Cross-validation: Repeated training and validation across data folds.

Baseline: A simple reference prediction method.

Hyperparameter: A setting chosen outside the model's learned parameters.

Accuracy: Proportion of all predictions that are correct.

Precision: Proportion of predicted positives that are true positives.

Recall: Proportion of actual positives detected.

F1 score: Harmonic mean of precision and recall.

MAE: Mean absolute prediction error.

RMSE: Square root of mean squared error.

Bias: Systematic error from restrictive assumptions.

Variance: Sensitivity to the particular training sample.

Regularisation: Penalising complexity to improve generalisation.

Permutation importance: Performance change after randomly shuffling a feature.

Local explanation: An explanation of one individual prediction.

Review Questions

5 Short Answer Questions

1. What is the distinct purpose of a test set?
2. Why can accuracy be misleading for an imbalanced classification problem?
3. State one difference between precision and recall.
4. What is the purpose of regularisation?
5. Why must preprocessing occur inside cross-validation folds?

5 Long Answer Questions

1. Explain a complete train–validation–test workflow and discuss how data leakage can occur.
2. Compare k-fold, stratified, grouped, and time-series cross-validation.
3. Explain the bias–variance trade-off and relate it to underfitting and overfitting.
4. Compare ridge and lasso regression, including their penalties and practical uses.
5. Discuss the benefits and limitations of feature importance and local explainability methods.

3 Analytical / Application Questions

1. A fraud classifier has very high accuracy but detects only 8% of fraudulent transactions. Diagnose the evaluation problem and recommend suitable metrics and validation procedures.
2. A hospital model uses repeated records from the same patient. Design a split and cross-validation strategy that avoids overly optimistic results, and justify it.
3. A churn model performs well overall but has much lower recall for one customer group. Propose a systematic investigation and identify technical, ethical, and governance actions.

Multiple Choice Questions with Answers

Question 1

Which dataset should be used only for the final unbiased performance estimate?

A. Training set

B. Validation set

C. Test set

D. Feature-engineering set

Correct Answer: C. Test set

Question 2

Which metric is the proportion of predicted positive cases that are actually positive?

A. Recall

B. Precision

C. Specificity

D. Accuracy

Correct Answer: B. Precision

Question 3

What is the main purpose of stratified cross-validation?

A. To preserve approximate class proportions in folds

B. To remove all missing values

C. To guarantee causal interpretation

D. To increase the number of features

Correct Answer: A. To preserve approximate class proportions in folds

Question 4

A model has excellent training performance but poor validation performance. Which issue is most likely?

A. Underfitting

B. High bias only

C. Overfitting and high variance

D. Insufficient test leakage

Correct Answer: C. Overfitting and high variance

Question 5

Which penalty is associated with ridge regression?

A. Sum of absolute coefficient values

B. Sum of squared coefficient values

C. Number of validation folds

D. Classification accuracy

Correct Answer: B. Sum of squared coefficient values

Question 6

Why is a majority-class classifier useful as a baseline?

A. It always gives state-of-the-art performance

B. It provides a simple reference for judging model value

C. It eliminates class imbalance

D. It tunes hyperparameters automatically

Correct Answer: B. It provides a simple reference for judging model value

Question 7

Which metric is especially sensitive to large regression errors because errors are squared?

A. MSE

B. MAE

C. Recall

D. Precision

Correct Answer: A. MSE

Question 8

What does permutation importance usually measure?

A. The causal effect of a feature

B. The change in performance after shuffling a feature

C. The number of missing values in a feature

D. The model's training duration

Correct Answer: B. The change in performance after shuffling a feature

Question 9

In nested cross-validation, what is the primary role of the outer loop?

A. To estimate generalisation after inner-loop tuning

B. To calculate feature means using all data

C. To replace the target variable

D. To guarantee fairness

Correct Answer: A. To estimate generalisation after inner-loop tuning

Question 10

Why may a feature-importance ranking not establish causation?

A. Importance reflects model reliance and may involve correlations

B. Importance is always calculated without data

C. Causation requires a larger test set only

D. Feature importance applies only to labels

Correct Answer: A. Importance reflects model reliance and may involve correlations