Model Evaluation, Validation and Tuning

Introduction

A machine-learning model is useful only when its performance is measured reliably and when it performs well on data that were not used to build it. Model evaluation is therefore more than calculating a single accuracy value. It involves designing a valid experimental procedure, selecting metrics appropriate to the business or scientific problem, identifying errors, and improving the model without allowing information from the future or the test set to influence development.

This chapter presents a practical framework for evaluating and improving supervised-learning models. It covers train–validation–test splitting, cross-validation, classification metrics, threshold selection, calibration, hyperparameter tuning, and error analysis. Although examples emphasize classification, many principles also apply to regression.

Learning Objectives

By the end of this chapter, you should be able to:

• explain the purpose of training, validation and test sets;
• distinguish data leakage, overfitting and underfitting;
• use cross-validation appropriately;
• construct and interpret a confusion matrix;
• calculate and interpret precision, recall, F1-score, ROC-AUC and calibration measures;
• select classification thresholds according to practical costs and benefits;
• tune hyperparameters using a valid evaluation procedure;
• perform structured error analysis; and
• communicate model performance responsibly.

1. Generalisation and the Evaluation Problem

A model learns a relationship between input variables and a target variable. Let the input be represented by a vector x and the target by y. A fitted model produces a prediction \hat{y} = f(x;\theta), where \theta represents learned parameters.

The central objective is not to reproduce the training data, but to perform well on new observations drawn from the same target population. This ability is called generalisation. A model may have very low training error but high error on new data; this is overfitting. Conversely, a model that is too simple may perform poorly on both training and unseen data; this is underfitting.

For a loss function L(y,\hat{y}), the empirical training risk is:

Here, n is the number of training observations, yᵢ is the observed target, and ŷᵢ is the model prediction. The corresponding population risk is the expected loss on future observations:

The population risk is usually unknown, so a carefully designed sample is used to estimate it. The test set should provide this estimate only after model choices have been completed.

Data leakage

Data leakage occurs when information unavailable at prediction time, or information derived from the evaluation data, influences model training. Examples include:

• scaling the full dataset before splitting;
• imputing missing values using statistics calculated from the test set;
• using a feature recorded after the outcome occurred;
• selecting features using the complete dataset before cross-validation; and
• repeatedly checking test performance while choosing models.

Leakage can produce impressive but misleading results. Every operation that learns from data—imputation, scaling, feature selection, dimensionality reduction and resampling—must be fitted only within the training portion of each evaluation split.

2. Train–Validation–Test Splits

A standard workflow divides labelled data into three subsets:

A parameter is learned directly by the algorithm, such as a linear-regression coefficient or a decision-tree split. A hyperparameter is selected by the practitioner, such as tree depth, regularisation strength or the number of neighbours.

A common split is 60% training, 20% validation and 20% test, although the appropriate proportions depend on dataset size. With limited data, a separate validation set may waste observations; cross-validation can provide a more efficient alternative.

Practical splitting principles

1. Random splitting is appropriate when observations are independent and identically distributed and the future resembles the past.
2. Stratified splitting preserves class proportions and is useful for classification, particularly with imbalanced classes.
3. Group splitting keeps related observations together—for example, all records from one patient, customer or machine. Otherwise, the model may see nearly identical information in both training and test sets.
4. Time-based splitting trains on earlier observations and tests on later observations when deployment predicts the future.
5. A fixed random seed improves reproducibility, but it does not remove sampling uncertainty.

A test set is not a source of feedback during development. If the test score is repeatedly consulted to make decisions, it effectively becomes another validation set and loses its independence.

3. Cross-Validation

In k-fold cross-validation, the training data are divided into k approximately equal folds. The model is trained on k−1 folds and evaluated on the remaining fold. This process is repeated until each fold has served as validation data once.

The cross-validation estimate of a metric M is commonly calculated as:

where Mⱼ is the score from fold j. The variability of fold scores can be reported using their sample standard deviation:

The standard deviation describes variation across folds; it is not automatically a confidence interval for future performance.

Common choices are five-fold or ten-fold cross-validation. Larger k gives each training run more data but requires more computation and can produce more correlated validation results. For classification, stratified k-fold cross-validation attempts to preserve class proportions in each fold. For time series, ordinary random folds are usually invalid because they can allow future information to enter the past; rolling or expanding time windows are more appropriate.

Nested cross-validation

When many hyperparameters or model families are compared, the same cross-validation process can become optimistic because it is used both to select and assess the model. Nested cross-validation addresses this by using:

• an inner loop to select hyperparameters; and
• an outer loop to estimate performance of the entire selection procedure.

For a final production model, practitioners often use cross-validation for selection, refit the selected configuration on all available training data, and evaluate it once on the untouched test set.

4. Classification Evaluation

The confusion matrix

For binary classification, predictions can be compared with actual labels using four counts:

A true positive is a correctly detected positive case. A false positive is an incorrect positive alert. A false negative is a missed positive case. A true negative is a correctly rejected negative case.

The following measures assume that the positive class has been defined clearly.

Accuracy

Accuracy is the proportion of all predictions that are correct. It is appropriate when classes and error costs are reasonably balanced. It can be misleading with class imbalance. If only 1% of transactions are fraudulent, always predicting “not fraud” gives 99% accuracy but detects no fraud.

Precision

Precision answers: among cases predicted positive, what proportion are actually positive? It matters when false alarms are costly, such as investigating alerts or contacting customers.

Recall or sensitivity

Recall answers: among all actual positive cases, what proportion did the model find? It matters when missing a positive case is costly, such as failing to identify a dangerous medical condition.

Specificity

Specificity is the proportion of actual negatives correctly identified. The false-positive rate is:

F1-score

The F1-score is the harmonic mean of precision and recall:

It is defined when precision plus recall is non-zero. The harmonic mean gives a low score when either component is low, so F1 is useful when both false positives and false negatives matter. It does not directly include true negatives and should not be treated as a universal replacement for accuracy.

Worked example

Suppose a fraud detector produces TP = 36, FP = 9, FN = 14 and TN = 941. There are 1,000 transactions.

The accuracy of 97.7% looks high, partly because negatives dominate. Precision of 80% means that one in five alerts is false, while recall of 72% means that 28% of fraud cases are missed. These values support a more informative discussion than accuracy alone.

Balanced accuracy

For binary classification, balanced accuracy is the average of sensitivity and specificity:

It gives equal importance to the two classes and is often more informative than ordinary accuracy under class imbalance.

5. Probability Scores, ROC-AUC and Precision–Recall

Many classifiers produce a score or estimated probability rather than a final class. A threshold converts this value into a decision. For example, with threshold t, predict positive when p(y=1|x) ≥ t. The default threshold of 0.5 is not universally optimal.

ROC curve and ROC-AUC

A receiver operating characteristic (ROC) curve plots true-positive rate against false-positive rate over all possible thresholds. The area under this curve, ROC-AUC, measures the model’s ranking ability. Informally, it is the probability that a randomly selected positive receives a higher score than a randomly selected negative, subject to handling of ties.

An AUC of 0.5 corresponds to random ranking, while 1.0 represents perfect ranking. AUC is threshold-independent, but it does not tell an organisation which threshold to use, nor does it guarantee useful probability estimates. ROC-AUC can also appear strong when the positive class is very rare.

Precision–recall curves

A precision–recall curve displays precision against recall as the threshold changes. It is often more informative than the ROC curve when positive cases are rare because it focuses on the quality of positive predictions. The baseline precision for a random classifier is approximately the prevalence of the positive class.

Calibration

A classifier is calibrated when predicted probabilities correspond to observed frequencies. For example, among cases assigned a probability near 0.70, approximately 70% should be positive over a sufficiently large and representative group.

Calibration is distinct from discrimination. A model may rank positives above negatives well but produce probabilities that are systematically too high or too low. Calibration can be assessed with reliability diagrams, calibration curves and summary measures such as the Brier score:

Here, pᵢ is the predicted probability for observation i, yᵢ is 0 or 1, and lower values are better. The Brier score depends on class prevalence and combines aspects of calibration and refinement; it should therefore be interpreted with suitable baselines.

Calibration methods include Platt scaling, which fits a logistic transformation to model scores, and isotonic regression, a flexible non-parametric monotonic mapping. Calibration must be fitted on data separate from the data used to train the original model, commonly through cross-validation. Poor calibration can lead to harmful decisions when probabilities determine resource allocation or risk treatment.

6. Threshold Selection

Threshold selection is a decision problem, not merely a technical default. Lowering a threshold usually increases recall and false-positive rate, while raising it usually decreases recall and increases precision, although exact changes depend on the score distribution.

A threshold can be chosen using:

• a required minimum recall or precision;
• a capacity limit, such as the number of cases an investigation team can review;
• an expected-cost calculation;
• a policy or safety requirement; or
• a validation-set metric such as F1, selected cautiously.

If false-positive cost is C_FP and false-negative cost is C_FN, an expected cost for a confusion matrix can be written as:

This simplified expression assumes that the costs are additive and that true-positive and true-negative benefits are either zero or represented separately. The threshold should be selected on validation data, not on the test set. If costs differ by person or circumstance, a single global threshold may be inappropriate.

7. Regression Evaluation

Although this chapter focuses on classification, evaluation principles also apply to regression. Common measures include mean absolute error (MAE), mean squared error (MSE) and root mean squared error (RMSE):

MAE is expressed in the same units as the target and gives errors a linear penalty. MSE and RMSE penalise large errors more strongly; RMSE has the target’s units. R² = 1 − SSE/SST compares residual variation with total variation, but it can be misleading when the data distribution changes and does not directly express prediction error in practical units.

8. Hyperparameter Tuning

Hyperparameter tuning is the systematic selection of settings that control learning. Examples include regularisation strength, tree depth, minimum samples per leaf, learning rate and number of estimators.

Search strategies

Grid search evaluates every combination in a predefined grid. It is transparent but can be expensive and inefficient when many parameters are irrelevant.

Random search samples combinations from specified distributions. It often explores high-impact parameters more effectively under a fixed computational budget.

Bayesian or sequential optimisation uses previous trial results to choose promising new configurations. It can be efficient, but it adds methodological and implementation complexity.

Successive halving allocates small resources to many configurations, eliminates weak candidates, and gives more resources to promising ones. The resource might be training iterations, data volume or number of trees.

A sound tuning procedure should:

1. define the target metric before searching;
2. establish a simple baseline;
3. specify realistic parameter ranges;
4. use pipelines so preprocessing occurs inside each fold;
5. use stratified, grouped or temporal validation as appropriate;
6. record random seeds, data versions and software versions;
7. avoid using the test set to choose settings; and
8. compare the tuned model with the baseline, including variability and operational cost.

Tuning for several metrics can produce different winners. The selected metric should reflect the actual decision objective rather than the easiest score to improve.

9. Error Analysis

A metric summarises performance, but error analysis explains it. A practical error-analysis cycle is:

1. inspect false positives and false negatives;
2. group errors by meaningful segments, such as region, product, device, age group or time period;
3. examine missing values, outliers and unusual feature combinations;
4. check label quality and ambiguous cases;
5. compare performance at different score thresholds;
6. identify systematic patterns; and
7. propose and test one improvement at a time.

A useful error table may contain the record identifier, true label, predicted label, probability, model version, important features and reason for human review. Care is required with personally identifiable or sensitive data.

Error analysis can reveal that the problem is not the algorithm. Labels may be delayed or inconsistent; a feature may be unavailable at deployment; one subgroup may be underrepresented; or the operating threshold may be unsuitable. Performance should therefore be reported overall and, where justified, by relevant subgroup. Small subgroup samples produce uncertain estimates, so subgroup results should include counts and appropriate uncertainty descriptions.

10. Illustrative Case: Hospital Readmission Risk

Background

A hospital develops a model to identify patients at high risk of readmission within 30 days. The model will support discharge planning, not automatically deny treatment.

Problem

Readmission occurs in 12% of the historical population. A false negative may result in a patient missing useful follow-up support, while a false positive consumes limited care-coordination capacity. Patient records contain repeated visits, so records from the same patient must not be randomly distributed across training and test sets.

Analysis

The data science team uses a patient-level group split. The most recent period is reserved as a temporal test set to approximate future deployment. Within the earlier data, grouped stratified cross-validation is used for hyperparameter selection. All imputation and scaling are fitted within each training fold.

The team reports recall, precision, specificity, PR-AUC, ROC-AUC and calibration. Accuracy is not used as the primary measure because most patients are not readmitted. A reliability diagram shows that the original probabilities are overconfident: cases assigned probabilities around 0.8 are readmitted less often than 80% of the time.

Approach and solution

The threshold is selected on validation data to achieve at least 80% recall while remaining within the care team’s weekly capacity. A calibration model is fitted without using the final test period. Errors are reviewed by discharge unit and by patient characteristics relevant to care access. The review finds that missing follow-up documentation is associated with both false positives and false negatives; this variable is removed because it is inconsistently available at deployment.

Outcome

On the untouched temporal test set, the calibrated model achieves lower raw accuracy than an always-negative baseline but substantially higher recall and useful precision at the operational threshold. The team reports confidence intervals or repeated-period variability, documents the threshold and capacity assumptions, and monitors performance after deployment.

Learning points

The case demonstrates that valid evaluation requires more than selecting the highest AUC. The split must represent deployment, preprocessing must be leakage-free, probability quality matters when risk scores guide action, and threshold selection must reflect capacity and harm. Ethical review is also necessary because unequal access to care or unequal data quality can create disparate error rates.

11. Limitations, Governance and Responsible Use

No metric captures every consequence of deployment. A high score on historical data may not persist after policy, population or technology changes. Dataset shift can occur through changes in prevalence, feature distributions or the relationship between features and outcomes.

Important governance practices include:

• documenting the data source, inclusion criteria and label definition;
• recording the split strategy and all preprocessing steps;
• reporting sample sizes, class prevalence and uncertainty;
• publishing the chosen threshold and its operational rationale;
• testing for subgroup performance differences where legally and ethically appropriate;
• protecting privacy and restricting access to sensitive evaluation records;
• retaining human oversight for high-impact decisions; and
• monitoring drift, calibration and error patterns after deployment.

Fairness cannot be established by one universal metric. Different fairness criteria may conflict, particularly when base rates differ. Evaluation should therefore involve domain experts, affected stakeholders, legal requirements and an explicit statement of the harms being considered.

12. Contemporary Practice

Modern machine-learning workflows increasingly use automated experiment tracking, reproducible pipelines, model registries and monitoring systems. Automated machine-learning tools can search models and hyperparameters, but they do not decide whether a split is valid, whether a label is meaningful, or whether a metric reflects social and operational priorities. Explainability tools can assist error analysis, but explanations are not substitutes for causal evidence or independent validation.

For deployed systems, evaluation becomes continuous. New labels may arrive late, thresholds may change, and monitoring may detect drift before ground-truth outcomes are available. A responsible lifecycle therefore includes pre-deployment testing, staged release, post-deployment monitoring and a documented process for rollback or retraining.

Chapter Summary

Reliable model evaluation begins with a split that reflects the intended use of the model. Training data fit parameters, validation data support model and threshold choices, and an untouched test set provides a final estimate of generalisation. Cross-validation improves the use of limited data, but it must respect class balance, groups and time ordering.

Confusion-matrix measures answer different questions: precision concerns the reliability of positive alerts, recall concerns the detection of actual positives, specificity concerns rejection of negatives, and F1 balances precision and recall. ROC-AUC assesses ranking, while precision–recall analysis is especially useful for rare positives. Calibration assesses whether predicted probabilities correspond to observed frequencies.

Hyperparameter tuning must be nested within a valid evaluation design and performed through leakage-free pipelines. Thresholds should reflect costs, capacity and safety requirements rather than defaulting automatically to 0.5. Finally, error analysis, subgroup assessment, documentation and post-deployment monitoring are essential for technically sound and responsible data-science practice.

Key Terms

Generalisation: Performance of a model on previously unseen data.

Overfitting: Learning patterns specific to training data that do not generalise.

Underfitting: Using a model that is too limited to capture important patterns.

Data leakage: Unintended use of information unavailable at prediction time or reserved for evaluation.

Training set: Data used to learn model parameters.

Validation set: Data used to compare models and select hyperparameters or thresholds.

Test set: Held-out data used for final performance estimation.

Cross-validation: Repeated training and validation across several data folds.

Stratification: Preserving class proportions across data splits.

Confusion matrix: Table of TP, FP, FN and TN counts.

Precision: Proportion of predicted positives that are actual positives.

Recall: Proportion of actual positives that are detected.

Specificity: Proportion of actual negatives correctly identified.

F1-score: Harmonic mean of precision and recall.

ROC-AUC: Area under the receiver operating characteristic curve.

Calibration: Agreement between predicted probabilities and observed frequencies.

Hyperparameter: Practitioner-selected setting controlling model learning.

Threshold: Cut-off used to convert a score or probability into a class decision.

Error analysis: Systematic investigation of correct and incorrect predictions.

Dataset shift: Change in data distributions or relationships after model development.

Review Questions

Short Answer Questions

1. What is the principal purpose of a test set?
2. Distinguish precision from recall.
3. Why can accuracy be misleading for an imbalanced classification problem?
4. What does calibration mean for a probabilistic classifier?
5. Give two examples of data leakage.

Long Answer Questions

1. Explain the roles of training, validation and test sets, including circumstances in which cross-validation is preferable to a fixed validation set.
2. Describe how a confusion matrix supports the calculation and interpretation of accuracy, precision, recall, specificity and F1-score.
3. Compare ROC-AUC with precision–recall analysis and explain when each is most informative.
4. Present a complete, leakage-free procedure for hyperparameter tuning.
5. Explain why threshold selection should be treated as a decision problem rather than a default software setting.

Analytical / Application Questions

1. A screening model has high recall but very low precision. Identify two operational consequences and propose two ways to investigate or improve the situation.
2. Design an evaluation strategy for a customer-churn model in which each customer has many monthly records and the model will predict future churn. Justify the split and validation method.
3. A model has ROC-AUC = 0.91 but poorly calibrated probabilities. Explain why this can occur and recommend an evaluation and remediation plan.

Multiple Choice Questions with Answers

Question 1

Which dataset should normally remain untouched until model selection and threshold decisions are complete?

A. Training set

B. Validation set

C. Test set

D. Bootstrap sample

Correct Answer: C. Test set

Question 2

A fraud model produces TP = 40 and FP = 10. What is its precision?

A. 0.40

B. 0.50

C. 0.75

D. 0.80

Correct Answer: D. 0.80

Question 3

Which metric measures the proportion of actual positive cases that are detected?

A. Precision

B. Recall

C. Specificity

D. Accuracy

Correct Answer: B. Recall

Question 4

Why is stratification commonly used in classification cross-validation?

A. To preserve class proportions across folds

B. To remove all outliers

C. To guarantee perfect calibration

D. To eliminate the need for a test set

Correct Answer: A. To preserve class proportions across folds

Question 5

What does ROC-AUC primarily measure?

A. Calibration of predicted probabilities

B. Ranking ability across thresholds

C. The number of false negatives at one fixed threshold

D. Training time

Correct Answer: B. Ranking ability across thresholds

Question 6

A classifier predicts probabilities near 0.80, but only about 0.55 of those cases are positive. What property is poor?

A. Calibration

B. Specificity

C. Feature scaling

D. Cross-validation fold size

Correct Answer: A. Calibration

Question 7

If false negatives are much more costly than false positives, which change will generally increase recall?

A. Raise the classification threshold

B. Lower the classification threshold

C. Remove the validation set

D. Use accuracy as the only metric

Correct Answer: B. Lower the classification threshold

Question 8

Which procedure is most appropriate when records from the same patient appear repeatedly?

A. Randomly split all records without restrictions

B. Keep patient groups together during splitting

C. Use only the training accuracy

D. Duplicate each patient in every fold

Correct Answer: B. Keep patient groups together during splitting

Question 9

What is the main purpose of hyperparameter tuning?

A. To select settings such as tree depth or regularisation strength

B. To change the true labels after prediction

C. To guarantee zero test error

D. To replace error analysis

Correct Answer: A. To select settings such as tree depth or regularisation strength

Question 10

A model has TP = 30 and FN = 10. What is its recall?

A. 0.25

B. 0.30

C. 0.75

D. 0.90

Correct Answer: C. 0.75