Model Evaluation, Selection and Tuning

Introduction

A machine-learning model is useful only when its performance is measured reliably and when the measurement reflects the intended application. A model may achieve excellent accuracy on data used during training yet fail on new cases because it has memorized noise, benefited from data leakage, or been evaluated with an unsuitable metric. Model evaluation therefore concerns more than calculating a score: it involves designing a credible experimental procedure, selecting metrics aligned with business and social costs, comparing models fairly, and understanding the sources of error.

This chapter develops a practical framework for evaluating regression and classification models. It explains train-validation-test splitting, cross-validation, baselines, hyperparameter tuning, underfitting and overfitting, calibration, error analysis, and the trade-offs involved in model selection.

Learning Objectives

By the end of this chapter, you should be able to:

1. Distinguish training, validation, and test data and explain their proper use.

2. Apply suitable evaluation metrics to regression and classification problems.

3. Construct meaningful baseline models.

4. Use cross-validation and hyperparameter search without contaminating the test set.

5. Diagnose underfitting, overfitting, data leakage, and poor calibration.

6. Analyse error trade-offs and select a model for a real application.

7. Explain limitations, governance concerns, and best practices in model evaluation.

1. Generalisation and the Evaluation Problem

The central goal of supervised learning is generalisation: performing well on previously unseen examples drawn from the target population. Let a dataset consist of observations \((x_i,y_i)\), where \(x_i\) is a feature vector and \(y_i\) is a target. A learning algorithm uses training data to produce a predictor \(f\). Its empirical risk is the average loss on observed training examples:

\[

\hat R(f)=\frac{1}{n}\sum_{i=1}^{n}L(y_i,f(x_i)).

\]

Here, \(L\) is a loss function and \(n\) is the number of training observations. The quantity of practical interest is expected, or population, risk:

\[

R(f)=E[L(Y,f(X))],

\]

where the expectation is over the population or data-generating distribution. Because the population distribution is unknown, performance must be estimated using observations that were not used to fit or repeatedly select the model.

A model with low training error but high unseen-data error is overfitting. A model with high error on both training and unseen data is likely underfitting, although poor features, noisy labels, or an unsuitable metric may also be responsible.

Data leakage

Data leakage occurs when information unavailable at prediction time influences training or evaluation. Examples include:

• scaling the complete dataset before splitting;
• using a feature created after the prediction event;
• allowing duplicate individuals to occur in both training and test sets;
• selecting features using the test labels;
• imputing missing values using statistics calculated from all records.

Leakage usually produces overly optimistic results. Preprocessing must therefore be fitted only on training data, preferably inside a pipeline that is evaluated within each cross-validation fold.

2. Train, Validation and Test Splits

A conventional workflow separates data into three roles.

Training set

The training set is used to estimate model parameters, such as regression coefficients or neural-network weights. It may also be used to fit preprocessing operations such as standardisation and imputation.

Validation set

The validation set is used to compare model families, select hyperparameters, choose thresholds, and make design decisions. Hyperparameters are settings chosen by the practitioner rather than learned directly from each training example, such as tree depth, regularisation strength, or the number of neighbours.

Test set

The test set is held back until the modelling process is complete. It provides an approximately unbiased estimate of performance on future data, provided that it is representative and has not influenced model decisions. Repeatedly checking the test score turns the test set into an informal validation set and makes the reported estimate optimistic.

A common initial split is 60–80% for training, 10–20% for validation, and 10–20% for testing. These are guidelines rather than rules. With limited data, cross-validation can replace a fixed validation set. In time-dependent applications, random splitting is often inappropriate: earlier observations should train the model and later observations should evaluate it.

Stratification and grouping

For imbalanced classification, stratified splitting preserves approximately the same class proportions in each partition. If observations belong to people, households, hospitals, or devices, all observations from a group should generally remain in one partition. Otherwise, the model may recognise the group rather than learn a general relationship. Grouped and time-aware splits are essential forms of leakage control.

3. Baselines and Appropriate Metrics

A model should not be judged only by whether its score appears large or small. It should be compared with a simple baseline and evaluated using metrics that reflect the decision problem.

Regression metrics

Suppose the actual values are \(y_i\), predictions are \(\hat y_i\), and there are \(n\) observations.

Mean absolute error (MAE) is

\[

MAE=\frac{1}{n}\sum_{i=1}^{n}|y_i-\hat y_i|.

\]

MAE has the same units as the target and gives errors a linear penalty. It is relatively less sensitive to extreme errors than squared-error measures.

Mean squared error (MSE) is

\[

MSE=\frac{1}{n}\sum_{i=1}^{n}(y_i-\hat y_i)^2.

\]

Root mean squared error (RMSE) is

\[

RMSE=\sqrt{MSE}.

\]

RMSE also has the target's units but penalises large errors more strongly. It is useful when occasional large errors are particularly costly.

Coefficient of determination, or \(R^2\), is commonly defined as

\[

R^2=1-\frac{\sum_i(y_i-\hat y_i)^2}{\sum_i(y_i-\bar y)^2},

\]

where \(\bar y\) is the mean of the observed targets in the evaluated sample. The denominator is the squared-error total from predicting the sample mean. Relative to this mean baseline, \(R^2=1\) indicates perfect predictions, \(R^2=0\) indicates equal squared error to the mean predictor, and negative values are possible when the model is worse than that baseline. \(R^2\) should not be interpreted as the percentage of all real-world variation explained in every context.

For skewed targets, percentage-based metrics can be problematic when actual values are zero or near zero. A logarithmic transformation, MAE on the original scale, or a domain-specific loss may be preferable.

Classification metrics

For binary classification, predictions can be represented by a confusion matrix:

• True positive (TP): positive case correctly predicted;
• true negative (TN): negative case correctly predicted;
• false positive (FP): negative case incorrectly predicted positive;
• false negative (FN): positive case incorrectly predicted negative.

Accuracy is

\[

Accuracy=\frac{TP+TN}{TP+TN+FP+FN}.

\]

Accuracy is often unsuitable under severe class imbalance. A classifier that labels every case as the majority class may obtain high accuracy while identifying no minority cases.

Precision is

\[

Precision=\frac{TP}{TP+FP},

\]

provided \(TP+FP>0\). It answers: among predicted positives, how many are actually positive?

Recall, also called sensitivity or true-positive rate, is

\[

Recall=\frac{TP}{TP+FN},

\]

provided \(TP+FN>0\). It answers: among actual positives, how many were detected?

Specificity, or true-negative rate, is

\[

Specificity=\frac{TN}{TN+FP},

\]

provided \(TN+FP>0\).

The F1 score is the harmonic mean of precision and recall:

\[

F1=\frac{2\times Precision\times Recall}{Precision+Recall},

\]

when the denominator is non-zero. F1 is useful when both precision and recall matter, but it ignores true negatives and does not directly represent financial or clinical cost.

A classifier often produces a score or probability, which must be converted into a class using a threshold. Changing the threshold generally creates a trade-off between recall and precision, or between sensitivity and specificity.

The receiver operating characteristic (ROC) curve plots true-positive rate against false-positive rate across thresholds. The area under this curve, ROC-AUC, measures ranking ability across thresholds. It can appear impressive when the negative class is extremely large. The precision-recall curve is often more informative for rare positive events because it focuses on positive predictions.

When costs are known, expected cost is more useful than a generic metric. If \(C_{FP}\) and \(C_{FN}\) are the costs of false positives and false negatives, respectively, a simple total error cost is

\[

Cost=C_{FP}FP+C_{FN}FN,

\]

possibly with additional costs for investigation, delay, or intervention. The cost values and their assumptions must be documented.

Probabilistic calibration

A model is calibrated if, among cases assigned a predicted probability of approximately 0.7, about 70% are positive in a sufficiently large comparable group. Discrimination and calibration are different. A model can rank cases correctly but produce probabilities that are systematically too high or too low.

Calibration can be examined with reliability diagrams, calibration curves, and measures such as the Brier score. For binary outcomes with predicted probabilities \(p_i\) and labels \(y_i\in\{0,1\}\), the Brier score is

\[

Brier=\frac{1}{n}\sum_{i=1}^{n}(p_i-y_i)^2.

\]

Lower values are better, although the score depends on class prevalence. Calibration methods such as Platt scaling or isotonic regression must themselves be fitted on data separate from the data used to train the original model.

4. Cross-Validation

K-fold cross-validation divides the available development data into \(K\) approximately equal folds. For each fold, the model is trained on the other \(K-1\) folds and evaluated on the held-out fold. The reported score is usually the mean of the \(K\) scores:

\[

\bar s=\frac{1}{K}\sum_{j=1}^{K}s_j.

\]

The variability of the scores is also important. A large mean score with high fold-to-fold variation may be less reliable than a slightly lower but stable score. Standard cross-validation assumes that observations are sufficiently independent and identically distributed. Alternatives include:

• Stratified K-fold: preserves class proportions.
• Group K-fold: keeps related groups together.
• Time-series cross-validation: trains on earlier data and validates on later data.
• Repeated K-fold: repeats the partitioning to assess sensitivity to random splits.

Cross-validation is a model-selection tool, not a substitute for a final untouched test set. A robust workflow is nested cross-validation: an inner loop selects hyperparameters and an outer loop estimates generalisation. After selection, the chosen procedure may be refitted on all development data and evaluated once on the final test set.

5. Model Selection and Hyperparameter Tuning

A hyperparameter controls the learning procedure. Examples include the regularisation coefficient in ridge regression, maximum tree depth, learning rate, number of trees, and the number of neighbours in k-nearest neighbours.

A practical tuning process is:

1. Define the prediction task, deployment conditions, metric, and constraints.

2. Create an appropriate development/test split.

3. Establish a simple baseline.

4. Construct leakage-safe preprocessing and modelling pipelines.

5. Specify a plausible search space.

6. Use cross-validation on development data to compare settings.

7. Examine both mean performance and variability.

8. Select a model using performance, interpretability, latency, fairness, and maintenance considerations.

9. Lock the model and evaluate once on the test set.

10. Document results and monitor performance after deployment.

Grid search evaluates every combination in a specified finite grid. It is transparent but inefficient when many hyperparameters are irrelevant or continuous. Random search samples configurations and often explores high-value dimensions more efficiently. Bayesian optimisation uses previous results to propose promising configurations but introduces additional modelling assumptions and complexity.

The search space must respect computational and statistical constraints. Extremely flexible searches can overfit the validation process, especially with small datasets. A simpler model within a performance tolerance is often preferable because it is easier to explain, audit, and maintain.

Regularisation controls complexity. For example, ridge regression minimises

\[

\sum_i(y_i-\beta_0-x_i^T\beta)^2+\lambda\sum_j\beta_j^2,

\]

where \(\lambda\geq0\) controls the penalty and the intercept is commonly excluded from the penalty. Larger \(\lambda\) shrinks coefficients toward zero, often reducing variance at the cost of increased bias. Features should generally be standardised when coefficient penalties are used so that the penalty treats scales comparably.

6. Diagnosing Underfitting and Overfitting

The bias-variance trade-off describes two sources of prediction error. High bias occurs when the model is too simple to represent important structure. High variance occurs when the model is excessively sensitive to the particular training sample.

Typical signs of underfitting include poor training performance, poor validation performance, and residual patterns indicating missing structure. Possible responses include richer features, a more flexible model, reduced regularisation, or improved data quality.

Typical signs of overfitting include very strong training performance but materially weaker validation performance, unstable cross-validation scores, and large gains from regularisation or early stopping. Possible responses include more data, stronger regularisation, reduced model complexity, feature selection, augmentation where valid, or better leakage control.

Learning curves plot training and validation performance against training-set size. A persistent gap suggests variance; both curves remaining poor and close together suggests bias. These diagnoses are not mechanical laws: label noise, distribution shift, and metric mismatch can produce similar patterns.

Residual analysis is particularly valuable for regression. Plot residuals against fitted values and important features, inspect large errors, and compare performance across meaningful subgroups. For classification, examine confusion matrices by subgroup, threshold-specific precision and recall, and false-positive and false-negative examples.

7. Worked Example: Choosing a Model for Loan-Default Screening

Consider an illustrative bank developing a model to identify applicants who may default within twelve months. The positive class is default, and defaults represent 8% of historical applications. Missing a likely defaulter has a higher financial cost than manually reviewing a low-risk applicant.

The bank first removes a post-approval field that records whether collections activity occurred. Although highly predictive historically, it is unavailable at application time and would cause leakage. Applications from the same customer are grouped into the same partition. The data are split into development and final test sets using stratification.

A majority-class baseline predicts no default for every applicant. It obtains approximately 92% accuracy, but recall for default is 0% and therefore it is operationally unacceptable. A logistic-regression model and a gradient-boosted-tree model are trained in pipelines. Imputation and scaling are fitted separately within each training fold.

Five-fold stratified cross-validation produces the following illustrative results:

| Model | ROC-AUC | Recall at selected threshold | Precision at selected threshold |

|---|---:|---:|---:|

| Majority baseline | 0.50 | 0.00 | Not applicable |

| Logistic regression | 0.78 | 0.66 | 0.31 |

| Boosted trees | 0.84 | 0.70 | 0.35 |

The boosted model ranks applicants better, but the bank also needs reliable probabilities for pricing and review allocation. A calibration curve shows that the boosted model is overconfident. Calibration is fitted using validation predictions, not the final test set. The decision threshold is then selected using documented review capacity and the relative costs of false positives and false negatives.

On the untouched test set, the selected model achieves ROC-AUC 0.83, recall 0.68, and precision 0.34. These values are not interpreted as universal quality labels. The bank also evaluates performance by age group, region, and income band; investigates whether missing data patterns create disparate error rates; and monitors drift after deployment.

The final decision is not simply “choose the highest AUC.” The model must satisfy minimum recall, produce usable probabilities, operate within review capacity, meet fairness and regulatory requirements, and be explainable enough for adverse-action processes. If the more complex model offers little operational benefit, the logistic model may be preferred.

8. Practical, Ethical and Governance Considerations

Evaluation data should represent the population and conditions expected after deployment. Historical labels may reflect institutional decisions rather than objective outcomes. For example, past loan approvals and defaults reveal only what happened among previously approved applicants, creating selection bias.

Fairness cannot be established by one universal metric. Possible analyses include group-specific false-positive and false-negative rates, calibration by group, equal opportunity, and outcome disparities. Some fairness criteria cannot generally be satisfied simultaneously when base rates differ, so the organisation must state its normative and legal objectives clearly.

Privacy and security also matter. Sensitive attributes may be needed for fairness auditing even when they are excluded from prediction, requiring controlled access and lawful processing. Evaluation reports should document data provenance, exclusions, missingness, population limitations, metric definitions, confidence or uncertainty information, and the exact model version.

After deployment, performance may change because of concept drift, population shift, policy changes, or adversarial behaviour. Monitoring should include input distributions, missingness, prediction distributions, delayed outcome metrics, calibration, subgroup performance, and operational impacts. A model should have a rollback or human-review procedure when monitoring indicates unsafe degradation.

9. Best Practices

• Define the prediction time, available information, and decision consequences before modelling.
• Select metrics from the real cost of errors rather than convention alone.
• Use a simple, interpretable baseline.
• Keep the test set untouched until the final evaluation.
• Put every learned preprocessing step inside a leakage-safe pipeline.
• Use stratified, grouped, or temporal validation as appropriate.
• Report uncertainty and variation, not only a single average score.
• Tune thresholds separately from model fitting and validate threshold choices.
• Assess both discrimination and calibration when probabilities are used.
• Inspect errors by subgroup and by important operating conditions.
• Prefer the simplest model meeting documented performance and governance requirements.
• Preserve reproducibility through versioned data, code, features, configurations, and random seeds.
• Monitor the deployed system and reassess it when the environment changes.

Chapter Summary

Reliable model evaluation estimates performance on future data rather than rewarding memorisation of training examples. Training data fit parameters, validation data guide design choices, and a final test set provides an honest end-stage estimate. Cross-validation improves use of limited development data, but it must respect time, groups, and class proportions and must not contaminate the test set.

Regression metrics include MAE, MSE, RMSE, and \(R^2\), each reflecting different error preferences. Classification requires analysis of the confusion matrix, precision, recall, specificity, threshold trade-offs, ranking metrics, and—when probabilities matter—calibration. Baselines reveal whether a complex model provides meaningful improvement. Hyperparameter tuning should use leakage-safe pipelines, cross-validation, and a search strategy appropriate to the problem.

Underfitting and overfitting can be investigated through training-validation comparisons, learning curves, residual analysis, and subgroup error analysis. Final model selection must consider accuracy, cost, fairness, interpretability, latency, privacy, and maintainability. Evaluation continues after deployment through monitoring for drift, calibration failure, and changing subgroup performance.

Key Terms

Generalisation: Performance on new data from the target population.

Training set: Data used to fit model parameters.

Validation set: Data used to compare models and tune design choices.

Test set: Held-out data used for final performance estimation.

Data leakage: Use of information that would not be available at prediction time.

Baseline: A simple reference method against which a model is compared.

MAE: Mean absolute prediction error.

MSE: Mean squared prediction error.

RMSE: Square root of mean squared error.

R-squared: Relative squared-error performance compared with a mean predictor.

Precision: Proportion of predicted positives that are actually positive.

Recall: Proportion of actual positives that are detected.

Specificity: Proportion of actual negatives correctly rejected.

F1 score: Harmonic mean of precision and recall.

Calibration: Agreement between predicted probabilities and observed frequencies.

Cross-validation: Repeated training and validation across data folds.

Hyperparameter: A learning-system setting selected outside parameter fitting.

Overfitting: Learning sample-specific patterns that do not generalise.

Underfitting: Failure to capture important structure in the data.

Concept drift: Change over time in relationships between inputs and outcomes.

Review Questions

Short Answer Questions

1. What is the purpose of a final test set?

2. Why can accuracy be misleading for an imbalanced classification problem?

3. Distinguish precision from recall.

4. What is data leakage? Give one example.

5. What does calibration mean for a probabilistic classifier?

Long Answer Questions

1. Explain a complete train-validation-test workflow, including the role of cross-validation.

2. Compare MAE, RMSE, and \(R^2\), and describe situations in which each may be useful.

3. Explain how hyperparameter tuning can overfit the validation process and how nested cross-validation helps.

4. Discuss the symptoms and remedies of underfitting and overfitting.

5. Explain why model selection should consider operational cost, fairness, interpretability, and monitoring in addition to predictive scores.

Analytical / Application Questions

1. A fraud detector has 99.5% accuracy but detects only 10% of fraudulent transactions. Analyse why the score is inadequate and recommend metrics and threshold analyses.

2. A time-dependent demand model performs well under random cross-validation but poorly after deployment. Identify likely methodological problems and design a better evaluation procedure.

3. Two models have identical ROC-AUC, but one is well calibrated and the other is overconfident. Which should be selected for probability-based risk pricing, and what additional evidence would you request?

Multiple Choice Questions with Answers

Question 1

Which dataset should normally be used only after model and hyperparameter decisions are complete?

A. Training set

B. Validation set

C. Test set

D. Resampling set

Correct Answer: C. Test set

Question 2

Which metric is most directly expressed in the same units as a regression target and gives a linear penalty to errors?

A. MAE

B. MSE

C. ROC-AUC

D. Accuracy

Correct Answer: A. MAE

Question 3

A classifier has TP = 40 and FN = 10. What is its recall?

A. 40%

B. 50%

C. 75%

D. 80%

Correct Answer: D. 80%

Question 4

Which validation method is most appropriate when records from the same patient must not appear in both training and validation folds?

A. Random shuffle split

B. Group cross-validation

C. Leave-label-out validation

D. Majority-class validation

Correct Answer: B. Group cross-validation

Question 5

What does a calibrated predicted probability of 0.7 mean approximately?

A. The model ranks the case seventh among ten cases

B. The case has a 70% probability under every possible population

C. About 70% of comparable cases assigned that probability are positive

D. The model has 70% accuracy

Correct Answer: C. About 70% of comparable cases assigned that probability are positive

Question 6

Which situation is a clear example of data leakage?

A. Using a feature available at prediction time

B. Fitting an imputer separately within each training fold

C. Evaluating a final model once on held-out data

D. Using a post-outcome collections field to predict loan default at application time

Correct Answer: D. Using a post-outcome collections field to predict loan default at application time

Question 7

If false negatives are much more costly than false positives, which action is generally likely to increase recall?

A. Lowering the positive-class decision threshold

B. Raising the positive-class decision threshold

C. Removing all validation data

D. Replacing recall with training accuracy

Correct Answer: A. Lowering the positive-class decision threshold

Question 8

What is a typical sign of overfitting?

A. Poor training and poor validation performance with no gap

B. Strong training performance and substantially weaker validation performance

C. Identical predictions for every observation

D. A test set that is never created

Correct Answer: B. Strong training performance and substantially weaker validation performance

Question 9

Which metric is often especially informative for rare positive events?

A. Precision-recall analysis

B. Training accuracy only

C. Mean absolute error

D. R-squared

Correct Answer: A. Precision-recall analysis

Question 10

Why should preprocessing such as scaling be included inside a cross-validation pipeline?

A. To guarantee perfect accuracy

B. To make the test set larger

C. To prevent information from validation folds influencing preprocessing fitted on training folds

D. To eliminate the need for a baseline

Correct Answer: C. To prevent information from validation folds influencing preprocessing fitted on training folds