Model Evaluation, Selection and Experimentation

Introduction

A machine-learning model is not valuable merely because it can fit a training set. Its practical value depends on how reliably it predicts previously unseen cases, whether its errors are acceptable for the application, and whether the reported evidence can be reproduced. Model evaluation is therefore an inferential activity: from finite observations, we estimate performance on a future population. Model selection extends this activity by comparing algorithms, representations, and hyperparameters. Experimentation adds disciplined control of data, randomness, software, and reporting.

The central danger is overfitting to the evaluation process. A model may overfit training data, but an analyst may also overfit a validation set by repeatedly trying alternatives and retaining the one that happens to perform best. A sound experimental design separates data used for learning, data used for decisions, and data reserved for final confirmation.

Learning objectives

By the end of this chapter, you should be able to:

• distinguish training, validation, and test data;
• select appropriate metrics for classification, regression, ranking, and imbalanced problems;
• use cross-validation, learning curves, and baselines to diagnose models;
• formulate and execute hyperparameter searches without leakage;
• compare models using uncertainty estimates and suitable statistical procedures;
• design reproducible, auditable machine-learning experiments;
• interpret performance results in technical, operational, and ethical contexts.

1. The logic of generalization

Let a dataset contain input–outcome pairs (x_i,y_i), where x_i is a feature vector and y_i is the observed target. A learning algorithm uses training data D_train to produce a predictor f. For a loss function L(y,f(x)), the population risk is

Here P is the unknown joint distribution of future cases, and the expectation represents average future loss. Because P is unknown, it is estimated using finite samples. The empirical risk on a sample of size n is

The empirical risk is useful only when the observations represent the intended deployment population and the evaluation data were not used improperly during model development. Standard statistical assumptions include independent and identically distributed observations for ordinary random splitting. Time series, clustered observations, repeated measurements, and grouped individuals violate this assumption and require specialized splitting schemes.

Bias, variance, and irreducible error

Prediction error can arise from:

• Irreducible noise: uncertainty in the data-generating process that no predictor can remove;
• Bias: systematic error caused by an overly restricted model or representation;
• Variance: sensitivity to the particular sample used for training;
• Estimation error: uncertainty because performance is measured on a finite evaluation sample;
• Distribution shift: a difference between development and deployment populations.

For squared-error prediction, a useful conceptual decomposition at a fixed input is

where σ² is conditional noise variance, Bias[f̂(X)] is the expected prediction minus the true conditional mean, and Var[f̂(X)] is variation across training samples. This decomposition assumes squared loss and is an explanatory framework rather than a direct recipe for measuring each term in ordinary data.

2. Train–validation–test designs

A conventional design divides data into three mutually exclusive portions:

A common split might be 60% training, 20% validation, and 20% test, but proportions depend on sample size, data cost, and variance. With limited data, cross-validation is generally preferable to sacrificing a large fixed validation set. The test set should be locked away until the analysis plan and model-selection decisions are complete.

After selecting hyperparameters using training and validation information, practitioners often refit the chosen configuration on the combined training and validation data, then evaluate once on the untouched test set. This is valid only if the test set has not influenced the choice of configuration, preprocessing, threshold, or stopping point.

Data leakage

Data leakage occurs when information unavailable at prediction time enters training or evaluation. Examples include:

• standardizing the full dataset before splitting;
• imputing missing values using test-set statistics;
• allowing a patient to appear in both training and test sets;
• using a post-outcome variable as a predictor;
• selecting features using all labels before cross-validation;
• tuning a classification threshold on the test set.

The safe pattern is to place preprocessing and the estimator in one pipeline. Within each fold, fit imputation, scaling, encoding, feature selection, and model parameters only on that fold’s training portion. Grouped or temporal leakage requires an appropriate split, not merely a pipeline.

Split strategies

• Stratified splitting approximately preserves class proportions and is useful for classification.
• Group splitting keeps all observations from an individual, household, hospital, or device in one partition.
• Time-based splitting trains on earlier observations and evaluates on later observations.
• Nested cross-validation separates hyperparameter selection from performance estimation when data are limited or many configurations are tested.

Random splitting is inappropriate when the deployment task is forecasting, when observations are correlated, or when future information can be inferred from identifiers or temporal structure.

3. Metrics must match the decision

A metric is a formalization of what counts as error. No metric is universally best.

Classification metrics

For binary classification, define:

• true positives (TP): positive cases correctly predicted;
• true negatives (TN): negative cases correctly predicted;
• false positives (FP): negative cases predicted positive;
• false negatives (FN): positive cases predicted negative.

Accuracy is

It is informative when class proportions and error costs are reasonably balanced. Otherwise, it can be misleading. Precision and recall are

Precision answers, “Among predicted positives, how many are positive?” Recall answers, “Among actual positives, how many were found?” The F1 score is the harmonic mean:

These ratios are undefined when their denominators are zero; software must specify a zero-division policy.

Specificity, also called the true-negative rate, is TN/(TN+FP). Balanced accuracy is the mean of recall and specificity:

For probabilistic classifiers, log loss rewards calibrated probabilities and strongly penalizes confident wrong predictions. For binary outcomes with y_i∈{0,1} and predicted probability p_i,

Probabilities are typically clipped away from exactly 0 and 1 for numerical stability. Brier score is the mean squared probability error, (1/n)∑(p_i−y_i)², with lower values better.

ROC–AUC measures ranking across classification thresholds and can remain high in highly imbalanced data even when precision is poor. Precision–recall curves are often more informative when the positive class is rare. The selected operating threshold should reflect costs, capacity, and safety requirements rather than defaulting to 0.5.

Regression metrics

For observed values y_i and predictions ŷ_i,

MAE is in the target’s units and is relatively robust to outliers. Mean squared error is

and RMSE is √MSE, also in target units but more sensitive to large errors. Mean absolute percentage error can be problematic when actual values are zero or near zero. R² = 1 − SSE/SST compares residual sum of squares with total sum of squares; it can be negative on test data and does not by itself establish practical usefulness.

Metrics should be reported with confidence intervals where feasible, alongside subgroup performance, calibration, and a confusion matrix or error distribution. A small improvement in a metric may not justify greater latency, cost, complexity, or fairness risk.

4. Cross-validation and uncertainty

In k-fold cross-validation, the development data are divided into k folds. Each fold is used once for validation while the remaining k−1 folds are used for training. The cross-validation estimate is

where s_j is the score on fold j. The fold standard deviation is

This describes variation across folds; it is not automatically a valid standard error because fold scores are dependent through overlapping training sets. A rough standard error is s/√k, but confidence intervals require care, particularly for model comparisons.

Stratified k-fold cross-validation is common for classification. Repeated cross-validation changes the partitions and can reveal instability. Nested cross-validation has an outer loop for unbiased performance estimation and an inner loop for hyperparameter selection:

1. Hold out one outer fold.
2. Search hyperparameters using only the remaining outer-training data and inner cross-validation.
3. Fit the selected configuration on all outer-training data.
4. Score it on the outer test fold.
5. Aggregate outer-fold scores.

Nested validation is computationally expensive but protects against optimistic estimates caused by reporting the best inner-fold result.

5. Baselines, learning curves, and diagnostic analysis

A baseline establishes whether complexity is justified. Examples include majority-class prediction, stratified random prediction, linear or logistic regression, a seasonal naive forecast, or a current operational rule. A baseline should use the same evaluation protocol as the proposed model.

A learning curve plots training and validation performance against training-set size. Typical interpretations are:

• high training error and similar validation error: underfitting; improve features or model capacity;
• low training error but much higher validation error: high variance; obtain more data, regularize, or simplify;
• both errors improving with more data: additional data may help;
• a persistent gap despite more data: investigate noise, leakage, distribution shift, or representation limits.

Validation curves vary one hyperparameter while holding others under a defined search procedure. Error analysis should inspect examples, segments, confidence, missingness, and systematic failure modes—not only aggregate scores.

6. Hyperparameter optimization and model selection

Model parameters are learned from data; hyperparameters are choices controlling the learning process, such as regularization strength, tree depth, learning rate, or number of neighbors. A search space should be specified before evaluation. Logarithmic scales are appropriate for quantities spanning orders of magnitude, such as a regularization coefficient.

Common strategies include:

• Grid search: evaluates a predetermined Cartesian grid; transparent but inefficient in high dimensions.
• Random search: samples configurations and often explores important dimensions more effectively.
• Bayesian optimization: uses results from previous trials to select promising configurations, often through a surrogate model and acquisition rule.
• Successive halving or Hyperband: allocates increasing resources to promising trials and stops weak ones early.

The optimization objective must be explicit: metric, direction, resource budget, random seeds, and constraints. Searching many configurations increases selection bias. If hundreds of trials are run, the final test estimate must remain untouched, and nested validation or a second holdout may be necessary.

A model should not be selected solely by the largest point estimate. Consider uncertainty, operational constraints, interpretability, training and inference cost, robustness, calibration, fairness, maintainability, and the consequences of failure.

7. Statistical comparison of models

Suppose two models produce paired predictions for the same test cases. The meaningful object is often the per-case loss difference

A positive mean d̄ indicates lower average loss for model B under this definition. Paired comparisons are more powerful than treating model scores from the same cases as independent.

For classification errors on paired cases, McNemar’s test uses the numbers of cases where A is correct and B is wrong, and vice versa. For repeated cross-validation, corrected procedures or hierarchical models may be needed because scores are correlated. Bootstrap resampling can estimate uncertainty in a metric or difference, but resampling units must respect groups and time. Permutation tests are useful when exchangeability assumptions are plausible.

Statistical significance is not practical significance. A tiny improvement can be statistically detectable in a huge dataset yet operationally irrelevant. Conversely, a meaningful improvement may be uncertain in a small sample. Report effect size, uncertainty interval, sample size, and decision relevance.

8. Reproducible experimentation

A reproducible experiment records:

1. dataset versions, provenance, inclusion criteria, and split assignments;
2. preprocessing code and fitted transformations;
3. model, hyperparameter search space, objective, and stopping rules;
4. software versions, hardware, random seeds, and environment configuration;
5. all trials, not only the winning trial;
6. evaluation scripts, metric definitions, confidence intervals, and subgroup results;
7. model artifacts, predictions, and a concise experiment report.

Random seeds improve repeatability but do not guarantee identical results across hardware, parallel libraries, or nondeterministic operations. Reproducibility has levels: repeatability within the same environment, reproducibility by another team using the same materials, and replicability under an independently implemented study. Version control, data validation, immutable artifacts, automated tests, and experiment-tracking systems support all three.

A preregistered analysis plan can specify primary metrics, exclusions, subgroup analyses, and stopping rules before results are inspected. This reduces researcher degrees of freedom and makes negative results informative.

9. Case study: hospital readmission risk

The following is an illustrative case.

Background and problem

A hospital wishes to identify patients at high risk of readmission within 30 days. Missing a high-risk patient may reduce the opportunity for follow-up care, while excessive alerts burden clinicians. The dataset contains multiple encounters per patient, laboratory measurements, diagnoses, and discharge information.

Analysis and design

A random row split would leak patient-specific patterns because the same patient could occur in training and test sets. Further, a variable recorded after discharge would not be available at prediction time. The team therefore defines a prediction timestamp, removes post-timestamp variables, and uses a patient-grouped, time-ordered split: earlier patients and encounters for development, later encounters for testing, with no patient overlap.

The baseline is a regularized logistic regression. A gradient-boosted tree is a candidate model. Imputation, encoding, and feature selection are implemented within each training fold. Hyperparameters are tuned with grouped inner cross-validation. The primary metric is recall at a fixed alert capacity of 15% of discharges; secondary metrics are precision, PR–AUC, calibration, subgroup recall, and alert latency.

Approach and outcome

The boosted model has higher PR–AUC, but at the operational threshold its recall is only slightly greater than the baseline and its calibration is poorer. After calibration using development data, the model achieves a more reliable risk interpretation. Subgroup analysis finds lower recall for a language group with sparse documentation. The team adds a missingness audit, consults clinical stakeholders, and declines deployment until the disparity is investigated and the workflow is tested prospectively.

Learning points

The technically highest aggregate score is not automatically the best system. Group-aware splitting prevented leakage; capacity-aware evaluation connected metrics to operations; calibration supported meaningful risk communication; and subgroup analysis exposed a governance issue invisible in the overall score. A prospective monitoring plan is required because prevalence, clinical practice, and documentation may change.

10. Limitations, ethics, and governance

Evaluation results are conditional on the sample, labels, time period, and measurement process. Label noise can make a model appear poor or reward it for reproducing historical decisions. Missing data may encode access barriers. A high-performing model can still be unfair if errors are unevenly distributed or if its use changes the data-generating process.

Governance should address data minimization, lawful use, consent where applicable, privacy, access controls, documentation, auditability, human oversight, contestability, and retention. Protected attributes may be needed for fairness auditing even when they are excluded from prediction; this requires careful legal and ethical review. Fairness criteria can conflict—for example, equal calibration and equal error rates are not generally simultaneously achievable when base rates differ. Therefore, fairness is a sociotechnical decision, not a single metric optimization problem.

Contemporary practice increasingly emphasizes distribution-shift monitoring, conformal prediction for uncertainty sets under stated assumptions, data-centric evaluation, foundation-model benchmarking, and continuous post-deployment evaluation. These methods supplement rather than replace sound splits, clear metrics, and independent testing.

Best-practice checklist

• Define the deployment population, prediction time, and decision cost.
• Choose splits that reflect dependence and chronology.
• Fit every learned preprocessing step inside the training data of each fold.
• Establish simple, credible baselines.
• Predefine primary metrics and operational thresholds.
• Use nested validation when selection is extensive or data are limited.
• Report uncertainty, subgroup results, calibration, and resource costs.
• Preserve the test set for final confirmation.
• Log every experiment and retain unsuccessful trials.
• Monitor performance, drift, fairness, and calibration after deployment.

Chapter Summary

Model evaluation estimates future performance, while model selection chooses among algorithms and configurations under uncertainty. Train–validation–test designs protect against optimistic conclusions; cross-validation improves data efficiency but must respect groups and time. Metrics should reflect the decision, class balance, costs, probability quality, and operational constraints. Baselines, learning curves, and error analysis reveal whether more capacity, data, or better representation is needed. Hyperparameter optimization must be isolated from final testing, and extensive searching calls for nested validation or an independent holdout. Statistical comparisons should use paired predictions, uncertainty intervals, and practical effect sizes. Reproducible experiments require versioned data, pipelines, environments, seeds, search records, and evaluation protocols. Finally, fairness, privacy, calibration, drift, and human consequences are part of model quality rather than optional additions.

Key Terms

Generalization: Performance on previously unseen cases from the intended population.

Empirical risk: Average observed loss on a finite sample.

Training set: Data used to estimate model parameters.

Validation set: Data used to make development and selection decisions.

Test set: Held-out data used for final performance estimation.

Data leakage: Unintended use of information unavailable at prediction time.

Cross-validation: Repeated partitioning of data into training and validation folds.

Nested cross-validation: Cross-validation with separate inner selection and outer evaluation loops.

Baseline: A simple reference method against which a proposed model is compared.

Hyperparameter: A learning-process setting selected outside parameter fitting.

Calibration: Agreement between predicted probabilities and observed frequencies.

Precision: Proportion of predicted positives that are truly positive.

Recall: Proportion of actual positives that are detected.

PR–AUC: Area under the precision–recall curve.

Learning curve: Performance as a function of training-set size.

Distribution shift: Change between development and deployment data distributions.

Effect size: Magnitude of a performance difference, independent of significance alone.

Reproducibility: Ability to obtain results again using the documented materials and procedure.

Review Questions

5 Short Answer Questions

1. Why should the test set remain untouched during hyperparameter tuning?
2. State one situation in which random splitting is inappropriate.
3. Distinguish precision from recall.
4. What does a large training–validation performance gap suggest?
5. Why is a baseline necessary in model evaluation?

5 Long Answer Questions

1. Explain the roles of training, validation, and test data, including the conditions under which training and validation data may be combined for final fitting.
2. Compare accuracy, F1, PR–AUC, log loss, MAE, and RMSE, and identify circumstances in which each may be useful.
3. Describe nested cross-validation and explain why ordinary cross-validation can produce an optimistic estimate after extensive tuning.
4. Discuss the sources of data leakage in a clinical or financial prediction system and propose controls.
5. Explain how reproducibility, statistical uncertainty, and practical significance should shape a model-selection report.

3 Analytical / Application Questions

1. A fraud dataset has 1% positive cases. A classifier obtains 99% accuracy but detects only 10% of fraud. Which additional metrics and evaluation design would you recommend, and why?
2. A forecasting model performs well under random five-fold cross-validation but poorly on the next calendar quarter. Diagnose the likely causes and design a more valid experiment.
3. Two models have mean test losses of 0.42 and 0.40, but their per-case errors are correlated and the difference is uncertain. Design a statistical comparison and explain what evidence would justify deployment.

Multiple Choice Questions with Answers

Question 1

Which partition should be reserved for the final, one-time estimate of generalization?

A. Training set

B. Validation set

C. Test set

D. Feature-selection set

Correct Answer: C. Test set

Question 2

Which metric is usually most informative for a rare-positive classification problem when identifying positives is important?

A. Precision–recall analysis

B. Raw accuracy alone

C. Mean absolute error

D. R-squared

Correct Answer: A. Precision–recall analysis

Question 3

A model has very low training error but much higher validation error. What is the most likely diagnosis?

A. High variance or overfitting

B. Strong underfitting

C. Perfect calibration

D. Data standardization

Correct Answer: A. High variance or overfitting

Question 4

What is the main purpose of nested cross-validation?

A. To eliminate all measurement noise

B. To separate hyperparameter selection from performance estimation

C. To guarantee the best possible model

D. To replace the need for a test metric

Correct Answer: B. To separate hyperparameter selection from performance estimation

Question 5

Which operation must be fitted separately within each training fold to avoid leakage?

A. A fixed mathematical definition of accuracy

B. An imputer that learns feature medians

C. The name of the target variable

D. A deployment hardware specification

Correct Answer: B. An imputer that learns feature medians

Question 6

If TP = 80 and FN = 20, what is recall?

A. 20%

B. 40%

C. 80%

D. 100%

Correct Answer: C. 80%

Question 7

What does calibration assess for a probabilistic classifier?

A. Whether predicted probabilities correspond to observed frequencies

B. Whether the model has the shortest training time

C. Whether all classes have equal size

D. Whether every feature is normally distributed

Correct Answer: A. Whether predicted probabilities correspond to observed frequencies

Question 8

Which split is most appropriate when multiple records belong to the same patient?

A. A row-wise random split allowing patient overlap

B. A group split keeping each patient in one partition

C. A split based only on feature magnitude

D. A split after using all labels for feature selection

Correct Answer: B. A group split keeping each patient in one partition

Question 9

Why can a statistically significant performance improvement still be unsuitable for deployment?

A. Significance always indicates data leakage

B. The improvement may be too small to justify operational cost or risk

C. Statistical tests cannot compare models

D. Significance means the model is uncalibrated

Correct Answer: B. The improvement may be too small to justify operational cost or risk

Question 10

Which practice most directly supports reproducible experimentation?

A. Reporting only the winning trial

B. Changing the evaluation metric after viewing results

C. Versioning data, code, environments, and experiment records

D. Using a test set repeatedly until performance stabilizes

Correct Answer: C. Versioning data, code, environments, and experiment records