Model Evaluation, Optimization and Experimentation

Introduction

A machine-learning model is useful only when its reported performance supports a reliable decision in the environment where the model will operate. High training accuracy is not evidence of generalization; a statistically significant improvement may be too small to matter operationally; and an apparently successful deployment may reflect confounding, leakage, or a change in user behaviour. Model evaluation therefore combines statistical design, optimization, domain knowledge, and responsible experimentation.

This chapter develops a practical framework for evaluating predictive systems. It covers train–validation–test design, cross-validation, hyperparameter optimization, uncertainty and statistical testing, causal awareness, cost-sensitive metrics, and online A/B experiments. The emphasis is on making defensible comparisons rather than merely producing attractive scores.

Learning Objectives

By the end of this chapter, you should be able to:

1. Design train, validation, and test partitions that protect against leakage.
2. Select and interpret cross-validation procedures for ordinary, grouped, temporal, and imbalanced data.
3. Distinguish model parameters from hyperparameters and apply suitable optimization strategies.
4. Quantify uncertainty and use statistical tests without confusing significance with practical importance.
5. Recognize causal threats in predictive evaluation and experimentation.
6. Choose cost-sensitive metrics and decision thresholds for real operational objectives.
7. Design and interpret an online A/B experiment.
8. Communicate model-performance evidence transparently and responsibly.

1. The Evaluation Problem

Let a predictor be written as f_θ(x), where x is an input and θ denotes learned parameters. For a loss function L(y, f_θ(x)), the empirical risk on a dataset of size n is:

Here, yᵢ is the observed outcome and R̂(θ) is an average, so its units are the units of the loss. The quantity of interest is usually expected risk on future observations:

The distribution P represents the target population and operating conditions. Evaluation estimates R(θ) using finite data. This estimate is credible only when the evaluation data resemble future data and have not influenced model development improperly.

Three questions should be separated:

• Generalization: How well will the model perform on new cases?
• Comparison: Is one model or configuration better than another?
• Decision value: Does the improvement justify its cost, risk, and operational consequences?

A model can be best on a benchmark and still be unsuitable because it is slow, unfair, poorly calibrated, difficult to monitor, or optimized for the wrong business objective.

2. Train–Validation–Test Design

A standard supervised-learning workflow divides labelled data into three conceptually different sets.

A common split might be 60% training, 20% validation, and 20% test, but the proportions depend on sample size, model complexity, prevalence, and data-generation process. With limited data, nested cross-validation can replace a fixed validation set.

Leakage and dependence

Data leakage occurs when information unavailable at prediction time enters training or evaluation. Examples include using a diagnosis code recorded after treatment to predict treatment response, normalizing data using the full dataset before splitting, or allowing duplicate customers to appear in both training and test sets.

Preprocessing must be fitted inside the training portion of each resampling split. For example, a mean used for imputation or a scaling standard deviation must be calculated from training data only. A pipeline should perform preprocessing and model fitting as one reproducible object.

The unit of splitting must reflect dependence. If several records belong to the same patient, household, machine, or customer, related records should normally remain in one partition. Otherwise, the model may identify the entity rather than learn a general pattern.

Temporal data require chronological evaluation. A suitable design is to train on earlier observations and test on later observations. Random shuffling can leak future information through trends, seasonality, policy changes, or delayed labels.

The final test set

The test set should be sealed until the analysis plan, preprocessing, model family, hyperparameters, and primary metric have been fixed. Repeatedly consulting the test set turns it into a de facto validation set and causes optimistic bias. If many final models are compared on the same test set, a new confirmatory test set or fresh prospective data may be required.

3. Cross-Validation

In k-fold cross-validation, the development data are divided into k approximately equal folds. Each fold is used once for validation while the remaining folds are used for training. If MSE_j is the validation mean squared error in fold j, the cross-validation estimate is:

The folds should be created before preprocessing, and every operation that learns from data should be repeated within each training fold. The choice of k involves a bias–variance and computational trade-off. Five- or ten-fold cross-validation is common; leave-one-out cross-validation can be computationally expensive and may have high estimate variance.

Appropriate variants

• Stratified k-fold: approximately preserves class proportions in each fold; useful for classification, especially with imbalanced outcomes.
• Group k-fold: keeps all observations from a group in one fold.
• Time-series or rolling validation: respects temporal order and may use expanding or sliding training windows.
• Repeated k-fold: repeats partitioning to assess sensitivity to the particular split.
• Nested cross-validation: uses an inner loop for hyperparameter selection and an outer loop for performance estimation.

Nested cross-validation is important when comparing many configurations. In the outer loop, the inner loop selects hyperparameters using only the outer training data. The selected model is then assessed on the untouched outer validation fold. The outer scores estimate performance of the entire selection procedure, not merely of a conveniently chosen model.

Cross-validation is not automatically valid. It can fail when observations are dependent, when deployment involves distribution shift, or when labels become available with delay. Resampling answers a question about the assumed data-generating process; it cannot repair a poor design.

4. Metrics, Calibration and Thresholds

Classification metrics

For binary classification, a confusion matrix contains true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN). Important metrics include:

Precision is undefined when TP + FP = 0, and recall is undefined when TP + FN = 0. Accuracy can be misleading under severe class imbalance. The F1 score is the harmonic mean of precision and recall:

The F1 score is appropriate only when balancing precision and recall is meaningful. It does not account for true negatives and is not a universal replacement for accuracy.

For ranking systems, ROC–AUC summarizes ranking across classification thresholds, whereas precision–recall curves are often more informative when positive cases are rare. Neither metric by itself establishes that predicted probabilities are accurate.

Regression metrics

For errors eᵢ = yᵢ − ŷᵢ, mean absolute error and root mean squared error are:

MAE has the same units as the target and weights errors linearly. RMSE also has the target's units but penalizes large errors more strongly. MAPE can be unstable or undefined when actual values are zero or near zero; it should not be used without checking the scale and meaning of the target.

Calibration and decision thresholds

A classifier is calibrated if, among cases assigned probability approximately p, the observed event frequency is approximately p. Calibration can be assessed with reliability diagrams, calibration intercept and slope, or proper scoring rules such as log loss and the Brier score:

A probability threshold converts a score into an action. The threshold should reflect consequences, capacity, and prevalence—not simply the default value 0.5. Threshold tuning must occur on validation data or within cross-validation, never on the final test set.

5. Cost-Sensitive Evaluation

Suppose a decision produces costs C_FP for a false positive and C_FN for a false negative. If a population contains N_FP false positives and N_FN false negatives, classification error cost is:

If each case also has a processing cost or benefit, those terms must be included. A threshold should minimize expected operational cost subject to constraints such as review capacity, safety limits, or minimum recall.

For a predicted probability p and two actions—treat and do not treat—the expected costs may be represented as:

The action with lower expected cost is selected, provided probabilities are calibrated and the costs accurately represent the decision context. Cost-sensitive learning can also assign weights during training, but the resulting probabilities may require recalibration.

6. Hyperparameter Optimization

Model parameters are learned from training data; hyperparameters are chosen outside the fitting process. Examples include tree depth, regularization strength, learning rate, number of neighbours, and neural-network architecture.

A search procedure should be defined by:

1. An objective metric and direction of improvement.
2. Valid ranges and distributions for each hyperparameter.
3. Resource limits, such as time, memory, or number of trials.
4. A resampling strategy that respects groups, time, and preprocessing.
5. A rule for selecting a final configuration.

Grid search evaluates a Cartesian product and is simple but inefficient when only a few hyperparameters strongly influence performance. Random search samples configurations and often explores important dimensions more effectively. Bayesian optimization builds a surrogate model of validation performance and selects promising trials using an acquisition rule. Early-stopping methods can allocate more resources to promising configurations.

Optimization must not be allowed to overfit the validation process. Searching thousands of configurations increases the chance that one performs well by chance. Track all trials, including failed or discarded ones, and use nested validation or a fresh test set for honest reporting.

A useful selection principle is the one-standard-error rule: if several configurations have statistically or practically similar performance, prefer the simpler, faster, more interpretable, or more stable model.

7. Uncertainty and Statistical Comparison

A reported score is an estimate, not a fact. Uncertainty may arise from sampling variation, split variation, label noise, and deployment drift. Confidence intervals can be obtained using appropriate analytic methods or bootstrap resampling. A percentile bootstrap interval is formed by repeatedly resampling cases, recalculating the metric, and taking suitable empirical quantiles. Resampling must preserve dependence—for example, sample patients rather than individual visits when visits are correlated.

When two classifiers are evaluated on the same cases, their predictions are paired. Comparing two independent means with an inappropriate test wastes information. For paired continuous losses, a paired test or bootstrap interval for the mean loss difference is more appropriate. For paired binary predictions, McNemar's test focuses on discordant cases. For ranking metrics, paired bootstrap or permutation methods can be used, with attention to dependence and multiple comparisons.

A p-value is the probability, under a specified null model, of observing data at least as inconsistent with that null as the result obtained. It is not the probability that the null hypothesis is true, nor a measure of effect size. Report the estimated difference and confidence interval as well as the p-value.

If many models, subgroups, or metrics are examined, the probability of at least one apparently positive result increases. Pre-specification, correction for multiple comparisons, independent confirmation, and transparent reporting reduce this risk. Statistical significance does not imply practical significance: a large dataset can make a negligible improvement statistically significant.

8. Causal Awareness in Predictive Evaluation

Prediction and causation answer different questions. A model may accurately predict hospital readmission without identifying which intervention will prevent readmission. A feature can be predictive because it is a proxy for access, policy, or historical discrimination.

Common causal threats include:

• Confounding: a third variable influences both an exposure and the outcome.
• Selection bias: inclusion in the dataset depends on variables related to exposure or outcome.
• Post-treatment adjustment: controlling for a variable affected by the intervention can remove or distort part of the causal effect.
• Collider bias: conditioning on a common consequence of two variables can create a misleading association.
• Treatment leakage: a feature recorded after an action encodes the outcome pathway.

If the question is causal—such as whether a message increases repayment—randomized assignment is generally the strongest design. Observational analyses require explicit assumptions, a time-aware causal diagram or directed acyclic graph, careful confounder selection, and sensitivity analysis. Predictive performance alone cannot establish intervention effectiveness.

9. Online A/B Experiments

An A/B experiment randomly assigns eligible units to a control or treatment condition. The unit may be a user, account, session, store, or region. The unit of randomization must match the interference structure: if users influence one another, individual randomization may produce contamination and cluster randomization may be preferable.

A sound experiment specifies:

• primary outcome and its measurement window;
• assignment mechanism and eligibility criteria;
• minimum detectable effect and sample-size rationale;
• guardrail metrics, such as latency, complaints, or safety incidents;
• experiment duration and stopping rule;
• treatment exposure and implementation checks;
• analysis population, including intention-to-treat principles.

For a binary outcome, the difference in conversion rates is:

where p_T and p_C are treatment and control proportions. For independent observations, a large-sample standard error for the difference is approximately:

where n_T and n_C are group sizes. This approximation assumes independent units and sufficiently large samples. Clustered assignment, repeated observations, or variance reduction methods require appropriate standard errors.

Repeatedly checking results and stopping when a favourable p-value appears inflates false-positive risk. Use a predeclared stopping rule, sequential-testing method, or Bayesian decision procedure. Analyze users according to assigned group for the primary intention-to-treat estimate; treatment-on-the-treated analyses require stronger assumptions because non-compliance is not random.

A statistically positive experiment can still be unacceptable if it harms a minority subgroup, increases long-term churn, violates privacy, or produces a small benefit at excessive cost. Monitor heterogeneous effects and guardrails before deployment.

10. Reproducible Evaluation Workflow

A professional workflow can be organized as follows:

1. Define the decision, prediction time, eligible population, and operational constraints.
2. Specify the primary metric, secondary metrics, costs, and guardrails before modelling.
3. Audit data provenance, labels, missingness, duplicates, groups, and temporal coverage.
4. Create leakage-resistant splits and an immutable test set.
5. Establish a simple baseline, such as a majority classifier, linear model, or historical rule.
6. Build preprocessing and estimation in a single pipeline.
7. Use suitable cross-validation and hyperparameter optimization.
8. Quantify uncertainty and compare models using paired, decision-relevant analyses.
9. Check calibration, subgroup performance, robustness, and computational requirements.
10. Run a prospective or randomized experiment when the deployment effect is uncertain.
11. Document versions, random seeds, trial histories, exclusions, and deviations from the plan.
12. Monitor drift, calibration, data quality, fairness, and business outcomes after release.

A model card or evaluation report should state the intended use, population, metrics, validation design, limitations, subgroup results, and conditions under which performance should not be trusted.

11. Illustrative Case: Predicting Missed Medical Appointments

Background

A clinic develops a model to identify appointments at high risk of non-attendance so that reminders can be offered. This is an illustrative case, not a report of a real clinic.

Problem

The clinic has appointment records from three years. The goal is not simply to rank risk; it is to choose a reminder policy that reduces missed appointments without overburdening patients or staff. False positives may annoy patients, while false negatives leave unused clinical capacity.

Analysis

The team discovers that the field reminder_sent was recorded after some scheduling decisions and is strongly associated with attendance. It is excluded because it would not be available at the prediction time. Patients have multiple appointments, so random row-level splitting would place the same patient in training and test data. The team uses a patient-grouped, chronological design: earlier appointments for eligible patients form development data, and a later period forms the final test set.

The primary metric is expected net benefit, combining recovered appointment value, reminder cost, and the cost of unnecessary reminders. Recall is a secondary operational metric because the clinic wants to identify many high-risk appointments. Calibration is assessed because the probability is used to prioritize staff effort.

Approach and solution

A logistic-regression baseline and a gradient-boosted model are evaluated using grouped time-aware cross-validation. Imputation and scaling are fitted within each training fold. Hyperparameters are selected in an inner loop. The final model is retrained on all development data and evaluated once on the later test period.

The model's threshold is chosen from validation data by minimizing the expected cost:

Here, t is the threshold, R(t) is the number of reminders, and C_R is the cost per reminder. The clinic also imposes a maximum daily reminder capacity. The final test result reports the threshold, confusion matrix, calibration, uncertainty interval, subgroup performance, and capacity impact.

Before full deployment, the clinic randomizes eligible high-risk appointments to standard practice or the new reminder policy. The primary outcome is attendance, while complaint rate and cancellation timing are guardrails. Assignment is at the patient level for the experiment period to reduce cross-condition contamination.

Outcome and learning points

The predictive model identifies risk more accurately than the historical rule, but the A/B experiment is required to establish whether reminders cause improved attendance. The team learns that:

• leakage control depends on understanding when fields are created;
• grouped and temporal validation can materially change estimated performance;
• cost and capacity determine the useful threshold;
• calibrated probabilities support resource allocation;
• predictive accuracy does not prove that an intervention works;
• subgroup and guardrail outcomes are part of model quality.

12. Challenges, Ethics and Governance

Evaluation data may encode unequal access, historical exclusion, or measurement error. A model can have similar overall accuracy across groups while producing different false-negative rates or calibration. Fairness criteria can conflict mathematically when base rates differ, so the appropriate criterion must be connected to the use case, law, and harms.

Privacy requires data minimization, access control, retention limits, and secure handling of identifiers. Experimentation requires informed governance, especially when treatment can affect health, employment, credit, safety, or essential services. Monitoring should detect drift, feedback loops, gaming, and changes in the label-generation process.

Regulated settings may require audit trails, explainability appropriate to the decision, human oversight, validation evidence, and change control. Explainability does not substitute for valid evaluation, and a transparent model is not automatically fair or accurate.

Contemporary Developments and Best Practice

Modern evaluation increasingly addresses distribution shift, uncertainty-aware prediction, conformal prediction, adaptive experimentation, and continuous monitoring. Conformal methods can provide prediction sets or intervals under exchangeability assumptions, but coverage may fail under strong drift or dependence. Foundation models and automated machine-learning systems increase the number of hidden choices, making data provenance, trial logging, and independent evaluation more important.

Best practice is to evaluate the complete decision system rather than the model in isolation. The system includes data collection, preprocessing, human review, thresholding, interventions, user responses, and downstream outcomes. A reliable evaluation is therefore technically rigorous, temporally realistic, cost-aware, statistically cautious, and explicit about its assumptions.

Chapter Summary

Reliable model evaluation begins with a clear prediction or decision question and a data split that mirrors deployment. Training data estimate parameters, validation data guide choices, and a sealed test set provides final evidence. Cross-validation is powerful when its folds respect groups, time, and preprocessing boundaries; nested cross-validation protects against selection bias when many configurations are searched.

Metrics must reflect the decision. Accuracy, precision, recall, regression losses, calibration, and cost-sensitive measures answer different questions. Thresholds should be selected using operational costs and constraints. Hyperparameter optimization requires controlled search, resource limits, and protection against validation overfitting.

Uncertainty intervals, paired comparisons, effect sizes, and appropriate statistical tests are more informative than isolated p-values. Prediction is not causation: causal claims require appropriate design and assumptions. A/B experiments estimate the effect of interventions when randomization, interference, stopping rules, and guardrails are handled correctly. Finally, fairness, privacy, reproducibility, monitoring, and governance are essential components of trustworthy evaluation.

Key Terms

Calibration: Agreement between predicted probabilities and observed outcome frequencies.

Causal inference: Methods for estimating effects of interventions or exposures.

Cross-validation: Resampling procedure that repeatedly trains and validates models on different data partitions.

Data leakage: Use of information in model development that would not be available at prediction time.

Effect size: Magnitude of a difference or association, independent of statistical significance.

F1 score: Harmonic mean of precision and recall.

False negative: A negative prediction when the true class is positive.

False positive: A positive prediction when the true class is negative.

Generalization: Performance on new observations from the target population.

Guardrail metric: Outcome monitored to detect unacceptable side effects of an intervention.

Hyperparameter: Model or training setting selected outside parameter fitting.

Intention-to-treat: Analysis according to randomized assignment, regardless of compliance.

Nested cross-validation: Outer evaluation loop containing an inner model-selection loop.

Precision: Proportion of predicted positives that are truly positive.

Recall: Proportion of actual positives that are detected.

Statistical significance: Evidence against a specified null model under stated assumptions.

Test set: Data reserved for final performance estimation.

Validation set: Data used to select models, hyperparameters, features, or thresholds.

Variance: Expected squared deviation of an estimator or random quantity from its mean.

Threshold: Score or probability cutoff used to convert predictions into actions.

Review Questions

Short Answer Questions

1. What is the principal difference between a validation set and a test set?
2. Why can random splitting be inappropriate for patient or customer records?
3. Define calibration and explain why it matters for resource allocation.
4. What is the difference between a model parameter and a hyperparameter?
5. Why does statistical significance not necessarily imply practical significance?

Long Answer Questions

1. Explain how leakage can enter preprocessing and feature construction, and describe a leakage-resistant pipeline.
2. Compare stratified, grouped, temporal, and nested cross-validation, including an appropriate use case for each.
3. Discuss the advantages and limitations of accuracy, F1, ROC–AUC, precision–recall analysis, and calibration.
4. Explain how cost-sensitive threshold selection differs from choosing a default probability threshold of 0.5.
5. Describe the design and interpretation of a rigorous A/B experiment, including randomization, stopping rules, primary outcomes, and guardrails.

Analytical / Application Questions

1. A fraud model has high accuracy because only 1% of transactions are fraudulent. Design a metric and validation strategy that would better support deployment, and justify your choices.
2. A model performs well under random cross-validation but poorly on the next month's data. Identify plausible causes and propose a revised evaluation and monitoring plan.
3. A reminder intervention improves attendance overall but increases complaints among one subgroup. Explain how you would analyze the result and decide whether to continue, modify, or stop the intervention.

Multiple Choice Questions with Answers

Question 1

Which dataset should remain untouched while hyperparameters and thresholds are selected?

A. Training set

B. Validation set

C. Test set

D. Bootstrap sample

Correct Answer: C. Test set

Question 2

Which cross-validation method keeps observations from the same patient in a single fold?

A. Stratified k-fold

B. Group k-fold

C. Leave-one-out only

D. Random subsampling

Correct Answer: B. Group k-fold

Question 3

If false negatives are substantially more costly than false positives, which action is generally appropriate?

A. Always use threshold 0.5

B. Ignore prevalence

C. Consider a lower threshold, subject to validation and capacity

D. Optimize accuracy only

Correct Answer: C. Consider a lower threshold, subject to validation and capacity

Question 4

What does precision measure?

A. The proportion of actual positives detected

B. The proportion of predicted positives that are truly positive

C. The proportion of actual negatives detected

D. The proportion of all predictions that are correct

Correct Answer: B. The proportion of predicted positives that are truly positive

Question 5

Which practice most directly prevents preprocessing leakage during cross-validation?

A. Fit preprocessing once on all records

B. Fit preprocessing separately within each training fold

C. Remove the validation folds entirely

D. Use a larger test set

Correct Answer: B. Fit preprocessing separately within each training fold

Question 6

What is the primary purpose of nested cross-validation?

A. To eliminate all sampling error

B. To estimate performance while accounting for hyperparameter selection

C. To guarantee causal validity

D. To replace the need for a target variable

Correct Answer: B. To estimate performance while accounting for hyperparameter selection

Question 7

A p-value is best described as which of the following?

A. The probability that the null hypothesis is true

B. The probability that the alternative hypothesis is true

C. The probability, under a specified null model, of results at least as extreme as observed

D. The practical size of an effect

Correct Answer: C. The probability, under a specified null model, of results at least as extreme as observed

Question 8

Why might a temporal split be preferable to a random split?

A. It always produces higher accuracy

B. It reflects the direction of time and reduces future-information leakage

C. It removes the need for validation

D. It guarantees stationarity

Correct Answer: B. It reflects the direction of time and reduces future-information leakage

Question 9

What does an intention-to-treat analysis preserve in an A/B experiment?

A. Analysis according to assigned group

B. Analysis only among users who complied

C. Analysis only among users with positive outcomes

D. Analysis after removing all non-users

Correct Answer: A. Analysis according to assigned group

Question 10

Which statement correctly distinguishes prediction from causation?

A. A predictive feature necessarily causes the outcome

B. High predictive accuracy proves an intervention is effective

C. Prediction estimates outcomes, whereas causal analysis concerns effects of interventions

D. Causal claims require no assumptions when a model is accurate

Correct Answer: C. Prediction estimates outcomes, whereas causal analysis concerns effects of interventions