AI, Predictive Analytics and Emerging Evaluation Methods

Introduction

Health care evaluation is increasingly conducted in environments in which data are generated continuously by electronic health records, claims systems, registries, remote-monitoring devices, patient portals, genomic platforms and social care systems. Artificial intelligence (AI) and predictive analytics can identify patterns that are difficult for humans to detect, support clinical and managerial decisions, and enable evaluation to move from retrospective reporting toward prospective learning. However, computational sophistication does not guarantee valid evidence or beneficial decisions. A highly accurate model may worsen inequity, fail after implementation, encourage inappropriate treatment, or merely reproduce historical patterns of unequal care.

This chapter examines AI-enabled decision support, predictive modelling, real-world evidence (RWE) and learning health systems (LHSs) as evaluation methods. It emphasizes the distinction between prediction and causal inference, the importance of validation in the intended setting, monitoring for performance drift, explainability, fairness and human oversight. It also considers regulatory and methodological developments relevant to health care managers, evaluators and policy leaders.

Learning Objectives

By the end of this chapter, the reader should be able to:

1. Distinguish prediction, diagnosis, risk stratification, causal inference and decision support.
2. Evaluate the design and performance of AI-enabled health care interventions.
3. Interpret discrimination, calibration, clinical utility and fairness measures.
4. Explain why temporal, geographic and external validation are necessary.
5. Design governance arrangements for drift, explainability, accountability and human oversight.
6. Assess the strengths and limitations of real-world evidence and learning health systems.
7. Anticipate emerging regulatory and methodological requirements.

1. From Conventional Evaluation to Computational Evaluation

Traditional evaluation often compares outcomes before and after an intervention, or between exposed and unexposed groups. AI introduces an additional object of evaluation: the model or algorithm itself. The evaluator must ask at least four separate questions:

• Technical validity: Does the model estimate or classify what it claims to estimate?
• Clinical or operational validity: Does it improve a decision or process in the target setting?
• Equity and ethical validity: Are benefits and harms distributed acceptably across groups?
• System validity: Does implementation produce net benefit after workflow, costs, adoption and unintended consequences are considered?

These questions are related but not interchangeable. A model can have excellent area under the receiver operating characteristic curve (AUROC) while being poorly calibrated. A model can be statistically valid but clinically useless if available interventions do not change outcomes. An intervention can improve average outcomes while widening disparities.

Prediction is not causation

A predictive model estimates an outcome conditional on observed information. A causal evaluation asks what would happen under alternative actions. For example, a model may predict hospital admission accurately, but this does not establish that admitting every high-risk patient will reduce mortality. Prediction describes risk; causal inference estimates the effect of an intervention.

A useful distinction is:

• Predictive question: Who is likely to experience outcome Y?
• Causal question: What is the effect of intervention A compared with intervention B on outcome Y?
• Decision question: For which patients will choosing A rather than B produce sufficient benefit relative to harm and cost?

Confusing these questions is a major source of inappropriate AI implementation.

2. AI-Enabled Decision Support

AI-enabled decision support includes rule-based systems, statistical risk scores, machine-learning models, natural-language processing, computer vision and generative systems. Their outputs may be alerts, rankings, recommendations, summaries, forecasts or automatically triggered actions.

A decision-support system should be evaluated as a socio-technical intervention, not merely as software. Its effect depends on the model, interface, workflow, user interpretation, organizational incentives, data quality and available response. Evaluation should therefore examine the full chain:

Data → model → output → human interpretation → action → patient or system outcome

Common applications include:

• deterioration and sepsis risk prediction;
• readmission and avoidable admission prediction;
• appointment and bed-demand forecasting;
• diagnostic image interpretation;
• medication safety alerts;
• population-health segmentation;
• fraud and waste detection; and
• personalized treatment or follow-up recommendations.

The relevant comparator is not always “no algorithm.” It may be usual clinician judgment, an existing score, a simpler statistical model or an alternative allocation policy. A complex model should earn its complexity by producing better decisions or outcomes.

3. Predictive Model Evaluation

3.1 Data quality and target definition

Evaluation begins before model fitting. The target outcome must be defined precisely, including its time horizon, measurement source and clinical meaning. A label such as “sepsis” may represent a clinician diagnosis, a laboratory-defined phenotype, an administrative code or a treatment proxy. Each definition has different validity and bias.

Important data questions include:

• Who is represented and who is missing?
• Were data generated for care, billing, research or operational purposes?
• Are predictors available at the moment the decision is made?
• Does the dataset contain leakage from future information?
• Are missing values informative rather than random?
• Are outcome measurements equally reliable across demographic groups?

Data leakage occurs when information unavailable at prediction time is inadvertently used in model development. It can produce impressive internal performance and disappointing real-world performance.

3.2 Discrimination

Discrimination concerns the ability to rank individuals according to outcome risk. For a binary outcome, sensitivity and specificity at a selected threshold are defined as:

Here, TP denotes true positives, FN false negatives, TN true negatives and FP false positives. Sensitivity is the proportion of people with the outcome who are correctly identified; specificity is the proportion without the outcome who are correctly classified as negative. These measures depend on the decision threshold.

The AUROC summarizes ranking performance across thresholds. It can be interpreted as the probability that a randomly selected person with the outcome receives a higher predicted risk than a randomly selected person without the outcome. AUROC is useful but incomplete: it may appear strong when the outcome is rare, and it does not establish whether predictions are numerically accurate or clinically useful.

3.3 Calibration

Calibration assesses agreement between predicted probabilities and observed frequencies. If a model assigns a group a mean risk of 0.20, approximately 20% should experience the outcome under comparable conditions. A calibration plot, calibration intercept and calibration slope can reveal systematic overprediction or underprediction.

The Brier score for a binary outcome is:

where n is the number of observations, pᵢ is the predicted probability for person i, and yᵢ is the observed outcome coded 0 or 1. Lower values indicate better probabilistic accuracy, although the score depends partly on outcome prevalence and should not be interpreted alone.

3.4 Clinical utility and threshold choice

A prediction is valuable only if it improves a decision. Decision-curve analysis evaluates net benefit across clinically relevant probability thresholds. A commonly used expression is:

Here, pₜ is the risk threshold at which a decision-maker would choose intervention, and n is the sample size. The expression assumes that the threshold represents a trade-off between the harm of a false positive and the harm of a false negative. It should be used only when the outcome, intervention and threshold have a defensible clinical interpretation.

A model may have modest discrimination yet improve care if it identifies a group for whom an effective intervention is available. Conversely, a highly discriminative model may have little utility when treatment is ineffective, burdensome or inaccessible.

3.5 Validation

Validation should progress from internal to external assessment:

1. Internal validation: resampling methods such as bootstrapping or cross-validation estimate optimism within the development data.
2. Temporal validation: testing on later data assesses transport across time.
3. Geographic or institutional validation: testing in another setting assesses transport across organizations and populations.
4. Prospective silent evaluation: the model runs without influencing care, allowing workflow and performance assessment.
5. Impact evaluation: a controlled or carefully designed observational study assesses effects on decisions and outcomes.

Randomly splitting one institution’s data into training and test sets is not equivalent to external validation. It may fail to reveal changes in coding, staffing, disease prevalence, treatment practice or population composition.

4. Fairness, Explainability and Human Oversight

4.1 Fairness is a governance choice as well as a technical property

Fairness cannot be reduced to one universal metric. Relevant questions include whether groups have comparable access to care, whether error rates differ, whether predictions are calibrated, and whether an intervention addresses structural disadvantage.

For a protected or relevant group g, group-specific sensitivity can be expressed as:

Equal sensitivity across groups is one possible fairness objective. Other objectives include equal specificity, equal positive predictive value or group calibration. These objectives can conflict, particularly when outcome prevalence differs between groups. Therefore, an organization must state which harms it is prioritizing and why, rather than claiming that an algorithm is simply “fair.”

Fairness assessment should include:

• performance by race, ethnicity, sex, age, disability, language, socioeconomic position and geography where legally and ethically appropriate;
• intersectional groups rather than only one characteristic at a time;
• differential access to the intervention triggered by the model;
• differences in false reassurance, alert burden and downstream treatment; and
• whether the target itself reflects unequal measurement or access.

Removing a protected attribute does not necessarily remove discrimination because correlated variables may act as proxies.

4.2 Explainability

Explainability has several meanings. A model may be intrinsically interpretable, such as a simple score, or supported by post-hoc explanations such as feature attribution. Explanations may be intended for clinicians, patients, auditors, developers or regulators. A technical explanation does not automatically provide a clinically meaningful reason.

Useful questions are:

• Does the explanation accurately represent the model’s behavior?
• Is it stable when inputs change slightly?
• Does it help a user detect error or improve a decision?
• Does it disclose uncertainty and missing information?
• Can affected people challenge or correct the result?

Feature importance should not be interpreted as causation. If a model uses prior service utilization as a predictor, high importance may reflect access patterns rather than biological need.

4.3 Human oversight

“Human in the loop” is not sufficient by itself. Humans may over-trust automation, ignore useful alerts, or lack time and authority to challenge outputs. Effective oversight requires:

• clearly defined responsibility for accepting, rejecting or escalating recommendations;
• training in model limitations and uncertainty;
• interfaces that display relevant context rather than an isolated score;
• mechanisms for override and documentation;
• monitoring of automation bias and alert fatigue; and
• organizational protection for staff who appropriately challenge the system.

For high-risk decisions, human review should be meaningful rather than ceremonial. The reviewer needs access to sufficient evidence, adequate time and authority to alter the decision.

5. Model Drift and Post-Implementation Evaluation

A model is developed under a data-generating process that may change. Covariate drift occurs when predictor distributions change. Outcome drift occurs when outcome prevalence changes. Concept drift occurs when the relationship between predictors and outcome changes. For example, a treatment innovation may alter the relationship between a clinical marker and mortality.

Monitoring should include:

• input-data completeness and plausibility;
• changes in predictor distributions;
• calibration and discrimination where outcomes are available;
• subgroup performance;
• alert volume and override rates;
• time to response;
• patient outcomes and unintended consequences; and
• cybersecurity and data-integrity events.

A pre-specified monitoring plan should define warning thresholds, review frequency, responsible personnel and actions such as recalibration, retraining, suspension or retirement. Recalibration may correct probability estimates without changing ranking, but it is not always sufficient when clinical relationships have changed.

Post-implementation evaluation should compare outcomes with an appropriate counterfactual. Options include stepped-wedge trials, cluster-randomized trials, interrupted time series, difference-in-differences designs and target-trial emulation. The choice depends on feasibility, rollout structure and the plausibility of assumptions.

6. Real-World Evidence

Real-world evidence is evidence about the use, benefits, risks or value of health interventions derived from real-world data. Sources include electronic records, claims, registries, pharmacy data, patient-generated data, pragmatic trials and linked social care datasets.

RWE can answer questions that conventional trials may not address efficiently:

• How does an intervention perform in routine practice?
• Which patients receive it and who is excluded?
• What are long-term safety outcomes?
• How does effectiveness vary by setting or subgroup?
• What are utilization, cost and equity consequences?

However, real-world data are not automatically representative or unbiased. Confounding by indication, missing outcomes, coding changes, immortal-time bias, selection bias and differential follow-up can distort results.

A causal contrast should specify the population, treatment strategies, assignment or treatment rule, outcome, follow-up period and estimand. For example, “the effect of initiating treatment within seven days compared with not initiating it within seven days on 90-day readmission” is more evaluable than “the impact of treatment.”

Propensity-score methods can help address measured confounding. A propensity score is the probability of receiving treatment conditional on observed covariates:

where A indicates treatment exposure and X is a vector of measured pre-treatment covariates. Matching, weighting or stratification based on e(X) can improve comparability, but cannot remove unmeasured confounding. Correct temporal ordering, adequate overlap and careful outcome definition remain essential.

7. Learning Health Systems

A learning health system integrates care delivery, data generation, analysis and improvement so that routine practice produces knowledge and knowledge is rapidly translated into practice. Its cycle is:

1. care generates data;
2. data are transformed into evidence;
3. evidence informs a change in practice;
4. outcomes of the change are measured; and
5. the cycle repeats.

The concept is attractive because it reduces the separation between research and service improvement. Yet rapid learning creates ethical and methodological challenges. Patients may not know whether their data are being used for quality improvement, research or algorithm development. Continuous adaptation may make the intervention and comparator unstable. Repeated testing can increase false-positive findings if analytical plans are not disciplined.

A mature LHS requires data interoperability, trustworthy identity management, transparent governance, patient participation, reproducible analytics and protected resources for evaluation. It should also preserve a distinction between quality improvement and research when consent, oversight or generalizable knowledge makes that distinction relevant.

8. Illustrative Case: Predicting Emergency Department Deterioration

This is a fictional illustrative case.

Background

A regional hospital introduces a machine-learning model that predicts the risk of clinical deterioration within six hours of emergency-department arrival. The model uses vital signs, laboratory results, demographics, arrival mode and selected electronic-record variables. A high-risk alert prompts nurse review and consideration of senior assessment.

Problem

During the first month, the model shows an AUROC of 0.86 in retrospective testing. After implementation, clinicians report too many alerts. Preliminary review finds that older patients and patients from a deprived neighborhood receive alerts more often, while calibration appears worse among patients whose laboratory results are delayed.

Analysis

The evaluation team separates four issues. First, the retrospective test may have benefited from leakage or a population unlike current patients. Second, alert frequency is not equivalent to benefit. Third, subgroup differences may reflect both true risk and unequal data completeness. Fourth, the workflow intervention—not merely the prediction—must be evaluated.

The team conducts prospective silent validation for four weeks, assesses calibration by subgroup, audits missingness, and interviews clinicians. It finds that the model is reasonably discriminative but overpredicts risk in one subgroup because historical admission patterns were used as a strong predictor. Nurses also report that alerts arriving during shift change are less likely to receive timely review.

Approach and solution

The hospital removes the problematic utilization variable, recalibrates the model using recent data, introduces a maximum alert frequency per patient, and displays the main clinical factors and data-timeliness warnings. A nurse-led response protocol defines when senior review is required. The hospital then conducts a stepped implementation across units, measuring response time, avoidable deterioration, intensive-care transfers, false-alert burden and subgroup outcomes.

Outcome

The revised system reduces alert volume and improves calibration. However, the first impact analysis shows no clear reduction in intensive-care transfers, partly because staffing constraints limit response capacity. The hospital therefore treats the model as an incomplete intervention and invests in escalation resources before expanding it.

Learning points

1. Strong retrospective performance does not establish implementation benefit.
2. A model can expose workflow weaknesses rather than solve them.
3. Fairness requires examining data quality, thresholds and downstream action.
4. Model modification should be followed by renewed validation.
5. The correct unit of evaluation is often the socio-technical care pathway.

9. Regulatory, Legal and Methodological Developments

Regulation is moving toward risk-based oversight, lifecycle management and accountability for developers and deployers. Requirements vary by jurisdiction, but common expectations include intended-use definition, quality management, documentation, cybersecurity, performance evidence, post-market monitoring, transparency and change control. Software that influences diagnosis, treatment or resource allocation may be regulated differently from low-risk administrative software.

Health care organizations should maintain an algorithm inventory containing intended purpose, population, data sources, version, owner, validation evidence, known limitations, fairness assessment, approval status and retirement criteria. Procurement contracts should address data rights, auditability, model updates, incident reporting, security and responsibility for harm.

Methodologically, future evaluation is likely to place greater emphasis on:

• prospective and pragmatic trials of AI-enabled pathways;
• reporting standards for prediction models and AI interventions;
• transportability and subgroup performance;
• causal machine learning for heterogeneous treatment effects;
• privacy-preserving analysis and federated learning;
• synthetic data assessment rather than uncritical use;
• patient-reported outcomes and experience measures;
• environmental and computational cost; and
• evaluation of generative AI, including hallucination, provenance and automation bias.

Generative AI requires particular caution. Fluency is not evidence of accuracy. Evaluation should test factuality, omissions, unsafe recommendations, bias, confidentiality, reproducibility and the effect on user behavior. Human review remains necessary where errors can cause material harm.

10. Managerial Best Practices

A health care organization considering AI should:

1. Define the decision problem and intended benefit before selecting a technology.
2. Identify affected patients, professionals and communities through participatory governance.
3. Establish a valid target outcome and an appropriate comparator.
4. Audit data provenance, missingness, representativeness and leakage.
5. Require internal, temporal and external validation where feasible.
6. Report discrimination, calibration, clinical utility and subgroup performance.
7. Test the complete workflow prospectively before routine use.
8. Specify human responsibilities, override processes and escalation pathways.
9. Monitor drift, safety, equity, alert burden and outcomes after deployment.
10. Document model versions, changes, incidents and retirement decisions.
11. Use proportional governance: higher-risk applications require stronger evidence and oversight.
12. Treat patients and communities as partners in decisions about data use and acceptable risk.

Challenges and Limitations

AI evaluation faces persistent limitations. Outcomes may be rare, delayed or measured inconsistently. External validation may be difficult when institutions use different coding systems. Fairness metrics can conflict, and group categories may be incomplete or socially constructed. Randomized trials may be expensive or ethically complex, while observational studies remain vulnerable to confounding. Models can also change clinician behavior, invalidating assumptions used in their original evaluation.

There are wider political and organizational concerns. Commercial opacity can limit independent auditing. Concentration of technology suppliers may create dependency. Digital systems may increase workload or surveillance. Energy use and hardware supply chains create environmental costs. Most importantly, algorithmic intervention cannot substitute for adequate staffing, accessible services or attention to structural determinants of health.

Chapter Summary

AI and predictive analytics expand the tools available for health care evaluation, but their value depends on the decision and system in which they operate. Prediction is distinct from causal inference, and statistical performance is distinct from clinical impact. Evaluators should examine discrimination, calibration, clinical utility, subgroup fairness, explainability and human oversight. Validation must extend beyond random data splits to temporal, geographic, prospective and impact evaluation.

Real-world evidence can describe effectiveness and safety in routine practice, but requires careful control of bias, confounding and data quality. Learning health systems connect care, data, evidence and improvement, while creating important governance and methodological responsibilities. Deployed models require lifecycle monitoring for drift, alert burden, equity and unintended harm. Sound governance includes transparent accountability, version control, participatory oversight, proportional regulation and meaningful opportunities for human challenge.

Key Terms

Artificial intelligence: Computational methods that perform tasks associated with human intelligence, such as classification, prediction or language generation.

Calibration: Agreement between predicted probabilities and observed outcome frequencies.

Causal inference: Estimation of the effect of an intervention or exposure under specified counterfactual conditions.

Concept drift: Change over time in the relationship between predictors and the outcome.

Decision support: Information or recommendations intended to improve a human or organizational decision.

Discrimination: A model’s ability to distinguish between people with and without an outcome.

External validation: Testing a model in data different from those used for development.

Fairness: The ethically and socially justified distribution of errors, benefits and burdens across groups.

Learning health system: A system that continuously converts routine care data into evidence and improvement.

Model drift: Degradation or change in model behavior caused by changes in data or context.

Net benefit: A threshold-based measure of clinical usefulness that weighs true and false positives.

Predictive analytics: Use of data and statistical or machine-learning methods to estimate future or unknown outcomes.

Real-world data: Data generated during routine health care, public health or daily-life activities.

Real-world evidence: Evidence derived from analysis of real-world data about interventions or health outcomes.

Sensitivity: The proportion of people with an outcome correctly identified as positive.

Specificity: The proportion of people without an outcome correctly identified as negative.

Target leakage: Use of information in model development that would not be available at the intended prediction time.

Temporal validation: Testing performance on data from a later period.

Explainability: The extent to which model behavior or outputs can be understood and communicated.

Human oversight: Meaningful human responsibility and authority to review, challenge or change an AI-supported decision.

Review Questions

Short Answer Questions

1. Distinguish predictive validity from causal validity.
2. What is calibration, and why is it important for risk communication?
3. Define concept drift and give one health care example.
4. Why is external validation stronger than a random train-test split for assessing transportability?
5. Identify two requirements for meaningful human oversight.

Long Answer Questions

1. Critically discuss the evaluation of an AI-enabled clinical decision-support system as a socio-technical intervention.
2. Compare discrimination, calibration and clinical utility as measures of predictive-model performance.
3. Explain why fairness objectives may conflict and describe how an organization should make these choices explicit.
4. Evaluate the opportunities and limitations of real-world evidence for assessing treatment effectiveness.
5. Discuss the principles and governance requirements of a learning health system.

Analytical / Application Questions

1. A readmission model has an AUROC of 0.89 but substantially overpredicts risk in one ethnic group. What additional analyses and governance actions should precede deployment?
2. A hospital introduces an AI deterioration alert, but response capacity is unchanged and alert overrides increase. Design an evaluation that distinguishes model failure from workflow failure.
3. A health authority wants to use claims data to estimate the causal effect of early treatment. Specify the treatment, comparator, population, outcome, follow-up period and major sources of bias that must be addressed.

Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes prediction from causal inference?

A. Prediction estimates likely outcomes, whereas causal inference estimates effects of alternative actions.

B. Prediction always requires randomization, whereas causal inference never does.

C. Prediction is concerned only with costs, whereas causal inference is concerned only with mortality.

D. Prediction is qualitative, whereas causal inference is always qualitative.

Correct Answer: A. Prediction estimates likely outcomes, whereas causal inference estimates effects of alternative actions.

Question 2

A model assigns a group an average risk of 0.20, and approximately 20% of that group experience the outcome. This primarily demonstrates:

A. High specificity

B. Calibration

C. Concept drift

D. Data leakage

Correct Answer: B. Calibration

Question 3

Which measure is most directly concerned with ranking people according to outcome risk?

A. Calibration intercept

B. Specificity at one threshold

C. AUROC

D. Brier score only

Correct Answer: C. AUROC

Question 4

In the net-benefit expression, pₜ represents:

A. The observed outcome prevalence

B. The model’s average predicted probability

C. The probability threshold at which intervention is chosen

D. The proportion of missing observations

Correct Answer: C. The probability threshold at which intervention is chosen

Question 5

Why does removing race or ethnicity from a model not necessarily eliminate unfairness?

A. Models cannot operate without demographic variables.

B. Correlated variables may act as proxies.

C. Calibration automatically becomes perfect.

D. External validation becomes impossible.

Correct Answer: B. Correlated variables may act as proxies.

Question 6

Which example is most consistent with concept drift?

A. The proportion of missing laboratory results increases.

B. The distribution of patient ages changes slightly.

C. A new treatment changes the relationship between a biomarker and mortality.

D. The model file is renamed.

Correct Answer: C. A new treatment changes the relationship between a biomarker and mortality.

Question 7

Which form of validation most directly assesses performance in a different institution?

A. Bootstrap validation

B. Geographic external validation

C. Repeated training-set evaluation

D. Feature selection

Correct Answer: B. Geographic external validation

Question 8

What is a central limitation of propensity-score methods?

A. They cannot use any clinical variables.

B. They eliminate the need to define an outcome.

C. They cannot remove confounding from unmeasured variables.

D. They are valid only for randomized trials.

Correct Answer: C. They cannot remove confounding from unmeasured variables.

Question 9

Which is the strongest example of meaningful human oversight?

A. Displaying an unchallengeable score to staff

B. Allowing review by a trained professional with authority to override the recommendation

C. Automatically applying every high-risk recommendation

D. Removing explanations to prevent disagreement

Correct Answer: B. Allowing review by a trained professional with authority to override the recommendation

Question 10

Why should an AI system be evaluated as a socio-technical intervention?

A. Its effects depend on data, workflow, users, actions and organizational context.

B. Software performance is unrelated to clinical outcomes.

C. Human users never influence algorithmic results.

D. Technical validation is impossible in health care.

Correct Answer: A. Its effects depend on data, workflow, users, actions and organizational context.