Evaluation Designs and Causal Inference

Introduction

Health-care managers routinely ask causal questions: Did a new discharge programme reduce readmissions? Did a payment reform improve quality without increasing inequity? Did a staffing intervention shorten waiting times? These questions cannot be answered reliably by simply comparing outcomes before and after an intervention or between users and non-users. The central challenge is to determine what would have happened to the same people, at the same time, had they not received the intervention.

Evaluation design is the architecture used to produce credible evidence about an intervention. Experimental designs create comparison groups through random allocation. Quasi-experimental designs use institutional rules, timing, thresholds or other external features that approximate randomization. Observational designs analyse naturally occurring exposure patterns and attempt to control statistically for differences between groups. Each design involves trade-offs among internal validity, feasibility, ethics, cost, timeliness and generalizability.

This chapter develops the counterfactual framework and applies it to randomized controlled trials, interrupted time series, difference-in-differences and regression discontinuity designs. It also examines bias, confounding, attribution, implementation and managerial interpretation.

Learning Objectives

By the end of this chapter, readers should be able to:

1. Define causal effects using counterfactual outcomes.
2. Distinguish experimental, quasi-experimental and observational designs.
3. Identify selection bias, confounding, information bias, contamination and threats to validity.
4. Select an evaluation design appropriate to a health-care decision.
5. Explain assumptions underlying randomized trials, interrupted time series, difference-in-differences and regression discontinuity.
6. Interpret effect estimates responsibly and assess attribution, equity and implementation consequences.

1. The Counterfactual Foundation of Causal Inference

For a person or organization i, let Y_i(1) denote the outcome if exposed to an intervention and Y_i(0) the outcome if not exposed. The individual causal effect is:

Here, τ_i is the causal effect for individual i. The fundamental problem is that only one of the two potential outcomes can be observed for that individual. A person cannot simultaneously receive and not receive the same intervention at the same time. Evaluation therefore estimates average effects across groups or populations.

The average treatment effect (ATE) is:

where E denotes the population mean. The average treatment effect on the treated (ATT) is:

where D = 1 identifies those who received the intervention. ATE and ATT are equal only under particular conditions, such as homogeneous effects or exchangeability between treated and untreated populations.

A simple difference in observed means is:

This equals a causal effect only when the treated and comparison groups are exchangeable with respect to potential outcomes. In observational health-care data, this condition is often implausible because treatment is influenced by illness severity, income, access, clinician judgment and patient preference.

Internal and external validity

Internal validity concerns whether the estimated effect is credible for the evaluated population and setting. External validity concerns whether it transports to other populations, organizations, periods or delivery systems. A tightly controlled trial may have strong internal validity but limited applicability to routine care. Conversely, a large administrative-data study may be highly representative but vulnerable to confounding.

Estimand before estimator

An evaluation should specify its estimand before selecting a statistical model. Important dimensions include:

• Population: Which patients, facilities or communities are included?
• Treatment: What precisely constitutes the intervention, including dose and duration?
• Comparator: Usual care, no programme, delayed implementation or an alternative programme?
• Outcome: Clinical, operational, financial, patient-reported or equity outcome?
• Time horizon: Immediate, one-year or lifetime effect?
• Effect measure: Difference, ratio, risk difference, risk ratio, odds ratio or cost difference?

For a binary outcome, the risk difference is p₁ − p₀, where p₁ and p₀ are intervention and comparison risks. The relative risk is p₁/p₀, provided p₀ > 0. Absolute and relative measures answer different managerial questions and should not be treated as interchangeable.

2. Sources of Bias and Threats to Attribution

Selection bias and confounding

Selection bias occurs when inclusion, treatment, follow-up or analysis depends on factors related to outcomes. Confounding occurs when a third variable affects both treatment assignment and outcome. For example, patients receiving a telemonitoring service may have more severe disease than non-users. A crude comparison could make the service appear ineffective even if it improves outcomes.

A confounder is not simply a variable associated with the outcome. It must also be related to exposure and must not be a consequence of the exposure. Adjusting for a mediator can remove part of the intervention effect; adjusting for a collider can create bias.

Directed acyclic graphs (DAGs) help distinguish these roles. If socioeconomic status affects both programme uptake and readmission, it is a confounder. If the programme affects follow-up attendance and attendance affects readmission, attendance is a mediator. If both treatment and an unmeasured cause affect selection into the analytic sample, conditioning on selection may induce collider bias.

Other important biases

• Performance bias: participants or providers alter behaviour because they know allocation.
• Detection bias: outcome measurement differs between groups.
• Attrition bias: loss to follow-up differs by treatment or prognosis.
• Contamination: comparison participants receive elements of the intervention.
• History bias: external events coincide with implementation.
• Maturation: outcomes change naturally over time.
• Regression to the mean: unusually high or low measurements tend to move toward average on repeat measurement.
• Publication and selective-reporting bias: favourable outcomes are more likely to be reported.
• Immortal-time bias: a period during which an individual must remain event-free is incorrectly attributed to treatment.

Confounding control

Design-based methods include randomization, restriction, matching, eligibility rules and natural experiments. Analytic methods include stratification, regression adjustment, inverse-probability weighting, standardization and propensity-score methods. These methods reduce bias only when relevant confounders are measured adequately and the model is appropriate. Statistical adjustment cannot recover information about a truly unmeasured confounder without additional assumptions.

A useful sensitivity question is: How strong would an unmeasured confounder need to be to explain the observed association? This does not prove causality, but it helps assess robustness.

3. Experimental Designs: Randomized Controlled Trials

In a randomized controlled trial (RCT), eligible units are assigned by a chance mechanism to intervention or comparator. Randomization makes allocation independent of potential outcomes in expectation. It therefore balances measured and unmeasured baseline characteristics across sufficiently large groups.

Basic structure

An RCT requires a defined population, intervention, comparator, allocation procedure, outcome measurement plan and follow-up period. Allocation concealment prevents recruiters from predicting the next assignment. Blinding, where feasible, reduces performance and detection bias. An intention-to-treat analysis estimates the effect of assignment, preserving the benefits of randomization even when adherence is imperfect.

The difference in mean outcomes is:

where Ȳ₁ and Ȳ₀ are observed group means. Under valid randomization, adequate follow-up and appropriate outcome measurement, this estimates the effect of assignment. For a binary outcome, the corresponding estimate is the difference in observed risks.

A per-protocol effect describes the effect among those who adhere to assigned treatment, but estimating it generally requires additional adjustment because adherence is not randomized. It may therefore be more clinically intuitive but less straightforward causally.

Cluster randomized trials

Hospitals, clinics or communities may be randomized instead of individuals. Cluster randomization reduces contamination and is often operationally practical, but observations within a cluster are correlated. The effective sample size is smaller than the raw patient count. A common planning quantity is the design effect:

where m is the average cluster size and ρ is the intracluster correlation coefficient (ICC), the similarity of outcomes within clusters. The approximate effective sample size is raw sample size divided by DE. This formula assumes approximately equal cluster sizes and a common ICC; unequal cluster sizes require more careful planning.

Strengths and limitations

RCTs provide strong internal validity and transparent attribution. However, they may be expensive, slow or ethically impossible. Excluding complex patients, using unusually intensive implementation and measuring short-term outcomes can limit generalizability. In health-care management, pragmatic trials often address these concerns by evaluating interventions under routine conditions.

4. Quasi-Experimental and Observational Designs

Quasi-experimental designs do not use random assignment but exploit a credible source of variation in exposure. They are especially valuable when policy is introduced to all eligible patients or when withholding an intervention is unacceptable.

Observational designs include cohort, case-control, cross-sectional and ecological studies. They can describe patterns, generate hypotheses and estimate associations. With strong design and assumptions, longitudinal observational studies can also support causal inference, but causal claims must be proportionate to the identification strategy.

The key distinction is not whether a study uses sophisticated statistics. It is whether the design creates a credible comparison that approximates the missing counterfactual.

5. Interrupted Time Series

An interrupted time series (ITS) examines repeated outcome measurements before and after a defined intervention point. It estimates whether implementation changed the level or slope of the outcome beyond the pre-existing trend.

A segmented regression model is:

where Y_t is the outcome at time t; β₀ is the baseline level; β₁ is the pre-intervention slope; I_t equals 0 before and 1 at or after intervention; T₀ is the intervention time; β₂ is the immediate level change; β₃ is the change in slope; and ε_t is the error term. The post-intervention slope is β₁ + β₃.

For valid inference, the analyst should examine seasonality, autocorrelation, sufficiently many pre- and post-intervention observations, concurrent interventions and changes in measurement. Standard errors must account for serial correlation where present. A single before-and-after comparison is not an ITS because it cannot distinguish intervention effects from ongoing trends.

ITS is particularly useful for national policies, electronic-record changes and hospital-wide programmes. A controlled ITS, using an unaffected outcome or comparison site, provides stronger protection against history bias.

6. Difference-in-Differences

Difference-in-differences (DiD) compares change over time in an intervention group with change over time in a comparison group. The basic estimator is:

where T denotes the treated group and C the comparison group. The first difference removes time-invariant group differences; the second difference removes common time shocks.

A two-period regression is:

Here, δ is the DiD estimate under the identifying assumptions. α is the comparison-group baseline mean, β captures baseline group differences, and γ captures common time change.

The central assumption is parallel trends: absent the intervention, treated and comparison groups would have experienced the same average change. This assumption cannot be proven from post-intervention data. Analysts should inspect pre-intervention trends, conduct placebo tests and use event-study specifications where appropriate. Parallel trends in observed outcomes do not guarantee parallel trends in unobserved potential outcomes, but substantial pre-period evidence increases credibility.

Other threats include anticipation, spillovers, staggered implementation and treatment effects that vary over time. With staggered adoption, conventional two-way fixed-effects estimators can be difficult to interpret when effects are heterogeneous. Modern estimators should be considered, particularly when treatment timing differs across sites.

7. Regression Discontinuity Designs

A regression discontinuity design (RDD) applies an intervention when a continuous assignment variable crosses a known threshold. For example, patients with a risk score of 70 or more may qualify for intensive case management. Patients just above and just below 70 are likely to be similar except for treatment assignment.

A sharp RDD has deterministic assignment:

where X_i is the assignment score and c is the cutoff. The local causal effect is the discontinuity in expected outcomes at the threshold:

This estimate applies locally to units near the cutoff. In a fuzzy RDD, crossing the threshold changes the probability of treatment rather than determining treatment perfectly; the discontinuity in outcome is then interpreted relative to the discontinuity in treatment probability.

Validity requires continuity of potential outcomes at the cutoff, no precise manipulation of the assignment score, and sufficient observations near the threshold. Analysts should test for manipulation, inspect covariate continuity, justify bandwidth and polynomial specification, and report sensitivity to alternative bandwidths. High-order global polynomials are generally discouraged because they can behave erratically at boundaries. RDD estimates may not apply to patients far from the cutoff.

8. Choosing a Design

Design choice should also reflect the decision’s stakes. For a high-cost national reform, a controlled ITS or phased cluster trial may be preferable to a simple descriptive comparison. For an eligibility rule, RDD may be more credible than regression adjustment across the entire population. For rare outcomes, a long follow-up or linked administrative data may be necessary.

9. Worked Application: A Hospital Readmission Programme

Background

This illustrative case concerns 12 hospitals implementing a pharmacist-led discharge programme intended to reduce 30-day readmissions among older adults with heart failure. Six hospitals began in January 2024; six comparison hospitals planned implementation in January 2025.

Problem

Managers initially compared readmissions in 2024: 18% in programme hospitals versus 16% in comparison hospitals. They concluded that the programme increased readmissions. This conclusion is inappropriate because hospitals differed before implementation, and the comparison ignores trends over time.

Analysis

Suppose the following rates were observed:

The programme-group change is 18% − 20% = −2 percentage points. The comparison-group change is 16% − 17% = −1 percentage point. Therefore:

The estimated programme effect is a reduction of 1 percentage point in readmissions relative to the comparison trend, not the raw 2-point reduction. If the baseline programme-group rate was 20%, the relative reduction against that baseline is approximately 1/20 = 5%, although the confidence interval and model specification are essential before making a formal claim.

The evaluation team should examine several pre-intervention years for parallel trends; verify that comparison hospitals did not introduce similar medication-reconciliation services; account for patient mix and clustering by hospital; examine mortality and emergency visits as possible unintended outcomes; and assess whether pharmacists actually delivered the planned intervention.

Approach and outcome

The preferred analysis is a controlled DiD with hospital and time effects, robust inference at the hospital level where feasible, and a pre-specified primary outcome. A process evaluation should measure medication reconciliation completion, patient understanding, follow-up attendance and fidelity. If the programme is later expanded to all hospitals, an ITS can assess whether the effect persists at system level.

Learning points

The case shows why a counterfactual is necessary, why baseline differences are not automatically bias, why trends matter, and why causal evaluation must combine outcome analysis with implementation evidence. It also illustrates that a statistically significant effect may not be practically important, and a small average effect may conceal larger benefits for high-risk or underserved patients.

10. Implementation, Ethics and Governance

Evaluation is not separate from ethical health-care management. An RCT requires ethical review, informed consent when applicable, risk minimization, privacy protection and fair participant selection. A waiver of consent may be considered for minimal-risk service evaluation only under applicable legal and institutional standards.

Data governance should specify lawful access, data minimization, secure linkage, retention, role-based access and disclosure controls. Analysts should pre-specify outcomes and analysis plans where possible, distinguish confirmatory from exploratory analyses, document deviations and report adverse outcomes. Subgroup analyses should be clinically justified and interpreted cautiously because multiple comparisons can generate chance findings.

Equity must be designed into evaluation. Report effects by relevant dimensions such as age, disability, sex, ethnicity, language, geography and deprivation when ethically and statistically appropriate. An intervention can improve the average outcome while widening disparities if access or adherence is uneven.

Artificial intelligence and digital health systems create additional evaluation challenges. Predictive models may change clinical behaviour, making historical validation insufficient. Evaluation should assess calibration, discrimination, workflow effects, automation bias, subgroup performance, privacy and unintended consequences. A model’s predictive accuracy does not establish that using it improves patient outcomes; causal evaluation of the deployment strategy is required.

11. Best Practices for Credible Evaluation

1. State the causal question, estimand, population, treatment, comparator and time horizon before analysing data.
2. Use a design feature that makes the counterfactual credible; do not rely on statistical sophistication alone.
3. Draw a causal diagram to identify confounders, mediators and colliders.
4. Measure outcomes consistently and blind assessors where feasible.
5. Examine baseline balance, pre-trends, missingness, contamination and implementation fidelity.
6. Use an appropriate unit of analysis and account for clustering and serial correlation.
7. Report effect sizes with uncertainty, not only p-values.
8. Present absolute effects alongside relative effects.
9. Test plausible alternative explanations and conduct sensitivity analyses.
10. Interpret generalizability, equity, costs and unintended effects explicitly.

Challenges and Limitations

No design eliminates uncertainty. Randomization may be infeasible or may estimate an effect under conditions unlike routine practice. Quasi-experiments depend on assumptions that cannot be tested completely. Administrative data may contain coding changes, missing clinical detail and inconsistent outcome definitions. Small numbers of hospitals limit statistical power and make conventional standard errors unreliable. Treatment effects may differ by patient risk, provider skill and context, so an average estimate can conceal clinically important heterogeneity.

Causal inference also involves transportability. An effect identified among patients near an RDD threshold, or in early-adopting hospitals, may not apply to the entire health system. Decision-makers should therefore combine causal estimates with clinical judgment, implementation evidence, cost-effectiveness analysis and stakeholder perspectives.

Chapter Summary

Causal evaluation asks what outcome would have occurred under an alternative exposure. Because individual potential outcomes cannot both be observed, credible designs construct comparison groups that approximate the counterfactual. Randomized trials achieve this through chance allocation, while quasi-experimental designs exploit timing, comparison trends or eligibility thresholds. Interrupted time series estimates intervention-related changes in level and slope; difference-in-differences estimates an additional change in treated units under the parallel-trends assumption; and regression discontinuity estimates a local effect at an assignment cutoff.

Selection bias, confounding, measurement error, attrition, contamination, history and regression to the mean can undermine attribution. Statistical adjustment is useful but cannot replace a credible design or correct for unmeasured confounding automatically. Professional evaluation requires explicit estimands, transparent assumptions, appropriate uncertainty, implementation assessment, ethical governance and attention to equity and generalizability.

Key Terms

Counterfactual: The outcome that would have occurred under an alternative treatment condition.

Potential outcome: An outcome defined under a particular exposure level.

Average treatment effect: The average difference between intervention and non-intervention potential outcomes.

Average treatment effect on the treated: The causal effect among those who received the intervention.

Exchangeability: The condition that treatment groups have comparable potential outcomes, apart from treatment.

Confounding: Distortion caused by a variable related to both exposure and outcome.

Selection bias: Bias caused by systematic differences in inclusion, treatment, follow-up or analysis.

Randomized controlled trial: An experiment using chance to allocate units to intervention or comparator.

Allocation concealment: Preventing recruiters from predicting assignment.

Intention-to-treat: Analysis according to assigned group, regardless of adherence.

Cluster randomized trial: A trial in which groups such as clinics or hospitals are randomized.

Intracluster correlation coefficient: Similarity of outcomes among units in the same cluster.

Interrupted time series: Repeated outcome observations analysed around an intervention time.

Difference-in-differences: A comparison of changes over time between treated and comparison groups.

Parallel trends: The assumption that groups would have changed similarly without intervention.

Regression discontinuity: A design using treatment assignment determined by a cutoff.

Spillover: An intervention effect experienced by members of the comparison group.

Internal validity: Credibility of causal inference within the evaluated setting.

External validity: Applicability of findings to other settings or populations.

Estimand: The precisely defined causal quantity an evaluation seeks to estimate.

Review Questions

Short Answer Questions

1. Why can an individual causal effect not usually be observed directly?
2. Distinguish the ATE from the ATT.
3. What is the central identifying assumption of a basic difference-in-differences design?
4. In an interrupted time series, what do an immediate level change and a slope change represent?
5. Why is a regression discontinuity estimate usually considered local?

Long Answer Questions

1. Compare randomized controlled trials, quasi-experimental designs and observational studies in terms of attribution, feasibility, ethics and generalizability.
2. Explain how selection bias, confounding, mediation and collider bias differ, using health-care examples.
3. Describe the assumptions, analysis and threats to validity of an interrupted time-series evaluation.
4. Explain how cluster randomization changes trial planning and analysis.
5. Discuss how an evaluation can incorporate ethics, data governance, equity and implementation fidelity.

Analytical / Application Questions

1. A hospital introduces an electronic prescribing alert in April. Monthly prescribing errors were falling before April, and a national safety campaign began in May. Which evaluation design would you recommend, and how would you address the competing explanations?
2. A case-management programme is offered to patients with a risk score of 70 or higher. Explain how you would assess whether a regression discontinuity design is credible and what population the estimate would describe.
3. Two regions introduce a payment reform in different years, while outcomes change heterogeneously over time. Explain why a simple two-way fixed-effects difference-in-differences model may be problematic and propose diagnostic or alternative approaches.

Multiple Choice Questions with Answers

Question 1

What is the fundamental problem of causal inference?

A. Outcomes cannot be measured in health care

B. Both potential outcomes for the same unit cannot be observed simultaneously

C. Randomization always produces unequal groups

D. Regression models cannot include multiple variables

Correct Answer: B. Both potential outcomes for the same unit cannot be observed simultaneously

Question 2

Which quantity is represented by E[Y(1) − Y(0)]?

A. Average treatment effect

B. Odds ratio

C. Intracluster correlation coefficient

D. Prevalence ratio

Correct Answer: A. Average treatment effect

Question 3

In a randomized controlled trial, the primary purpose of allocation concealment is to:

A. Increase the sample size

B. Prevent recruiters from predicting assignments

C. Guarantee participant adherence

D. Eliminate outcome measurement

Correct Answer: B. Prevent recruiters from predicting assignments

Question 4

In the segmented regression model for an interrupted time series, the coefficient for the post-intervention indicator usually represents:

A. The pre-intervention slope

B. The immediate level change

C. The sample mean

D. The intracluster correlation

Correct Answer: B. The immediate level change

Question 5

The key assumption in a basic difference-in-differences analysis is:

A. Treatment must be assigned randomly

B. The treated group must have a higher baseline outcome

C. Treated and comparison groups would have followed parallel trends without treatment

D. There can be no time trend in either group

Correct Answer: C. Treated and comparison groups would have followed parallel trends without treatment

Question 6

A programme-group outcome falls from 20% to 18%, while a comparison-group outcome falls from 17% to 16%. What is the difference-in-differences estimate?

A. −3 percentage points

B. −2 percentage points

C. −1 percentage point

D. +1 percentage point

Correct Answer: C. −1 percentage point

Question 7

In a sharp regression discontinuity design, treatment assignment is determined by:

A. A random number generator only

B. Crossing a known cutoff on an assignment variable

C. The outcome measured after treatment

D. The average outcome in the comparison group

Correct Answer: B. Crossing a known cutoff on an assignment variable

Question 8

Which variable is most likely a mediator rather than a confounder if a discharge programme increases follow-up attendance, which then reduces readmissions?

A. Baseline age

B. Socioeconomic status measured before treatment

C. Follow-up attendance after programme enrolment

D. Pre-intervention disease severity

Correct Answer: C. Follow-up attendance after programme enrolment

Question 9

For a cluster randomized trial, the design effect increases when:

A. The intracluster correlation is larger

B. The outcome is measured more accurately

C. The intervention is blinded

D. The number of outcome categories decreases

Correct Answer: A. The intracluster correlation is larger

Question 10

Which statement best describes an RDD estimate?

A. It necessarily applies equally to every patient in the health system

B. It estimates a causal effect for units near the eligibility cutoff, under continuity assumptions

C. It requires no examination of the assignment variable

D. It is equivalent to a cross-sectional correlation

Correct Answer: B. It estimates a causal effect for units near the eligibility cutoff, under continuity assumptions