Evidence-Based Healthcare, Program Evaluation and Decision-Making

Introduction and strategic context

Health-care managers make decisions under conditions of uncertainty, scarcity and competing values. They must decide which interventions to fund, how to implement them, whether services are producing benefit, and when an established programme should be redesigned or discontinued. Evidence-based healthcare provides a disciplined approach to these decisions by combining the best available research evidence with professional expertise, patient and population preferences, feasibility, equity and opportunity cost.

Evidence-based decision-making is broader than applying the result of a clinical trial. A trial may demonstrate efficacy under controlled conditions, while managers must determine whether an intervention is effective in routine practice, affordable, acceptable, equitable and sustainable. Consequently, this chapter links critical appraisal with implementation science, programme evaluation, health technology assessment (HTA), economic evaluation and quality improvement (QI).

Learning objectives

By the end of this chapter, the learner should be able to:

1. Distinguish major forms and hierarchies of health evidence.
2. Critically appraise primary research, systematic reviews and clinical guidelines.
3. Construct a logic model and select appropriate evaluation designs and indicators.
4. Explain implementation-science frameworks and diagnose implementation barriers.
5. Apply the principles of HTA and economic evaluation to priority-setting.
6. Compare quality-improvement methods and interpret outcome, process and balancing measures.
7. Integrate evidence, values, feasibility, equity and cost into defensible management decisions.

1. The foundations of evidence-based healthcare

Evidence-based healthcare integrates three domains:

• Research evidence: estimates of benefits, harms, diagnostic accuracy, prognosis, cost and implementation effects.
• Contextual and professional knowledge: clinical expertise, organisational capability, local epidemiology and service constraints.
• Values and preferences: what patients, communities, staff, funders and policy-makers consider important.

A useful decision is not necessarily the decision supported by the largest study. It is the decision that best balances expected health benefit, harm, equity, acceptability, affordability and feasibility for the population concerned.

Types of evidence

Experimental studies, particularly randomised controlled trials (RCTs), are often strong for estimating causal effects. Cohort and case-control studies can examine prognosis, aetiology and adverse effects, but are more vulnerable to confounding. Qualitative studies explain experience, acceptability and mechanisms. Economic studies estimate resource consequences. Routine data and surveillance describe reach, variation and population need.

The hierarchy of evidence is question-dependent. An RCT may be appropriate for treatment efficacy, but qualitative research may be more informative for understanding why patients do not attend a service. A systematic review of poor-quality studies does not automatically provide high-certainty evidence.

PICO and question formulation

A focused question commonly identifies:

• Population or problem
• Intervention or exposure
• Comparator
• Outcomes

For implementation questions, PICOT adds time, and frameworks may also specify setting and implementation strategy. Outcomes should be meaningful to patients and decision-makers, not merely easy to measure. Mortality, functional status and quality of life are usually more important than an isolated surrogate marker, although validated surrogate outcomes can be useful when strongly linked to patient benefit.

2. Critical appraisal of research evidence

Critical appraisal asks three questions: Is the study trustworthy? What is the size and precision of the effect? Is the evidence applicable to this decision?

Internal validity

Important threats include selection bias, information bias, confounding, attrition and selective reporting. Randomisation reduces confounding when successfully implemented. Allocation concealment prevents foreknowledge of assignment; it is distinct from blinding. Blinding may reduce performance and ascertainment bias but is not always possible.

For an intervention study, intention-to-treat analysis preserves the benefits of randomisation and estimates the effect of assignment. Per-protocol analysis may estimate the effect among adherent participants but can introduce bias because adherence is often related to prognosis.

Effect measures

For a binary outcome, risk is the probability of the outcome during a defined period. If a events occur among n people, then:

The risk ratio is:

The risk difference is:

These measures require comparable outcome definitions and follow-up periods. For an undesirable outcome, a risk ratio below 1 suggests benefit. Absolute effects are particularly useful for planning because they incorporate baseline risk.

If an intervention reduces risk, the relative risk reduction is:

When the risk difference is expressed as a positive absolute reduction, the number needed to treat is:

NNT is dimensionless and must be accompanied by the outcome, follow-up period and comparator. It should not be interpreted without considering harms, costs and baseline risk. For example, if stroke risk falls from 10% to 6%, the absolute reduction is 4 percentage points, or 0.04, and NNT = 1 ÷ 0.04 = 25. Thus, 25 comparable patients would need the intervention during the stated period to prevent one additional stroke, assuming the estimate is valid.

For continuous outcomes, the mean difference is appropriate when studies use the same scale. The standardised mean difference is useful when different scales measure the same construct, but its clinical interpretation can be less direct.

Precision and uncertainty

A confidence interval describes uncertainty in an estimate under repeated-sampling assumptions. A narrow interval indicates greater statistical precision, not necessarily absence of bias. Statistical significance does not establish clinical importance. A small benefit may be statistically persuasive but not worth the burden or cost; conversely, an important effect may be imprecisely estimated in a small study.

Applicability and equity

External validity requires comparison of participants, intervention delivery, setting, comparator and outcomes with the proposed use. Managers should ask whether effects differ by age, sex, disability, socioeconomic position, ethnicity, geography or comorbidity. An intervention can improve average outcomes while widening inequalities if access is concentrated among advantaged groups.

3. Systematic reviews and clinical guidelines

A systematic review uses an explicit question, reproducible search, eligibility criteria, appraisal and synthesis. Meta-analysis statistically combines results, but pooling is justified only when populations, interventions, comparators and outcomes are sufficiently comparable.

Heterogeneity may be clinical, methodological or statistical. A random-effects model allows that true effects may differ across studies; it does not solve poor study quality or unexplained heterogeneity. Reviewers should examine forest plots, confidence intervals, publication bias, selective outcome reporting and the credibility of subgroup analyses.

The GRADE approach commonly assesses certainty across risk of bias, inconsistency, indirectness, imprecision and publication bias. “High certainty” means that further research is unlikely to change confidence in the estimate, not that the intervention is necessarily beneficial or affordable.

Clinical guidelines translate evidence into recommendations. High-quality guidelines disclose conflicts of interest, use systematic evidence review, distinguish evidence from expert judgment, explain recommendation strength and address applicability, equity and resource use. A strong recommendation usually implies that desirable effects clearly outweigh undesirable effects for most people; a conditional recommendation indicates that choices depend substantially on patient preferences, context or resource constraints.

Guidelines should not be treated as inflexible rules. Local adaptation should preserve the evidential rationale while accounting for epidemiology, capacity, law, culture and patient priorities. Deviation may be appropriate when individual circumstances differ, but it should be transparent and documented.

4. Programme evaluation and logic models

A programme is an organised set of resources and activities intended to produce specified changes. Evaluation is the systematic collection and interpretation of information to judge merit, improve implementation and support accountability.

A logic model links:

Inputs → Activities → Outputs → Short-term outcomes → Intermediate outcomes → Long-term impact

Inputs include staff, funding, facilities, data and partnerships. Activities are what the programme does. Outputs are immediate products, such as consultations completed or staff trained. Outcomes are changes in behaviour, access, quality or health. Impact refers to sustained population-level effects. Assumptions and external factors should be stated because the causal chain is never automatic.

A logic model is not a proven theory of change. It is a testable representation of how change is expected to occur. It should identify mechanisms, risks and conditions necessary for success.

Evaluation questions and designs

A credible evaluation begins with decisions that the evidence must inform. Typical questions include:

• Needs: Who requires the intervention and why?
• Process: Was it delivered as intended and to whom?
• Outcome: Did relevant outcomes change?
• Impact: Can change plausibly be attributed to the programme?
• Economic: Was value produced relative to resources used?
• Equity: Who benefited, who was excluded and were disparities altered?
• Sustainability: Can benefits and delivery be maintained?

Before-and-after comparisons are simple but vulnerable to secular trends, regression to the mean and concurrent changes. Stronger quasi-experimental designs include controlled before-and-after studies, interrupted time series and difference-in-differences. Difference-in-differences estimates whether the change over time in an intervention group exceeds the change in a comparison group:

Here, Y is the outcome, I denotes intervention, C denotes comparison, and pre/post indicate periods before and after implementation. A key assumption is parallel trends: without the intervention, both groups would have experienced comparable changes. This assumption must be examined rather than asserted.

Mixed methods are often essential. Quantitative data may show that uptake is low, while interviews explain whether the cause is workload, mistrust, transport, digital exclusion or poor communication.

Indicators and data quality

A balanced indicator set includes outcome, process, output, experience, equity and balancing measures. For example, a primary-care access programme might track appointment waiting time (outcome), proportion of requests triaged within one day (process), patient-reported access (experience), access by deprivation quintile (equity), and staff overtime or emergency referrals (balancing measures).

Indicators should be valid, reliable, sensitive to change, feasible to collect and resistant to gaming. Numerators and denominators must be explicit, and missing data should be reported. A rate requires a defined population and time period; comparisons require attention to case mix and measurement changes.

5. Implementation science

Implementation science studies methods for integrating evidence-based interventions into routine practice. It distinguishes intervention effectiveness from implementation effectiveness. A programme may be clinically effective but poorly implemented; alternatively, implementation may be excellent while the intervention itself has little benefit.

Common determinants include:

• intervention complexity, adaptability and evidence strength;
• organisational resources, leadership, culture and learning climate;
• staff knowledge, beliefs, motivation and workload;
• patient needs, preferences and social conditions; and
• external policy, financing, regulation and professional norms.

Frameworks such as CFIR organise determinants; RE-AIM examines Reach, Effectiveness, Adoption, Implementation and Maintenance; the Consolidated Framework for Implementation Research supports structured diagnosis. The frameworks are aids to thinking, not substitutes for causal reasoning.

Implementation strategies may include audit and feedback, educational outreach, reminders, champions, workflow redesign, technical assistance, financial incentives and participatory adaptation. Strategies should be matched to diagnosed barriers. Training alone is unlikely to solve shortages of staff, incompatible information systems or unsafe workflows.

Fidelity should be assessed carefully. Excessive insistence on fidelity may prevent adaptation, while uncontrolled adaptation may remove the active ingredients. A useful distinction is between core functions that must be preserved and adaptable forms through which they are delivered.

6. Health technology assessment and economic evaluation

HTA is a multidisciplinary process that examines the clinical, economic, ethical, social, organisational and legal consequences of a health technology. “Technology” includes medicines, devices, diagnostics, procedures, digital tools and service models.

HTA asks whether a technology works, for whom, compared with what, at what cost, and with what distributional consequences. It should consider the full care pathway, implementation requirements and disinvestment from alternatives.

Economic evaluation

Cost-minimisation analysis is appropriate only when outcomes are demonstrated to be equivalent. Cost-effectiveness analysis compares costs with outcomes in natural units, such as cases detected or life-years gained. Cost-utility analysis commonly uses quality-adjusted life-years (QALYs). Cost-benefit analysis values both costs and benefits in monetary terms.

The incremental cost-effectiveness ratio is:

where C₁ and C₀ are costs of the new and comparator options, and E₁ and E₀ are their effects. Units may be currency per QALY or currency per life-year gained. The denominator must represent the same outcome metric in both strategies. The ICER is meaningful only when the comparator is relevant and dominated options have been excluded.

An intervention is dominant if it costs less and produces greater benefit. It is dominated if it costs more and produces less benefit. Decision-makers may compare the ICER with a willingness-to-pay threshold, but thresholds are not universal laws. They reflect budget constraints, opportunity costs, social preferences and institutional policy.

Costs should be identified, measured and valued from a stated perspective, such as health-system, payer or societal. Relevant costs may include healthcare use, patient time, productivity, training, infrastructure and adverse events. Time horizons should capture important costs and effects. Discounting future costs and outcomes may be required by national guidance; the chosen rate and sensitivity assumptions must be reported.

Sensitivity analysis tests uncertainty. One-way analysis varies parameters individually; probabilistic sensitivity analysis assigns probability distributions and repeatedly samples values to estimate the probability of cost-effectiveness. Results should be presented alongside clinical uncertainty, equity considerations and affordability. Cost-effective does not mean affordable immediately.

7. Quality improvement and decision-making

Quality improvement seeks to make care safer, more effective, timely, efficient, equitable and patient-centred. Quality assurance often checks compliance against standards, whereas QI tests changes in real settings and learns from variation.

The Model for Improvement asks: What are we trying to accomplish? How will we know that a change is an improvement? What changes may result in improvement? The Plan–Do–Study–Act cycle then tests a change on a small scale, studies data and adapts the next cycle. It is a learning method, not proof that every observed change was caused by the intervention.

Statistical process control (SPC) displays data over time with a centre line and limits estimated from process variation. Control limits are not performance targets. A process can be stable but poor, or unstable while having a desirable average. Special-cause signals should prompt investigation; indiscriminate reaction to every fluctuation creates tampering.

Root-cause analysis examines system contributors after an adverse event. It should avoid hindsight bias and individual blame. Tools such as process mapping, the five whys and failure mode and effects analysis can identify vulnerabilities before harm occurs. Safety improvement requires just culture, psychological safety, reliable reporting and proportionate accountability.

8. Integrated priority-setting

Managers should use a transparent decision process:

1. Define the population, problem and decision.
2. Establish need, baseline outcomes and inequalities.
3. Formulate an answerable question.
4. Appraise evidence and its certainty.
5. Estimate local effectiveness, resources, cost and feasibility.
6. Assess implementation determinants and risks.
7. Compare options, including the opportunity cost of not funding alternatives.
8. Consider equity, ethics, acceptability and legal duties.
9. Agree indicators, targets, review dates and stop or adapt rules.
10. Communicate rationale and revise decisions when evidence changes.

Multi-criteria decision analysis can make trade-offs explicit by scoring options against criteria such as health gain, equity, cost, feasibility and strategic fit. Its weights are normative rather than objective facts; stakeholder participation and sensitivity analysis are therefore essential.

Illustrative case: reducing avoidable hospital admissions

Background. A regional health authority observes rising admissions among older adults with heart failure. A nurse-led home-monitoring and rapid-review service is proposed.

Problem. A small pilot reports fewer admissions, but it has no comparison group, short follow-up and substantial staff overtime. Leaders must decide whether to scale it.

Analysis. The team formulates a PICO question and reviews RCTs and systematic reviews. It maps the proposed pathway: trained nurses and monitoring devices are inputs; daily review and medication reconciliation are activities; completed reviews are outputs; improved self-management and earlier treatment are intermediate outcomes; admissions and quality of life are long-term outcomes. Interviews identify digital exclusion, unreliable broadband and uncertainty about escalation responsibilities.

Approach. The authority pilots the service in four clinics and uses four matched clinics as comparators. It pre-specifies admissions, mortality, quality of life, patient experience, reach by deprivation and staff overtime. A difference-in-differences analysis is planned, with examination of pre-intervention trends. Implementation strategies include staff champions, simplified escalation protocols, loan devices and telephone alternatives. An economic evaluation includes device costs, staff time, admissions avoided and patient time.

Outcome. After one year, admissions fall modestly in intervention clinics, but benefit is concentrated among digitally connected patients. Telephone support improves reach, while overtime remains high. The authority therefore expands the adapted model only where staffing and escalation capacity are secured, and redesigns the workforce model before further expansion.

Learning points. A promising clinical signal is not sufficient for scale-up. Evaluation must examine attribution, reach, equity, workload, cost and sustainability. Implementation adaptation can improve access without abandoning the intervention’s core function.

9. Ethics, governance and contemporary developments

Evidence-based management involves ethical choices because allocating resources benefits some groups more than others. Decision-makers should address autonomy, privacy, fairness, non-discrimination, proportionality and accountability. Patient and public involvement should influence questions, outcomes and interpretation rather than serve merely as consultation after decisions are made.

Data governance requires lawful collection, purpose limitation, security, data minimisation, retention controls and responsible linkage. De-identification reduces but does not eliminate re-identification risk. Evaluation involving people may require ethics review, consent procedures or institutional governance even when classified as service improvement.

Artificial intelligence can support evidence retrieval, risk stratification, workflow prediction and evidence synthesis. However, managers must assess dataset shift, bias, calibration, explainability, automation bias, cybersecurity, vendor transparency and human oversight. An algorithm should be evaluated in the intended population and workflow, not only on retrospective test data. Performance metrics should include subgroup calibration and consequences of false positives and false negatives.

Contemporary practice increasingly emphasises living systematic reviews, real-world evidence, pragmatic trials, adaptive designs, patient-reported outcomes, learning health systems and equity-focused implementation. These developments increase decision speed and relevance but also require stronger governance, version control and methods for distinguishing signal from noise.

Challenges and limitations

Evidence may be unavailable, contradictory, outdated or poorly applicable. Publication and reporting biases can exaggerate benefits. Routine data may contain coding changes, missingness and unequal capture. Programme effects may take years to appear, while budgets operate annually. Economic models depend on assumptions about extrapolation, utilities, adherence and future costs. Implementation studies can overemphasise organisational factors and understate political economy, power and structural inequality.

No framework removes judgement. Transparent assumptions, independent challenge, sensitivity analysis, stakeholder engagement and periodic review are therefore central safeguards. The aim is not certainty, but decisions that are proportionate to evidence, explicit about values and capable of learning.

Chapter Summary

Evidence-based healthcare integrates research evidence with expertise, context, patient preferences, feasibility, equity and opportunity cost. Critical appraisal examines validity, effect size, precision and applicability. Risk ratios and absolute effects answer different questions, and NNT must be interpreted with its outcome and time horizon. Systematic reviews require attention to search methods, bias, heterogeneity and certainty; guidelines require transparent methods and contextual adaptation.

Programme evaluation uses logic models to connect resources, activities, outputs and outcomes. Strong evaluations select designs that address attribution, combine quantitative and qualitative evidence, and use valid indicators with explicit denominators. Implementation science diagnoses determinants and matches strategies to barriers. HTA and economic evaluation examine value, affordability and broader consequences; the ICER is useful only when costs, effects, comparators and assumptions are appropriate. QI uses iterative testing, time-series learning and systems-based safety methods. Strategic decisions should be transparent, equity-sensitive, ethically governed and revisable as evidence changes.

Key Terms

Evidence-based healthcare: Integration of research, expertise, context and values in health decisions.

Critical appraisal: Systematic assessment of validity, importance and applicability of evidence.

Confounding: Distortion of an exposure–outcome relationship by a third variable.

Risk ratio: Risk in an intervention group divided by risk in a comparison group.

Absolute risk reduction: Difference in risk between comparator and intervention when risk is reduced.

Number needed to treat: Number receiving an intervention to produce one additional specified beneficial outcome.

Systematic review: Reproducible synthesis of evidence addressing a focused question.

GRADE: Framework for rating certainty of evidence and strength of recommendations.

Clinical guideline: Evidence-informed recommendations for practice.

Logic model: Diagrammatic or narrative account of how programme resources and activities are expected to produce outcomes.

Process evaluation: Assessment of implementation, delivery, reach and context.

Implementation science: Study of methods for integrating effective interventions into routine practice.

RE-AIM: Framework addressing Reach, Effectiveness, Adoption, Implementation and Maintenance.

Health technology assessment: Multidisciplinary assessment of clinical, economic, ethical and organisational consequences of technology.

QALY: Quality-adjusted life-year combining length and quality of life.

ICER: Incremental cost per additional unit of health outcome gained.

Opportunity cost: Benefit forgone when resources are used for one option rather than the next best option.

Quality improvement: Systematic effort to improve healthcare processes and outcomes.

Statistical process control: Use of time-series methods to distinguish common from special causes of variation.

Balancing measure: Measure used to detect unintended effects of an improvement.

Review Questions

Short Answer Questions

1. What are the three principal domains integrated by evidence-based healthcare?
2. Distinguish allocation concealment from blinding.
3. Why is absolute risk often more useful than relative risk for planning services?
4. What is the purpose of a logic model?
5. Why is cost-effective not necessarily equivalent to affordable?

Long Answer Questions

1. Critically discuss how a manager should appraise a systematic review before applying its findings locally.
2. Explain how implementation science complements evidence of clinical effectiveness.
3. Compare before-and-after evaluation, interrupted time series and difference-in-differences designs.
4. Discuss the ethical and managerial implications of using QALYs in priority-setting.
5. Explain how QI, programme evaluation and HTA can be integrated in a health-system decision.

Analytical / Application Questions

1. A screening programme reduces disease risk from 8% to 5% over five years. Calculate the absolute risk reduction and NNT, and explain what additional information a manager needs before adoption.
2. A hospital introduces a digital discharge tool, but uptake is low among older adults and rural patients. Use an implementation framework to diagnose determinants and propose matched strategies and indicators.
3. Two clinics receive an intervention. Outcomes change from 40 to 30 per 1,000 patients in the intervention clinic and from 50 to 45 per 1,000 in the comparison clinic. Calculate the difference-in-differences estimate and state the key causal assumption.

Multiple Choice Questions with Answers

Question 1

Which feature most directly improves the internal validity of a randomised controlled trial?

A. Random allocation with concealed assignment

B. A large number of study sites without randomisation

C. Use of a surrogate outcome only

D. Exclusion of all participants with comorbidity

Correct Answer: A. Random allocation with concealed assignment

Question 2

An intervention lowers risk from 10% to 6%. What is the approximate number needed to treat over the stated follow-up period?

A. 4

B. 10

C. 25

D. 60

Correct Answer: C. 25

Question 3

Which is an essential assumption for a standard difference-in-differences interpretation?

A. The intervention group has zero baseline risk

B. The groups have parallel trends without the intervention

C. Both groups have identical sample sizes

D. The outcome is always normally distributed

Correct Answer: B. The groups have parallel trends without the intervention

Question 4

Which component of a logic model refers to immediate products of programme activities?

A. Inputs

B. Outputs

C. Intermediate outcomes

D. External factors

Correct Answer: B. Outputs

Question 5

What does GRADE primarily assess?

A. Hospital staffing levels

B. Certainty of evidence and strength of recommendations

C. The monetary value of all health outcomes

D. The reliability of an electronic health record

Correct Answer: B. Certainty of evidence and strength of recommendations

Question 6

Which statement about an ICER is correct?

A. It is incremental cost divided by incremental effect

B. It is total cost divided by total patient number

C. It proves that an intervention is affordable

D. It can be interpreted without specifying a comparator

Correct Answer: A. It is incremental cost divided by incremental effect

Question 7

Which is a balancing measure in a quality-improvement project?

A. The intended reduction in waiting time

B. The number of staff trained

C. Unintended staff overtime after workflow redesign

D. The programme budget approved at initiation

Correct Answer: C. Unintended staff overtime after workflow redesign

Question 8

A clinical guideline with a conditional recommendation usually indicates that:

A. The intervention is legally mandatory

B. The evidence is always high certainty

C. The choice depends substantially on preferences or context

D. The intervention has no possible benefit

Correct Answer: C. The choice depends substantially on preferences or context

Question 9

Which implementation strategy is most directly matched to a knowledge gap among clinicians?

A. Educational outreach with feedback

B. Removing all outcome measurement

C. Increasing device costs

D. Ignoring workflow constraints

Correct Answer: A. Educational outreach with feedback

Question 10

An intervention costs 200,000 currency units and produces 100 additional QALYs compared with standard care. What is the ICER?

A. 2,000 currency units per QALY

B. 20,000 currency units per QALY

C. 100,000 currency units per QALY

D. 200,000 currency units per QALY

Correct Answer: B. 20,000 currency units per QALY