Process, Implementation and Improvement Evaluation

Introduction and context

Health-care evaluation is concerned not only with whether an intervention produces desired outcomes, but also with how it is delivered, to whom, under what conditions, and whether it can be maintained. A programme may appear ineffective because its underlying clinical theory is weak; equally, it may fail because it was not implemented as designed. Conversely, favourable outcomes may result from adaptations, contextual advantages, or unrelated changes in the health system. Process, implementation and improvement evaluation provide the analytical tools needed to distinguish these explanations.

This chapter examines the evaluation of implementation quality and organisational improvement. It focuses on fidelity, reach, dose, adoption, feasibility, acceptability and sustainability; introduces implementation-science frameworks; and explains how PDSA and Lean methods support iterative improvement. The central managerial question is not simply, “Did the intervention work?” It is also, “Was it delivered, received and sustained in a way that makes a fair test of the intervention possible?”

Learning objectives

By the end of this chapter, the reader should be able to:

1. Distinguish intervention outcomes from implementation and process outcomes.
2. Define and measure fidelity, reach, dose, adoption, feasibility, acceptability and sustainability.
3. Apply implementation-science frameworks to health-care evaluation.
4. Diagnose whether poor results reflect implementation failure, intervention failure, or both.
5. Use PDSA and Lean approaches for structured service improvement.
6. Design mixed-method process evaluations that combine quantitative and qualitative evidence.
7. Interpret implementation evidence ethically, critically and for managerial decision-making.

1. From outcome evaluation to implementation evaluation

An intervention is the clinical, organisational or policy strategy intended to produce change. Examples include a falls-prevention bundle, a digital appointment system, a new workforce model or a population-screening programme. Implementation is the process of integrating that intervention into routine practice. Process evaluation examines what actually happened during delivery and how participants experienced it.

A useful distinction is between three classes of outcomes:

These outcomes are related but not interchangeable. High fidelity does not guarantee clinical benefit, and a positive clinical result does not prove that implementation was faithful. A process evaluation therefore reconstructs the causal pathway from intervention resources and activities to intermediate service changes and ultimate outcomes.

A simple logic model is:

Inputs → Activities → Immediate outputs → Intermediate outcomes → Patient outcomes

Implementation evaluation asks whether each arrow functioned as expected. For example, a medication-reconciliation programme may provide training and software (inputs), require reconciliation at admission (activity), produce completed medication lists (output), reduce prescribing discrepancies (intermediate outcome), and ultimately reduce adverse drug events (patient outcome).

2. Core implementation constructs

Fidelity

Fidelity is the extent to which delivery conforms to the intervention’s intended design. It has several dimensions:

• Content fidelity: whether the required components were delivered.
• Frequency or dose fidelity: whether activities occurred often enough.
• Quality of delivery: whether activities met the expected standard.
• Participant responsiveness: whether users engaged with the intervention.
• Adaptation fidelity: whether local modifications preserved essential functions.

Fidelity should not be confused with rigid uniformity. Some adaptations are necessary for cultural, logistical or clinical fit. The evaluator should identify core components that must remain unchanged and adaptable periphery components that may be modified.

A basic fidelity proportion can be calculated as:

This measure assumes that components are clearly specified and equally weighted. If components differ substantially in importance, a weighted score may be more valid, but the weights must be justified in advance.

Reach

Reach describes the proportion and characteristics of the eligible population that participates in or is exposed to an intervention. It includes both coverage and representativeness. A programme reaching 70% of eligible patients may still worsen inequity if the remaining 30% are concentrated among people with limited transport, language barriers or complex illness.

The denominator must be defined carefully. For example, “eligible” may mean all patients identified in a registry, while “receiving” may mean attendance at one session or completion of the full programme. Evaluators should report reach by relevant subgroups rather than only as an overall percentage.

Dose

Dose refers to the amount of intervention delivered and received. Delivered dose may include the number of sessions, contacts, minutes or messages provided by staff. Received dose concerns what participants actually attended, used or understood. These can diverge: a clinic may schedule six education sessions, but patients may attend only two.

Dose is often expressed using counts, means, medians or completion proportions. Its interpretation depends on the intervention theory. More dose is not always better; excessive contacts can create burden, while insufficient exposure may prevent benefit.

Adoption

Adoption is the decision by individuals, teams, organisations or systems to initiate use of an intervention. It is important to specify the level of adoption. A hospital may formally adopt a digital triage system, while only a small proportion of clinical teams use it consistently.

Adoption should therefore be assessed at multiple levels:

• organisational commitment and resourcing;
• team enrolment and participation;
• professional use in routine work; and
• patient uptake.

Feasibility

Feasibility concerns whether an intervention can be delivered successfully within available resources, workflows, skills, infrastructure and regulatory constraints. Feasibility is not identical to effectiveness. A highly effective intervention may be infeasible in a rural clinic if it requires specialist staff or unreliable broadband.

Feasibility evidence may include completion rates, staff time, equipment availability, workflow compatibility, recruitment performance and implementation costs. Feasibility studies are especially important before large-scale evaluation.

Acceptability

Acceptability is the extent to which stakeholders consider an intervention agreeable, appropriate or satisfactory. It may differ between patients, clinicians, managers and commissioners. A tool can be acceptable to managers because it produces useful reports but unacceptable to clinicians because it increases documentation burden.

Acceptability should be assessed through validated questionnaires where available, interviews, focus groups, complaints, opt-out rates and observed use. Stated acceptability is weaker evidence than sustained voluntary use, but actual use may also reflect coercion or lack of alternatives; both forms of evidence require interpretation.

Sustainability

Sustainability is the continuation of an intervention and its benefits after initial implementation support or research funding declines. It has at least three dimensions:

1. Continued delivery of core activities.
2. Continued organisational capacity and resources.
3. Maintenance of intended outcomes and equity.

A programme may continue nominally while quality deteriorates. Sustainability evaluation should therefore examine performance trends, staff turnover, financing, leadership, infrastructure, policy alignment and the ability to adapt without losing essential functions.

3. Implementation-science frameworks

Implementation science studies methods for promoting the uptake of evidence-informed practices in routine settings. Frameworks do not replace judgement; they structure inquiry and prevent evaluators from focusing only on individual behaviour.

The RE-AIM framework

RE-AIM evaluates five dimensions:

• Reach: who participates and whether participants represent the target population;
• Effectiveness: impact on important outcomes, including adverse and equity effects;
• Adoption: uptake by settings and staff;
• Implementation: consistency, quality, costs and adaptations;
• Maintenance: continuation over time.

RE-AIM is useful for programme planning and reporting because it combines individual and organisational levels. Its limitation is that the dimensions may be difficult to measure consistently and do not automatically explain why implementation succeeded or failed.

The Consolidated Framework for Implementation Research

The Consolidated Framework for Implementation Research (CFIR) organises determinants into domains such as intervention characteristics, outer setting, inner setting, characteristics of individuals and implementation processes. Examples include perceived relative advantage, compatibility with existing workflows, leadership engagement, available resources, staff knowledge and the quality of planning.

CFIR helps evaluators ask why outcomes differ between sites. However, it is a determinant framework rather than a complete causal model. It identifies potentially influential conditions but does not by itself establish which condition caused an outcome.

The Normalization Process Theory perspective

Normalization Process Theory focuses on how new practices become routinely embedded. It examines whether staff understand the intervention, see its value, participate in its operation, and appraise its effects. This is particularly relevant for complex interventions requiring repeated coordination across professional boundaries.

Choosing and combining frameworks

A practical evaluation may use CFIR to identify determinants, RE-AIM to define outcomes, and Normalization Process Theory to examine routinisation. Framework selection should be driven by the decision problem, not by a desire to mention multiple theories. The evaluator should specify constructs before data collection, operationalise them in measurable indicators, and retain space for unexpected contextual explanations.

4. Measuring implementation: a mixed-method design

No single data source adequately measures implementation. A robust design triangulates at least three types of evidence:

Quantitative indicators should have explicit numerators, denominators, time periods and data owners. Qualitative data should be sampled purposively to include high- and low-performing sites, different professional groups and underserved populations.

A process-evaluation protocol should specify:

• the intervention theory and core components;
• implementation outcomes and operational definitions;
• data sources and collection frequency;
• fidelity and adaptation rules;
• subgroup and equity analyses;
• procedures for handling missing data;
• confidentiality and governance arrangements; and
• how findings will be returned to teams without creating blame.

Where implementation is compared across sites, evaluators should account for clustering and baseline differences. Simple percentages can be misleading when one site contributes many more patients than another or when case mix changes over time. Statistical analysis may include multilevel models, interrupted time-series analysis or controlled before-and-after designs, but the method must match the data structure and assumptions.

5. Distinguishing implementation failure from intervention failure

This distinction is central to responsible evaluation. Suppose a new discharge-coordination intervention does not reduce readmissions. At least four explanations are possible:

1. The intervention was not delivered adequately.
2. It was delivered adequately but the causal theory was wrong.
3. It worked for some groups but not others.
4. External conditions overwhelmed its effect.

A practical diagnostic matrix is useful:

This matrix is not a substitute for causal analysis. Low fidelity may itself result from an intervention that is too complex, costly or incompatible with practice. Thus implementation failure and intervention failure can be interdependent.

Evaluators should distinguish failure to implement from implementation of a failed intervention. The first requires redesign, training, resourcing or contextual change. The second requires reconsideration of the intervention’s mechanism, target population or components. Prematurely abandoning an intervention with poor implementation wastes potentially useful innovation; indefinitely improving implementation of an ineffective intervention wastes resources and may expose patients to harm.

6. Improvement evaluation: PDSA and Lean

PDSA cycles

Plan–Do–Study–Act is an iterative improvement method. In the Plan phase, a team identifies a problem, states a change hypothesis, predicts results and defines measures. In Do, the change is tested on a small scale and observations are recorded. In Study, results are compared with predictions and unintended effects are examined. In Act, the team adopts, adapts or abandons the change and plans the next cycle.

PDSA is strongest when cycles are sequential, documented and based on explicit predictions. It is weakened when teams implement a broad intervention once, measure only final outcomes, and label the activity a PDSA cycle.

Measures normally include:

• Outcome measures: whether the system’s result improved;
• Process measures: whether key actions occurred; and
• Balancing measures: whether improvement caused harm or burden elsewhere.

For example, reducing emergency-department waiting time may require a process measure for rapid triage and a balancing measure for incomplete assessment or staff overtime.

Lean thinking

Lean is a management philosophy concerned with delivering value from the patient’s perspective while reducing waste. Common forms of waste include defects, overproduction, waiting, unused talent, transport, inventory, motion and unnecessary processing. Lean tools include value-stream mapping, standard work, visual management, root-cause analysis and error-proofing.

Lean should not be reduced to cost cutting or staff intensification. In health care, “value” includes clinical benefit, safety, experience, equity and appropriate resource use. Removing an apparent non-value-adding step may create risk if that step provides a safety check.

Combining implementation science with improvement methods

Implementation science explains determinants and outcomes across contexts; PDSA supports local learning and adaptation; Lean analyses flow and waste. A strong programme can use implementation frameworks to identify barriers, Lean to redesign the process, and PDSA to test changes incrementally. The evaluator should still preserve a clear distinction between planned intervention components and local adaptations.

7. Worked application: improving medication reconciliation

Consider an illustrative case in which a 350-bed hospital introduces pharmacist-supported medication reconciliation at admission. The intended intervention requires four components: obtaining the patient’s medication history, verifying it with at least one reliable source, documenting discrepancies, and communicating the reconciled list to the admitting team.

During the first month, 240 eligible admissions are reviewed. The four components are completed for 156 patients. The hospital therefore reports:

The result is a patient-level complete-fidelity measure. It does not show whether one component is responsible for most failures, so component-level audit is required. Suppose reliable-source verification occurred for 210 patients, while communication to the admitting team occurred for only 170. The bottleneck is likely communication rather than history taking.

The team conducts interviews and observations. Pharmacists report that the electronic record has no prominent alert for unresolved discrepancies. Nurses report uncertainty about who owns communication during night shifts. Patients with limited English proficiency are less likely to have a complete history because interpreter access is delayed.

The team develops a PDSA cycle. The prediction is that a standard electronic handoff field and a night-shift escalation rule will increase complete reconciliation from 65% to at least 85% without increasing medication delays. The first test is limited to one ward for one week. Measures include complete reconciliation, time per admission, unresolved discrepancies at 24 hours, and staff-reported burden.

After testing, complete reconciliation rises to 84%, but time per admission increases substantially. The team then redesigns the form to remove duplicate fields and uses a trained pharmacy technician for initial information gathering. The next cycle reaches 88%, with no increase in unresolved discrepancies. However, interpreter-dependent patients remain below 70% completion. The improvement is therefore effective overall but inequitable.

The appropriate conclusion is not simply that the intervention “worked.” Implementation improved through workflow redesign, but a reach and equity problem persists. Further adaptation should address interpreter availability and monitor whether the core safety function—accurate reconciliation and communication—is preserved. Sustainability planning should include role ownership, training for new staff, electronic-record maintenance and ongoing monthly sampling after the improvement team withdraws.

8. Governance, ethics and equity

Implementation evaluation frequently uses patient records, staff performance data and digital activity logs. Governance must address lawful access, data minimisation, secure storage, role-based permissions and retention periods. Where evaluation is research rather than service improvement, appropriate ethical review and consent requirements apply according to local regulation.

Evaluation can unintentionally become punitive. Reporting a low-fidelity ward without examining staffing, workload, leadership and system design may unfairly blame individuals. Feedback should support learning and psychological safety while retaining accountability for serious neglect or unsafe practice.

Equity should be treated as an implementation outcome, not an optional subgroup analysis. Reach, dose, acceptability and outcomes should be examined by relevant characteristics such as age, disability, ethnicity, socioeconomic position, language, geography and digital access, subject to lawful and ethical data use. Digital interventions require particular attention to exclusion, algorithmic bias, accessibility and the possibility that automation shifts work to patients or lower-paid staff.

9. Contemporary developments and best practice

Contemporary implementation evaluation increasingly uses routinely generated digital data, electronic dashboards, natural-language processing and predictive analytics. These tools can identify delays, variation and drop-off in real time. They do not eliminate the need for human interpretation: a low click rate may mean rejection of a tool, workflow circumvention, poor interface design or incomplete data capture.

Artificial intelligence may support implementation by identifying patients needing follow-up, summarising barriers from free text or forecasting capacity requirements. Governance must address model validity, explainability, drift, privacy, bias and responsibility for decisions. AI should augment—not replace—clinical and managerial judgement.

Best practice includes:

1. Define the intervention and its essential functions before measuring fidelity.
2. Specify numerators, denominators, timeframes and data sources for every indicator.
3. Combine quantitative trends with qualitative explanation.
4. Measure reach and equity, not only average performance.
5. Record adaptations and distinguish beneficial adaptation from loss of core function.
6. Use balancing measures to detect unintended harm.
7. Test changes on a small scale before broad adoption, but plan for scale early.
8. Monitor sustainability after implementation support ends.
9. Interpret poor outcomes in light of implementation evidence.
10. Share findings transparently with those who deliver and receive care.

Challenges and limitations

Implementation measures are vulnerable to inconsistent definitions, incomplete records, self-report bias and changing denominators. Fidelity scores can reward superficial compliance and ignore quality. Reach can conceal inequity. Adoption may reflect formal policy rather than actual use. Sustainability is difficult to assess when follow-up is short. PDSA cycles may generate local learning without establishing whether results generalise to other settings. Lean initiatives may unintentionally prioritise throughput over relational care.

Complex interventions also interact with context. Staffing, leadership, financing, policy, professional norms and patient populations change over time. Consequently, implementation evaluation should avoid claiming universal effects from a single site or short observation period. A transparent account of uncertainty is a mark of quality, not weakness.

Chapter Summary

Process, implementation and improvement evaluation explain how health-care interventions function in real settings. Fidelity concerns conformity to intended core components; reach concerns coverage and representativeness; dose concerns the amount delivered and received; adoption concerns uptake; feasibility concerns practical deliverability; acceptability concerns stakeholder judgement; and sustainability concerns continuation of delivery, capacity and benefit.

RE-AIM, CFIR and Normalization Process Theory provide complementary ways to structure evaluation. Mixed methods are generally necessary because routine data show what happened while interviews and observation help explain why. Poor outcomes with low fidelity suggest implementation failure, whereas poor outcomes despite adequate implementation raise stronger concerns about intervention effectiveness. PDSA supports disciplined iterative testing, while Lean examines value, flow and waste. Both must be used with outcome, process and balancing measures.

Effective evaluation is equitable, ethically governed and sensitive to context. The goal is not merely to label programmes successful or unsuccessful, but to produce actionable knowledge about what should be retained, adapted, resourced, scaled or discontinued.

Key Terms

Adoption: The decision or behaviour of a person, team or organisation to begin using an intervention.

Acceptability: The perceived appropriateness, satisfaction or agreeableness of an intervention.

Balancing measure: A measure used to detect unintended consequences of an improvement.

CFIR: A framework for organising determinants of implementation across multiple domains.

Dose: The amount of intervention delivered or received.

Feasibility: The extent to which an intervention can be practically delivered in a setting.

Fidelity: The degree to which delivery matches the specified intervention design.

Implementation failure: Failure to deliver or embed an intervention adequately.

Implementation outcome: An outcome describing uptake, delivery, use or maintenance of an intervention.

Implementation science: The study of methods for integrating evidence-informed practices into routine care.

Intervention failure: Failure of an adequately implemented intervention to produce intended effects.

Lean: An improvement philosophy focused on patient value and reduction of waste.

Maintenance: Continued implementation and benefit over time.

PDSA: Plan–Do–Study–Act, an iterative method for testing and refining change.

Process evaluation: Systematic assessment of implementation, context, delivery and participant experience.

Reach: The proportion and characteristics of the eligible population exposed to an intervention.

Sustainability: The capacity to maintain intervention activities, capability and outcomes.

Triangulation: Use of multiple methods or data sources to strengthen interpretation.

Review Questions

Short Answer Questions

1. Define fidelity and identify two dimensions of fidelity.
2. How does reach differ from adoption?
3. What is the purpose of a balancing measure in PDSA work?
4. Name the five dimensions of the RE-AIM framework.
5. Why should implementation outcomes be analysed by equity-relevant subgroups?

Long Answer Questions

1. Explain how a process evaluation can distinguish implementation failure from intervention failure.
2. Compare CFIR and RE-AIM and discuss how they could be used together.
3. Describe a mixed-method evaluation plan for a new digital health intervention.
4. Explain the strengths and limitations of PDSA and Lean in health-care improvement.
5. Discuss the ethical and governance issues involved in using electronic records and AI for implementation evaluation.

Analytical / Application Questions

1. A screening programme has high adoption by hospitals but low reach among rural patients and declining completion after the first visit. Diagnose the likely implementation problems and propose measures to investigate them.
2. A clinical bundle is delivered with 95% fidelity, yet patient outcomes do not improve. What evidence would you examine before concluding that the intervention is ineffective?
3. Design two PDSA cycles to improve medication reconciliation while protecting staff workload, patient safety and equity.

Multiple Choice Questions with Answers

Question 1

Which statement best defines fidelity?

A. The proportion of eligible patients who participate

B. The extent to which delivery conforms to the intended intervention

C. The extent to which an intervention remains funded

D. The number of organisations that formally approve an intervention

Correct Answer: B. The extent to which delivery conforms to the intended intervention

Question 2

A programme is offered to 500 eligible patients and received by 350. What is its reach?

A. 35%

B. 50%

C. 70%

D. 85%

Correct Answer: C. 70%

Question 3

Which RE-AIM dimension concerns continuation after initial implementation support ends?

A. Reach

B. Adoption

C. Implementation

D. Maintenance

Correct Answer: D. Maintenance

Question 4

Which measure is most clearly a balancing measure for reducing emergency-department waiting time?

A. Median waiting time

B. Percentage receiving rapid triage

C. Rate of incomplete safety assessments

D. Number of patients arriving

Correct Answer: C. Rate of incomplete safety assessments

Question 5

A programme has low fidelity and poor outcomes. What is the most appropriate initial conclusion?

A. The intervention is proven ineffective

B. Implementation failure is plausible, so effectiveness remains uncertain

C. The programme is definitely equitable

D. The outcome must be caused by patient preference

Correct Answer: B. Implementation failure is plausible, so effectiveness remains uncertain

Question 6

Which framework organises determinants such as inner setting, outer setting and intervention characteristics?

A. CFIR

B. RE-AIM

C. PDSA

D. Lean

Correct Answer: A. CFIR

Question 7

In a PDSA cycle, what occurs during the Study phase?

A. The team first defines the problem

B. The change is tested without recording observations

C. Results are compared with predictions and unintended effects are assessed

D. The intervention is automatically scaled nationally

Correct Answer: C. Results are compared with predictions and unintended effects are assessed

Question 8

Which statement about dose is correct?

A. Dose always means the programme budget

B. Dose concerns the amount delivered or received

C. Dose is identical to adoption

D. Dose can only be measured through interviews

Correct Answer: B. Dose concerns the amount delivered or received

Question 9

A hospital formally adopts a digital tool, but clinicians rarely use it. Which distinction is most important?

A. Formal adoption may not represent actual professional use

B. Reach is automatically 100%

C. Fidelity is necessarily high

D. Sustainability has already been demonstrated

Correct Answer: A. Formal adoption may not represent actual professional use

Question 10

Which approach primarily focuses on patient value and reducing waste in workflows?

A. Lean

B. CFIR

C. Normalization Process Theory

D. Reach analysis

Correct Answer: A. Lean