Evaluation Theory, Models and Systems Thinking

Introduction

Health-care interventions rarely operate as isolated technical projects. A screening programme depends on public awareness, provider competence, laboratory capacity, referral pathways, financing, trust and political commitment. A hospital quality-improvement initiative may produce different results across facilities because leadership, workload, professional norms and patient populations differ. Consequently, evaluation must do more than ask whether an intervention worked. It must examine how, for whom, under what conditions and with what consequences it worked.

Evaluation theory provides the concepts and assumptions used to answer these questions. Evaluation models provide structured approaches for planning inquiry, selecting evidence and making judgments. Systems thinking helps evaluators understand interdependence, feedback, adaptation, emergence and unintended effects. Together, these approaches support more credible decisions about complex health interventions.

This chapter examines theory of change, logic models, results chains, realist evaluation, utilization-focused evaluation, the CIPP model and systems-thinking approaches. The emphasis is not on treating these as competing templates. In advanced practice, they are often combined: a theory of change may explain causal assumptions; a logic model may communicate implementation; realist evaluation may investigate context and mechanisms; utilization-focused evaluation may shape the process around intended users; and systems thinking may reveal interactions that linear models conceal.

Learning objectives

By the end of this chapter, the reader should be able to:

1. Distinguish evaluation theory, evaluation models and programme theories.
2. Construct and critically assess a theory of change, logic model and results chain.
3. Apply realist evaluation concepts of context, mechanism and outcome.
4. Design an evaluation around intended use and intended users.
5. Use the CIPP model to evaluate health programmes across their life cycle.
6. Apply systems-thinking concepts to complex health interventions.
7. Integrate complementary approaches into a coherent evaluation design.
8. Identify ethical, managerial and governance implications of theory-informed evaluation.

Evaluation theory and programme theory

Evaluation is the systematic and transparent assessment of the merit, value, significance or performance of an intervention. It differs from routine monitoring, which generally tracks implementation and indicators, although monitoring data may be important evidence for evaluation.

Evaluation theory refers to ideas about what evaluation is for, how value should be judged, what counts as credible knowledge and how evaluators should relate to stakeholders. Programme theory refers more specifically to the explanation of how an intervention is expected to produce change. In practice, a programme theory may include:

• the problem definition and its causes;
• the intervention activities;
• the expected causal processes;
• the assumptions and contextual conditions required;
• the outcomes and longer-term impacts; and
• the values or standards used to judge success.

A programme theory is not merely a list of activities. It is a reasoned account of why those activities should produce particular results. For example, distributing blood-pressure monitors does not automatically reduce cardiovascular risk. The intended causal explanation may be: patients receive monitors; patients understand how to use them; measurements are taken regularly; clinicians review readings; treatment is adjusted when necessary; adherence improves; and blood pressure is controlled. Each link contains assumptions that can fail.

Evaluation models should be treated as decision aids rather than rigid recipes. A model helps organize questions, evidence and judgments, but it does not remove the need for professional reasoning. The appropriate model depends on the intervention’s maturity, complexity, purpose, stakeholders, available data and ethical context.

Theory of change

A theory of change is a systematic explanation of how and why a desired change is expected to occur in a particular context. It normally connects inputs, activities, outputs, outcomes and impact, while making assumptions and external influences explicit.

A strong theory of change distinguishes between:

• Activities: what the programme does.
• Outputs: immediate products or services delivered.
• Outcomes: changes in behaviour, capability, access, quality or health status.
• Impact: broader and usually longer-term changes in population health, equity or system performance.
• Assumptions: conditions believed to be true for the causal pathway to operate.
• External factors: influences outside programme control, such as economic change, epidemics, legislation or competing services.

The theory should be developed with relevant stakeholders but should not simply reproduce stakeholder opinions. It requires evidence, challenge and explicit discussion of uncertainty. A useful theory of change includes causal links such as “if–then” statements. For example: if community health workers receive training, supervision and reliable supplies, then they are more likely to provide accurate counselling and referrals; if referral facilities are accessible and responsive, then referred patients are more likely to complete care; if treatment is completed, health outcomes may improve.

The theory of change is particularly useful during programme design, evaluation planning and interpretation of mixed results. If outputs are achieved but outcomes are not, the evaluator can examine whether the mechanism failed, whether assumptions were incorrect or whether external factors intervened.

Constructing a theory of change

A practical process is:

1. Define the problem using epidemiological, service-use and equity evidence.
2. Specify the desired outcomes and the population that should benefit.
3. Work backwards from outcomes to identify necessary preconditions.
4. Map activities and outputs that can plausibly produce those preconditions.
5. State assumptions, contextual conditions and possible negative effects.
6. Identify indicators and evidence sources for each important link.
7. Test the theory against research, stakeholder experience and implementation realities.
8. Revise it as evidence and context change.

The evaluator should distinguish causal claims from descriptive statements. “Ten workshops were delivered” is descriptive. “The workshops improved clinical practice because they changed knowledge, confidence and peer norms” is a causal proposition that requires evidence.

Logic models and results chains

A logic model is a structured representation of the relationship between resources, activities, outputs and results. It is often presented as a table or flow diagram. A results chain is a closely related, usually more linear representation of the progression from inputs to activities, outputs, outcomes and impact.

A results chain is useful for concise communication. A logic model is often more detailed and may include assumptions, risks, indicators and responsible actors. Neither should be interpreted as proof that change is linear or automatic.

Logic models are valuable for clarifying programme boundaries and aligning indicators with intended results. They also expose weak designs. A programme may have many outputs but no plausible pathway to health outcomes. Conversely, an outcome may be desirable but too distant or confounded to attribute to the programme.

A common error is to confuse outputs with outcomes. The number of patients counselled is an output; improved treatment adherence is an outcome. The number of clinics receiving software is an output; improved continuity of care may be an outcome. Another error is to assume that an impact indicator can be interpreted without measuring intermediate steps. If mortality does not change, the programme may still have improved access or quality, or the follow-up period may be too short.

Limits of linear models

Linear models can understate feedback and adaptation. For instance, increased detection of disease may initially raise recorded prevalence, even while the programme is improving case finding. A new referral pathway may increase demand at hospitals, causing congestion that later reduces patient satisfaction. These effects are not necessarily programme failure; they may be transitional or emergent consequences requiring management.

A logic model should therefore be accompanied by narrative explanation, risk analysis and attention to unintended outcomes. It is a communication device, not a substitute for causal investigation.

Realist evaluation

Realist evaluation is concerned with explaining what works, for whom, in what circumstances and how. It is especially relevant when interventions are complex and outcomes vary across settings. Its central explanatory unit is commonly expressed as context–mechanism–outcome configuration, or C–M–O.

• Context includes organizational, social, economic, cultural, political and historical conditions.
• Mechanism refers to the responses, reasoning, opportunities or resources through which an intervention generates change. A mechanism is not simply the programme activity itself. Training is an activity; increased professional confidence or altered expectations may be mechanisms.
• Outcome is the observed result, intended or unintended.

For example, an electronic prescribing system may improve medication safety in a hospital with reliable internet, supportive leadership and protected training time. The mechanism may involve clinicians trusting alerts and changing prescribing habits. In another hospital, frequent interruptions and alert fatigue may produce little improvement or even workarounds. The same intervention can therefore produce different outcomes without either site being anomalous.

Realist evaluation commonly proceeds through an iterative cycle:

1. Develop initial programme theories from literature, policy and stakeholder knowledge.
2. Collect qualitative and quantitative evidence about contexts, mechanisms and outcomes.
3. Compare cases or subgroups to identify patterns of variation.
4. Refine the programme theories.
5. Test the refined explanations in additional settings.

Realist methods may include interviews, observations, document analysis, administrative data, surveys and comparative case studies. The aim is not merely to calculate an average effect but to explain patterned differences. This makes realist evaluation valuable for scale-up, adaptation and implementation decisions.

A limitation is that mechanisms are often difficult to observe directly and may be interpreted differently by different stakeholders. Strong realist work therefore makes the reasoning explicit, considers rival explanations and uses evidence from multiple sources.

Utilization-focused evaluation

Utilization-focused evaluation, associated particularly with Michael Quinn Patton, begins with the principle that an evaluation should be judged by its usefulness to intended users. The evaluator works with primary intended users to clarify the decisions they face, the information they need, the timing of the evaluation and the form in which findings can be acted upon.

Potential users may include a health ministry, hospital executives, programme managers, clinicians, community representatives, commissioners or patient organizations. “Stakeholder” is a broad category; an intended user is a person or group expected to interpret and use findings for a real decision.

A utilization-focused process may ask:

• What decision will be made?
• Who has authority or influence over that decision?
• What evidence would change the decision?
• When is the evidence needed?
• What degree of certainty is feasible and sufficient?
• How will findings be communicated and followed up?

This approach does not mean producing whatever a powerful stakeholder requests. Evaluators retain obligations to independence, methodological integrity, transparency and affected communities. Utilization is improved when users participate meaningfully in framing questions, interpreting findings and planning response, while the evaluator challenges unsupported assumptions.

Its strengths include relevance, ownership and a greater likelihood that findings will influence practice. Risks include political pressure, instrumental use of evaluation to justify a predetermined decision and neglect of less powerful groups. These risks can be addressed through explicit terms of reference, transparent methods, documentation of dissenting views and attention to equity.

The CIPP model

The CIPP model organizes evaluation around Context, Input, Process and Product. It is associated with Daniel Stufflebeam and is designed to support improvement and accountability across a programme’s life cycle.

Context evaluation supports needs assessment and priority setting. Input evaluation compares strategies, resources and implementation plans. Process evaluation monitors implementation and identifies deviations. Product evaluation assesses outcomes, impacts, unintended consequences and sustainability.

CIPP is useful for managerial decision-making because it supports decisions about planning, structuring, implementing and continuing or revising a programme. It can be combined with a theory of change: the theory explains causal expectations, while CIPP ensures that needs, resources, implementation and results are all examined.

A limitation is that the model can become an administrative checklist if evaluators collect descriptive information without making judgments. Evaluation criteria should therefore be established in advance where possible, and findings should be interpreted in relation to standards, feasibility, equity and stakeholder values.

Systems thinking in health-care evaluation

Systems thinking views a health intervention as part of a dynamic network rather than as an isolated treatment. A health system includes patients, families, professionals, organizations, technologies, financing arrangements, regulations, communities and political institutions. These elements interact across levels and over time.

Important systems concepts include:

• Interdependence: a change in one component affects others.
• Feedback: consequences influence future behaviour or conditions.
• Non-linearity: small changes can have large effects, while large investments may produce little change.
• Emergence: system-level patterns arise from interactions and cannot be understood by examining one component alone.
• Adaptation: actors modify behaviour in response to incentives, constraints and results.
• Path dependence: previous decisions and institutional histories shape current options.
• Boundaries: every evaluation defines what is included and excluded, but boundaries are analytical choices rather than natural facts.
• Equifinality: different pathways may produce similar outcomes.
• Unintended consequences: interventions can generate effects outside their stated objectives.

Systems mapping may use causal-loop diagrams, stock-and-flow models, network analysis, process maps or rich pictures. A causal-loop diagram, for example, may show how appointment availability affects waiting time, patient satisfaction, missed appointments, demand and staff workload. The purpose is not visual complexity for its own sake; it is to identify feedback relationships and leverage points.

Figure: An illustrative outpatient-access feedback structure. The arrows represent hypothesized relationships, not established causal effects.

Systems thinking changes evaluation questions. Instead of asking only “Did the intervention increase attendance?”, an evaluator may ask whether increased attendance changed workload, waiting time, staff retention, equity and future demand. This broadens outcome assessment and helps identify unintended effects.

Integrating the approaches in an evaluation design

The approaches can be combined in a disciplined sequence:

1. Define purpose and users. Utilization-focused evaluation clarifies decisions, users and timing.
2. Understand the problem and system. Context analysis and systems mapping identify actors, dependencies and boundaries.
3. Develop the programme theory. A theory of change states causal assumptions and desired results.
4. Structure implementation and indicators. A logic model or results chain specifies activities, outputs and outcomes.
5. Investigate variation. Realist evaluation examines how context and mechanisms shape results across groups or sites.
6. Review the programme life cycle. CIPP ensures attention to context, inputs, process and products.
7. Adapt and communicate. Findings are interpreted with users, documented transparently and used to revise the intervention and its theory.

This integration should not become a burdensome accumulation of frameworks. The evaluator should select only concepts that answer the evaluation purpose. A small community intervention may need a simple results chain and process evaluation. A national digital-health reform may require systems mapping, realist comparison, equity analysis and multiple feedback cycles.

Applied case: an illustrative hypertension-control programme

Background

A regional health authority introduces a hypertension-control programme in 20 primary-care facilities. The intervention includes standardized screening, electronic reminders, nurse-led counselling, medication protocols and referral of high-risk patients. After 18 months, screening rates rise substantially, but blood-pressure control improves in only some facilities.

Problem

A conventional output report might conclude that implementation was successful because screening and reminder targets were met. However, the variation in control rates suggests that the causal pathway differs across facilities.

Analysis through multiple models

The theory of change proposes that screening identifies untreated patients; reminders and counselling improve follow-up; protocols support appropriate prescribing; and regular review improves control. Its assumptions include medicine availability, continuity of care, patient trust, clinician adherence and affordable transport.

The logic model distinguishes training and software installation as inputs or activities, completed screenings and follow-up visits as outputs, medication initiation and adherence as intermediate outcomes, and controlled blood pressure as a longer-term outcome.

CIPP analysis finds that the context differs: high-performing facilities serve communities with relatively stable transport and have supportive clinical leaders. Input analysis finds that some facilities lack calibrated equipment and have insufficient nursing time. Process data show that reminders are generated but not consistently acted upon. Product data show improved screening overall, mixed follow-up and unequal control by facility and socioeconomic group.

A realist interpretation suggests the electronic reminder works when clinicians regard it as actionable, workloads permit review and medicines are available. In facilities with frequent stock-outs, the reminder may create frustration rather than effective treatment. The mechanism is therefore not “the software”; it is the combination of a prompt, clinical confidence, available treatment and an organizational routine for response.

A systems perspective identifies a possible balancing effect: improved screening increases the number of patients requiring follow-up, which increases workload and waiting time. If staffing does not expand, the increased demand may reduce the quality of counselling and undermine control.

Approach and solution

The evaluation team works with primary-care managers, clinicians, patient representatives and the regional authority to identify decisions for the next funding cycle. It combines routine data, patient interviews, facility observations, medicine-stock records and comparative case analysis. Rather than recommending identical expansion everywhere, it recommends:

• repairing and calibrating equipment;
• linking reminders to clear responsibility and protected review time;
• strengthening medicine procurement and stock monitoring;
• adapting counselling for language, literacy and transport constraints;
• using nurse-led follow-up where clinically appropriate; and
• monitoring workload, waiting time and equity alongside blood-pressure control.

Outcome and learning points

The evaluation does not treat average improvement as sufficient evidence of success. It identifies conditions under which the programme is effective and shows how implementation can create pressure elsewhere in the system. The principal learning is that intervention components, mechanisms, context and system responses must be evaluated together. A technically sound programme can underperform when organizational conditions do not support its causal pathway.

Critical issues, ethics and governance

Evaluation can affect funding, employment, professional reputation and access to care. Ethical practice therefore requires respect for persons, minimization of harm, confidentiality, fair inclusion and responsible communication. Evaluation teams should obtain appropriate ethical and governance approvals, particularly when collecting identifiable patient information or conducting research-like activities.

Important governance practices include:

• clearly defining the evaluation’s purpose and authority;
• separating service improvement from disciplinary investigation where possible;
• protecting personal data through data minimization, access controls and secure storage;
• explaining consent, confidentiality and limits to confidentiality;
• including groups likely to be disadvantaged or overlooked;
• reporting uncertainty, missing data and contradictory findings;
• declaring conflicts of interest; and
• ensuring that findings are not used to stigmatize communities or penalize facilities without considering context.

Equity should be incorporated into the programme theory and analysis, not added only at the reporting stage. Outcomes should be examined across relevant dimensions such as socioeconomic position, sex, age, disability, geography, ethnicity, language and digital access, subject to lawful and ethical data collection.

Technology, artificial intelligence and contemporary developments

Digital platforms can support real-time dashboards, process tracing, patient-reported outcomes and adaptive evaluation. Artificial intelligence may assist with record classification, anomaly detection or synthesis of large qualitative datasets. However, these tools do not replace causal reasoning or stakeholder judgment. Risks include biased training data, opaque algorithms, automation bias, privacy breaches and the use of proxy measures that misrepresent patient benefit.

When AI is used in evaluation, teams should document the model’s purpose, data provenance, performance limitations, human oversight, security controls and procedures for correcting errors. Algorithmic performance should be examined across relevant population groups. A technically accurate prediction may still be ethically inappropriate if it reinforces unequal access or shifts accountability away from decision-makers.

Contemporary evaluation increasingly emphasizes adaptive learning, developmental evaluation, participatory approaches, implementation science, equity-oriented evaluation and mixed-methods evidence. These developments reflect recognition that health interventions evolve and that knowledge must be useful in changing environments. Nevertheless, methodological flexibility should not mean weak standards. Evaluation designs should remain explicit about questions, assumptions, evidence quality and limitations.

Best-practice principles

A high-quality evaluation of a complex health intervention should:

1. Begin with a clearly defined decision or learning purpose.
2. Identify intended users and affected populations.
3. Make the programme theory and assumptions explicit.
4. Distinguish activities, outputs, outcomes and impacts.
5. Examine context, implementation fidelity, adaptation and mechanisms.
6. Combine quantitative and qualitative evidence when each answers different questions.
7. Assess unintended effects, equity and sustainability.
8. Use systems mapping where interdependence and feedback are important.
9. Treat models as tools for reasoning rather than proof of causation.
10. Communicate uncertainty and make recommendations actionable.
11. Protect participants and govern data responsibly.
12. Revise the evaluation and programme theory as learning accumulates.

Chapter Summary

Evaluation theory provides the assumptions and values that shape questions, evidence and judgments. Programme theory explains how an intervention is expected to produce change. A theory of change makes causal pathways, assumptions and external influences explicit. Logic models and results chains organize inputs, activities, outputs, outcomes and impacts, but their apparent linearity must not conceal feedback or adaptation.

Realist evaluation explains variation through context–mechanism–outcome configurations. Utilization-focused evaluation aligns evaluation with intended users and decisions while preserving methodological integrity. The CIPP model examines context, input, process and product across the programme life cycle. Systems thinking expands analysis to interdependence, feedback, non-linearity, emergence and unintended consequences.

For complex health interventions, the most useful design is often integrative. It begins with purpose and users, develops a defensible programme theory, structures indicators, investigates variation, examines system effects and supports transparent adaptation. Ethical governance, equity, privacy and responsible use of technology are essential components of credible evaluation.

Key Terms

Evaluation theory: Ideas about the purpose, values, knowledge and conduct of evaluation.

Programme theory: An explanation of how and why an intervention is expected to produce change.

Theory of change: A detailed causal account linking intervention actions to desired results and identifying assumptions.

Logic model: A structured representation of resources, activities, outputs and results.

Results chain: A usually linear sequence from inputs through activities and outputs to outcomes and impact.

Input: A resource required for programme implementation.

Activity: An action undertaken by a programme.

Output: A direct product or service delivered by an intervention.

Outcome: A change resulting from intervention activities.

Impact: A broad, usually longer-term change in health or system conditions.

Assumption: A condition believed necessary for a causal link to operate.

Context: Conditions that shape how an intervention operates and produces results.

Mechanism: The reasoning, response, resource or process through which an intervention generates change.

Realist evaluation: An approach asking what works, for whom, in what circumstances and how.

Utilization-focused evaluation: Evaluation designed around the needs and decisions of intended users.

CIPP model: A framework assessing context, input, process and product.

Systems thinking: Analysis of relationships, feedback and interdependence within a wider system.

Emergence: System-level behaviour arising from interactions among components.

Unintended consequence: An effect not included among the intervention’s stated objectives.

Review Questions

Short Answer Questions

1. Distinguish a programme theory from a logic model.
2. What is the difference between an output and an outcome?
3. Define context, mechanism and outcome in realist evaluation.
4. What is the central principle of utilization-focused evaluation?
5. Name and briefly explain the four dimensions of the CIPP model.

Long Answer Questions

1. Explain how to construct and critically assess a theory of change for a complex health intervention.
2. Compare logic models, results chains and systems-thinking approaches, including their strengths and limitations.
3. Discuss how realist evaluation can explain why the same intervention produces different results in different facilities.
4. Explain how utilization-focused evaluation can improve the relevance and use of evaluation findings while preserving evaluator independence.
5. Develop an integrated evaluation design combining theory of change, CIPP and systems thinking for a national health programme.

Analytical / Application Questions

1. A digital appointment system increases attendance but also produces longer waiting times and staff dissatisfaction. Use systems-thinking concepts to propose evaluation questions and potential explanations.
2. Two primary-care clinics receive the same hypertension intervention; one improves control rates and the other does not. Develop two plausible context–mechanism–outcome configurations and identify evidence needed to test them.
3. A health ministry asks for an evaluation report three weeks before a funding decision, but key outcome data are incomplete. Using utilization-focused principles and ethical standards, propose a defensible evaluation strategy and explain its limitations.

Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes an outcome from an output?

A. An outcome is a resource, whereas an output is a budget.

B. An outcome is a change resulting from an intervention, whereas an output is a direct product or service delivered.

C. An outcome is always short term, whereas an output is always long term.

D. An outcome describes an activity, whereas an output describes an assumption.

Correct Answer: B. An outcome is a change resulting from an intervention, whereas an output is a direct product or service delivered.

Question 2

In a theory of change, an assumption is best understood as:

A. A condition believed necessary for a causal link to operate

B. A final population-health impact

C. A numerical target for service delivery

D. A list of programme staff

Correct Answer: A. A condition believed necessary for a causal link to operate

Question 3

Realist evaluation primarily asks:

A. Whether a programme used its entire budget

B. Whether all sites implemented an identical protocol

C. What works, for whom, in what circumstances and how

D. Whether outputs are always greater than impacts

Correct Answer: C. What works, for whom, in what circumstances and how

Question 4

Which CIPP dimension examines whether implementation is occurring as intended?

A. Context

B. Input

C. Process

D. Product

Correct Answer: C. Process

Question 5

Utilization-focused evaluation begins by emphasizing:

A. The evaluator’s preferred statistical method

B. The decisions and information needs of intended users

C. The elimination of stakeholder participation

D. The use of only national mortality data

Correct Answer: B. The decisions and information needs of intended users

Question 6

A hospital introduces an electronic reminder system. The reminder changes prescribing only when clinicians trust the alerts and have time to review them. In realist terms, clinician trust and protected review time are examples of:

A. Mechanisms and contextual conditions

B. Outputs and impacts

C. Products and budgets

D. Sampling errors

Correct Answer: A. Mechanisms and contextual conditions

Question 7

Which systems-thinking concept describes patterns arising from interactions among system components?

A. Emergence

B. Standardization

C. Randomization

D. Attrition

Correct Answer: A. Emergence

Question 8

Why can a logic model be misleading if interpreted too literally?

A. It cannot list any programme activities.

B. It may conceal feedback, adaptation and unintended consequences behind a linear sequence.

C. It is only applicable to financial audits.

D. It prevents the measurement of outputs.

Correct Answer: B. It may conceal feedback, adaptation and unintended consequences behind a linear sequence.

Question 9

A programme achieves high screening coverage but no improvement in treatment control. Which evaluation response is most appropriate?

A. Conclude immediately that the programme had no effect.

B. Examine intermediate links, assumptions, implementation and contextual barriers.

C. Ignore treatment control because screening is an output.

D. Replace all outcome measures with activity counts.

Correct Answer: B. Examine intermediate links, assumptions, implementation and contextual barriers.

Question 10

Which practice best supports ethical evaluation governance?

A. Reporting only favourable findings to protect programme reputation

B. Using identifiable patient data without access controls

C. Declaring conflicts of interest and reporting uncertainty and limitations

D. Excluding disadvantaged populations to simplify analysis

Correct Answer: C. Declaring conflicts of interest and reporting uncertainty and limitations