Causal Inference, Experimentation and Program Evaluation

Introduction

HR analytics often begins with an association: employees who attend leadership training may receive higher performance ratings, flexible-work teams may show lower absence, or employees with supportive managers may be less likely to resign. Such findings are useful for generating hypotheses, but they do not by themselves establish that an HR intervention caused the observed outcome. The employees who attend training may already be more motivated; flexible-work teams may have more experienced managers; and supportive managers may lead units with better working conditions.

Causal inference addresses a different question from prediction. Prediction asks, “Who is likely to leave?” Causal analysis asks, “What would happen to this employee, team or organisation if a particular intervention were introduced?” This distinction is central to evidence-based HR because interventions consume money, managerial attention, employee time and organisational trust.

A rigorous evaluation therefore compares observed outcomes with a credible estimate of what would have happened without the intervention. This unobserved alternative is called the counterfactual. Randomized experiments create comparable treatment and control groups by design. Quasi-experimental methods attempt to approximate that comparison when randomization is impractical or unethical.

Learning Objectives

By the end of this chapter, learners should be able to:

1. Distinguish correlation, prediction and causation.
2. Explain counterfactuals, potential outcomes, treatment effects and causal estimands.
3. Design and interpret randomized HR experiments.
4. Select and evaluate quasi-experimental designs, including difference-in-differences.
5. Construct appropriate control groups and assess threats to validity.
6. Calculate and interpret average treatment effects and program impacts.
7. Develop an ethical and operational evaluation plan for an HR intervention.
8. Communicate causal findings without overstating the evidence.

1. From Association to Causation

Correlation is not causation

A correlation describes how two variables vary together. If employees receiving more coaching have higher productivity, coaching and productivity are positively associated. The association may reflect a causal effect, but it may also arise from:

• Confounding: a third variable affects both treatment receipt and the outcome.
• Reverse causality: the outcome, or an early indicator of it, influences treatment assignment.
• Selection bias: employees choose, or managers nominate, themselves for an intervention.
• Measurement effects: the intervention changes reporting behaviour rather than the underlying outcome.
• Chance: an apparent pattern appears in a particular sample but would not replicate.

For example, suppose trained employees have a turnover rate of 8%, while untrained employees have a rate of 14%. The raw difference is 6 percentage points. It is tempting to conclude that training reduced turnover by 6 points. However, if high-potential employees were more likely to be selected for training and were also less likely to leave, the difference combines the training effect with pre-existing differences.

The counterfactual framework

For person i, let Y_i(1) denote the outcome if the person receives the intervention and Y_i(0) the outcome if the person does not. The individual treatment effect is:

The symbols Y_i(1) and Y_i(0) are potential outcomes, and τ_i is the individual causal effect. The fundamental problem of causal inference is that, for a given person, only one potential outcome is observed. We cannot observe the same employee simultaneously receiving and not receiving training under identical conditions.

For a population of N employees, the average treatment effect (ATE) is:

where E denotes the population mean. The ATE is appropriate when the question concerns the average effect of assigning the intervention to the target population. The average treatment effect on the treated (ATT) is:

where T = 1 indicates that an employee actually received treatment. ATE and ATT are equal only under additional assumptions, such as homogeneous effects or treatment assignment that is independent of potential outcomes.

A causal estimand must be specified before analysis. Examples include:

Causal diagrams and assumptions

A directed acyclic graph can clarify assumptions. Suppose manager nomination affects participation in training, and manager nomination is also influenced by prior performance. Prior performance affects later performance. Prior performance is therefore a confounder that should be measured and addressed when randomization is absent.

Figure: A confounding structure in which prior performance affects both training participation and later performance.

A diagram is not proof of causation; it represents a substantive theory that must be assessed with institutional knowledge and data. Conditioning on a variable can also create bias if that variable is a mediator or collider. For example, controlling for post-training promotion may block part of the training effect, while controlling for a variable influenced by both training and motivation may induce a spurious association.

2. Randomized Experiments in HR

Logic and design

In a randomized controlled trial (RCT), eligible employees, teams or locations are assigned by chance to treatment or control conditions. If randomization is correctly implemented and the sample is sufficiently large, treatment assignment is independent, in expectation, of both observed and unobserved potential outcomes. The simple difference in mean outcomes estimates the intention-to-treat effect:

Here, Ȳ_T is the mean outcome for the assigned treatment group and Ȳ_C is the mean outcome for the assigned control group. The formula applies to a properly randomized comparison, with consistent outcome measurement and no differential loss to follow-up that destroys comparability.

For a binary outcome such as turnover, the difference is measured in percentage points. If turnover is 10% in treatment and 14% in control, ITT = 0.10 − 0.14 = −0.04, or a reduction of 4 percentage points. A relative reduction is a different quantity:

where p_T and p_C are treatment and control proportions. In this example the relative reduction is 0.04 ÷ 0.14 ≈ 28.6%. Both measures should be reported because percentage-point and percentage changes can lead to different managerial interpretations.

Unit of randomization

The randomization unit must match the risk of spillover. Individual randomization is efficient when employees can receive different treatments without affecting one another. Team or site randomization is preferable when employees share a manager, workflow or information environment. Cluster randomization reduces contamination but usually requires a larger sample because outcomes within a cluster are correlated.

A simplified design-effect expression is:

where m is the average number of employees per cluster and ρ is the intracluster correlation coefficient (ICC), the proportion of outcome variance attributable to between-cluster differences. The effective sample size is approximately the nominal sample size divided by DE. This formula is a planning approximation; power calculations should also account for the number of clusters, unequal cluster sizes, covariates and the chosen statistical model.

Core experimental choices

A credible HR experiment should specify:

1. Eligibility: who can participate and why.
2. Treatment: the intervention, dosage, timing and provider.
3. Control: no intervention, usual practice, wait-list or an active alternative.
4. Assignment: the randomization procedure and allocation ratio.
5. Primary outcome: the outcome chosen before examining results.
6. Follow-up period: sufficient time for effects to emerge.
7. Analysis population: usually all assigned units for an intention-to-treat analysis.
8. Implementation measures: participation, take-up, fidelity and adverse effects.

The control condition should be ethically acceptable and operationally meaningful. A no-contact control may exaggerate the effect of a program because treatment employees receive additional attention. An active control, such as existing manager support, may provide a more realistic comparison.

Compliance, contamination and attrition

Employees may not comply with assignment. Some assigned employees may not attend training, while control employees may obtain similar training elsewhere. The ITT effect estimates the effect of being offered or assigned the intervention, which is often the relevant policy effect.

A per-protocol analysis estimates effects among those who follow the protocol, but treatment adherence is generally not random. It can therefore introduce selection bias. Instrumental-variable methods may estimate a local average treatment effect for compliers, but require strong assumptions: assignment affects outcomes only through treatment, assignment has no effect on treatment in the opposite direction, and there is sufficient variation in compliance.

Attrition is another threat. If employees with poor experiences are more likely to leave the study, the observed treatment effect may be biased. Researchers should report attrition by arm, compare baseline characteristics of retained and lost units, and conduct sensitivity analyses.

3. Quasi-Experimental Designs

Randomization is not always feasible. Organisations may need to roll out a policy to all employees, legal or ethical constraints may prohibit withholding a beneficial intervention, or managers may reject random allocation. Quasi-experimental designs use institutional rules, timing, thresholds or comparison groups to approximate randomization.

Before-and-after comparisons

A pre-post comparison measures the outcome before and after an intervention:

It is easy to implement but vulnerable to history, seasonality, regression to the mean, changing workforce composition and general trends. A fall in absence after a wellness program may instead reflect a seasonal decline in illness or a change in absence recording.

Matched comparison designs

A treated group can be compared with observationally similar untreated employees using matching, propensity scores or weighting. The propensity score is:

where X is a vector of pre-treatment covariates. Matching can improve balance on observed characteristics, but it cannot correct for unmeasured confounding. Covariates must be measured before treatment; matching on post-treatment variables can block causal pathways or create bias.

Regression discontinuity

A regression-discontinuity design applies when treatment is assigned using a threshold. For example, employees with a performance score below a defined cut-off may receive intensive coaching. Units just above and below the threshold can be compared if they are similar apart from treatment status. The causal interpretation is local to the threshold and depends on the assumption that individuals cannot precisely manipulate the assignment score.

Instrumental variables

An instrument affects treatment receipt but has no direct effect on the outcome except through treatment. Examples may arise from administrative assignment rules or randomized encouragement. Instruments are difficult to defend in practice because the exclusion restriction is substantive, not testable from the outcome data alone. The resulting estimate often applies only to compliers rather than the entire workforce.

Interrupted time series

An interrupted time series examines whether the level or trend of an outcome changes after an intervention. It is stronger than a single pre-post comparison when many observations are available before and after implementation. Analysts should model seasonality, autocorrelation, concurrent interventions and anticipation effects. A comparison series unaffected by the policy strengthens the design.

4. Difference-in-Differences

Difference-in-differences (DiD) compares the change over time in a treated group with the change over time in a comparison group. Let Y denote an outcome, T indicate treatment group and Post indicate the post-intervention period. The basic DiD estimator is:

The first difference removes time-invariant differences between groups. The second difference removes common time shocks. The key identifying assumption is parallel trends: absent the intervention, treatment and comparison groups would have experienced the same average change over time.

Worked example

A company introduces predictable scheduling in five warehouses. Five similar warehouses do not receive the policy during the study period. Monthly absence rates are:

Treatment change: 9% − 12% = −3 percentage points.

Comparison change: 10% − 11% = −1 percentage point.

Therefore:

The estimated policy effect is a 2-percentage-point reduction in absence, assuming parallel trends and no differential concurrent shocks. If each warehouse employed 200 people, the estimated reduction corresponds to approximately 4 percentage points ×? No: the correct calculation uses the estimated 2-point effect: 200 × 0.02 = 4 fewer absent employees per comparable period per warehouse, assuming the rate is measured over the same period and the workforce size is stable.

Regression form

A standard two-period DiD regression is:

Here, Y_it is the outcome for unit i at time t; T_i identifies the treated group; Post_t identifies the post-period; T_i × Post_t is the interaction; and ε_it is the error term. The coefficient δ is the DiD estimate. The intercept and main effects capture baseline group differences and common time changes. Standard errors should account for serial correlation and clustering at the level of treatment assignment, especially with repeated observations.

The parallel-trends assumption cannot be proven because the untreated potential outcome after treatment is unobserved. Researchers can examine pre-intervention trends, use an event-study specification, conduct placebo tests and investigate whether composition or other policies changed differentially. Similar pre-trends support—but do not guarantee—the assumption.

5. Measuring Program Impact

A sound evaluation begins with a theory of change:

Inputs → Activities → Outputs → Short-term outcomes → Long-term outcomes

For a manager-coaching program, inputs may include coach time and software; activities include coaching sessions; outputs include completed sessions; short-term outcomes include improved feedback quality; and long-term outcomes may include retention, performance and employee well-being.

The theory of change helps distinguish:

• Implementation measures: Was the program delivered as designed?
• Process measures: Did employees participate and engage?
• Intermediate outcomes: Did expected mechanisms change?
• Final outcomes: Did organisationally important results change?
• Harms and unintended effects: Did workload, inequity or stress increase?

Effect sizes and uncertainty

An estimate should be accompanied by uncertainty. For a difference in means, a confidence interval can be expressed as:

where SE is the standard error and the critical value depends on the desired confidence level and sampling distribution. A 95% confidence interval is a repeated-sampling statement: under the model and procedure, 95% of intervals constructed in this way would contain the true parameter. It is not the probability that a particular fixed parameter lies in the interval.

Statistical significance does not equal practical importance. An effect of 0.2 percentage points may be statistically detectable in a very large workforce but operationally immaterial. Conversely, a meaningful effect may fail to reach conventional significance in a small pilot. Decision-makers should consider magnitude, uncertainty, cost, feasibility, equity and strategic relevance.

For binary outcomes, risk difference, relative risk and odds ratio answer different questions:

These measures require probabilities between 0 and 1 and a clearly defined observation period. Odds ratios can appear large when outcomes are common, so they should not automatically be described as percentage changes in risk.

Heterogeneous treatment effects

An average effect can conceal important variation. A learning intervention may benefit new managers but have no effect on experienced managers. Heterogeneity should be examined using pre-specified, theoretically justified subgroups such as tenure, role or baseline skill. Post hoc subgroup searches increase false-positive risk. Organisations should avoid using sensitive attributes to deny development opportunities without compelling legal, ethical and evidence-based justification.

6. An Evaluation Workflow for HR Professionals

A practical evaluation follows these steps:

1. Define the decision. Specify what action will follow a positive, null or adverse result.
2. State the causal question. Identify population, intervention, comparator, outcome and time horizon.
3. Build a theory of change. Identify mechanisms, implementation requirements and possible harms.
4. Select the strongest feasible design. Prefer randomization when ethical and operationally viable; otherwise justify a quasi-experimental design.
5. Choose the unit and sample. Account for clustering, minimum detectable effects, attrition and statistical power.
6. Pre-register the protocol. Record primary outcomes, analysis rules, exclusions and subgroup hypotheses before inspecting results.
7. Protect data quality. Establish definitions, data lineage, consistent measurement and secure linkage.
8. Monitor implementation. Distinguish failure of the theory from failure of delivery.
9. Estimate effects transparently. Report point estimates, uncertainty, assumptions and sensitivity analyses.
10. Assess external validity. Explain whether results generalize across locations, roles, employment arrangements and time periods.
11. Make a proportionate decision. Scale, adapt, pause or discontinue based on evidence and consequences.

Power analysis should be undertaken before recruitment. For a simple two-group comparison, the required sample depends on the baseline variance, desired minimum detectable effect, significance level, power and allocation ratio. In cluster trials, the design effect increases the required sample. Analysts should not inflate the sample after seeing interim results without a documented sequential design, because repeated unplanned testing increases false-positive risk.

7. Ethical, Legal and Governance Issues

HR experiments involve real livelihoods and power relationships. Participation should not expose employees to unreasonable risk, retaliation or loss of ordinary employment rights. Evaluation plans should address informed consent where required, voluntariness, confidentiality, data minimization, access controls and retention periods.

A control group is not automatically unethical. It may receive usual practice, a delayed intervention or an active alternative. However, withholding a proven essential benefit may be unacceptable. Researchers and HR leaders should consider distributive justice: who receives potential benefits, who bears burdens and whether selection rules disadvantage protected groups.

Algorithmic interventions require additional scrutiny. A model may assign coaching or predict attrition, but predictive accuracy does not establish that coaching causes retention. Automated recommendations can reproduce historical bias through unequal measurement, selective manager action or proxy variables. Impact evaluation should therefore examine outcomes and error patterns across relevant groups, provide human oversight and preserve avenues for explanation and appeal.

Privacy law and employment regulation vary by jurisdiction. Sensitive data should be used only for legitimate, proportionate purposes and in accordance with applicable law, organisational policy and collective agreements. De-identification reduces but does not eliminate re-identification risk, particularly in small teams.

8. Illustrative Case: Evaluating a Manager-Coaching Program

Background

A national services organisation reports high voluntary turnover among first-line supervisors. It proposes a six-month coaching program focused on feedback, workload planning and career conversations. Senior leaders want evidence before expanding the program to 1,200 supervisors.

Problem

Managers volunteer for coaching, so a simple comparison between participants and non-participants would be confounded by motivation and manager support. The organisation also worries that coaching may improve retention while increasing short-term workload.

Analysis and approach

The evaluation team defines the primary outcome as voluntary supervisor turnover within twelve months. Secondary outcomes include employee engagement, quality of documented feedback, sickness absence and coaching completion. The organisation agrees to randomize eligible supervisors within regions to immediate coaching or a wait-list control. Randomization within region improves balance where labour-market conditions differ. The analysis will follow intention-to-treat principles.

Baseline data include tenure, role, prior turnover history, team size and engagement. The team pre-specifies a subgroup analysis by baseline engagement and monitors adverse outcomes. Because supervisors within the same team may influence one another, the team checks for contamination and considers cluster-robust inference. Monthly administrative data are linked using protected identifiers, with access restricted to the evaluation team.

Suppose the twelve-month turnover rate is 11% in the coaching group and 15% in the wait-list group. The estimated ITT is:

Thus, coaching is associated with a 4-percentage-point reduction in turnover under the randomized assignment. If the 95% confidence interval is [−7, −1] percentage points, the estimate is compatible with a reduction of between 1 and 7 points under the study assumptions. The organisation should also examine cost per retained supervisor, completion rates and whether benefits are concentrated in a particular group.

Outcome and learning points

The program is scaled conditionally rather than automatically. The organisation expands coaching in regions where implementation fidelity is high, improves scheduling where completion is low and continues monitoring workload. The case demonstrates that a causal estimate is only one part of a decision: implementation, costs, equity, sustainability and potential harms also matter.

Challenges and Limitations

Causal inference in HR has several persistent limitations. First, interventions may change during implementation, making the evaluated treatment different from the eventual program. Second, employees and managers may anticipate assignment and alter behaviour before the intervention. Third, organisational systems are interconnected, so spillovers can violate the assumption that one unit’s treatment does not affect another unit’s outcome. Fourth, outcomes such as culture, belonging or leadership quality may be measured imperfectly. Fifth, short evaluation windows may miss delayed effects or long-term harms.

External validity is particularly important. A program that works in a technology firm with voluntary participation may not work in a manufacturing plant with shift work. Transporting an estimate requires comparison of populations, treatment delivery, institutional context and outcome definitions. Replication across settings is stronger evidence than a single successful pilot.

The most credible evaluation is therefore not necessarily the most statistically elaborate. It is the design whose assumptions are plausible, whose implementation is observable, whose measures are valid and whose results support a defensible decision.

Contemporary Developments and Best Practices

Modern HR analytics increasingly combines administrative records, surveys, workflow data and experimentation platforms. Digital interventions make phased rollouts and randomized encouragement designs more feasible, but they also create risks of excessive monitoring and opaque experimentation. Synthetic controls, Bayesian hierarchical models and machine-learning-assisted heterogeneity analysis can improve estimation, but they do not remove the need for a credible identification strategy.

Best practice is to:

• separate prediction from causal claims;
• define the estimand before selecting a method;
• prefer randomization where feasible;
• inspect pre-treatment balance and pre-trends;
• avoid controlling for post-treatment variables without a causal rationale;
• report absolute effects, relative effects and uncertainty;
• test implementation, mechanisms and unintended consequences;
• document missing data, attrition and deviations from protocol;
• involve employees and relevant representatives in governance; and
• communicate assumptions as clearly as results.

Chapter Summary

Causal inference asks what would happen under an intervention compared with a credible counterfactual. Correlation and prediction are not sufficient to establish causation because confounding, selection, reverse causality and measurement changes can produce misleading associations. The potential-outcomes framework defines individual and average treatment effects, while the estimand clarifies the population and intervention being evaluated.

Randomized experiments provide strong causal evidence when assignment is implemented correctly, spillovers and attrition are managed, and outcomes are measured consistently. The intention-to-treat effect captures the consequence of assignment or offer, whereas per-protocol and complier effects answer narrower questions and require stronger assumptions. When randomization is not feasible, quasi-experimental designs—including matching, regression discontinuity, instrumental variables, interrupted time series and difference-in-differences—can be useful, but each depends on explicit identifying assumptions.

Difference-in-differences estimates the difference between treated and comparison-group changes over time. Its central assumption is parallel trends in the absence of treatment. Program evaluation should connect inputs, activities, outputs, mechanisms and outcomes; report uncertainty and practical importance; investigate heterogeneous effects; and consider costs, equity, privacy, employee welfare and generalizability. Responsible HR analytics treats causal evidence as a foundation for judgement, not as a substitute for ethical and managerial accountability.

Key Terms

Counterfactual: The outcome that would have occurred for the same unit under an alternative treatment condition.

Potential outcome: An outcome defined under a particular treatment state, such as Y(1) or Y(0).

Individual treatment effect: The difference between a unit’s treated and untreated potential outcomes.

Average treatment effect (ATE): The average causal effect in a defined target population.

Average treatment effect on the treated (ATT): The average causal effect among units that received treatment.

Intention-to-treat (ITT): The effect of being assigned or offered an intervention, regardless of compliance.

Confounding: Distortion of a treatment–outcome relationship by a common cause of treatment and outcome.

Selection bias: Bias arising when treated and untreated units differ systematically in ways related to outcomes.

Randomized controlled trial: An experiment in which eligible units are assigned to conditions by chance.

Control group: A comparison group that does not receive the focal intervention or receives an alternative condition.

Spillover: An effect of one unit’s treatment on another unit’s outcome.

Contamination: Receipt of the intervention by units assigned to the control condition.

Quasi-experiment: A design that estimates causal effects without researcher-controlled random assignment.

Difference-in-differences: An estimator comparing changes over time between treated and comparison groups.

Parallel trends: The assumption that treated and comparison groups would have followed the same outcome trend without treatment.

Regression discontinuity: A design exploiting treatment assignment at a known threshold.

Instrumental variable: A variable that affects treatment but influences the outcome only through treatment, under specified assumptions.

Theory of change: A structured explanation linking program resources and activities to outcomes.

External validity: The extent to which findings generalize to other populations, settings or periods.

Implementation fidelity: The degree to which an intervention is delivered as designed.

Review Questions

Short Answer Questions

1. Why does a correlation between training participation and retention not by itself prove that training caused retention?
2. Define the counterfactual and explain why it is central to causal inference.
3. Distinguish ATE, ATT and ITT.
4. State the parallel-trends assumption in difference-in-differences analysis.
5. Identify two ethical issues that should be considered before conducting an HR experiment.

Long Answer Questions

1. Explain the potential-outcomes framework and discuss the fundamental problem of causal inference in an HR context.
2. Compare individual-level and cluster-level randomization, including the role of spillovers and intracluster correlation.
3. Explain how a difference-in-differences study should be designed and assessed.
4. Compare randomized experiments, matching, regression discontinuity and interrupted time series in terms of assumptions and practical use.
5. Develop a complete evaluation plan for a proposed employee well-being intervention.

Analytical / Application Questions

1. A training group’s turnover falls from 16% to 10%, while a comparison group’s turnover falls from 14% to 12%. Calculate the difference-in-differences estimate and explain its interpretation.
2. An organisation offers coaching to volunteers and observes higher productivity among participants. Identify at least three plausible threats to causal interpretation and propose a stronger design.
3. A randomized manager-coaching program reduces absence but only 40% of assigned managers complete it. Explain what the ITT estimate means, why a per-protocol estimate may be biased, and what additional implementation evidence should be reported.

Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes causation from correlation?

A. Causation requires that two variables have different means.

B. Causation concerns the change in an outcome under an intervention relative to a credible counterfactual.

C. Causation is established whenever the correlation is statistically significant.

D. Causation applies only to randomized survey data.

Correct Answer: B. Causation concerns the change in an outcome under an intervention relative to a credible counterfactual.

Question 2

In the potential-outcomes framework, the individual treatment effect is:

A. Y(1) + Y(0)

B. Y(1) ÷ Y(0)

C. Y(1) − Y(0)

D. Y(1) × Y(0)

Correct Answer: C. Y(1) − Y(0)

Question 3

An intervention group has a turnover rate of 10% and a control group has a rate of 14%. What is the risk difference for the intervention group?

A. −4 percentage points

B. 4 percentage points

C. 28.6 percentage points

D. 71.4 percentage points

Correct Answer: A. −4 percentage points

Question 4

What does an intention-to-treat estimate measure in a randomized experiment?

A. The effect among only those who completed every treatment activity

B. The effect of assignment or offer, regardless of compliance

C. The effect among employees who were initially most motivated

D. The effect after removing all employees with missing data

Correct Answer: B. The effect of assignment or offer, regardless of compliance

Question 5

Which condition is central to a basic difference-in-differences design?

A. Treatment and comparison groups must have identical outcomes at baseline.

B. Treatment must be assigned by a manager.

C. Absent treatment, the groups would have followed parallel trends.

D. The treatment group must be twice as large as the comparison group.

Correct Answer: C. Absent treatment, the groups would have followed parallel trends.

Question 6

Before an intervention, absence is 12% in the treatment group and 11% in the comparison group. Afterward, the rates are 9% and 10%, respectively. What is the difference-in-differences estimate?

A. −1 percentage point

B. −2 percentage points

C. 1 percentage point

D. 2 percentage points

Correct Answer: B. −2 percentage points

Question 7

Why can matching fail to eliminate confounding?

A. It can balance observed covariates but cannot generally balance unmeasured confounders.

B. It always removes treatment variation.

C. It requires random assignment by definition.

D. It can only be used with binary outcomes.

Correct Answer: A. It can balance observed covariates but cannot generally balance unmeasured confounders.

Question 8

What is a spillover in an HR experiment?

A. A missing outcome observation

B. An effect of one unit’s treatment on another unit’s outcome

C. A treatment effect that is statistically insignificant

D. A change in the treatment definition after analysis

Correct Answer: B. An effect of one unit’s treatment on another unit’s outcome

Question 9

Which measure primarily assesses whether an intervention was delivered as designed?

A. Implementation fidelity

B. External validity

C. Relative risk

D. Sampling variance

Correct Answer: A. Implementation fidelity

Question 10

Why should practical importance be considered alongside statistical significance?

A. Statistical tests cannot be used with HR data.

B. A statistically detectable effect may be too small to matter operationally, while a meaningful effect may be imprecisely estimated in a small sample.

C. Practical importance is always numerically identical to the p-value.

D. Practical importance removes the need to report uncertainty.

Correct Answer: B. A statistically detectable effect may be too small to matter operationally, while a meaningful effect may be imprecisely estimated in a small sample.