9. Causal Inference and Experimental Designs

Introduction and Context

Business research frequently asks causal questions: Does a price reduction increase demand? Does remote work improve productivity? Does a leadership-development programme reduce employee turnover? Does a new compliance system reduce operational risk? These questions cannot be answered reliably by observing associations alone. Causal inference concerns the identification and estimation of what would happen to an outcome if a unit, organization, market, or policy were exposed to a different condition.

The central difficulty is counterfactual. For a firm receiving an intervention, researchers can observe the outcome under treatment, but not the outcome that the same firm would have experienced simultaneously without treatment. Causal research therefore constructs credible comparisons that approximate this unobserved counterfactual. Randomised experiments achieve this through assignment by chance. Quasi-experimental designs use institutional rules, timing, thresholds, or other sources of plausibly exogenous variation. Observational methods such as matching and instrumental variables attempt to reproduce, under assumptions, the balance or variation that an experiment would provide.

For doctoral researchers, causal inference is not merely a collection of statistical techniques. It is a logic of design, measurement, identification, estimation, and interpretation. Statistical sophistication cannot compensate for a weak research design or an implausible causal assumption. This chapter develops that logic and considers its implications for evidence-based managerial decisions.

Learning Objectives

By the end of this chapter, readers should be able to:

1. Distinguish causal effects from associations and formulate counterfactual research questions.

2. Explain the logic of randomised, field, natural, and quasi-experimental designs.

3. Evaluate difference-in-differences, regression discontinuity, matching, and instrumental-variable strategies.

4. Diagnose threats to internal validity, external validity, construct validity, and statistical conclusion validity.

5. Interpret treatment effects, assumptions, uncertainty, and heterogeneous effects for managerial decisions.

6. Design ethically defensible and practically useful causal studies.

From Association to Causal Effect

Let Y(1) denote the outcome a unit would experience under treatment and Y(0) the outcome under control. The individual treatment effect is Y(1) − Y(0). Because only one potential outcome is observed for each unit, the individual effect is generally unobservable. Researchers instead estimate an average treatment effect (ATE), commonly expressed as E[Y(1) − Y(0)]. For a treated subgroup, the average treatment effect on the treated (ATT) may be more relevant.

A difference in observed means, E[Y|D=1] − E[Y|D=0], equals a causal effect only when treatment assignment makes the treated and untreated groups comparable with respect to potential outcomes. This condition is often called exchangeability, conditional independence, or absence of confounding. In a randomised experiment, assignment is independent of potential outcomes by design. In observational research, comparability must be argued through assumptions and evidence.

Causal claims also require clarity about the treatment, population, time period, outcome, and estimand. “The effect of training” is incomplete unless the researcher specifies which training, delivered to whom, over what period, compared with what alternative, and measured through which outcome. A managerial intervention may influence sales directly, alter employee behaviour, or operate through customer perceptions. A sound design distinguishes the treatment from mechanisms and downstream consequences.

Important Terminology

A treatment is the intervention or exposure whose effect is being studied; a control or comparison condition represents the relevant alternative. A unit may be an employee, customer, store, firm, transaction, or region. A confounder affects both treatment assignment and the outcome. A mediator lies on the causal pathway between treatment and outcome, whereas a moderator changes the size or direction of an effect. Internal validity concerns whether the estimated effect is causal for the studied units and setting. External validity concerns transferability to other populations, contexts, or times. SUTVA, the stable unit treatment value assumption, generally requires well-defined treatment conditions and no interference between units. Spillover occurs when one unit’s treatment affects another unit’s outcome.

Randomised Experiments: The Benchmark Design

In a completely randomised experiment, eligible units are assigned by a random mechanism to treatment or control. Randomisation balances observed and unobserved characteristics in expectation, making the assignment independent of potential outcomes. When compliance is high and outcomes are measured consistently, the difference in mean outcomes estimates the ATE.

Randomisation does not eliminate every problem. Small samples may remain imbalanced by chance, attrition may differ between groups, and participants may alter behaviour because they know they are being studied. Researchers therefore use blocking or stratification when important characteristics—such as firm size or prior performance—should be balanced deliberately. Cluster randomisation assigns groups such as branches or schools rather than individuals when contamination is likely. Because observations within clusters are correlated, analysis must account for the design, often through cluster-robust standard errors or multilevel models.

Power analysis should precede data collection. It considers the expected effect size, outcome variability, significance level, desired power, and number of independent units. In business settings, the number of customers or transactions may appear large while the number of independently assigned stores or firms is small. Treating correlated observations as independent produces overstated precision.

An experiment may estimate an intention-to-treat (ITT) effect—the effect of being assigned to treatment—even when participants do not comply. The treatment-on-the-treated effect concerns the effect among those who actually receive treatment and requires additional assumptions. ITT is often the most policy-relevant estimate because it reflects implementation realities.

Laboratory, Field, and Natural Experiments

Laboratory experiments provide control over stimuli, timing, and measurement. They are useful for testing mechanisms, such as whether perceived fairness mediates the response to an algorithmic price. Their limitations include artificiality and limited behavioural stakes.

Field experiments take place in real organizational or market settings. A retailer might randomly vary promotional messages across stores, or a platform might randomise the order in which users see interface features. Field experiments often have stronger practical relevance, but implementation can create noncompliance, spillovers, selective attrition, and measurement problems.

A natural experiment occurs when an external event or institutional rule allocates exposure in a manner that approximates random assignment, although researchers did not control the assignment. Examples may include a policy introduced at different times across jurisdictions or an eligibility rule based on an administrative threshold. “Natural” does not mean automatically valid. The researcher must establish why exposure is plausibly unrelated to potential outcomes and examine competing explanations.

Quasi-Experimental Designs

Quasi-experiments are especially valuable when randomisation is infeasible, unethical, or operationally unacceptable. Their credibility depends on the institutional setting and the identifying assumption rather than on the label attached to the method.

Difference-in-Differences

Difference-in-differences (DiD) compares changes over time in a treated group with changes in a comparison group that did not receive the intervention. With two groups and two periods, the estimator is:

(ȲT,after − ȲT,before) − (ȲC,after − ȲC,before).

The key assumption is parallel trends: absent treatment, the treated and comparison groups would have experienced similar outcome changes. This assumption cannot be proven from post-treatment data, but researchers can examine pre-treatment trends, use institutional knowledge, conduct placebo tests, and select a credible comparison group. A regression implementation may include group and time fixed effects, treatment indicators, covariates, and clustered standard errors.

Modern applications require caution when treatment begins at different times for different units. Conventional two-way fixed-effects estimators can produce misleading weighted averages when treatment effects vary across groups or time. Event-study specifications and estimators designed for staggered adoption can reveal dynamic effects, pre-trends, anticipation, and treatment-effect heterogeneity.

Regression Discontinuity

Regression discontinuity (RD) exploits a rule assigning treatment when a continuous running variable crosses a known cutoff. For example, firms with a risk score above a threshold may receive mandatory review. Units just below and just above the cutoff are assumed to be similar except for treatment, yielding a local causal effect.

A sharp RD assigns treatment deterministically at the threshold; a fuzzy RD permits imperfect compliance and uses the discontinuity as an instrument for actual treatment. Credible RD analysis requires no precise manipulation of the running variable around the cutoff, continuity of potential outcomes at the threshold, appropriate bandwidth selection, and careful functional-form choices. Density tests, covariate continuity checks, graphical inspection, and robustness across bandwidths strengthen the analysis. The estimate is local: it applies primarily to units near the cutoff, not necessarily to all firms or customers.

Matching and Weighting

Matching attempts to construct a comparison group resembling treated units on observed characteristics. Propensity-score matching estimates the probability of treatment conditional on covariates and pairs or weights units with similar scores. Mahalanobis matching, coarsened exact matching, and inverse-probability weighting are related approaches.

Matching can improve balance on measured variables but cannot remove bias from unobserved confounders. Researchers should assess common support, inspect covariate balance, avoid conditioning on post-treatment variables, and conduct sensitivity analyses. A propensity score is not a substitute for substantive theory: the covariate set should be guided by a causal model, often represented through a directed acyclic graph. Matching is most persuasive when treatment selection is plausibly explained by rich pre-treatment information and when overlap is adequate.

Instrumental Variables

An instrumental variable (IV) affects treatment assignment but has no direct effect on the outcome except through treatment. For instance, an administrative eligibility rule or random encouragement may influence programme participation without independently changing performance. IV estimation requires relevance, exclusion, and independence assumptions. Relevance means the instrument changes treatment probability. Exclusion means it has no pathway to the outcome other than treatment. Independence means it is unrelated to unobserved determinants of the outcome.

With heterogeneous treatment effects, IV commonly identifies a local average treatment effect (LATE): the effect for compliers whose treatment status responds to the instrument. It does not necessarily estimate the effect for never-takers or always-takers. Weak instruments create unstable estimates and misleading conventional inference. Researchers should report first-stage strength, explain the institutional basis of exclusion, and avoid presenting LATE as a universal effect.

A Framework for Identifying Causal Designs

A useful design process begins with a causal estimand rather than a preferred technique. The researcher should:

1. State the intervention, comparison condition, population, outcome, time horizon, and estimand.

2. Draw a causal diagram identifying confounders, mediators, colliders, and possible spillovers.

3. Determine whether randomisation is feasible and ethically acceptable.

4. Identify institutional variation that could support DiD, RD, IV, or another quasi-experiment.

5. Assess measurement quality, treatment compliance, attrition, interference, and timing.

6. Specify the analysis before examining outcomes, including primary outcomes and robustness checks.

7. Report assumptions, uncertainty, heterogeneous effects, and limitations in language appropriate to the evidence.

This process prevents method-driven research in which a technique is selected because it is fashionable rather than because it matches the assignment mechanism.

Validity Threats and Diagnostic Reasoning

Internal validity may be threatened by selection bias, history, maturation, regression to the mean, differential attrition, contamination, treatment misclassification, and researcher or participant reactivity. In pre-post studies, a change after an intervention may reflect economic conditions or seasonal demand rather than treatment. In voluntary programmes, motivated participants may improve even without the programme. In a sales experiment, employees may share treatment information with control colleagues.

External validity involves more than sample size. Effects may vary by organizational culture, market competition, managerial capability, regulation, treatment intensity, or time. A highly controlled experiment may estimate a clean effect in a narrow setting but fail to represent normal implementation. Conversely, a field study may be realistic but difficult to replicate. Transportability requires explicit reasoning about effect modifiers and the similarity between study and target contexts.

Construct validity concerns whether treatment and outcome measures represent the intended concepts. A training programme may increase test scores without improving job performance. Statistical conclusion validity concerns power, model specification, dependence, missing data, and multiplicity. Pre-registration, transparent code, replication, sensitivity analysis, and clear distinction between confirmatory and exploratory analysis support credible inference.

Ethics, Governance, and Responsible Use

Causal experiments must respect informed consent where required, privacy, fairness, and the principle of minimizing harm. Organizations should not withhold essential services merely to create a control group. Where an intervention is believed to be beneficial but evidence is uncertain, stepped-wedge designs may provide eventual treatment to all participating units while supporting comparison over time.

Digital experimentation raises additional concerns. Online platforms may randomise prices, recommendations, or information at massive scale, but users may not understand how their data are used. Automated eligibility rules can reproduce discrimination if treatment allocation correlates with protected characteristics. Governance should address data minimization, secure access, pre-specified stopping rules, auditability, adverse-event monitoring, and the authority to halt an experiment. Ethical approval and organizational review are not bureaucratic substitutes for sound design; they are part of responsible causal research.

Applied Case: Evaluating a Managerial Coaching Programme

Background

A national service organization introduces an eight-week coaching programme for branch managers. Senior leadership expects coaching to improve employee retention and customer satisfaction. Participation is initially voluntary, and high-performing branches are more likely to nominate managers.

Problem

After six months, participating branches report lower turnover. The executive team considers expanding the programme but asks whether coaching caused the improvement.

Analysis

A simple comparison is biased because participating branches differ before treatment. They may have more motivated managers, stronger local labour markets, or better resources. Researchers first define the estimand as the effect of offering coaching to eligible branch managers on six-month employee retention.

The organization then redesigns recruitment. Eligible branches are stratified by region and baseline turnover and randomly assigned either to immediate coaching or to a wait-list control. The analysis follows the ITT principle. Baseline retention, branch size, manager tenure, and labour-market indicators are recorded. Employee outcomes are measured through administrative records rather than manager reports. Because employees within branches are correlated, inference is clustered at branch level.

Researchers also monitor attendance and employee movement between branches. If employees transfer from treated to control branches, interference may dilute the estimate. A secondary analysis examines whether effects differ by baseline turnover and manager experience, while treating these subgroup findings as exploratory unless pre-specified.

Outcome

Suppose immediate-offer branches show a modest reduction in turnover relative to wait-list branches, with a confidence interval that excludes zero. Attendance is incomplete, so the ITT effect is smaller than the effect among participants, but it is more relevant to the decision of whether the organization should offer the programme routinely. The organization also finds no clear short-term improvement in customer satisfaction, suggesting that retention and customer outcomes may operate on different time scales.

Learning Points

The case illustrates why voluntary before-and-after comparisons are inadequate, why the treatment must be defined as an offer or actual participation, and why the unit of assignment determines analysis. It also demonstrates that a statistically credible effect does not automatically justify universal adoption. Decision-makers must consider implementation cost, subgroup effects, duration, unintended consequences, and whether the trial context resembles ordinary operations.

Managerial Use of Causal Evidence

Managers should ask four questions when interpreting a study. First, is the comparison credible? Second, what population and treatment does the estimate describe? Third, how precise and practically important is the effect? Fourth, would implementation conditions reproduce the study’s mechanism and compliance?

A small statistically significant effect may have little economic value if implementation is expensive. A large effect with wide uncertainty may justify a carefully monitored pilot rather than immediate scale-up. Heterogeneous effects can support targeted deployment, but subgroup analysis must not become a post hoc search for favourable results. Cost-effectiveness analysis should combine causal estimates with treatment costs, opportunity costs, risks, and distributional consequences.

Digital analytics and machine learning can support causal work through automated data linkage, anomaly detection, heterogeneous-effect estimation, and adaptive experimentation. Prediction, however, is not causation. A model that accurately predicts employee turnover does not show that changing the predicted risk factor will reduce turnover. Machine-learning methods should therefore be integrated with explicit identification strategies, transparent validation, and human oversight.

Current Developments and Best Practices

Contemporary causal research increasingly combines experiments with administrative and platform data, synthetic controls, interrupted time-series designs, mediation analysis, and transportability methods. Researchers are also paying greater attention to staggered treatment adoption, interference across networks, treatment-effect heterogeneity, and causal effects of algorithms themselves.

Best practice includes:

• defining the estimand before selecting the estimator;
• using pre-treatment variables and avoiding post-treatment adjustment;
• documenting treatment assignment and implementation fidelity;
• checking balance, pre-trends, overlap, discontinuities, and sensitivity to specifications;
• accounting for clustering, missing data, attrition, and multiple testing;
• reporting effect sizes and confidence intervals rather than relying only on significance;
• distinguishing ITT, treatment-on-the-treated, ATE, ATT, and LATE interpretations;
• making data, code, protocols, and deviations available where ethically and legally possible; and
• connecting statistical evidence to operational feasibility and stakeholder consequences.

Chapter Summary

Causal inference seeks to estimate counterfactual differences between treatment and comparison conditions. Randomised experiments provide the strongest general design for causal identification because random assignment balances potential outcomes in expectation, although implementation, attrition, interference, and ethical issues remain. Field experiments improve practical relevance, while natural and quasi-experiments exploit institutional variation.

Difference-in-differences depends on parallel trends; regression discontinuity identifies local effects near a cutoff; matching balances observed covariates but cannot solve unmeasured confounding; and instrumental variables identify effects for units whose treatment responds to a credible instrument. Each design is defined by its assumptions, not merely its statistical formula. Internal, external, construct, and statistical conclusion validity must be evaluated together. For managerial decisions, causal estimates should be interpreted with attention to uncertainty, heterogeneity, cost, implementation, ethics, and transferability.

Key Terms

Causal effect: Difference between potential outcomes under treatment and control.

Counterfactual: The unobserved outcome a unit would have experienced under an alternative condition.

Average treatment effect: Average causal effect across a defined population.

Treatment effect on the treated: Average causal effect among units receiving treatment.

Randomisation: Assignment by chance to treatment conditions.

Intention-to-treat: Effect of being assigned or offered treatment, regardless of compliance.

Confounder: Variable influencing both treatment assignment and the outcome.

Mediator: Variable through which a treatment affects an outcome.

Moderator: Variable that changes the size or direction of a treatment effect.

SUTVA: Assumption of well-defined treatments and no interference between units.

Internal validity: Credibility of a causal claim for the studied setting and units.

External validity: Transferability of findings to other settings or populations.

Difference-in-differences: Comparison of outcome changes across treated and comparison groups.

Parallel trends: Assumption that groups would have followed similar untreated trends.

Regression discontinuity: Design using treatment assignment at a threshold on a running variable.

Matching: Construction of comparable treated and untreated units using observed covariates.

Instrumental variable: Variable affecting treatment but influencing the outcome only through treatment.

Local average treatment effect: IV-identified effect for treatment compliers.

Spillover: Effect of one unit’s treatment on another unit’s outcome.

Common support: Overlap in observed characteristics or treatment probabilities across groups.

Review Questions

Short Answer Questions

1. Why is a causal effect described through potential outcomes?

2. Distinguish an intention-to-treat effect from a treatment-on-the-treated effect.

3. What is the parallel-trends assumption in difference-in-differences?

4. Why is an RD estimate generally considered local?

5. What is the difference between a confounder and a mediator?

Long Answer Questions

1. Explain how randomisation supports causal inference and discuss the main threats that can remain after random assignment.

2. Compare difference-in-differences, regression discontinuity, matching, and instrumental variables in terms of identification assumptions and interpretation.

3. Discuss internal and external validity using examples from organizational field experiments.

4. Explain why treatment-effect heterogeneity matters for managerial decisions.

5. Develop a causal research design for evaluating an employee-development intervention.

Analytical / Application Questions

1. A firm introduces a compliance system in high-risk branches while low-risk branches remain untreated. Branches using the system show fewer incidents after one year. Identify at least three threats to a causal interpretation and propose a stronger design.

2. A government subsidy is offered to firms with productivity scores above a fixed threshold. Explain how an RD study could be designed and what evidence would be needed to support its assumptions.

3. A digital platform randomly encourages some users to adopt a new productivity tool, but not all encouraged users comply. Explain how IV analysis could be used and which causal effect would be identified.

22. Multiple Choice Questions with Answers

Question 1

Which condition makes the difference in mean outcomes between treatment and control groups a credible estimate of the average treatment effect in a randomized experiment?

A. Treatment groups have identical post-treatment outcomes

B. Assignment is independent of potential outcomes

C. The sample contains only high-performing units

D. All participants comply perfectly with every measurement procedure

Correct Answer: B. Assignment is independent of potential outcomes

Question 2

A difference-in-differences design is most directly threatened when treated and comparison groups would have followed different outcome trends even without the intervention. This is a violation of:

A. The exclusion restriction

B. The common-support condition

C. The parallel-trends assumption

D. The relevance condition

Correct Answer: C. The parallel-trends assumption

Question 3

What does an intention-to-treat estimate measure?

A. The effect among only those who fully complied

B. The effect of being assigned or offered treatment

C. The effect for units near a treatment threshold

D. The effect after adjusting for all mediators

Correct Answer: B. The effect of being assigned or offered treatment

Question 4

In a regression-discontinuity design, the most defensible causal interpretation usually concerns:

A. All units in the national population

B. Only units that refuse treatment

C. Units close to the assignment cutoff

D. Units selected after observing outcomes

Correct Answer: C. Units close to the assignment cutoff

Question 5

Which statement best describes a limitation of propensity-score matching?

A. It cannot balance any observed characteristics

B. It requires treatment to be randomly assigned

C. It cannot remove bias from unmeasured confounders

D. It always estimates a local average treatment effect

Correct Answer: C. It cannot remove bias from unmeasured confounders

Question 6

For an instrumental variable to support causal inference, the exclusion restriction requires that the instrument:

A. Be strongly correlated with the outcome directly

B. Affect the outcome only through treatment

C. Be assigned after the outcome is measured

D. Be unrelated to treatment assignment

Correct Answer: B. Affect the outcome only through treatment

Question 7

A study finds that coaching participants have lower turnover, but participation was voluntary and motivated managers were more likely to enrol. The most immediate threat is:

A. Selection bias

B. Regression discontinuity

C. Instrument relevance

D. Measurement invariance

Correct Answer: A. Selection bias

Question 8

Why should branch-level randomisation generally be analyzed with attention to clustering?

A. Employees in the same branch may have correlated outcomes

B. Clustering guarantees treatment compliance

C. It eliminates the need for randomisation

D. It converts an observational study into an experiment

Correct Answer: A. Employees in the same branch may have correlated outcomes

Question 9

An IV estimate with heterogeneous treatment effects most commonly identifies:

A. The effect for every unit regardless of treatment behaviour

B. The effect for compliers affected by the instrument

C. The effect only for never-treated units

D. The unconditional association between treatment and outcome

Correct Answer: B. The effect for compliers affected by the instrument

Question 10

A predictive machine-learning model identifies employees likely to resign. Which conclusion is justified without a causal design?

A. Changing the predicted risk factor will reduce resignations

B. The risk factor is a proven treatment target

C. The model predicts resignation risk, but does not by itself establish causation

D. The model automatically satisfies the exclusion restriction

Correct Answer: C. The model predicts resignation risk, but does not by itself establish causation