9. Experimental Design and Causal Reasoning

Introduction and Context

Data science is often used to answer two different kinds of questions. The first is predictive: What is likely to happen? The second is causal: What would happen if we changed something? A predictive model may identify customers who are likely to leave, while a causal analysis asks whether offering those customers a discount would reduce departures. The distinction is fundamental. A variable can be highly useful for prediction without being an effective point of intervention.

Causal reasoning is therefore concerned with explaining outcomes through interventions, not merely identifying patterns. It supports decisions such as whether a public-health programme improves vaccination, whether a new product feature increases engagement, whether employee training raises productivity, or whether a policy reduces unemployment. These decisions require evidence that separates the effect of an action from the effects of other factors occurring at the same time.

Experimental design provides the strongest general framework for learning about causal effects. In a well-designed experiment, units are assigned to treatment and control conditions in a manner that makes the groups comparable, at least in expectation. When randomisation is feasible and implemented correctly, differences in outcomes can be attributed to the intervention with substantially greater confidence. When randomisation is not possible, observational causal methods attempt to approximate the logic of an experiment, but their conclusions depend more heavily on assumptions.

This chapter develops the conceptual and practical foundations of causal reasoning, with particular attention to experiments, A/B tests, confounding, selection bias, interpretation, ethics, and organisational decision-making.

Learning Objectives

By the end of this chapter, a student should be able to:

1. Distinguish association, prediction, and causation.

2. Define treatment, outcome, unit of analysis, potential outcomes, and causal estimands.

3. Explain how randomisation supports causal inference.

4. Design and evaluate an A/B test, including its sampling, power, metrics, and implementation.

5. Identify confounding, selection bias, measurement bias, interference, and other threats to validity.

6. Use observational methods such as matching, regression adjustment, difference-in-differences, and instrumental variables appropriately.

7. Interpret causal evidence for business, policy, and professional decisions.

8. Recognise ethical, governance, and operational responsibilities in experimentation.

Association Is Not an Intervention

An association describes a statistical relationship between variables. If users who receive more product notifications purchase more frequently, notification exposure and purchasing are associated. This finding does not establish that notifications caused the purchases. Frequent purchasers may be more active users and therefore more likely both to receive notifications and to buy. This alternative explanation is a form of confounding.

Causation concerns a comparison between possible worlds: the outcome that would occur for the same unit if it received treatment versus the outcome that would occur if it did not. Since the same unit cannot generally be observed simultaneously under both conditions, causal inference is fundamentally a missing-data problem. Experimental design constructs a credible comparison group to estimate the unobserved counterfactual.

Prediction and causation can diverge in several ways. A variable may be a strong predictor but not a useful intervention target. For example, a customer’s prior complaints may predict cancellation, but eliminating complaints is not necessarily a feasible or desirable strategy. Conversely, a variable may have a causal effect but weak predictive power if the effect is modest or heterogeneous. A causal model must therefore be designed around a clearly stated intervention and outcome, rather than around correlations alone.

A Language for Causal Questions

A causal study begins by specifying its elements precisely.

• The unit is the entity receiving treatment, such as a customer, patient, school, employee, household, or geographic region.
• The treatment is the intervention or exposure whose effect is being studied. It may be binary, such as receiving an email, or multivalued, such as receiving different doses.
• The outcome is the result measured after treatment, such as revenue, retention, blood pressure, or examination performance.
• A covariate is a pre-treatment characteristic that may help explain treatment assignment or outcome variation.
• The treatment effect is the difference in an outcome attributable to treatment under a specified definition and time horizon.

The potential-outcomes framework represents two possible outcomes for unit i: Yᵢ(1), if the unit receives treatment, and Yᵢ(0), if it does not. The individual treatment effect is Yᵢ(1) − Yᵢ(0), but only one of these potential outcomes is observed. Two common summaries are the average treatment effect (ATE),

ATE = E[Y(1) − Y(0)],

and the average treatment effect on the treated (ATT),

ATT = E[Y(1) − Y(0) | T = 1].

The ATE asks what would happen if treatment were assigned to the relevant population as a whole. The ATT asks what effect treatment had, or would have, for those who actually received it. These quantities need not be equal when treatment effects differ across people.

A causal question should also specify the estimand: the exact quantity to be estimated. “Does the programme work?” is too vague. A stronger question is: “What is the average change in six-month employment earnings among eligible applicants if they are offered the programme rather than receiving existing services?” The wording identifies the population, intervention, comparator, outcome, and time horizon.

Why Randomisation Works

Randomised controlled trials assign units to treatment or control using a chance mechanism. With a sufficiently large sample and sound implementation, randomisation balances both observed and unobserved pre-treatment characteristics in expectation. Consequently, the average difference in outcomes between groups estimates the causal effect of assignment.

The logic depends on several conditions. Consistency means that the treatment received corresponds to the treatment being studied and that the outcome is well defined. Exchangeability means that, conditional on the design or measured covariates, treated and untreated units are comparable with respect to potential outcomes. Positivity means that each relevant type of unit has a non-zero probability of receiving each treatment level. No interference, often called the Stable Unit Treatment Value Assumption, means that one unit’s treatment does not alter another unit’s outcome. These assumptions can fail in practical settings.

Randomisation does not automatically guarantee a good study. Problems may arise from inadequate sample size, faulty random-number generation, non-compliance, differential attrition, contamination between groups, or changes in measurement. Randomisation protects against systematic baseline differences, but it does not repair every operational or ethical weakness.

Designing an A/B Test

An A/B test is a controlled experiment in which eligible units are randomly assigned to version A, usually the control, or version B, the treatment. It is common in digital products because assignment and outcomes can often be recorded automatically. Nevertheless, an A/B test is a scientific study rather than merely a software deployment.

The design should proceed through a sequence of decisions.

Formulate the decision and hypothesis

Begin with the decision the evidence will inform. A product team might ask whether a redesigned checkout page should replace the existing page. The primary hypothesis could be that the redesign increases completed purchases without materially increasing cancellations or support contacts.

Define the unit and assignment mechanism

The unit might be a user, account, household, or session. Assignment should usually occur at the level at which outcomes and interference are most appropriately controlled. Assigning separate sessions from the same user to different versions can create contamination and misleadingly narrow uncertainty. Persistent user-level assignment is often preferable for a multi-visit product experience.

Specify primary and secondary outcomes

A primary outcome is selected before examining results. It should be operationally defined, measurable, and linked to the decision. Secondary outcomes may reveal mechanisms or unintended effects. For example, conversion rate may be primary, while average order value, refund rate, page load time, and customer complaints are secondary. A large collection of unplanned metrics increases the chance of finding a seemingly positive result by chance.

Determine sample size and duration

Sample-size planning depends on baseline outcome frequency, minimum practically important effect, desired statistical power, significance threshold, and allocation ratio. Statistical significance alone is not enough: a tiny improvement may be precisely estimated but economically irrelevant. The experiment must also run long enough to capture delayed outcomes, weekday variation, seasonality, and novelty effects.

Randomise, monitor, and analyse

The system should verify that assignment is balanced, treatment exposure is correctly logged, and users are not repeatedly reassigned. Monitoring should include data quality and safety metrics, not only the primary outcome. The main analysis should follow the pre-specified assignment rule. In an intention-to-treat analysis, units are analysed according to their assigned group, even if they did not comply. This preserves the benefit of randomisation and estimates the effect of being offered or assigned treatment.

A per-protocol analysis, restricted to units who followed the assigned treatment, may describe the effect among compliers but can introduce selection bias because compliance is often related to motivation or other characteristics. It should not casually replace intention-to-treat analysis.

Statistical Interpretation and Decision Value

An estimated treatment effect should be accompanied by uncertainty, commonly a confidence interval. A point estimate of a two-percentage-point increase with a confidence interval from 0.5 to 3.5 percentage points conveys more information than a statement that the result is “significant.” The interval describes values compatible with the data under the model and design assumptions; it is not a guarantee that the true effect lies inside it.

Statistical significance is affected by sample size and does not measure practical importance. Decision-makers should consider effect magnitude, uncertainty, implementation cost, reversibility, distributional consequences, and risks. A treatment may be worthwhile even when a study is inconclusive if it is inexpensive and reversible, whereas a high-risk policy may require stronger evidence.

Multiple testing creates another interpretive problem. If many outcomes, subgroups, or stopping points are examined, at least one apparently positive result may occur by chance. Pre-registration, a primary outcome, multiplicity adjustments where appropriate, and replication reduce this risk. Subgroup effects should be interpreted cautiously unless subgroups were specified in advance and the study has adequate power. An overall average effect can conceal beneficial effects for one group and harmful effects for another.

Threats to Causal Validity

Confounding

A confounder influences both treatment assignment and the outcome. For example, prior illness may affect whether a patient receives an intensive treatment and also affect recovery. Comparing treated and untreated patients without accounting for illness severity may attribute the effect of severity to treatment.

Confounding is not solved simply by controlling for every available variable. Adjusting for a pre-treatment confounder can help, but adjusting for a mediator—an intermediate consequence of treatment—may remove part of the effect being estimated. Adjusting for a collider, a variable influenced by treatment and outcome, can create a spurious association. Causal diagrams, particularly directed acyclic graphs, help researchers decide which variables should be adjusted for and clarify assumptions.

Selection bias

Selection bias occurs when inclusion in the analysed sample depends on treatment, outcome, or factors related to them. A survey of only highly engaged customers may overstate satisfaction. Analysing only employees who completed training can make training appear more effective if motivated employees are more likely to complete it. Loss to follow-up can similarly distort results when attrition differs between groups.

Measurement and implementation problems

An outcome measured differently across groups can generate apparent treatment effects. If assessors know which patients received treatment, their judgements may be influenced. Blinding, objective measures, standardised procedures, and audit trails reduce such risks. Treatment misclassification also weakens estimates: if some control users are exposed to the intervention, the contrast between groups becomes less clear.

Interference and spillovers

The treatment of one unit may affect another. A public-health campaign can influence untreated residents through social networks; a price promotion in one store can alter demand in nearby stores. Individual randomisation may then estimate a diluted or ambiguous effect. Cluster randomisation, network-aware designs, or explicit measurement of spillovers may be required.

External validity

Internal validity concerns whether the estimated effect is credible for the study population. External validity concerns whether it transports to another population, location, time, or implementation context. A controlled online experiment may not predict results after a feature is deployed to all customers, particularly if competitors respond or operational capacity changes. Generalisation requires substantive reasoning, not merely a large sample.

Observational Causal Methods

Experiments are not always feasible. Ethical restrictions, cost, legal constraints, rare exposures, and policies already in operation may require observational analysis. Such methods can be valuable, but their credibility depends on explicit assumptions and sensitivity analysis.

Regression adjustment estimates outcome differences while controlling for measured covariates. It is useful when the functional form is reasonable and important confounders are observed. Flexible machine-learning models can improve nuisance-function estimation, but predictive accuracy does not by itself establish causal validity.

Matching constructs treated and comparison groups with similar measured characteristics. Propensity-score methods model the probability of treatment given covariates, while methods such as coarsened exact matching or inverse-probability weighting create a more comparable pseudo-population. Balance should be assessed after matching or weighting. Matching cannot address unmeasured confounding.

Difference-in-differences compares changes over time in a treated group with changes in a comparison group. Its central assumption is parallel trends: in the absence of treatment, both groups would have followed similar trends. Pre-treatment trends can provide evidence relevant to this assumption, though they cannot prove it.

Instrumental variables use a variable that affects treatment assignment but influences the outcome only through treatment, except through specified pathways. Valid instruments are difficult to find. The resulting estimate may apply only to compliers affected by the instrument rather than to the entire population.

Regression discontinuity designs exploit a treatment threshold. Units just above and below a known cutoff are compared, assuming they cannot precisely manipulate their position and that other determinants change smoothly at the threshold. This can yield credible local causal effects, but not necessarily effects far from the cutoff.

Interrupted time-series analysis examines whether an outcome’s level or trend changes after an intervention. It requires sufficient pre- and post-intervention observations and careful consideration of concurrent events, seasonality, and autocorrelation.

These methods should not be selected because they produce a preferred result. The design should follow the institutional setting, treatment assignment process, timing, and most defensible assumptions.

An Illustrative Case: A Retention Intervention in a Subscription Service

Background

Consider a fictional subscription-based learning platform, LearnNow. Management observes that users who complete an orientation tutorial are more likely to remain subscribed after three months. The company proposes sending a personalised tutorial reminder to new users and wants to know whether the reminder causes higher retention.

Situation and problem

A simple comparison between reminded and non-reminded users is inappropriate because the messaging system currently targets users who have already shown high engagement. Those users may be more likely to retain regardless of the message. The decision is whether to deploy the reminder to all eligible new users.

Analysis and design

The analytics team defines the unit as a new user account and the intervention as receiving one reminder within 48 hours of registration. The primary outcome is active paid subscription at 90 days. Secondary outcomes include tutorial completion, support contacts, unsubscribe requests, and revenue. Users are randomly assigned at account level to control or treatment, with assignment retained across devices.

Before launch, the team specifies a minimum practically important retention increase, estimates the required sample size, and identifies safeguards against excessive messaging. It pre-registers the primary analysis and commits to intention-to-treat estimation. The experiment is scheduled to cover several full weeks to reduce calendar effects.

Results and interpretation

Suppose the treatment group shows a modest increase in 90-day retention, with an uncertainty interval that excludes zero. Tutorial completion also rises, while support contacts do not. However, the effect is concentrated among users who registered through mobile devices, and the estimate for desktop users is imprecise. The team should not conclude that the reminder benefits only mobile users solely from a subgroup comparison; the interaction should be formally assessed and replicated.

The estimated effect applies to users eligible under the experiment’s conditions and to the particular reminder message. It does not automatically establish that more frequent reminders, a different message, or deployment in another market will have the same effect. Management should combine the effect estimate with incremental revenue, message cost, potential fatigue, and privacy requirements before deployment.

Learning points

The case demonstrates why a causal question requires an explicit intervention, outcome, population, and time horizon. Randomisation addresses the original engagement-related confounding, but careful implementation remains necessary. Business value depends on both the size and uncertainty of the effect and the costs and risks of scaling it.

Ethics, Governance, and Organisational Practice

Experimentation can affect real people, not merely abstract observations. Researchers should assess foreseeable harms, informed-consent requirements, privacy implications, fairness, and the possibility that a treatment disadvantages vulnerable groups. In low-risk digital experiments, consent requirements may differ by jurisdiction and context, but legal compliance is not the same as ethical adequacy.

Data minimisation, access controls, retention limits, de-identification where appropriate, and transparent documentation should accompany experimentation. Sensitive attributes must not be used casually for targeting or subgroup analysis. Fairness evaluation should consider whether average gains conceal harms for protected or vulnerable populations.

Organisations also need governance for stopping rules, adverse-event monitoring, experiment registration, version control, and post-deployment review. The team should document the hypothesis, assignment unit, eligibility rules, outcomes, exclusions, analysis plan, deviations, and decision rationale. Independent review is especially important for medical, financial, employment, education, and public-sector interventions.

Emerging Directions

Modern experimentation increasingly combines automated allocation, real-time monitoring, and machine-learning-based personalisation. Multi-armed bandits can allocate more traffic to apparently better options while learning, but they change the estimand and complicate inference because allocation probabilities evolve. They are useful for optimisation when exploration and exploitation are explicit priorities, whereas fixed randomisation is often preferable for clean confirmatory evidence.

Causal machine learning seeks to estimate heterogeneous treatment effects and identify which interventions are likely to help which units. These methods can support targeted decisions, but subgroup discovery requires validation, adequate data, and safeguards against discrimination. Synthetic controls, federated data analysis, and privacy-preserving experimentation are also developing in policy and industry settings.

The enduring principle is that advanced computation cannot replace a credible design. Algorithms can estimate complex patterns, but the meaning of those patterns depends on treatment definition, assumptions, measurement, and institutional context.

Best Practices for Causal Work

A defensible causal study should:

• State the intervention, comparator, population, outcome, and time horizon precisely.
• Choose the unit of assignment to minimise contamination and interference.
• Use randomisation whenever feasible and ethically acceptable.
• Pre-specify primary outcomes, analysis rules, and stopping criteria.
• Distinguish intention-to-treat effects from effects among compliers.
• Report effect sizes and uncertainty, not only significance labels.
• Examine balance, missingness, attrition, implementation fidelity, and adverse outcomes.
• Use causal diagrams and substantive knowledge to guide adjustment.
• Test assumptions and conduct sensitivity analyses in observational studies.
• Separate exploratory findings from confirmatory evidence.
• Evaluate practical value, equity, privacy, and scalability before acting.
• Replicate important findings across time, populations, and implementation settings.

Chapter Summary

Causal reasoning asks what would happen under an intervention and therefore requires a credible counterfactual comparison. Association and prediction may reveal useful patterns, but neither alone establishes that changing a variable will change an outcome. The potential-outcomes framework defines treatment effects, while randomisation makes treatment groups comparable in expectation and supports causal interpretation.

A sound A/B test specifies the decision, unit, treatment, outcomes, sample size, duration, assignment mechanism, and analysis plan. Intention-to-treat analysis preserves the logic of randomisation, while confidence intervals and practical significance support responsible decisions. Confounding, selection bias, measurement error, attrition, interference, and limited external validity can threaten conclusions. When experiments are impossible, observational methods—including regression adjustment, matching, difference-in-differences, instrumental variables, regression discontinuity, and interrupted time series—offer alternatives whose assumptions must be examined rather than hidden.

Causal evidence has managerial, policy, ethical, and technical consequences. Responsible data scientists communicate uncertainty, assess heterogeneous and distributional effects, protect privacy, monitor harms, and avoid treating sophisticated algorithms as substitutes for sound research design.

Key Terms

Association: A statistical relationship between variables that does not necessarily imply causation.

Average Treatment Effect (ATE): The average difference between potential outcomes under treatment and no treatment for a defined population.

Average Treatment Effect on the Treated (ATT): The average causal effect among units that actually received treatment.

A/B Test: A randomised experiment comparing two versions or conditions.

Causal Effect: The change in an outcome attributable to an intervention or exposure.

Collider: A variable influenced by two other variables; conditioning on it can create spurious association.

Confounder: A pre-treatment variable related to both treatment assignment and the outcome.

Control Group: Units assigned to a comparison condition rather than the focal treatment.

Counterfactual: The outcome that would have occurred for a unit under an alternative treatment condition.

Exchangeability: The condition that treated and untreated units are comparable with respect to potential outcomes, given the design or adjustment set.

External Validity: The extent to which findings generalise to other populations or settings.

Intention-to-Treat: Analysis according to assigned treatment, regardless of compliance.

Interference: A situation in which one unit’s treatment affects another unit’s outcome.

Mediator: A variable lying on the pathway through which treatment affects an outcome.

Potential Outcome: The outcome a unit would experience under a specified treatment condition.

Positivity: The condition that relevant units have a non-zero probability of receiving each treatment level.

Selection Bias: Distortion caused by the process determining who enters, remains in, or is analysed in a study.

Treatment Effect: The difference between outcomes under two specified treatment conditions.

Review Questions

Short Answer Questions

1. Why can a strong predictor fail to be a useful causal intervention target?

2. Define confounding and provide an example from a business or policy setting.

3. What is the difference between ATE and ATT?

4. Why is intention-to-treat analysis commonly used in randomised experiments?

5. What does positivity mean in causal inference?

Long Answer Questions

1. Explain how randomisation supports causal inference and discuss circumstances in which randomisation may still produce misleading conclusions.

2. Describe the main stages in designing and analysing an A/B test for a digital product.

3. Compare association, prediction, and causation using a coherent applied example.

4. Explain how selection bias, measurement bias, and interference can threaten experimental validity.

5. Compare regression adjustment, matching, difference-in-differences, and instrumental variables as observational causal methods.

Analytical / Application Questions

1. A hospital finds that patients receiving a new therapy have lower recovery rates than patients receiving standard care. Identify at least two plausible explanations other than treatment harm and propose a more credible study design.

2. An online retailer tests a recommendation system by assigning sessions rather than users. Explain the threats created by this choice and recommend an improved assignment strategy.

3. A government evaluates a training programme using employment changes before and after participation, with no comparison group. Critique the design and propose an approach that would strengthen causal interpretation.

22. Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes a causal effect from an association?

A. A causal effect describes a relationship observed in a sample.

B. A causal effect describes the change produced by an intervention relative to a counterfactual.

C. A causal effect is any relationship with a small p-value.

D. A causal effect is a relationship found by a predictive model.

Correct Answer: B. A causal effect describes the change produced by an intervention relative to a counterfactual.

Question 2

What is the main purpose of randomisation in an experiment?

A. To guarantee that every individual benefits from treatment.

B. To eliminate all measurement error.

C. To make treatment groups comparable in expectation, including on unobserved factors.

D. To ensure that the treatment effect is statistically significant.

Correct Answer: C. To make treatment groups comparable in expectation, including on unobserved factors.

Question 3

A customer’s prior engagement affects both whether the customer receives a promotional email and whether the customer purchases. Prior engagement is most likely a:

A. Mediator

B. Collider

C. Confounder

D. Treatment outcome

Correct Answer: C. Confounder

Question 4

Why is intention-to-treat analysis valuable in an A/B test?

A. It analyses users according to assignment and preserves the benefits of randomisation.

B. It removes the need to define a primary outcome.

C. It guarantees a larger treatment effect.

D. It excludes users who do not comply with treatment.

Correct Answer: A. It analyses users according to assignment and preserves the benefits of randomisation.

Question 5

Which practice most directly reduces the risk of false positive findings from examining many outcomes?

A. Increasing the number of exploratory subgroups after seeing results

B. Pre-specifying a primary outcome and addressing multiplicity

C. Analysing only users who completed treatment

D. Stopping the test whenever the result looks favourable

Correct Answer: B. Pre-specifying a primary outcome and addressing multiplicity

Question 6

A study compares employment changes in a treated region with changes in a similar untreated region. Which assumption is central to a difference-in-differences analysis?

A. The treatment must be assigned by lottery.

B. The regions must have identical population sizes.

C. The groups would have followed parallel trends without treatment.

D. There can be no pre-treatment observations.

Correct Answer: C. The groups would have followed parallel trends without treatment.

Question 7

Why can adjusting for a mediator produce an inappropriate estimate of a total treatment effect?

A. It may remove part of the pathway through which treatment operates.

B. It always creates randomisation.

C. It guarantees positivity.

D. It converts an observational study into an experiment.

Correct Answer: A. It may remove part of the pathway through which treatment operates.

Question 8

An experiment reports a two-percentage-point increase with a confidence interval from 0.5 to 3.5 percentage points. Which interpretation is most appropriate?

A. The true effect must be exactly two percentage points.

B. The result is statistically and necessarily economically important.

C. The data are compatible with a positive effect within the stated uncertainty range, subject to design assumptions.

D. The intervention will have the same effect in every population.

Correct Answer: C. The data are compatible with a positive effect within the stated uncertainty range, subject to design assumptions.

Question 9

A treatment assigned to one person changes the outcome of another untreated person. This is an example of:

A. Positivity

B. Interference or spillover

C. Consistency

D. Measurement blinding

Correct Answer: B. Interference or spillover

Question 10

Which conclusion is most defensible when a machine-learning model predicts customer churn accurately?

A. The most important predictors are automatically valid intervention targets.

B. The model proves that changing every predictor will reduce churn.

C. Prediction may support targeting, but causal analysis is needed to estimate intervention effects.

D. Randomised experiments are unnecessary whenever prediction accuracy is high.

Correct Answer: C. Prediction may support targeting, but causal analysis is needed to estimate intervention effects.