Experimental Design and Causal Inference

Introduction

Business analytics frequently asks causal questions: Did a price reduction increase sales? Did a recommendation algorithm improve conversion? Did a training programme raise employee productivity? These questions differ from descriptive questions, such as how many customers purchased, and predictive questions, such as which customers are likely to purchase. Causal inference concerns what would happen to the same unit under different possible actions.

The central difficulty is that, for an individual customer, employee, store, or firm, we observe only one realized condition. A customer either receives an email or does not; we cannot directly observe that customer’s outcome under the alternative condition at the same time. Experimental design provides a systematic way to estimate these unobserved counterfactual outcomes. Randomized experiments are generally the strongest design for identifying causal effects, while quasi-experimental methods seek credible causal evidence when random assignment is impossible.

This chapter develops the logic of experiments and A/B tests, explains control groups and randomization, distinguishes correlation from causation, and evaluates confounding, endogeneity, internal validity, external validity, and ethical constraints.

Learning Objectives

By the end of this chapter, you should be able to:

1. Distinguish descriptive, predictive, and causal research questions.
2. Define treatments, outcomes, counterfactuals, average treatment effects, and estimands.
3. Design randomized experiments and A/B tests with appropriate control groups.
4. Explain how randomization addresses confounding and selection bias.
5. Evaluate quasi-experimental designs, including difference-in-differences, regression discontinuity, instrumental variables, and matching.
6. Diagnose threats to causal validity, including interference, noncompliance, attrition, and measurement error.
7. Interpret experimental findings responsibly and communicate their business implications.

1. From Association to Causation

An association is a systematic relationship between variables. For example, customers who use a mobile application may spend more than customers who do not. This relationship alone does not establish that application use causes higher spending. Frequent users may be more loyal, wealthier, or more technologically confident. These characteristics can influence both application use and spending.

A causal claim compares outcomes under alternative interventions. The potential-outcomes framework formalizes this idea. For unit i, let:

• Yᵢ(1) be the outcome if unit i receives the treatment;
• Yᵢ(0) be the outcome if unit i receives the control condition;
• Tᵢ be the treatment indicator, where Tᵢ = 1 denotes treatment and Tᵢ = 0 denotes control;
• Yᵢ be the observed outcome.

The observed outcome is:

The individual treatment effect is:

This quantity is usually unobservable because only one potential outcome is observed for each unit. The fundamental problem of causal inference is therefore a missing-data problem involving counterfactual outcomes.

The average treatment effect (ATE) is:

Here, E denotes the population expectation. The ATE is the average causal effect if the treatment were assigned to the relevant target population. A related quantity is the average treatment effect on the treated (ATT):

The ATT describes the average effect among units that actually received treatment. ATE and ATT are equal under particular conditions, such as homogeneous treatment effects or treatment assignment independent of potential outcomes. They should not be treated as interchangeable automatically.

Important Estimands

An estimand is the precise causal quantity a study intends to estimate. Examples include:

A well-designed study specifies the estimand before data analysis. Changing the estimand after seeing the results can produce misleading conclusions.

2. The Logic of Randomized Experiments

A randomized controlled trial assigns eligible units to treatment or control using a chance mechanism. In a simple two-arm experiment, each unit has a known probability of assignment to each condition. Randomization makes treatment assignment independent of potential outcomes in expectation:

The symbol ⟂ means statistical independence. In practical terms, randomization tends to balance both observed and unobserved characteristics between groups. It therefore removes systematic selection bias, provided the experiment is properly implemented.

Under random assignment, consistency, and adequate implementation, the difference in group means estimates the causal effect:

where Ȳ₁ is the sample mean outcome in the treatment group and Ȳ₀ is the sample mean outcome in the control group. The hat indicates an estimate rather than the true population parameter.

Randomization does not guarantee identical groups in every sample. Chance imbalances can occur, particularly in small samples. Randomization guarantees a valid assignment mechanism, not perfect balance. Researchers should examine baseline characteristics and may use pre-specified covariate adjustment to improve precision.

Control Groups

A control group provides the comparison needed to estimate what would have happened without the intervention. Possible controls include:

• No-treatment control: eligible units receive no new intervention.
• Business-as-usual control: units receive the existing process.
• Placebo or attention control: units receive an alternative experience that controls for attention or expectations.
• Active control: the new intervention is compared with an existing alternative.

The appropriate control depends on the causal question. If the question is whether a new training course is better than the current course, business-as-usual is usually more relevant than no training.

Blocking and Stratification

If a researcher expects outcomes to differ substantially across regions, customer segments, or store sizes, randomization can be performed within blocks. For example, customers may be divided into high-value and low-value strata, with treatment assigned randomly within each stratum. Blocking improves balance and can increase precision. The analysis must account for the design, usually by including block indicators or estimating effects within blocks and combining them appropriately.

Cluster Randomization

Sometimes treatment is assigned to groups rather than individuals. A retailer may assign entire stores to a new staffing policy, or a firm may assign departments to a training programme. This is cluster randomization. Observations within a cluster tend to be correlated, so treating them as independent can understate uncertainty.

The intracluster correlation coefficient (ICC), commonly denoted ρ, measures similarity among units within clusters. A rough design effect for equal cluster sizes is:

where m is the average number of units per cluster. The effective sample size is approximately the actual sample size divided by DE. This formula assumes approximately equal cluster sizes and a common ICC; more complex designs require more specialized calculations.

3. Designing an A/B Test

An A/B test is a randomized experiment comparing version A, often the control, with version B, the treatment. A rigorous A/B test involves the following sequence.

3.1 Define the Decision and Hypothesis

Begin with a decision, not merely a metric. For example: “Should the firm deploy the redesigned checkout page to all eligible customers?” State a primary hypothesis and a primary outcome, such as completed purchase rate within seven days.

A binary conversion outcome can be summarized as:

where x is the number of conversions and n is the number of eligible units. The difference in conversion rates is:

The ratio or percentage lift may also be reported:

The denominator must be clearly stated, and percentage-point differences must not be confused with percentage lift.

3.2 Define Eligibility, Unit, and Exposure

The unit of randomization might be a user, account, session, store, or day. The choice affects independence, interference, and implementation. If a user can appear in both groups through multiple devices, contamination may occur. A stable user identifier and persistent assignment are often preferable.

Eligibility criteria should be specified before assignment. Researchers should define when treatment begins, how long it lasts, and the observation window for outcomes. Outcomes measured too early may miss delayed effects; outcomes measured too late may include unrelated influences.

3.3 Randomize and Preserve Assignment

Use a reproducible randomization procedure. Treatment allocation should occur before outcomes are observed, and analysts should not selectively exclude inconvenient observations. A pre-analysis plan can specify hypotheses, primary outcomes, stopping rules, subgroup analyses, and statistical procedures.

3.4 Determine Sample Size and Precision

Sample size depends on the baseline outcome rate, minimum detectable effect, significance level, desired power, allocation ratio, and outcome variance. For a simple comparison of two independent means with equal group sizes, an approximate sample size per group is:

where σ² is the assumed outcome variance, δ is the minimum detectable difference, α is the two-sided Type I error rate, β is the Type II error rate, and z_q is the qth standard-normal quantile. This approximation assumes independent observations, equal variances, equal allocation, and a continuous outcome. Binary outcomes require a corresponding proportion-based calculation; cluster designs require inflation by the design effect.

Power is the probability of detecting an effect of a specified size under the assumed model. A non-significant result does not prove that the treatment has no effect; it may reflect low power, imprecise measurement, or a genuinely small effect.

3.5 Analyze According to Assignment

The intention-to-treat (ITT) analysis compares units according to their assigned condition, even if they do not comply. ITT estimates the effect of offering or assigning the intervention and preserves the benefit of randomization.

Per-protocol analysis includes only units who followed the protocol. It may be useful for operational questions but can be biased because compliance is often related to motivation or other characteristics. Therefore, it should not replace ITT without careful justification.

A confidence interval communicates uncertainty. For an estimated effect θ̂, a conventional approximate 95% confidence interval is:

where SE(θ̂) is the standard error. This interval relies on appropriate sampling or large-sample assumptions. It is not a statement that there is a 95% probability that a fixed parameter lies in the calculated interval; rather, the procedure would cover the true parameter in approximately 95% of repeated samples under its assumptions.

4. Worked Example: A Conversion Experiment

A digital retailer randomly assigns 20,000 eligible visitors equally to two checkout pages. The control page produces 1,000 purchases; the redesigned page produces 1,100 purchases.

The control conversion rate is:

The treatment conversion rate is:

The estimated effect is:

Thus, the redesign increases conversion by 1 percentage point. Relative lift is:

The correct interpretation is “a 1-percentage-point increase, equivalent to a 10% relative lift,” not simply “conversion increased by 10 percentage points.”

The business decision should also consider implementation cost, margin, returns, customer complaints, and longer-term retention. Statistical significance alone does not establish commercial value.

5. Threats to Experimental Validity

Internal Validity

Internal validity concerns whether the estimated effect is causal for the units and conditions studied. Major threats include:

• Noncompliance: assigned units do not receive the assigned treatment.
• Contamination: control units are exposed to treatment.
• Attrition: outcomes are missing differently across groups.
• Interference: one unit’s treatment affects another unit’s outcome.
• Differential measurement: outcomes are measured differently by condition.
• Implementation changes: the intervention varies across locations or time.
• Peeking and optional stopping: repeated outcome inspection leads to unplanned termination or analysis choices.

The stable unit treatment value assumption (SUTVA) is often used to summarize two conditions: consistency of treatment and no interference between units. It may fail in referral programmes, marketplaces, social networks, and competing-store experiments.

External Validity

External validity concerns whether findings generalize to other populations, times, settings, or treatment versions. A highly controlled experiment may have strong internal validity but limited realism. Generalization should consider:

• differences in customer composition;
• changes in technology or market conditions;
• variation in implementation quality;
• treatment effect heterogeneity;
• differences between short-term and long-term outcomes.

Replication across relevant settings and analysis of pre-specified subgroups can improve generalizability, but subgroup estimates require adequate sample sizes and cautious interpretation.

Statistical Conclusion Validity

Statistical conclusion validity concerns whether the analysis supports the stated inference. Common problems include multiple testing, low power, model misspecification, outliers, and treating repeated observations as independent. If many outcomes or subgroups are tested, some apparently significant findings may occur by chance. Researchers should identify primary outcomes in advance or adjust for multiplicity when appropriate.

6. Quasi-Experimental Designs

When random assignment is infeasible or unethical, quasi-experimental designs exploit institutional rules, timing, geographic variation, or other sources of plausibly exogenous variation. They can provide strong evidence, but their identifying assumptions must be defended rather than assumed.

Difference-in-Differences

Difference-in-differences (DiD) compares changes over time in a treated group with changes in a comparison group:

The key assumption is parallel trends: in the absence of treatment, the treated and comparison groups would have experienced similar average changes. Pre-treatment trend analysis can provide evidence, but it cannot prove that parallel trends would have continued after treatment.

For example, if treated stores increase sales from 500 to 560 and comparison stores increase from 520 to 540, then:

The estimated treatment effect is 40 sales units, subject to the parallel-trends assumption and other design requirements.

Regression Discontinuity

A regression discontinuity (RD) design uses a cutoff rule. Units just above and just below a threshold are compared when treatment assignment changes discontinuously at the cutoff. For example, applicants with a credit score of 700 or above may receive an offer.

RD identifies a local causal effect near the threshold, not necessarily an effect for all applicants. Validity requires that units cannot precisely manipulate the assignment variable and that other determinants of outcomes do not also jump at the cutoff. Graphical inspection, continuity tests, bandwidth choices, and robustness checks are important.

Instrumental Variables

An instrumental variable (IV) affects treatment assignment but affects the outcome only through treatment, under the exclusion restriction. It must also be relevant: the instrument must predict treatment. Under additional assumptions, IV identifies a local average treatment effect for compliers rather than necessarily the overall ATE.

A common two-stage formulation is:

followed by:

where Zᵢ is the instrument, T̂ᵢ is predicted treatment, and β₁ is interpreted causally only under the IV assumptions. Weak instruments can produce unstable estimates, and the exclusion restriction is generally difficult to verify empirically.

Matching and Weighting

Matching methods compare treated units with observationally similar untreated units. Propensity scores summarize the probability of treatment given measured covariates:

Matching or weighting can reduce imbalance in observed covariates. It cannot remove bias from unmeasured confounders, and a credible analysis requires overlap: treated and untreated units must have comparable covariate patterns. Matching is therefore not equivalent to randomization.

7. Confounding and Endogeneity

A confounder is a variable that influences both treatment and outcome and is not itself caused by the treatment. If advertising spending is higher in regions with falling demand, a naive correlation between advertising and sales may be negative even if advertising is effective. Demand conditions confound the relationship.

Endogeneity is a broader condition in which a predictor is statistically related to the error term in a causal model. It can arise from omitted variables, reverse causality, measurement error, sample selection, or simultaneous determination. In a regression model:

τ has a causal interpretation only when the treatment is appropriately assigned or, conditional on X, is independent of the error-generating potential outcomes. Observational regression adjustment is useful, but the inclusion of many controls does not automatically establish causation.

Researchers should distinguish confounders from mediators. A mediator lies on the causal pathway from treatment to outcome. Controlling for a mediator can remove part of the effect being estimated. A collider is a variable influenced by two causes; conditioning on it can create a spurious association. Causal diagrams, grounded in subject-matter knowledge, help determine which variables should be adjusted for.

8. A Meaningful Applied Case

Illustrative Case: Personalized Recommendations at an Online Retailer

Background. An online retailer develops a machine-learning recommendation panel intended to increase basket value. Analysts observe that customers who already see recommendations spend more than other customers.

Problem. Product managers want to know whether the panel causes higher revenue and whether it should be deployed broadly. Observational comparisons are unreliable because loyal customers are more likely to visit pages where the panel is displayed.

Approach. The retailer defines the customer as the randomization unit and assigns eligible customers to either the existing page or the recommendation panel. Assignment is persistent across sessions. The primary outcome is revenue per eligible customer over 30 days; secondary outcomes include conversion, number of items, returns, and customer-service contacts. The analysis follows ITT principles. A power analysis is based on historical revenue variance and a minimum economically meaningful increase. Treatment is not rolled out to customers in a region where the recommendation model has known safety and privacy problems until those issues are resolved.

Analysis. The team checks assignment balance, exposure rates, attrition, and contamination. It reports the treatment effect with a confidence interval and separately reports effects on returns and complaints. It does not stop the test merely because an early daily result appears favorable. It also examines whether the effect differs for new and returning customers, with subgroup analyses specified in advance.

Outcome. Suppose the panel increases average revenue modestly but also increases returns enough to eliminate contribution-margin gains. The appropriate decision is not to deploy the original version simply because revenue rose. The firm may redesign recommendations, test quality constraints, or optimize for profit rather than gross revenue.

Learning points. The case illustrates that causal design must align with the business decision, that the primary metric should reflect value rather than convenience, and that unintended outcomes can change the managerial conclusion. It also demonstrates why machine-learning personalization does not remove the need for randomization; indeed, algorithmic targeting can intensify selection bias in observational data.

9. Ethics, Governance, and Responsible Experimentation

Business experiments affect real people and should meet ethical and governance standards. Researchers should assess foreseeable risks, privacy implications, fairness, and the possibility that treatment may disadvantage vulnerable groups. Data collection should follow applicable privacy laws and organizational policies, including purpose limitation, access control, retention rules, and appropriate consent or legal basis.

A/B testing should not knowingly expose participants to serious harm merely to obtain information. Experiments involving prices, credit, employment, health, safety, or essential services require heightened review. Treatment assignment should not create unlawful discrimination. Sensitive subgroup effects should be evaluated where legally and scientifically appropriate.

Governance practices include version-controlled analysis code, documented randomization, pre-specified decision rules, secure data handling, audit trails, and independent review for high-risk interventions. Reproducibility requires recording the population, assignment mechanism, treatment versions, exclusions, outcomes, analysis date, and deviations from the plan.

10. Technology and Contemporary Practice

Modern experimentation platforms can automate assignment, exposure logging, metric calculation, and sequential monitoring. These systems improve operational reliability but do not solve conceptual problems. A poorly defined estimand, contaminated control group, or biased outcome remains problematic even when analysis is automated.

Machine-learning systems create additional challenges. Adaptive experimentation can change allocation probabilities as evidence accumulates; analyses must account for the adaptive design. Personalization can produce heterogeneous treatment effects, but subgroup discovery should be validated on new data. Synthetic controls, geo-experiments, and platform-level experiments are useful when individual randomization is impossible, though their assumptions differ from those of ordinary A/B tests.

Causal machine learning methods can estimate conditional treatment effects, but prediction accuracy is not the same as causal validity. A model may accurately predict outcomes while recommending an intervention whose effect has not been identified. Analysts should separate outcome prediction from treatment-effect estimation and communicate uncertainty about both.

Best-Practice Checklist

Before launching an experiment, ask:

1. What exact causal decision and estimand are being studied?
2. What is the unit of assignment, and can units interfere with one another?
3. Is randomization feasible, concealed, reproducible, and persistent?
4. Is the control condition appropriate to the decision?
5. Are primary outcomes, observation windows, and stopping rules specified?
6. Is the sample large enough for the minimum meaningful effect?
7. How will noncompliance, attrition, contamination, and missing data be handled?
8. Are costs, harms, privacy, fairness, and unintended outcomes monitored?
9. If the design is quasi-experimental, what identifying assumption supports causality?
10. Does the estimated effect generalize to the proposed deployment population?

Chapter Summary

Causal inference asks what would happen under alternative interventions and therefore depends on counterfactual reasoning. The potential-outcomes framework defines individual effects, ATE, ATT, and other estimands. Randomized experiments identify causal effects by assigning treatment independently of potential outcomes, while control groups provide the counterfactual comparison. A/B tests require clear units, eligibility rules, outcomes, exposure windows, sample-size planning, persistent assignment, and analysis based on assignment.

Internal validity can be threatened by noncompliance, contamination, interference, attrition, measurement differences, and optional stopping. External validity concerns generalization across populations and contexts. When randomization is unavailable, difference-in-differences, regression discontinuity, instrumental variables, and matching can provide useful evidence, but each depends on explicit assumptions. Confounding and endogeneity explain why correlation alone is insufficient for causal claims. Effective business analytics combines sound design, transparent analysis, ethical governance, and attention to economic as well as statistical significance.

Key Terms

Causal inference: Reasoning about the effects of interventions or treatments.

Counterfactual: The outcome that would have occurred under an alternative condition.

Treatment: An intervention, exposure, or condition whose effect is being studied.

Control group: A comparison group representing the relevant untreated or alternative condition.

Potential outcome: An outcome a unit would have under a specified treatment condition.

Individual treatment effect: The difference between a unit’s treated and untreated potential outcomes.

Average treatment effect (ATE): The average causal effect in a target population.

Average treatment effect on the treated (ATT): The average causal effect among treated units.

Estimand: The precisely defined causal quantity a study seeks to estimate.

Randomization: Assignment by a chance mechanism.

A/B test: A randomized comparison between two versions or conditions.

Intention-to-treat effect: The effect of assignment to treatment, regardless of compliance.

Confounder: A variable associated with both treatment and outcome that distorts their relationship.

Endogeneity: Dependence between a predictor and the disturbance or unobserved determinants of an outcome.

Internal validity: Credibility of a causal conclusion for the studied units and conditions.

External validity: Applicability of findings to other populations, settings, or times.

Difference-in-differences: A design comparing changes in treated and comparison groups over time.

Regression discontinuity: A design exploiting a treatment change at an assignment cutoff.

Instrumental variable: A variable that shifts treatment but affects the outcome only through treatment, under specified assumptions.

Interference: A condition in which one unit’s treatment affects another unit’s outcome.

Review Questions

Short Answer Questions

1. Why is a causal effect considered a counterfactual quantity?
2. Distinguish the ATE from the ATT.
3. What is the purpose of a control group in an experiment?
4. Explain why ITT analysis is commonly preferred in randomized trials.
5. State the parallel-trends assumption in difference-in-differences analysis.

Long Answer Questions

1. Explain how randomization addresses confounding and discuss why it does not guarantee identical treatment and control groups in every sample.
2. Design an A/B test for a subscription business, specifying the unit of randomization, treatment, control, primary outcome, observation window, and analysis plan.
3. Compare internal validity and external validity, giving examples of threats to each.
4. Explain how noncompliance, contamination, attrition, and interference can bias experimental conclusions.
5. Compare matching, instrumental variables, regression discontinuity, and difference-in-differences as alternatives to randomized assignment.

Analytical / Application Questions

1. A retailer observes that customers receiving discount codes spend less than customers who do not receive them. Identify at least three plausible sources of confounding and propose an experiment or quasi-experiment to estimate the discount’s causal effect.
2. In a randomized test, the treatment group’s conversion rate is 12% and the control group’s rate is 10%, with 5,000 customers in each group. Calculate the absolute difference and relative lift, then explain why managerial evaluation should include costs and longer-term outcomes.
3. A policy is introduced in treated branches but not comparison branches. Treated branches rise from 800 to 900 sales, while comparison branches rise from 780 to 820. Calculate the difference-in-differences estimate and state the key assumption required for a causal interpretation.

Multiple Choice Questions with Answers

Question 1

Which quantity represents the average causal effect in a target population?

A. The correlation between treatment and outcome

B. The average treatment effect

C. The treatment-group mean alone

D. The control-group variance

Correct Answer: B. The average treatment effect

Question 2

Why does randomization generally reduce confounding?

A. It guarantees no measurement error

B. It makes treatment assignment independent of potential outcomes in expectation

C. It ensures every individual has the same outcome

D. It removes the need for a control group

Correct Answer: B. It makes treatment assignment independent of potential outcomes in expectation

Question 3

A control conversion rate is 10% and a treatment conversion rate is 11%. What is the relative lift?

A. 1%

B. 10%

C. 11%

D. 21%

Correct Answer: B. 10%

Question 4

Which analysis estimates the effect of being assigned to treatment regardless of compliance?

A. Per-protocol analysis

B. Intention-to-treat analysis

C. Post-treatment matching

D. Placebo-only analysis

Correct Answer: B. Intention-to-treat analysis

Question 5

The key identifying assumption in a basic difference-in-differences design is:

A. Equal sample sizes at all dates

B. Perfect compliance with treatment

C. Parallel trends absent treatment

D. No outcome variation over time

Correct Answer: C. Parallel trends absent treatment

Question 6

A regression discontinuity design primarily identifies:

A. A global effect for every population

B. A local effect near the assignment cutoff

C. Only a correlation between variables

D. An effect without any assumptions

Correct Answer: B. A local effect near the assignment cutoff

Question 7

Which problem occurs when a control unit is exposed to the treatment?

A. Contamination

B. Blocking

C. Stratification

D. Power analysis

Correct Answer: A. Contamination

Question 8

For equal cluster sizes, if m = 10 and ICC ρ = 0.10, what is the approximate design effect?

A. 1.01

B. 1.10

C. 1.90

D. 10.00

Correct Answer: C. 1.90

Question 9

Why can controlling for a mediator be problematic?

A. It always increases sample size

B. It can remove part of the treatment effect being estimated

C. It guarantees randomization

D. It eliminates all measurement error

Correct Answer: B. It can remove part of the treatment effect being estimated

Question 10

An instrumental variable must, among other conditions, be:

A. Unrelated to treatment

B. Relevant to treatment and subject to an exclusion restriction

C. A post-treatment outcome

D. Perfectly correlated with the outcome directly

Correct Answer: B. Relevant to treatment and subject to an exclusion restriction