Experimentation, Causal Analytics and Impact Evaluation

Introduction and context

Business analytics often describes association: customers who receive an offer may purchase more, and hospitals with shorter waiting times may report higher satisfaction. Association alone, however, does not establish that the offer caused the purchase or that shorter waits caused satisfaction. Decisions about pricing, marketing, product design, public policy and operations require estimates of causal effects—the change in an outcome attributable to an intervention, relative to a credible alternative in which the intervention did not occur.

Experimentation creates this alternative through random assignment. When randomization is impossible, analysts use quasi-experimental designs, such as matching and difference-in-differences, together with explicit assumptions and sensitivity analysis. This chapter develops the logic of causal inference, the design and analysis of A/B tests, causal diagrams, treatment-effect estimation, and practical impact evaluation.

Learning objectives

By the end of this chapter, you should be able to:

1. Distinguish correlation, prediction and causation.
2. Define treatment, control, outcomes, estimands and counterfactuals.
3. Design and analyze randomized A/B tests, including sample-size reasoning and uncertainty.
4. Use causal diagrams to identify confounding, mediation and inappropriate controls.
5. Explain matching and difference-in-differences, including their assumptions and limitations.
6. Interpret heterogeneous treatment effects and translate estimates into decisions.
7. Recognize ethical, privacy, operational and governance requirements for experimentation.

1. The causal question: potential outcomes and counterfactuals

Let unit i be a customer, employee, store, patient or region. Let Y_i(1) denote the outcome that would occur if unit i received treatment, and Y_i(0) the outcome that would occur under control. The individual treatment effect is

The two potential outcomes cannot normally be observed for the same unit at the same time. This is the fundamental problem of causal inference. An experiment estimates an average effect by creating groups that are comparable in expectation.

The average treatment effect (ATE) is

where E denotes the population mean. The average treatment effect on the treated (ATT) is

where T equals 1 for treated units and 0 otherwise. ATE is relevant when considering broad rollout; ATT is often more relevant when evaluating an intervention already adopted by a particular group. These definitions require a clearly defined population, treatment, outcome, timing and follow-up period.

A causal estimate should be distinguished from a descriptive difference:

This difference equals a causal effect only under conditions that make the treated and control groups comparable. For example, ambitious customers may both adopt a premium service and spend more, creating confounding.

2. Causal assumptions and diagrams

A causal analysis begins with a substantive model, not with software. A directed acyclic graph (DAG) represents proposed directional relationships. Consider an advertising example:

Figure: Customer intent confounds the relationship between advertising and purchase; engagement is a possible mediator.

A confounder affects both treatment and outcome. Adjusting for measured confounders can reduce bias. A mediator lies on the causal pathway from treatment to outcome. Controlling for a mediator estimates a direct effect rather than the total effect and may remove part of the effect of interest. A collider is a common consequence of two variables; conditioning on it can create a spurious association. Thus, “control for everything” is not a valid causal strategy.

Three central identification conditions are commonly stated as follows:

• Consistency: the observed outcome under the treatment received corresponds to the relevant potential outcome, and treatment versions are sufficiently well defined.
• Exchangeability: conditional on measured covariates X, potential outcomes are independent of treatment: Y(1),Y(0) ⟂ T | X. In observational studies this is also called no unmeasured confounding.
• Positivity: every relevant covariate pattern has a positive probability of receiving each treatment: 0 < P(T=1|X=x) < 1.

Randomization supports exchangeability in expectation without requiring all confounders to be measured. It does not automatically solve noncompliance, attrition, interference between units or poor outcome measurement.

3. Designing randomized experiments and A/B tests

An A/B test randomly assigns eligible units to version A (control) or version B (treatment). Before launch, specify the unit of randomization, eligibility rules, intervention, primary outcome, observation window, exclusion rules, analysis population and decision threshold. Pre-registration or an analysis plan reduces selective reporting and repeated experimentation.

Randomization and operational design

Randomize at the level at which treatment is delivered and contamination is limited. If a promotion is delivered to an account, account-level assignment is usually preferable to session-level assignment. If employees in one team influence one another, cluster randomization by team may be needed, although it reduces statistical efficiency because outcomes within a cluster are correlated.

Use a stable assignment mechanism, such as a reproducible hash of a customer identifier. Confirm balance on important pre-treatment variables, but do not repeatedly rerandomize until a preferred balance appears. Balance is expected probabilistically, not guaranteed exactly.

The primary estimand might be an intention-to-treat effect (the effect of being assigned to treatment), a per-protocol effect (the effect among those who comply with the protocol), or a treatment-on-the-treated effect. Intention-to-treat is usually the most credible operational effect because it preserves randomization, even when some users do not engage.

Estimating the effect

For a continuous outcome, the difference in sample means is

where Ȳ₁ and Ȳ₀ are treatment and control sample means. For a binary outcome, this is an absolute percentage-point effect. Relative lift is

where p₁ and p₀ are treatment and control rates and p₀ > 0. Absolute and relative effects should both be reported because a large relative lift may correspond to a small practical change when the baseline rate is low.

For independent observations, an approximate standard error for the difference in two means is

where s₁² and s₀² are sample variances and n₁, n₀ are sample sizes. A 95% large-sample confidence interval is

The approximation is most defensible when observations are independent, sample sizes are adequate and the estimator has an approximately normal sampling distribution. Clustered assignment requires cluster-robust methods or cluster-level analysis; treating every individual as independent can understate uncertainty.

For a binary outcome, an approximate standard error for the difference in proportions is

where p̂₁ and p̂₀ are observed proportions. Exact or regression-based methods may be preferable for rare outcomes or small samples.

Worked example: conversion experiment

An online retailer randomly assigns 10,000 visitors to a redesigned checkout and 10,000 to the existing checkout. There are 1,200 purchases in treatment and 1,100 in control. Therefore, p̂₁ = 0.120 and p̂₀ = 0.110. The estimated effect is 0.010, or 1.0 percentage point. Relative lift is 0.010 ÷ 0.110 = 0.0909, approximately 9.1%.

The standard error is

The approximate 95% interval is 0.010 ± 1.96(0.00457), or approximately [0.0010, 0.0190]. The interval barely excludes zero under this approximation. A responsible conclusion is that the redesign is associated with an estimated 1.0 percentage-point increase, with substantial uncertainty; business impact, implementation cost and effects on returns or customer support must also be assessed.

Power, precision and sequential decisions

Sample size should be based on a minimum detectable effect (MDE), baseline variance or conversion rate, desired power and significance level. Power is the probability of detecting an effect of a specified size under a chosen design; it is not the probability that a statistically significant result is true. Larger samples increase precision, but they do not correct systematic bias.

Repeatedly checking results and stopping when a p-value crosses 0.05 inflates the false-positive rate. Use a fixed stopping rule, alpha-spending method, group-sequential design or Bayesian decision protocol established in advance. Correct for multiple comparisons when many outcomes, segments or variants are explored. The primary outcome should be distinguished from exploratory findings.

4. Beyond the simple A/B test

Regression adjustment and covariate pre-treatment

In a randomized experiment, regression adjustment can improve precision:

Here Yᵢ is the outcome, Tᵢ is treatment assignment, Xᵢ is a vector of pre-treatment covariates, α is an intercept, β is a coefficient vector and εᵢ is an error term. The coefficient τ estimates an adjusted treatment contrast under suitable model specification. Covariates measured after treatment should not be added merely to improve fit, because they may be mediators or colliders.

Heterogeneous treatment effects

An average effect may conceal important differences. Estimate effects by pre-specified segments such as customer tenure or region, but avoid interpreting random noise as a segment strategy. An interaction model can be written as

where Gᵢ identifies a subgroup and δ represents the difference in treatment effects between groups. Heterogeneity claims require adequate sample size, credible theory and correction for extensive subgroup searching.

Interference and network effects

The standard potential-outcomes framework often assumes that one unit’s treatment does not affect another unit’s outcome, known as the stable unit treatment value assumption. Referral programs, social advertising and contagious products violate this assumption. Designs may need geographic clusters, network-aware randomization or estimands that explicitly include spillovers.

5. Quasi-experimental impact evaluation

Quasi-experiments are used when randomization is infeasible, unethical or not implemented. Their credibility comes from design and assumptions rather than from statistical sophistication alone.

Matching and propensity scores

Matching constructs a comparison group with similar observed pre-treatment characteristics. The propensity score is

where X contains measured covariates. Methods include exact matching, nearest-neighbour matching, propensity-score matching, inverse probability weighting and coarsened exact matching. After matching or weighting, assess covariate balance using standardized mean differences rather than relying only on p-values.

Matching identifies a causal effect only under conditional exchangeability, positivity and consistency. It cannot remove bias from unmeasured confounders. Poor overlap means that treated units have no credible controls; extrapolation should then be avoided. Matching can also increase variance if many observations are discarded or extreme weights are used.

Difference-in-differences

Difference-in-differences (DiD) compares changes over time in a treated group with changes in a comparison group. With two groups and two periods:

The key identifying assumption is parallel trends: absent treatment, the treated and comparison groups would have experienced the same average change. This assumption cannot be proven from post-treatment data. Examine several pre-treatment periods, plot trends and test whether differences are substantively plausible; pre-trend tests alone are not sufficient proof.

A basic regression form is

where δ is the DiD effect, i indexes units and t indexes time. Standard errors should account for serial correlation and the level of treatment assignment, often using clustered standard errors. With staggered adoption across units, conventional two-way fixed-effects estimators can be misleading when treatment effects vary over time; event-study or modern group-time estimators may be more appropriate.

Regression discontinuity and instrumental variables

A regression discontinuity design compares units just above and below a treatment threshold. It requires continuity of potential outcomes at the cutoff and no precise manipulation of the running variable. Its effect is local to the threshold.

An instrumental-variable design uses an instrument that affects treatment but affects the outcome only through treatment and is sufficiently strong. Under additional assumptions, it estimates a local average treatment effect for compliers, not necessarily the effect for everyone. These methods are powerful but require domain-specific justification; they should not be selected merely because they produce a desired result.

6. From effect estimate to business decision

Statistical significance is not economic significance. Convert effects into outcomes that decision-makers understand. If a campaign increases conversion by Δp among N eligible customers and contribution margin per incremental purchase is m, estimated incremental contribution is

This calculation assumes the estimated effect transports to the rollout population, the margin is measured consistently and costs are included. Include uncertainty by evaluating the financial result across the confidence interval or a decision-relevant probability distribution.

A useful evaluation report includes the estimand, population, treatment contrast, outcome definition, assignment mechanism, analysis sample, attrition, estimate, uncertainty interval, assumptions, subgroup findings, adverse outcomes and implementation recommendation. Report harms and secondary outcomes, not only the metric that improved.

7. Illustrative case: a delivery-platform service guarantee

Background. A fictional delivery platform considers guaranteeing a credit when orders arrive more than 45 minutes after promised time. Managers believe the guarantee may increase repeat orders but may also increase compensation costs and encourage risky promises.

Problem. The platform cannot simply compare customers who receive credits with those who do not: late orders are more likely to receive credits, and customers experiencing delays may differ in satisfaction and future demand.

Analysis and approach. The platform randomly assigns eligible orders, before delivery, to either the guarantee policy or the current policy. The primary outcome is repeat order within 30 days. Secondary outcomes are cancellation, support contact, delivery cost and customer complaints. Assignment is at the customer-order level, but interference is monitored because courier capacity may be shared across orders. The analysis follows intention-to-treat. A pre-specified subgroup analysis examines new versus existing customers.

After eight weeks, the guarantee group has a repeat-order rate of 18.4% and the control group 17.6%, an estimated increase of 0.8 percentage points. The finance team estimates contribution margin of $6 per incremental repeat order and average incremental compensation and administration cost of $0.20 per eligible order. For 100,000 eligible orders, the estimated incremental contribution is

The estimated behavioral effect is positive, but the program is not profitable under these assumptions. Management therefore considers a narrower guarantee for high-value orders, while monitoring whether targeting changes the causal effect and whether customer fairness is compromised.

Learning points. Randomization separated the policy effect from the fact that late orders are intrinsically different. The decision still required cost outcomes, not merely a statistically favorable customer metric. Operational interference, heterogeneous effects and the possibility of behavioral adaptation required continued evaluation.

8. Ethics, governance and contemporary practice

Experimentation must respect informed consent requirements where applicable, organizational policies and sector regulation. Minimize collection of personal data, use access controls, document retention periods and protect sensitive attributes. Do not expose participants to unreasonable risk or deliberately withhold legally required services. In employment, health, finance and education, fairness and power imbalances demand additional review.

Algorithms can automate assignment, targeting and heterogeneous-effect estimation, but automation does not establish causality. Machine-learning models may improve nuisance prediction or discover candidate subgroups, yet flexible search increases overfitting and false discoveries. Use holdout data, cross-validation where appropriate, pre-specified confirmatory analyses and human review. Monitor model drift and treatment effects after deployment: an intervention that worked in one market or period may not transport to another.

Best practice is to maintain an experiment registry, version treatment code, log eligibility and assignment, freeze primary metrics before peeking, monitor data quality and perform post-launch audits. When experimentation is impossible, use a transparent quasi-experimental design, disclose assumptions and conduct sensitivity analyses for unmeasured confounding, alternative windows, placebo outcomes and alternative comparison groups.

Challenges and limitations

Causal estimates can fail because of noncompliance, attrition, missing data, contamination, spillovers, interference, measurement changes, novelty effects and selective rollout. External validity is also limited: an internally valid experiment may not generalize across seasons, customer segments or countries. Statistical adjustment cannot rescue a poorly defined treatment or an outcome that does not represent the decision objective. Finally, causal analysis is partly empirical and partly substantive: institutional knowledge is needed to decide whether assumptions are plausible.

Chapter Summary

Causal analytics asks what would have happened under a credible alternative intervention. Potential outcomes define individual and average treatment effects, while randomized assignment supports comparability and makes A/B tests the strongest routine design for many business decisions. Sound experiments require an appropriate unit of randomization, pre-specified outcomes, adequate power, reliable implementation, uncertainty estimates and protection against repeated-testing errors.

Causal diagrams clarify confounders, mediators and colliders and prevent indiscriminate adjustment. Matching can balance observed covariates but cannot solve unmeasured confounding. Difference-in-differences relies primarily on a defensible parallel-trends assumption and careful treatment of timing and clustered uncertainty. Regression discontinuity and instrumental variables identify local effects under specialized assumptions. Ultimately, effect estimates must be combined with costs, benefits, harms, fairness, operational constraints and transportability before action is taken.

Key Terms

A/B test: Randomized comparison of two versions or policies.

Average treatment effect (ATE): Mean causal effect in a defined target population.

Average treatment effect on the treated (ATT): Mean causal effect among treated units.

Causal effect: Change in an outcome attributable to an intervention relative to a counterfactual.

Collider: A variable caused by two other variables; conditioning on it may create bias.

Confounder: A variable related to both treatment and outcome that distorts their association.

Counterfactual: The outcome that would have occurred under an alternative treatment condition.

Difference-in-differences: Comparison of outcome changes across treated and comparison groups.

Estimand: Precisely defined causal quantity targeted by an analysis.

Exchangeability: Comparability of potential outcomes across treatment conditions, conditional on relevant covariates.

Heterogeneous treatment effect: Variation in causal effect across units or subgroups.

Intention-to-treat: Effect of assignment to an intervention, regardless of compliance.

Mediator: Variable lying on the causal pathway from treatment to outcome.

Minimum detectable effect: Smallest effect a design is intended to detect with specified power.

Parallel trends: DiD assumption that groups would have changed similarly without treatment.

Positivity: Every relevant covariate pattern has a positive chance of each treatment.

Potential outcome: Outcome a unit would experience under a specified treatment condition.

Propensity score: Probability of treatment conditional on observed covariates.

Randomization: Assignment mechanism giving units known treatment probabilities.

Statistical power: Probability of detecting a specified effect under a chosen design.

Review Questions

Short Answer Questions

1. Why does correlation between treatment and outcome not necessarily establish causation?
2. Define ATE and ATT and explain when each may be useful.
3. What is the purpose of randomization in an A/B test?
4. Distinguish a confounder from a mediator.
5. State the parallel-trends assumption in difference-in-differences.

Long Answer Questions

1. Explain how potential outcomes, counterfactuals and random assignment support causal inference.
2. Design an A/B test for a new customer-retention intervention, including unit of randomization, outcome, analysis population and monitoring plan.
3. Compare matching and randomized experiments in terms of assumptions, strengths and limitations.
4. Explain why controlling for every available variable can create causal bias.
5. Discuss how statistical significance, economic significance and ethical considerations should jointly inform rollout decisions.

Analytical / Application Questions

1. A treatment group has 2,400 conversions among 20,000 users and control has 2,200 among 20,000 users. Calculate the absolute effect and relative lift, and explain what additional information is needed before rollout.
2. A policy is adopted in one region while a similar region remains untreated. Pre-policy outcomes are diverging rather than parallel. Assess whether DiD is credible and propose an alternative evaluation strategy.
3. An experiment increases purchases but also increases complaints and refund requests. Develop a decision framework that incorporates primary outcomes, adverse outcomes, costs, uncertainty and subgroup fairness.

Multiple Choice Questions with Answers

Question 1

Which quantity represents the average treatment effect?

A. The observed mean outcome among treated units only

B. E[Y(1) − Y(0)] in the target population

C. The correlation between treatment and outcome

D. The probability of treatment given covariates

Correct Answer: B. E[Y(1) − Y(0)] in the target population

Question 2

What is the main purpose of random assignment in an A/B test?

A. To guarantee that every individual treatment effect is identical

B. To eliminate all sampling variation

C. To create treatment groups that are comparable in expectation

D. To ensure the treatment has a positive effect

Correct Answer: C. To create treatment groups that are comparable in expectation

Question 3

Which variable is a mediator?

A. A pre-treatment factor affecting both treatment and outcome

B. A variable caused by treatment that lies on the pathway to the outcome

C. A variable unrelated to treatment and outcome

D. A variable used only to calculate sample size

Correct Answer: B. A variable caused by treatment that lies on the pathway to the outcome

Question 4

In difference-in-differences, the key identifying assumption is usually:

A. Treatment and control groups have identical outcomes at every time

B. The treatment effect is zero before treatment

C. Treated and comparison groups would have followed parallel trends without treatment

D. All observations are independently and identically distributed

Correct Answer: C. Treated and comparison groups would have followed parallel trends without treatment

Question 5

A control conversion rate is 0.10 and a treatment conversion rate is 0.12. What is the relative lift?

A. 0.02%

B. 2%

C. 12%

D. 20%

Correct Answer: D. 20%

Question 6

Which condition means that every relevant covariate pattern has a positive probability of receiving each treatment?

A. Consistency

B. Positivity

C. Interference

D. Mediation

Correct Answer: B. Positivity

Question 7

Why can conditioning on a collider introduce bias?

A. It can create an association between variables that were otherwise independent

B. It always increases sample size

C. It guarantees parallel trends

D. It converts an observational study into a randomized experiment

Correct Answer: A. It can create an association between variables that were otherwise independent

Question 8

What does an intention-to-treat estimate measure?

A. The effect among only those who complied perfectly

B. The effect of assignment to treatment, regardless of compliance

C. The effect after excluding all nonresponders

D. The effect for a subgroup selected after seeing outcomes

Correct Answer: B. The effect of assignment to treatment, regardless of compliance

Question 9

Which issue is matching unable to solve by itself?

A. Differences in measured pre-treatment covariates

B. Lack of overlap in observed propensity scores

C. Confounding from important unmeasured variables

D. Unequal observed sample sizes

Correct Answer: C. Confounding from important unmeasured variables

Question 10

Why should repeated checking of an experiment’s p-value be governed by a pre-specified rule?

A. To increase the treatment effect

B. To prevent inflation of the false-positive rate

C. To guarantee external validity

D. To eliminate the need for a control group

Correct Answer: B. To prevent inflation of the false-positive rate