9. Experimental Design, A/B Testing and Causal Inference

Introduction and Context

Data science is often presented as a discipline for prediction: estimating whether a customer will churn, forecasting demand, or classifying an image. Yet many of the most consequential decisions are causal. A product team wants to know whether a redesigned checkout page increases purchases; a public agency asks whether a training programme improves employment; a hospital evaluates whether a new treatment reduces readmissions. In each case, the central question is not merely what is associated with an outcome, but what would happen if a particular intervention were applied.

Causal questions are difficult because the same unit cannot ordinarily be observed both with and without treatment at the same time. For a given individual, we observe either the outcome under treatment or the outcome under control, but not both. Experimental design provides a disciplined way to approximate this missing counterfactual. Randomized experiments, including online A/B tests, use deliberate assignment to create comparable groups. When randomization is impossible, causal inference relies on carefully justified assumptions and quasi-experimental designs.

This chapter develops the principles required to design, analyse, and critique experiments. It connects classical analysis of variance (ANOVA) with modern experimentation, explains factorial and blocked designs, and introduces potential-outcomes reasoning, confounding, selection bias, and causal graphs. The emphasis is practical: a statistically significant result is not automatically a useful or credible causal conclusion.

Learning Objectives

By the end of this chapter, you should be able to:

1. Distinguish descriptive, predictive, and causal questions.

2. Explain counterfactuals, treatment effects, estimands, and randomization.

3. Design randomized, blocked, and factorial experiments.

4. Analyse treatment effects using means, confidence intervals, hypothesis tests, and ANOVA.

5. Design and interpret online A/B tests, including metrics, power, interference, and sequential monitoring.

6. Identify confounding, selection bias, attrition, measurement problems, and threats to validity.

7. Explain quasi-experimental approaches such as difference-in-differences, regression discontinuity, instrumental variables, and interrupted time series.

8. Evaluate causal claims using explicit assumptions, sensitivity analysis, and ethical standards.

From Association to Intervention

An association describes how variables vary together. If users who receive more support tickets have lower retention, this relationship may be real, but it does not establish that reducing support tickets would improve retention. The relationship may reflect reverse causality, omitted variables, or selection. For example, dissatisfied customers may both contact support frequently and be more likely to leave.

A causal effect is defined through a comparison with a counterfactual. Let \(Y_i(1)\) be the outcome for unit \(i\) if it receives treatment and \(Y_i(0)\) the outcome if it receives control. The individual treatment effect is

\[

\tau_i=Y_i(1)-Y_i(0).

\]

Only one of these two potential outcomes is observed. This is the fundamental problem of causal inference. Common population-level estimands include the average treatment effect (ATE),

\[

ATE=E[Y(1)-Y(0)],

\]

and the average treatment effect on the treated (ATT),

\[

ATT=E[Y(1)-Y(0)\mid T=1].

\]

The ATE asks what would happen if treatment were assigned to the target population. The ATT asks about those who actually received treatment. In practice, the estimand should be specified before analysis because a result can be accurate for one population and misleading for another.

A causal claim requires more than a difference in observed averages. It requires a design or set of assumptions that makes the treated and untreated outcomes comparable. Randomization is powerful because, in expectation, it balances both observed and unobserved characteristics between groups.

Designing a Credible Experiment

A sound experiment begins with a clearly defined unit of randomization: an individual, household, transaction, school, geographic region, or user account. The unit receiving treatment may differ from the unit at which outcomes are measured. For example, a marketing campaign may be assigned by household while purchases are measured for individuals. Such clustering affects standard errors and must be reflected in the analysis.

The treatment should be operationally precise. “Improved service” is not an adequate treatment definition; “a response within two hours using the revised script” is more reproducible. The control condition must also be specified. It may be business as usual, a placebo, or an alternative intervention. A well-defined primary outcome and a pre-established observation window prevent researchers from selecting favourable outcomes after examining results.

Random assignment can be represented by a treatment indicator \(T_i\), where \(T_i=1\) denotes treatment and \(T_i=0\) denotes control. Under complete randomization, every permissible assignment has a known probability. Under simple randomization, units are assigned independently, commonly using a random number generator. In restricted randomization, such as permuted blocks, the allocation is constrained to maintain balance.

Randomization does not automatically guarantee a useful experiment. The sample must be large enough to detect an effect that matters. Power depends on the significance level, desired power, outcome variability, treatment allocation, and minimum detectable effect. For a continuous outcome, greater variance or a smaller target effect requires a larger sample. For binary outcomes, baseline conversion rates and the practical meaning of a change are central to planning.

A confidence interval communicates both the estimated effect and its uncertainty. A narrow interval around a small effect may be more useful than a statistically significant but imprecise estimate. Statistical significance should therefore be separated from practical significance. A change can be detectable but economically irrelevant, or valuable but difficult to detect with a small sample.

Blocking and Stratification

Blocking improves precision when units can be grouped into relatively homogeneous blocks before treatment assignment. Suppose an online retailer expects treatment effects to differ by country and baseline activity. It can form blocks using country and activity category, then randomize treatment within each block. Comparisons are made among similar units rather than across highly heterogeneous groups.

Blocking is particularly valuable when a prognostic variable strongly predicts the outcome. It also protects against accidental imbalance in small samples. However, blocks must be defined before treatment and should not be based on post-treatment information. Excessive blocking can create very small cells and complicate implementation.

In analysis, a blocked experiment may be represented as

\[

Y_{ij}=\mu+\tau T_{ij}+\beta_j+\varepsilon_{ij},

\]

where \(\beta_j\) represents the effect of block \(j\). The treatment coefficient estimates the treatment difference after accounting for systematic block variation. Covariate adjustment can similarly improve precision in a randomized experiment, provided it is based on pre-treatment variables.

Factorial Experiments

A factorial design studies two or more factors simultaneously. In a 2 × 2 experiment, factor A might be email subject line and factor B might be delivery time. Participants are assigned to four combinations: A1B1, A1B2, A2B1, and A2B2. This design estimates two main effects and their interaction.

The main effect of A is the average difference between A1 and A2 across levels of B. The interaction effect asks whether the effect of A changes depending on B. For example, a concise subject line may improve open rates for morning delivery but reduce them for evening delivery. Examining only average effects would conceal this interaction.

Factorial designs are efficient because one experiment can address several decisions. Their main limitation is interpretive complexity, particularly when there are many factors or when interactions are substantial. Researchers should specify plausible interactions in advance and avoid treating every subgroup pattern as confirmatory evidence.

ANOVA as a Framework for Experimental Analysis

Analysis of variance decomposes outcome variability into systematic components attributable to experimental factors and residual variation. In a one-way ANOVA with \(k\) groups, the total sum of squares is partitioned as

\[

SS_T=SS_B+SS_W,

\]

where \(SS_B\) is between-group variation and \(SS_W\) is within-group variation. The F statistic compares the mean square between groups with the mean square within groups:

\[

F=\frac{MS_B}{MS_W}.

\]

The null hypothesis is that all group means are equal. A significant omnibus F test indicates that at least one mean differs, but it does not identify which groups differ. Planned contrasts or appropriately adjusted post-hoc comparisons are then required.

In a randomized experiment with two groups, the difference in means and the corresponding t test are equivalent to a one-factor ANOVA. ANOVA becomes especially useful for factorial designs, blocked experiments, and models containing covariates. Its assumptions include independent errors, appropriate treatment of the outcome scale, and reasonably stable variance. Robust standard errors, transformations, generalized linear models, or random-effects models may be preferable when these assumptions are doubtful.

An important distinction is between experimental factors and nuisance variables. A nuisance variable is not the principal focus but explains outcome variation, such as site, batch, day, or machine. Blocking and covariate adjustment account for nuisance variation without confusing it with treatment effects.

Online A/B Testing in Practice

An A/B test is a randomized experiment in which users are exposed to version A or version B of a product, message, algorithm, or interface. A typical workflow is:

1. State a decision-oriented hypothesis and primary metric.

2. Define eligibility, randomization unit, treatment variants, and exposure rules.

3. Estimate sample size and test duration.

4. Implement assignment using a stable randomization mechanism.

5. Verify balance, exposure, logging, and data quality.

6. Analyse the pre-specified primary outcome and relevant secondary outcomes.

7. Interpret effect size, interval, heterogeneity, costs, and risks.

8. Decide whether to launch, revise, continue, or reject the intervention.

The randomization unit must prevent contamination. If users share an account or influence one another, individual-level assignment may allow treatment to spill into the control group. Cluster randomization, in which entire teams, stores, or communities are assigned, may be more appropriate, although it usually requires a larger sample.

Metrics should correspond to the decision. A product change may increase click-through rate while decreasing completed purchases or increasing customer complaints. A guardrail metric, such as latency or cancellation rate, can prevent a narrow primary metric from masking harm. Ratio metrics require care because the numerator and denominator may be correlated; user-level aggregation or suitable regression methods are often preferable to treating every event as independent.

Repeated peeking at results can inflate the false-positive rate if ordinary fixed-sample tests are used. This does not mean that monitoring is prohibited. Sequential testing procedures, alpha-spending methods, or Bayesian decision rules can support continuous monitoring when specified appropriately. Stopping because an early estimate happens to be favourable is not equivalent to a planned stopping rule.

Multiple comparisons create another risk. Testing many metrics, segments, and variants increases the chance of finding a seemingly positive result by chance. Researchers should identify a primary outcome, control the family-wise error rate or false discovery rate when appropriate, and label exploratory findings clearly.

Internal and External Validity

Internal validity concerns whether the estimated effect is credible for the units and conditions studied. Randomization supports internal validity, but it can be weakened by noncompliance, differential attrition, faulty exposure, interference, or informative missingness. An intention-to-treat analysis estimates the effect of assignment, preserving the benefits of randomization. A per-protocol analysis estimates the effect among those who followed the protocol but may be confounded because compliance is not random.

External validity concerns generalization. An experiment among frequent users during a holiday period may not represent new users or ordinary weeks. Treatment effects can vary by population, setting, implementation quality, and competing interventions. Generalization should therefore be justified rather than assumed. Replication across relevant contexts and analysis of pre-specified effect modifiers improve confidence.

Heterogeneous treatment effects are scientifically and managerially important. An average positive effect may conceal harm for a subgroup. Conversely, a near-zero average may result from strong benefits for some users and harms for others. Subgroup analysis should use credible pre-treatment variables, adequate sample sizes, interaction models, and cautious interpretation.

When Randomization Is Not Available

Observational data do not provide random treatment assignment, but several designs can approximate a counterfactual under explicit assumptions.

Difference-in-differences compares changes over time in a treated group with changes in a comparison group:

\[

\hat\tau_{DiD}=(\bar Y_{T,after}-\bar Y_{T,before})-(\bar Y_{C,after}-\bar Y_{C,before}).

\]

Its central assumption is parallel trends: absent treatment, the treated and comparison groups would have changed similarly. Pre-treatment trends can provide evidence, but cannot prove this assumption. Anticipation, concurrent policies, and changing group composition threaten validity.

Regression discontinuity exploits a treatment rule based on a cutoff, such as eligibility above a score of 70. Units just above and below the threshold may be comparable. The effect is local to the cutoff, and credibility requires that units cannot precisely manipulate the assignment variable and that other outcomes do not jump at the threshold.

Instrumental variables use a variable that affects treatment but influences the outcome only through treatment. For example, an assignment lottery may serve as an instrument for programme participation. The exclusion restriction is strong and often difficult to defend. Under additional assumptions, the resulting estimate commonly applies to compliers rather than the entire population.

Interrupted time series examines whether the level or slope of an outcome changes after an intervention. It requires sufficient pre-intervention data and attention to seasonality, autocorrelation, concurrent shocks, and changes in measurement.

Matching and propensity-score methods can balance measured covariates, but they do not remove bias from unmeasured confounders. A propensity score is a design aid, not a guarantee of causality. Regression adjustment, weighting, matching, and doubly robust estimators all depend on assumptions about treatment assignment, overlap, and outcome modelling.

Causal Graphs and Confounding

A directed acyclic graph (DAG) makes causal assumptions visible. A confounder is a common cause of treatment and outcome. For example, customer tenure may influence both eligibility for a retention offer and the probability of renewal. Failing to adjust for tenure can create a spurious treatment association.

A backdoor path is a non-causal path connecting treatment and outcome. Adjustment should block relevant backdoor paths, usually by conditioning on pre-treatment confounders. Researchers should not adjust automatically for every available variable. A mediator lies on the causal pathway from treatment to outcome; controlling for it may remove part of the total treatment effect. A collider is caused by two variables; conditioning on it can create an association that was not previously present.

This framework clarifies why post-treatment variables are dangerous. If treatment changes engagement and engagement affects retention, controlling for engagement estimates a direct effect rather than the total effect and may introduce bias if engagement is measured with error or affected by other causes.

Illustrative Case Study: A Delivery Platform Experiment

Background. A fictional food-delivery platform considers a redesigned order-tracking screen. Product managers believe that clearer status information will reduce customer support contacts and increase repeat orders.

Situation and problem. The initial observational comparison shows that users who viewed the new screen had higher repeat-order rates. However, the new screen was initially enabled for highly engaged users, who were already more likely to return. The product team must determine whether the screen itself causes improvement.

Approach. The team defines the account as the randomization unit and assigns eligible accounts to the existing screen or the new screen using a persistent random hash. The primary outcome is repeat ordering within 30 days. Secondary outcomes are support contacts, delivery-related complaints, and app latency. Support contacts are designated as a guardrail because reducing contact by making help harder to find would be unacceptable.

Before launch, the team conducts a power analysis based on the historical repeat-order rate and a minimum effect considered commercially meaningful. Randomization is stratified by country and prior-month order frequency. Assignment remains fixed for the test period to avoid users experiencing both versions. Instrumentation records assignment, exposure, order events, support contacts, and latency.

Analysis. The intention-to-treat difference in repeat-order rates is estimated with a confidence interval. A regression model adjusts for the pre-treatment blocking variables to improve precision. The team checks balance and exposure rates, but does not remove users simply because they did not click the screen; doing so would change the estimand and potentially introduce selection bias. A factorial extension is considered for a future test combining the tracking screen with proactive delay notifications, because the effects may interact.

Outcome. Suppose the new screen produces a modest increase in repeat ordering with a confidence interval excluding zero, while support contacts fall and latency remains unchanged. The result supports deployment for the tested population. However, the team reports that the experiment does not establish the effect for users excluded from the test or for markets with different delivery processes. It also schedules monitoring after rollout because novelty effects and operational changes may alter the result.

Learning points. The case illustrates why causal evidence requires more than an observed association. Clear estimands, persistent randomization, pre-treatment stratification, guardrail metrics, intention-to-treat analysis, and attention to generalization are all part of responsible experimentation.

Common Failure Modes and Professional Judgement

A frequent error is randomizing after treatment has already been selectively offered. This creates the appearance of experimentation without its protection against confounding. Another is randomizing by event while analysing as if events were independent, even though many events come from the same user. Standard errors then become too small.

Attrition is problematic when dropout differs by treatment and is related to potential outcomes. Missing outcomes should be investigated rather than silently discarded. Noncompliance requires distinguishing assignment effects from treatment-received effects. Spillovers violate the assumption that one unit’s treatment does not affect another unit’s outcome, often called the stable unit treatment value assumption (SUTVA).

Analysts should avoid changing the primary metric, eligibility criteria, or stopping rule after seeing results without clearly labelling the analysis as exploratory. A statistically significant result can be produced through extensive searching. Conversely, failure to reject the null hypothesis is not proof that the treatment has no effect; it may reflect low power or a wide confidence interval.

Ethically, experiments require proportionality, respect for privacy, and protection from avoidable harm. Users should not be exposed to serious risk merely to improve a small business metric. Sensitive interventions may require informed consent, independent review, or regulatory approval. Data minimization, access control, retention limits, and transparent governance are essential when experimentation uses behavioural or personal data. Automated experimentation systems should preserve audit logs, version assignments, and reproducible analysis code.

Emerging Practice in Data-Driven Experimentation

Modern organizations increasingly use experimentation platforms that automate randomization, exposure logging, metric computation, and monitoring. Machine-learning systems can assist with heterogeneous treatment-effect estimation, adaptive allocation, and experiment prioritization. These tools can improve efficiency but also introduce risks: adaptive algorithms may complicate inference, optimization may favour short-term metrics, and black-box subgroup findings may be unstable.

Causal machine learning is most useful when it complements, rather than replaces, design. Double machine learning, causal forests, Bayesian hierarchical models, and uplift modelling can estimate treatment heterogeneity while controlling complex covariates. Their credibility still depends on data quality, overlap, treatment definition, and assumptions about confounding. The practical standard remains clear: state the causal question, make assumptions explicit, design comparison groups carefully, and communicate uncertainty.

Best Practices for Experimenters

A defensible experimental programme should:

• define the estimand, unit, treatment, control, population, and time horizon before data collection;
• randomize with a reproducible mechanism and prevent contamination where possible;
• use blocking or covariate adjustment based only on pre-treatment information;
• select one primary outcome and meaningful guardrail metrics;
• conduct power and minimum-detectable-effect planning;
• monitor data quality without repeatedly applying unplanned significance tests;
• report effect sizes and confidence intervals, not only p-values;
• distinguish intention-to-treat, per-protocol, and treatment-on-the-treated interpretations;
• investigate missingness, noncompliance, interference, and heterogeneous effects;
• document deviations, exploratory analyses, and limitations;
• assess ethical, privacy, operational, and generalization consequences before deployment.

Chapter Summary

Experimental design provides the foundation for credible causal claims. Potential outcomes define treatment effects through counterfactual comparisons, while randomization makes treatment groups comparable in expectation. Blocking improves precision and protects balance; factorial designs estimate main effects and interactions; ANOVA decomposes variation and supports multi-factor analysis. Online A/B testing extends these principles to digital products but requires careful attention to randomization units, metrics, power, contamination, sequential monitoring, and multiple comparisons.

When randomization is unavailable, difference-in-differences, regression discontinuity, instrumental variables, interrupted time series, matching, and adjustment can provide evidence under explicit assumptions. Causal graphs help distinguish confounders, mediators, and colliders. Across all designs, validity depends on implementation, measurement, missing data, compliance, interference, and the match between the target population and the study sample. Responsible data science treats causal inference as a combination of design, statistical analysis, domain knowledge, transparency, and ethical judgement.

Key Terms

Counterfactual: The potential outcome that would have occurred under an alternative treatment condition.

Potential outcome: The outcome a unit would experience under a specified treatment or control condition.

Average treatment effect (ATE): The average difference between treatment and control potential outcomes in a target population.

Average treatment effect on the treated (ATT): The average causal effect among units that received treatment.

Randomization: Assignment of units to conditions using a chance mechanism.

Treatment effect: A causal difference in outcomes attributable to an intervention.

Estimand: The precisely defined causal quantity a study seeks to estimate.

Blocking: Grouping similar units before random assignment to improve balance and precision.

Factorial design: An experiment that varies two or more factors simultaneously.

Interaction: A condition in which the effect of one factor depends on the level of another factor.

ANOVA: A method that partitions outcome variation into systematic and residual components.

A/B test: A randomized comparison between two versions of a product, message, or intervention.

Guardrail metric: A secondary measure used to detect harmful consequences of an intervention.

Confounder: A variable that causally influences both treatment assignment and outcome.

Selection bias: Distortion caused by systematic differences in who enters, remains in, or is observed in comparison groups.

Intention-to-treat effect: The effect of being assigned to treatment, regardless of compliance.

Difference-in-differences: An estimator comparing changes over time between treated and comparison groups.

Regression discontinuity: A quasi-experiment based on treatment assignment at a known threshold.

Instrumental variable: A variable that affects treatment but affects the outcome only through treatment under an exclusion assumption.

Collider: A variable caused by two other variables; conditioning on it can induce spurious association.

Review Questions

Short Answer Questions

1. Why does randomization help estimate a causal effect?

2. Distinguish the ATE from the ATT.

3. What is the purpose of blocking in an experiment?

4. Why can repeated unplanned peeking at A/B-test results be problematic?

5. Define a collider and explain why conditioning on one may create bias.

Long Answer Questions

1. Explain the potential-outcomes framework and the fundamental problem of causal inference. How do randomized experiments address it?

2. Describe how a factorial experiment estimates main effects and interactions. Give an example in a digital product setting.

3. Discuss the design and analysis requirements of a reliable online A/B test.

4. Explain ANOVA, including the decomposition of variation and interpretation of the F statistic.

5. Compare difference-in-differences, regression discontinuity, instrumental variables, and interrupted time series, including a key assumption for each.

Analytical / Application Questions

1. A company gives a new retention offer only to customers who have contacted support in the previous month. Customers receiving the offer renew more often. Identify possible sources of bias and propose a randomized design.

2. An A/B test increases click-through rate but also increases cancellations and latency. How should the product team evaluate the result and decide whether to deploy?

3. A policy is assigned to municipalities with unemployment rates above a fixed threshold. Explain how you would use a regression-discontinuity design and what validity checks would be necessary.

22. Multiple Choice Questions with Answers

Question 1

Which quantity is defined as E[Y(1) − Y(0)] for a target population?

A. Average treatment effect

B. Average treatment effect on the treated

C. Intention-to-treat effect only

D. Selection bias

Correct Answer: A. Average treatment effect

Question 2

A researcher randomizes users within country and prior-activity groups. What is the main purpose of this procedure?

A. To guarantee a statistically significant result

B. To improve balance and precision using blocking

C. To eliminate measurement error

D. To estimate only post-treatment effects

Correct Answer: B. To improve balance and precision using blocking

Question 3

In a 2 × 2 factorial experiment, an interaction exists when:

A. Both factors have identical sample sizes

B. The outcome has no residual variation

C. The effect of one factor depends on the level of the other

D. Each factor is tested in a separate experiment

Correct Answer: C. The effect of one factor depends on the level of the other

Question 4

Why is an intention-to-treat analysis often preferred in a randomized experiment with noncompliance?

A. It preserves the original benefits of random assignment

B. It guarantees a larger treatment effect

C. It excludes participants who did not follow instructions

D. It estimates only the effect among perfect compliers

Correct Answer: A. It preserves the original benefits of random assignment

Question 5

A product team repeatedly checks an ordinary fixed-sample p-value and stops whenever it falls below 0.05. What is the main concern?

A. Blocking bias

B. Inflated false-positive risk

C. Loss of external validity caused by randomization

D. Automatic violation of SUTVA

Correct Answer: B. Inflated false-positive risk

Question 6

A variable causes both treatment assignment and the outcome. In a causal analysis, this variable is most accurately called a:

A. Mediator

B. Collider

C. Confounder

D. Guardrail metric

Correct Answer: C. Confounder

Question 7

Difference-in-differences relies centrally on which assumption?

A. Treatment effects are identical for every individual

B. Treated and comparison groups would have followed parallel trends without treatment

C. The treatment is assigned by a random number generator

D. The outcome cannot vary over time

Correct Answer: B. Treated and comparison groups would have followed parallel trends without treatment

Question 8

A treatment is assigned to units scoring above 70, and outcomes are compared just above and below 70. Which design is being used?

A. Regression discontinuity

B. Factorial randomization

C. One-way blocking

D. Matched-pairs ANOVA

Correct Answer: A. Regression discontinuity

Question 9

Why might adjusting for a post-treatment mediator be inappropriate when estimating a total treatment effect?

A. It can remove part of the pathway through which treatment operates

B. It always increases the sample size

C. It converts a randomized study into a factorial design

D. It guarantees confounding

Correct Answer: A. It can remove part of the pathway through which treatment operates

Question 10

An A/B test shows a small statistically significant increase in clicks, but purchases decline and complaints rise. What is the most appropriate interpretation?

A. Deploy immediately because statistical significance settles the decision

B. Ignore all secondary outcomes because they were not randomized

C. Evaluate the primary and guardrail metrics together before deciding

D. Conclude that randomization failed without further investigation

Correct Answer: C. Evaluate the primary and guardrail metrics together before deciding