Experimental Design and Causal Analysis

Introduction

Business analytics is often used not merely to describe what happened, but to determine what is likely to happen if an organisation changes a policy, price, interface, promotion, process or resource allocation. This is a causal question. A descriptive analysis may show that customers exposed to a recommendation engine purchased more frequently. A causal analysis asks whether the recommendation engine caused the additional purchases, rather than merely identifying customers who were already more likely to buy.

Experimental design provides the strongest general framework for answering causal questions. By deliberately assigning units to treatment and control conditions, researchers can create comparable groups and estimate the effect of an intervention. A/B testing is the most familiar business application, but the same principles apply to operations, marketing, finance, human resources, public policy and product development.

This chapter develops the logic of controlled experiments, randomisation, blocking and factorial designs. It then examines treatment effects, confounding, correlation and causation, quasi-experimental designs, practical implementation, ethics and limitations.

Learning objectives

By the end of this chapter, you should be able to:

1. Distinguish descriptive, predictive and causal questions.
2. Define experimental units, treatments, outcomes, controls and estimands.
3. Design and evaluate A/B tests using randomisation and appropriate metrics.
4. Explain blocking, stratification, factorial designs and interaction effects.
5. Calculate and interpret simple treatment effects and uncertainty measures.
6. Identify confounding, selection bias, interference and other threats to validity.
7. Distinguish randomised experiments from quasi-experimental evidence.
8. Make responsible managerial decisions from experimental results.

1. From association to causation

A relationship between two variables is an association. For example, customers who use a mobile application may have higher retention than customers who do not. This association may reflect the application’s value, but it may also reflect age, income, digital confidence, loyalty or a pre-existing intention to remain with the company.

Causation concerns a comparison between possible outcomes under different conditions. For an individual unit, let Y_i(1) denote the outcome if the unit receives treatment and Y_i(0) denote the outcome if it does not. The individual causal effect is:

Here, τ_i is the causal effect for unit i. The fundamental problem of causal inference is that, for the same unit at the same time, only one of Y_i(1) and Y_i(0) can normally be observed. The missing outcome is the counterfactual.

Because individual effects are usually unobservable, analysts estimate an average treatment effect (ATE):

The symbol E denotes an average or expected value over the target population. The ATE is meaningful only when treatment and control conditions are clearly defined and the population to which the conclusion applies is specified.

A related quantity is the average treatment effect on the treated (ATT):

where D = 1 indicates that a unit actually received treatment. ATE and ATT are equal under particular conditions, such as random treatment assignment with appropriate compliance, but they may differ when treatment is targeted toward selected people.

Correlation is not causation

Correlation can arise through at least four mechanisms:

• direct causation: advertising increases demand;
• reverse causation: high demand causes more advertising;
• common causes: seasonality increases both advertising and demand;
• selection: customers who choose a premium service differ systematically from others.

A causal claim therefore requires more than a large correlation or a statistically significant coefficient. It requires a credible identification strategy explaining why the observed comparison represents a counterfactual comparison.

2. The architecture of a controlled experiment

A controlled experiment has several essential components.

The unit of randomisation must be chosen carefully. If an offer is delivered at store level, randomising individual customers within a store may cause contamination: customers in the same store can share information. In such a case, stores may be the experimental units, producing a cluster-randomised design.

Treatment and control conditions

A treatment should be operationally precise. “Improved service” is too vague; “a response within two hours during business days” is testable. The control condition should represent the relevant alternative, often the current process. A poorly defined control makes the estimated effect difficult to interpret.

Outcomes should be specified before observing results. Primary outcomes might include conversion rate, gross margin or customer retention. Secondary outcomes may include complaints, refunds and support contacts. A treatment that improves conversion but substantially increases refunds may not be commercially beneficial.

Randomisation

Randomisation assigns eligible units to treatment conditions using a chance mechanism. Under successful randomisation, treatment assignment is independent of potential outcomes in expectation:

The symbol ⟂ means statistical independence, and D is the treatment indicator. Randomisation balances both measured and unmeasured characteristics on average. It therefore removes systematic selection as an explanation for a treatment-control difference, provided implementation is sound.

Randomisation does not guarantee identical groups in every sample. It creates balance probabilistically. Larger samples generally reduce accidental imbalance, while blocking can improve balance for important known characteristics.

A simple difference in means

For a continuous outcome, a basic estimator is:

where Ȳ_T and Ȳ_C are the sample means for treatment and control, and Ŷτ estimates the ATE under random assignment and appropriate implementation.

For a binary outcome, such as conversion, the estimated effect is a difference in proportions:

The relative lift is:

The difference is measured in percentage points, whereas relative lift is a percentage. These must not be confused.

Worked example: an online checkout test

Suppose 10,000 eligible visitors are randomly assigned equally to an existing checkout and a redesigned checkout. The control group has 500 conversions and the treatment group has 560 conversions.

The conversion rates are:

• control: 500 ÷ 5,000 = 0.100, or 10.0%;
• treatment: 560 ÷ 5,000 = 0.112, or 11.2%.

The estimated absolute effect is:

Thus, the redesign increases conversion by 1.2 percentage points. Relative lift is 0.012 ÷ 0.100 = 0.12, or 12%. The estimated number of additional conversions among 5,000 treated visitors is 60, although the realised business effect depends on costs, margins and longer-term consequences.

For a difference in independent proportions, an approximate standard error is:

where n_T and n_C are group sizes. This approximation is suitable when observations are independent and sample sizes are sufficiently large for the normal approximation. In the example, the standard error is approximately 0.0065. A 95% approximate confidence interval is:

which is approximately 0.012 ± 0.0127, or from about −0.0007 to 0.0247. The interval includes zero, so the evidence is not conventionally conclusive at the 5% level, despite the positive point estimate. A decision should consider uncertainty, practical value and risk, not only a threshold-based p-value.

3. Planning an A/B test

A credible A/B test begins before data collection.

Formulate the decision and hypothesis

State the intervention, population, primary outcome, time horizon and decision rule. For example: “Among new visitors, does version B increase seven-day completed purchases without reducing contribution margin over the next 14 days?” The null hypothesis may be H₀: τ = 0, while a directional alternative may be H₁: τ > 0.

Determine the unit and sample size

Sample-size planning depends on baseline outcome rate, minimum detectable effect, significance level, power, allocation ratio and variance. For a simple two-group comparison of means with equal group sizes, an approximate per-group sample size is:

Here, σ² is the assumed common outcome variance, Δ is the practically important effect, α is the two-sided Type I error rate, 1−β is power, and z values are standard-normal quantiles. This formula assumes independent observations, equal allocation and a mean comparison. Binary outcomes, unequal allocation, clustering and attrition require modified calculations.

The minimum detectable effect should be commercially meaningful, not merely statistically detectable. A tiny improvement may be unprofitable if implementation costs are high.

Pre-analysis planning

Before launch, document eligibility rules, assignment logic, primary and secondary outcomes, exclusion rules, analysis period, stopping criteria and treatment of missing data. This reduces selective reporting and “p-hacking”, in which repeated searches through analyses produce apparently favourable results by chance.

Repeatedly checking results and stopping when a desired p-value appears changes the error rate unless a sequential design or corrected monitoring procedure is used. Similarly, testing many outcomes or segments increases the chance of a false positive. Pre-specification, multiplicity control and transparent reporting are best practice.

4. Blocking, stratification and cluster designs

Blocking

Blocking groups similar experimental units before randomisation. Within each block, units are randomised to treatment or control. Useful blocking variables are strongly related to the outcome, such as region, customer tenure, device type or store size.

Blocking improves precision because treatment and control are compared within relatively homogeneous groups. It does not automatically remove bias if blocks are defined after treatment or if assignment is not genuinely random within blocks.

In analysis, a blocked comparison can be estimated using weighted within-block effects:

Here, b indexes blocks, w_b are pre-specified weights, and Ȳ_Tb and Ȳ_Cb are treatment and control means in block b. Weights may reflect the target population or the block sample sizes, depending on the estimand.

Stratified randomisation

Stratification is closely related to blocking and is common in digital experiments. For example, a company may ensure that each treatment arm contains approximately equal proportions of mobile and desktop users. This protects against chance imbalance in important subgroups. Analysts should still estimate uncertainty using the actual design rather than treating the stratification as irrelevant.

Cluster-randomised experiments

When treatment is assigned to groups such as stores, schools or sales teams, observations within a cluster may be correlated. The effective sample size is therefore smaller than the raw number of individuals. A common planning concept is the design effect:

where m is the average number of individuals per cluster and ρ is the intracluster correlation coefficient. The design effect indicates how much variance is inflated relative to independent observations. Cluster-robust standard errors or multilevel models may be required. The number of clusters, not only the number of individuals, is crucial for inference.

5. Factorial designs and interaction

A factorial experiment studies multiple factors simultaneously. In a 2 × 2 design, factor A and factor B each have two levels, creating four conditions.

The main effect of A is the average outcome under discount pricing minus the average outcome under standard pricing: (120 + 150) ÷ 2 − (100 + 115) ÷ 2 = 27.5. The main effect of B is (115 + 150) ÷ 2 − (100 + 120) ÷ 2 = 22.5.

An interaction exists when the effect of one factor depends on the level of another. The effect of the message under standard pricing is 15, while under discount pricing it is 30. The interaction contrast is:

Using the table, this is 30 − 15 = 15. The positive interaction suggests that messaging is more effective when combined with discount pricing. In practice, interactions should be interpreted with predicted outcomes or plots, not isolated coefficients.

Factorial designs can be more efficient than running separate experiments, but they require adequate sample sizes in every cell and careful attention to operational complexity.

6. Threats to validity

Confounding

A confounder affects both treatment assignment and outcome and is not adequately controlled. If high-value customers are preferentially offered a loyalty programme, the raw difference between participants and non-participants may exaggerate the programme’s effect.

Randomisation addresses confounding in expectation. In observational studies, analysts may adjust for measured confounders, but adjustment cannot reliably remove unknown or poorly measured confounding.

Selection and attrition

Selection bias occurs when inclusion or continued participation depends on treatment or factors related to outcomes. If dissatisfied users leave a treatment group before the outcome is measured, an analysis of only completers may be biased. The intention-to-treat (ITT) effect compares units according to their original assignment, regardless of compliance:

ITT preserves the benefit of randomisation and estimates the effect of offering or assigning the intervention. A per-protocol analysis estimates the effect among those who complied but may reintroduce selection bias.

Noncompliance and contamination

Some assigned users do not receive treatment, while control users may access it. These problems dilute the ITT effect and can undermine the contrast between conditions. Assignment logs, exposure logs and intention-to-treat reporting should be maintained.

Interference and spillovers

The stable unit treatment value assumption (SUTVA) commonly requires that one unit’s outcome does not depend on another unit’s treatment assignment and that treatment is well defined. Social-media campaigns, network products and store-level promotions often violate this assumption. When spillovers matter, cluster randomisation, network designs or explicit measurement of exposure may be necessary.

Internal and external validity

Internal validity concerns whether the estimated effect is credible for the tested units and conditions. External validity concerns whether it generalises to other populations, periods, channels or settings. A tightly controlled experiment may have strong internal validity but limited generalisability. Replication across seasons, customer groups and operating environments strengthens the business case.

7. Quasi-experimental evidence

Randomisation is not always feasible. Regulation, ethics, cost, timing or operational constraints may require observational or quasi-experimental methods. These methods use design features that approximate a counterfactual, but their assumptions must be defended explicitly.

Before-and-after comparisons

A simple before-and-after estimate is:

This is weak evidence when other changes occur at the same time. Seasonality, economic conditions and unrelated campaigns may explain the difference.

Difference-in-differences

Difference-in-differences compares changes over time in a treated group with changes in a comparison group:

The key assumption is parallel trends: absent treatment, the treated and comparison groups would have experienced similar average changes. This assumption cannot be proven from post-treatment data, but pre-treatment trends can provide supporting evidence. Time-varying confounders and anticipation of treatment remain concerns.

Regression discontinuity

A regression-discontinuity design uses a treatment rule based on a threshold, such as eligibility for a loan above a credit score. Units just above and below the cutoff may be comparable. The estimated effect is local to the threshold and depends on no manipulation of the assignment variable and continuity of potential outcomes around the cutoff.

Instrumental variables

An instrumental variable affects treatment assignment but influences the outcome only through treatment, apart from specified assumptions. Instruments are difficult to find and must satisfy relevance, exclusion and independence conditions. The resulting effect often applies to compliers influenced by the instrument, not necessarily to the entire population.

8. An illustrative case: delivery-slot recommendations

The following is a fictional illustrative case.

Background and problem

A food-delivery platform believes that recommending less congested delivery slots will reduce late deliveries and improve repeat ordering. Operations managers propose an algorithm that displays a recommended slot at the top of the booking page. Product managers want evidence that the recommendation causes improvement rather than merely appearing to customers who are already flexible.

Analysis and approach

The platform defines the customer-session as the unit of randomisation. Eligible sessions are randomly assigned to the current booking page or the recommendation page. Randomisation is blocked by device type and geographic region because both are related to delivery performance. The primary outcome is on-time delivery within seven days; secondary outcomes are completed orders, cancellation, margin and customer support contacts.

The team pre-specifies an ITT analysis, excludes only sessions failing eligibility before randomisation, and records both assignment and actual exposure. It also checks whether customers share links that cause contamination. Because several sessions may belong to the same account, account-level clustering is considered in sensitivity analyses.

Results and interpretation

Suppose the recommendation increases on-time delivery by 2.0 percentage points but decreases completed orders by 0.3 percentage points. Customer support contacts decline, while margin is unchanged. The recommendation therefore improves one operational outcome but creates a possible demand cost. Management should not adopt it solely because the primary metric improved. A weighted decision may consider the financial value of reduced support, the value of retention and the lost contribution from fewer orders.

Segment analysis shows a larger effect on mobile users. However, this subgroup result was not a pre-specified primary hypothesis and involves many comparisons. It should be treated as exploratory and tested in a subsequent confirmatory experiment.

Learning points

The case demonstrates that causal design begins with a precise unit, intervention and counterfactual. Blocking can improve precision; ITT protects randomisation; multiple outcomes require disciplined interpretation; and operational success depends on net business value rather than one favourable metric.

9. Ethics, governance and responsible experimentation

Experiments can affect prices, access, employment conditions, credit decisions and customer welfare. Ethical review should consider informed consent where appropriate, risk, privacy, fairness, vulnerable populations and the reversibility of harm. A/B testing does not justify exposing users to unreasonable risk merely because assignment is random.

Data minimisation, access controls, retention limits and lawful processing are essential when experiments use personal data. Sensitive outcomes should be protected, and subgroup performance should be examined for disparate harm. Automated decision experiments require model and treatment documentation, audit trails and clear accountability.

A practical governance record should include the hypothesis, owner, data sources, eligibility rule, randomisation key, versioned treatment code, analysis plan, results, deviations and decision. Reproducible pipelines reduce accidental changes and make independent review possible.

10. Technology and contemporary practice

Modern experimentation platforms automate assignment, exposure logging, metric calculation and dashboards. Machine-learning systems can support variance reduction by adjusting for pre-treatment covariates, but covariates measured after treatment must not be used as if they were baseline variables. Automated experimentation also increases the risk of running many simultaneous tests, interference among tests and opaque decision rules.

Adaptive and sequential experiments can allocate more observations to promising treatments or stop early for benefit or harm. They require specialised statistical procedures because ordinary fixed-sample significance calculations may no longer be valid. Organisations should prefer transparent, documented methods over uncontrolled dashboard monitoring.

Causal machine learning can estimate heterogeneous treatment effects, such as whether an intervention works differently by customer segment. These methods are useful for targeting, but subgroup findings require out-of-sample validation, attention to fairness and protection against overfitting. Prediction of who responds is not itself proof that treatment causes the response.

Best-practice checklist

1. Define the causal question and estimand before collecting data.
2. Specify treatment, control, unit, outcome and time horizon precisely.
3. Randomise at the level at which interference is plausible.
4. Use blocking or stratification for important prognostic variables.
5. Plan sample size around a meaningful minimum effect.
6. Pre-specify primary outcomes, stopping rules and exclusions.
7. Analyse according to assignment and report noncompliance.
8. Use uncertainty intervals and distinguish percentage points from relative lift.
9. Investigate attrition, contamination, imbalance and implementation failure.
10. Treat subgroup and exploratory findings as provisional unless confirmed.
11. Evaluate financial, operational, ethical and distributional consequences.
12. Document the experiment so that it can be audited and replicated.

Chapter Summary

Experimental design converts a business intervention into a structured comparison of potential outcomes. The central causal challenge is the unobserved counterfactual: what would have happened to a treated unit without treatment. Randomisation provides the strongest general protection against confounding, while blocking and stratification improve balance and precision. A/B tests require clearly defined units, treatments, controls, outcomes, estimands, sample sizes and analysis plans.

Treatment effects can be expressed as differences in means or proportions, with confidence intervals used to represent sampling uncertainty. Factorial designs estimate multiple interventions efficiently and reveal interaction effects. Cluster designs must account for dependence within groups. ITT analysis preserves the benefits of randomisation when compliance is imperfect.

When experiments are impossible, difference-in-differences, regression discontinuity and instrumental variables can provide quasi-experimental evidence, but each depends on assumptions that must be examined rather than assumed. Confounding, selection, attrition, interference and limited generalisability can undermine conclusions. Responsible experimentation also requires privacy, fairness, documentation and attention to the total business and social consequences of a decision.

Key Terms

Causal effect: The difference between outcomes under treatment and under the relevant counterfactual condition.

Counterfactual: The unobserved outcome that would have occurred under an alternative condition.

Average treatment effect (ATE): The average causal effect across a defined target population.

Average treatment effect on the treated (ATT): The average causal effect among units receiving treatment.

Experimental unit: The entity assigned to a treatment or control condition.

Treatment: The intervention or condition whose effect is being evaluated.

Control: The comparison condition against which treatment is assessed.

Randomisation: Allocation by a chance mechanism to reduce systematic differences between groups.

Blocking: Grouping similar units before random assignment to improve balance and precision.

Stratification: Ensuring treatment allocation is balanced across defined subgroups.

Factorial design: An experiment that tests two or more factors and their combinations.

Interaction effect: A situation in which the effect of one factor depends on another factor.

Confounding: Distortion of a treatment-outcome relationship by a common or related variable.

Intention-to-treat: Analysis based on original assignment, regardless of treatment compliance.

Contamination: Exposure of a comparison unit to some or all of the treatment.

Interference: Dependence of one unit’s outcome on another unit’s treatment status.

Internal validity: Credibility of the causal estimate for the studied conditions.

External validity: Extent to which findings generalise beyond the study conditions.

Difference-in-differences: A comparison of changes over time between treated and comparison groups.

Quasi-experiment: A non-randomised design using a rule or structure to approximate causal inference.

Review Questions

Short Answer Questions

1. What is the fundamental problem of causal inference?
2. Distinguish an absolute treatment effect from relative lift.
3. Why can blocking improve an experiment?
4. What does intention-to-treat analysis estimate?
5. State the parallel-trends assumption in difference-in-differences analysis.

Long Answer Questions

1. Explain how randomisation supports causal interpretation and discuss circumstances in which it may fail operationally.
2. Describe the main stages in planning a rigorous A/B test for a digital product.
3. Compare individual randomisation with cluster randomisation, including implications for sample size and inference.
4. Explain factorial designs and use an example to distinguish main effects from interaction effects.
5. Compare internal validity and external validity, and explain why managers need to consider both.

Analytical / Application Questions

1. A retailer observes that loyalty-programme members spend more than non-members. Identify at least three possible explanations for this association and propose a design that could estimate the programme’s causal effect.
2. A policy is introduced in one region but not another. Sales data are available for twelve months before and six months after implementation. Explain how you would assess whether difference-in-differences is credible, and identify threats to the comparison.
3. Design an experiment to test a new employee scheduling algorithm. Specify the experimental unit, treatment, control, primary outcome, blocking variables, possible interference and ethical safeguards.

Multiple Choice Questions with Answers

Question 1

Which quantity compares the average potential outcome under treatment with the average potential outcome under control?

A. Average treatment effect

B. Intracluster correlation coefficient

C. Relative lift only

D. Selection probability

Correct Answer: A. Average treatment effect

Question 2

In a properly randomised experiment, the main purpose of random assignment is to:

A. Guarantee that every individual treatment effect is identical

B. Balance observed and unobserved characteristics in expectation

C. Eliminate all sampling variability

D. Ensure that treatment effects are statistically significant

Correct Answer: B. Balance observed and unobserved characteristics in expectation

Question 3

A control conversion rate is 10.0% and a treatment conversion rate is 11.2%. What is the absolute treatment effect?

A. 1.2 percentage points

B. 1.2% relative lift

C. 12 percentage points

D. 112 percentage points

Correct Answer: A. 1.2 percentage points

Question 4

What is the principal purpose of blocking before randomisation?

A. To guarantee a positive treatment effect

B. To increase similarity within comparison groups and improve precision

C. To replace the need for a control group

D. To permit post-treatment selection of participants

Correct Answer: B. To increase similarity within comparison groups and improve precision

Question 5

In a 2 × 2 factorial experiment, an interaction means that:

A. Both factors have exactly the same main effect

B. The sample contains two treatment groups only

C. The effect of one factor depends on the level of the other factor

D. No treatment condition is used

Correct Answer: C. The effect of one factor depends on the level of the other factor

Question 6

Which analysis preserves the original benefits of randomisation when some assigned participants do not comply?

A. Completers-only analysis

B. Intention-to-treat analysis

C. Post-treatment subgroup analysis

D. Before-and-after analysis

Correct Answer: B. Intention-to-treat analysis

Question 7

The key identifying assumption in a standard difference-in-differences design is:

A. Treatment and comparison groups have identical outcomes at every date

B. The groups would have followed parallel trends without treatment

C. The treatment group is twice as large as the control group

D. The outcome must be binary

Correct Answer: B. The groups would have followed parallel trends without treatment

Question 8

If individuals within the same store have correlated outcomes, analysing them as fully independent observations will generally:

A. Overstate the effective sample size and make uncertainty too small

B. Guarantee unbiased individual treatment effects

C. Remove interference automatically

D. Make blocking impossible

Correct Answer: A. Overstate the effective sample size and make uncertainty too small

Question 9

Which issue occurs when control customers gain access to the treatment through sharing?

A. Blocking

B. Contamination

C. External validity

D. Factorial balance

Correct Answer: B. Contamination

Question 10

A statistically significant experiment improves conversion but increases refunds enough to reduce profit. The most appropriate managerial conclusion is to:

A. Adopt the treatment because significance is decisive

B. Ignore refunds because they are secondary outcomes

C. Evaluate the net business effect, including costs and harms

D. Replace randomisation with correlation analysis

Correct Answer: C. Evaluate the net business effect, including costs and harms