Experimentation and Causal Decision Making

Introduction

Business analytics often begins with a descriptive question: What happened? It then progresses to prediction: What is likely to happen? Causal decision making asks a more demanding question: What would happen if the organization deliberately changed something?

Examples include:

• Would a discount increase profitable sales?
• Would a redesigned checkout page increase completed purchases?
• Would additional employee training reduce service errors?
• Would a fraud-detection rule reduce losses without unfairly denying legitimate customers?

These questions cannot usually be answered reliably by observing simple associations. Customers who receive discounts may differ from those who do not; employees selected for training may already be more motivated; and customers exposed to a new website may use different devices or arrive through different channels. Causal analysis is therefore concerned with constructing a credible comparison between what happened under a treatment and what would have happened without it.

The central tool for achieving this goal is the experiment, especially the randomized controlled experiment or A/B test. When randomization is impossible, analysts may use quasi-experimental designs that attempt to approximate the logic of random assignment.

Learning Objectives

By the end of this chapter, students should be able to:

1. Distinguish correlation, prediction, and causation.
2. Define potential outcomes, treatment effects, and counterfactuals.
3. Design and evaluate randomized experiments and A/B tests.
4. Identify confounding, selection bias, interference, and measurement problems.
5. Interpret average and heterogeneous treatment effects.
6. Explain major quasi-experimental designs and their assumptions.
7. Assess statistical uncertainty, practical significance, and external validity.
8. Communicate causal findings responsibly in business decisions.

1. The Logic of Causal Inference

1.1 Association is not causation

An association exists when two variables tend to vary together. For example, customers who use a loyalty application may spend more than customers who do not. This does not establish that the application caused the additional spending. Frequent shoppers may be more likely both to install the application and to spend more. Customer engagement is a possible confounder: it influences both application adoption and spending.

A causal claim states that changing an intervention would change an outcome, holding other relevant conditions constant. Causal claims therefore require a comparison involving alternative states of the same decision unit.

1.2 Potential outcomes and counterfactuals

Let treatment status be represented by T_i = 1 for a treated unit and T_i = 0 for an untreated unit. For unit i, define:

• Y_i(1): the outcome that would occur if unit i received treatment;
• Y_i(0): the outcome that would occur if unit i did not receive treatment.

The individual treatment effect is:

The difficulty is fundamental: for each unit, only one of the two potential outcomes can be observed. The unobserved alternative is the counterfactual. Randomized experiments do not reveal individual treatment effects directly; they estimate average effects by creating comparable groups.

The average treatment effect (ATE) is:

Here, E denotes the population average. The ATE is meaningful when treatment and control conditions are clearly defined and the target population is specified.

The average treatment effect on the treated (ATT) is:

The ATT describes the average effect among those who actually received treatment. ATE and ATT may differ when treatment effects vary across people.

1.3 Causal estimands

An estimand is the precise quantity a study seeks to estimate. Before collecting data, analysts should specify:

• the unit of analysis, such as customer, order, employee, store, or day;
• the treatment and its duration;
• the outcome and measurement window;
• the target population;
• the effect measure, such as difference in means, relative change, or effect on a probability;
• the decision threshold or business value of the effect.

For a binary outcome, the difference in conversion rates is often used:

where p₁ is the treatment-group conversion probability and p₀ is the control-group conversion probability. The relative lift is:

This measure requires p₀ > 0 and should not replace reporting the absolute difference, because a large percentage lift can correspond to a very small practical change.

2. Randomized Experiments and A/B Tests

2.1 Why randomization works

In a randomized experiment, eligible units are assigned to treatment or control using a random mechanism. Randomization makes assignment independent of both observed and unobserved characteristics in expectation. Thus, before treatment, the groups should be comparable apart from random variation.

Under the assumptions of consistency, no interference, and successful random assignment, the difference in average observed outcomes estimates the ATE:

where Ȳ₁ and Ȳ₀ are the sample mean outcomes for treatment and control groups. The formula is directly applicable when treatment assignment is randomized and the observed outcome is measured consistently.

Randomization does not guarantee identical groups in every sample. It guarantees balance in expectation. Baseline imbalance should therefore be assessed, but analysts should avoid mechanically dropping variables or changing the estimand after inspecting results.

2.2 Components of a well-designed experiment

A credible A/B test typically includes the following components.

Research question. State the decision and causal claim clearly. “Does the new recommendation algorithm increase 30-day contribution margin per active customer?” is stronger than “Is the new algorithm better?”

Experimental unit. Identify the entity assigned to treatment. Assignment may occur at the user, account, store, region, or time-period level. The unit must match the mechanism of treatment and the risk of spillover.

Treatment and control. Define exactly what each group receives. The control may be the current experience, a placebo, no intervention, or an alternative policy.

Randomization. Use a reproducible randomization procedure. Blocked or stratified randomization can improve balance across important categories such as geography, customer type, or device.

Outcome. Choose a primary outcome before analysis. Secondary outcomes can provide useful evidence but should be interpreted cautiously when many outcomes are examined.

Exposure and compliance. Distinguish assignment from actual exposure. An intent-to-treat analysis estimates the effect of being assigned to treatment, whereas a treatment-on-the-treated analysis concerns actual receipt and requires stronger assumptions.

Duration. Allow sufficient time for the treatment to operate, while considering novelty effects, seasonality, learning, and delayed outcomes.

Analysis plan. Specify exclusions, missing-data rules, subgroup analyses, stopping rules, and the primary statistical method in advance.

2.3 Intent-to-treat and per-protocol analysis

The intent-to-treat (ITT) effect compares units according to their assigned condition, even if some do not comply. For a randomized assignment indicator Z, the ITT estimator is:

ITT preserves the benefits of randomization and is usually the most policy-relevant estimate: What is the effect of offering or assigning the intervention?

A per-protocol analysis includes only units who followed the assigned protocol. It may be useful operationally, but excluding noncompliers can destroy comparability because compliance may be related to motivation or expected outcomes.

If assignment affects treatment receipt, assignment affects the outcome only through treatment, and there are no defiers, instrumental-variable methods can estimate a local average treatment effect for compliers. These assumptions are substantial and should not be treated as automatic.

2.4 Statistical uncertainty and power

An observed difference is an estimate, not a certainty. A confidence interval communicates sampling uncertainty. For independent groups with sample means Ȳ₁ and Ȳ₀, sample standard deviations s₁ and s₀, and sample sizes n₁ and n₀, an approximate standard error for the difference is:

This formula assumes independent observations and is commonly used with Welch’s method when variances may differ. An approximate 95% confidence interval is:

where t* is the appropriate critical value based on the desired confidence level and degrees of freedom. For large samples, t* is approximately 1.96 for a 95% interval.

Power analysis should be conducted before the experiment. It considers baseline outcome variability, minimum detectable effect, significance level, desired power, allocation ratio, and attrition. A statistically detectable effect may be commercially trivial, whereas a commercially important effect may be missed by an underpowered test.

2.5 Worked example: an online checkout test

An online retailer randomly assigns 20,000 visitors to either the existing checkout or a redesigned checkout. The control group contains 10,000 visitors and records 800 purchases. The treatment group contains 10,000 visitors and records 880 purchases.

Control conversion is:

p₀ = 800 ÷ 10,000 = 0.080 = 8.0%

Treatment conversion is:

p₁ = 880 ÷ 10,000 = 0.088 = 8.8%

The absolute increase is:

Δ = 0.088 − 0.080 = 0.008

Thus, conversion increased by 0.8 percentage points. Relative lift is:

0.008 ÷ 0.080 = 0.10 = 10%

The approximate standard error for two independent proportions is:

Substitution gives:

SE ≈ √(0.088×0.912÷10,000 + 0.080×0.920÷10,000) ≈ 0.00396

An approximate 95% interval is 0.008 ± 1.96×0.00396, or approximately (0.0002, 0.0158). The interval barely excludes zero under this approximation. The retailer should still consider implementation cost, margin per purchase, returns, customer support effects, and whether the test population represents future traffic.

3. Threats to Validity

3.1 Confounding

A confounder influences both treatment assignment and the outcome. For example, high-value customers may be more likely to receive a retention offer and less likely to churn. A naive comparison may incorrectly attribute their lower churn to the offer.

Randomization prevents confounding in expectation. In observational data, analysts may use design or statistical adjustment, but adjustment is credible only when the relevant confounders are measured adequately and the model is appropriate.

3.2 Selection bias

Selection bias occurs when inclusion in the analysis depends on treatment, outcome, or factors related to both. An experiment restricted to customers who voluntarily return to a website may not estimate the effect for all invited customers. Attrition can also create bias if dropout rates differ by treatment and are related to outcomes.

3.3 Post-treatment adjustment

Variables measured after treatment may be consequences of treatment. Controlling for them can remove part of the treatment effect or introduce bias. For example, controlling for “whether a customer opened the promotional email” may be inappropriate when the promotion itself affects opening behavior.

3.4 Interference and spillovers

The standard potential-outcomes framework often assumes that one unit’s treatment does not affect another unit’s outcome. This is called the stable unit treatment value assumption, or SUTVA. It may fail when customers share referrals, employees work in teams, or a marketplace intervention changes supply and demand for everyone.

Cluster randomization, geographic assignment, network-aware designs, or market-level experiments may be preferable when interference is substantial. The trade-off is reduced statistical efficiency and a need to account for within-cluster correlation.

3.5 Experiment contamination and novelty

Participants may encounter both versions through multiple devices, accounts, or channels. Treatment effects may also reflect initial excitement rather than durable behavior. Analysts should monitor assignment integrity, cross-exposure, treatment fidelity, and effects over time.

3.6 Multiple testing and peeking

Testing many outcomes, segments, or time windows increases the chance of finding an apparently positive result by chance. Repeatedly checking results and stopping when a desired significance level appears can also inflate false-positive risk. Pre-registration, correction methods, sequential testing procedures, and transparent reporting reduce these problems.

4. Quasi-Experimental Designs

When random assignment is infeasible or unethical, quasi-experimental methods use naturally occurring variation or institutional rules. Their validity depends on design-specific assumptions, not merely on applying a sophisticated estimator.

4.1 Difference-in-differences

Difference-in-differences compares changes over time in a treated group with changes in a comparison group:

The key assumption is parallel trends: absent treatment, the treated and comparison groups would have experienced similar average changes. Analysts should examine pre-treatment trends and consider time-varying confounders, anticipation, staggered adoption, and clustered standard errors.

For example, if sales in a pilot region rise from 100 to 130 units and sales in a comparison region rise from 120 to 140 units, then:

DiD = (130−100) − (140−120) = 30 − 20 = 10

The estimated treatment effect is 10 units, assuming parallel trends and no other differential shock.

4.2 Regression discontinuity

A regression discontinuity design exploits a known cutoff. Suppose firms with a risk score of 70 or higher receive intensive support, while firms below 70 do not. Firms very close to the threshold may be comparable, so a jump in outcomes at the cutoff can estimate a local causal effect.

The design requires that units cannot precisely manipulate the assignment score and that no other intervention changes at the same threshold. The resulting estimate is local: it may not apply to firms far from the cutoff.

4.3 Instrumental variables

An instrumental variable affects treatment receipt but has no direct effect on the outcome except through treatment. A valid instrument must satisfy relevance, independence, and the exclusion restriction. These requirements are difficult to defend in many business settings. Instrumental-variable estimates often apply to compliers rather than the entire population.

4.4 Matching and weighting

Matching treated and untreated units with similar observed characteristics can improve comparability. Propensity scores summarize the probability of treatment given measured covariates. Weighting and regression adjustment can then estimate effects under assumptions such as no unmeasured confounding, consistency, and positivity.

These methods cannot solve confounding caused by variables that were not measured or measured poorly. A matched comparison may look balanced on a dataset’s variables while remaining substantially different on motivation, need, or future expectations.

5. Heterogeneous Effects and Decision Value

Average effects can conceal important differences. A marketing campaign may increase purchases among new customers but reduce profit among existing customers because discounts are given to people who would have purchased anyway.

Effect heterogeneity may be examined by pre-specified subgroups, interaction models, or causal machine-learning methods. Subgroup analysis requires adequate sample size, a plausible mechanism, and caution about false discoveries. A subgroup result discovered after extensive searching is exploratory unless confirmed in new data.

Business decisions should combine causal impact with value. If treatment changes outcome Y and the monetary value per outcome is v, while treatment cost per unit is c, an illustrative incremental value per unit is:

This expression assumes a linear value per outcome and that costs and benefits are measured on the same unit and time basis. In practice, analysts may need to include implementation costs, cannibalization, refunds, operational capacity, risk, and discounting over time.

Statistical significance is not the same as economic significance. A tiny effect can be statistically precise in a very large dataset, while a meaningful effect may be uncertain in a small pilot.

6. External Validity and Scaling

Internal validity concerns whether the estimated effect is credible for the studied units under the studied conditions. External validity concerns whether it transports to other populations, settings, time periods, or implementation methods.

An experiment may not generalize because:

• participants are unusually engaged;
• the test occurs during a special season;
• the treatment is delivered by unusually skilled staff;
• competitors react after rollout;
• capacity constraints appear at scale;
• the outcome window is too short;
• the treatment interacts with geography, culture, regulation, or technology.

Generalization should be treated as an evidence-building process. Analysts can replicate experiments across locations and periods, report treatment-effect heterogeneity, compare sample composition with the target population, and conduct sensitivity analyses. Transporting an effect requires substantive judgment, not simply applying a statistical coefficient to a new population.

7. Ethical, Governance, and Technology Considerations

Experiments should respect informed consent requirements, privacy law, organizational policy, and sector-specific regulation. Risk, vulnerability, and fairness matter even when a test is statistically rigorous. An intervention should not be tested merely because randomization is possible.

Important governance practices include:

• define acceptable risk and stopping criteria;
• minimize collection of personal data;
• protect sensitive attributes and access controls;
• document assignment logic and analysis decisions;
• monitor differential effects across protected or vulnerable groups;
• distinguish experimentation from covert discrimination;
• preserve reproducible code, data definitions, and version history.

AI systems create additional experimentation challenges. Adaptive algorithms can change treatment exposure during a test, optimize toward short-term metrics, or amplify unequal outcomes. Analysts should separate model updates from treatment effects, log versions and decisions, and evaluate both average and distributional consequences. Automated experimentation does not eliminate the need for human oversight.

8. Applied Case: A Retention Offer at Northstar Telecom

The following is an illustrative case.

Background

Northstar Telecom observes that customers approaching contract renewal have elevated churn. The company proposes a one-month service credit for selected customers. Finance leaders want to know whether the credit reduces churn enough to justify its cost.

Problem

Historically, retention credits were offered by service agents to customers who complained or threatened to leave. Customers receiving credits therefore differed systematically from other customers. A simple comparison showed lower churn among credit recipients, but this could reflect agent attention or customer service history rather than the credit itself.

Analysis and approach

Northstar defines the experimental unit as the eligible customer account. It specifies:

• treatment: one-month credit, available for renewal-period use;
• control: usual renewal process without the credit;
• primary outcome: account churn within 90 days of renewal;
• secondary outcomes: contribution margin, complaints, usage, and later churn;
• assignment: customer-level randomization, stratified by region and contract type;
• analysis: intent-to-treat, with subgroup analysis by baseline risk specified in advance.

The company excludes accounts already scheduled for cancellation, because the offer cannot plausibly affect their renewal decision. It monitors whether agents override assignment and whether customers hold multiple accounts that could contaminate treatment.

Suppose 5,000 treatment accounts experience 600 churns and 5,000 control accounts experience 700 churns. Treatment churn is 12.0%; control churn is 14.0%. The estimated reduction is 2 percentage points, or approximately 14.3% relative to the control churn rate.

If the retained account produces expected contribution margin of $90 during the relevant period and the credit costs $20, the simplified incremental value per eligible account is:

90×0.02 − 20×0.??

However, the credit cost applies only when customers use it, so finance must use the actual redemption rate rather than assume every account incurs the full cost. If the average cost is $8 per eligible account, the estimated value is 90×0.02 − 8 = −$6.20 per account under this simplified calculation. The offer would not be justified on retention margin alone, though longer-term value or reduced service costs could change the conclusion. (The arithmetic is $1.80 − $8 = −$6.20.)

Outcome and learning points

Northstar finds that the overall effect is modest but concentrated among medium-risk customers. High-risk customers are unlikely to be retained by a one-month credit, while low-risk customers would usually remain anyway. The company therefore considers a targeted offer, but validates the subgroup result in a second experiment before operationalizing it.

The case illustrates five principles: historical comparisons were confounded; randomization created a credible counterfactual; ITT protected the analysis from agent noncompliance; financial value required cost measurement; and average effects did not identify the most valuable target group.

9. Best Practices for Causal Decision Making

1. Begin with a decision, not merely a metric.
2. Define the estimand, population, treatment, outcome, and time horizon before analysis.
3. Randomize whenever feasible and ethically appropriate.
4. Choose the assignment unit to limit interference and contamination.
5. Use ITT as the primary analysis for randomized assignment.
6. Pre-specify primary outcomes and stopping rules.
7. Report effect sizes and confidence intervals, not p-values alone.
8. Separate statistical significance from financial and operational significance.
9. Investigate attrition, noncompliance, missingness, and measurement changes.
10. Treat subgroup findings as confirmatory only when adequately planned and powered.
11. For quasi-experiments, state and test the design’s identifying assumptions.
12. Replicate important findings across populations, periods, and implementation settings.
13. Document data lineage, code, treatment versions, and deviations from the protocol.
14. Evaluate fairness, privacy, safety, and unintended consequences.
15. Communicate uncertainty honestly and avoid causal language unsupported by design.

Chapter Summary

Causal decision making asks what would happen if an intervention changed, rather than merely describing associations. Potential outcomes provide the conceptual foundation, but the unobserved counterfactual makes causal estimation difficult. Randomized experiments address this problem by assigning comparable units to treatment and control, allowing the difference in average outcomes to estimate an average treatment effect under appropriate assumptions.

A credible A/B test requires clear treatment definitions, suitable assignment units, pre-specified outcomes, adequate duration, attention to compliance and interference, and analysis of uncertainty. ITT preserves randomization and answers the practical question of assignment or offer effects. Statistical significance must be combined with effect size, economic value, feasibility, and risk.

When randomization is unavailable, difference-in-differences, regression discontinuity, instrumental variables, matching, and weighting can provide useful evidence, but each depends on strong, design-specific assumptions. Confounding, selection bias, post-treatment adjustment, contamination, attrition, and multiple testing can undermine conclusions. Finally, a causal effect estimated in one setting may not generalize elsewhere. Responsible analytics therefore combines rigorous design, transparent assumptions, ethical governance, replication, and managerial judgment.

Key Terms

Causal effect: The change in an outcome attributable to an intervention.

Counterfactual: The unobserved outcome that would have occurred under an alternative treatment condition.

Potential outcome: An outcome defined under a specified treatment state.

Average treatment effect (ATE): The average difference between treated and untreated potential outcomes in a target population.

Average treatment effect on the treated (ATT): The average treatment effect among units that received treatment.

Randomized controlled experiment: A study that assigns units to conditions using a random mechanism.

A/B test: A randomized comparison of two versions, policies, or experiences.

Intent-to-treat (ITT): The effect of being assigned to treatment, regardless of actual compliance.

Confounding: Distortion caused by a variable related to both treatment assignment and outcome.

Selection bias: Distortion arising when inclusion or retention depends on relevant characteristics.

Interference: A situation in which one unit’s treatment affects another unit’s outcome.

SUTVA: The assumption of no interference and consistent treatment definition.

Statistical power: The probability of detecting an effect of a specified size under defined conditions.

Difference-in-differences: A design comparing changes over time between treated and comparison groups.

Regression discontinuity: A design using a treatment rule based on a cutoff.

Instrumental variable: A variable that changes treatment receipt while satisfying assumptions for causal identification.

External validity: The extent to which findings generalize beyond the study setting.

Heterogeneous treatment effect: Variation in causal effects across units or subgroups.

Estimand: The precisely defined causal quantity a study seeks to estimate.

Review Questions

Short Answer Questions

1. Why does correlation between treatment and outcome fail to establish causation?
2. Define an individual treatment effect using potential-outcomes notation.
3. What is the practical interpretation of an intent-to-treat effect?
4. State the parallel-trends assumption in difference-in-differences analysis.
5. Why can statistical significance differ from economic significance?

Long Answer Questions

1. Explain how randomization addresses confounding and discuss conditions under which a randomized A/B test may still produce a misleading conclusion.
2. Compare ATE and ATT, explaining why they may differ in the presence of heterogeneous treatment effects.
3. Describe the design, assumptions, strengths, and limitations of difference-in-differences.
4. Explain how interference, contamination, and noncompliance affect experiment design and interpretation.
5. Discuss why external validity matters when scaling a successful business experiment.

Analytical / Application Questions

1. A retailer observes that loyalty-program members spend more than nonmembers. Identify at least three possible confounders and propose a causal design to estimate the program’s effect.
2. A training program is assigned to stores with unusually high error rates. Explain why a before-and-after comparison may be biased and propose a quasi-experimental alternative.
3. An A/B test increases conversion by 0.5 percentage points but lowers average order margin by $2. Design a decision analysis that determines whether rollout is justified.

Multiple Choice Questions with Answers

Question 1

Which quantity represents the average treatment effect?

A. E[Y(1) − Y(0)]

B. E[Y(1) + Y(0)]

C. E[Y(0) | T = 1]

D. E[T | Y = 1]

Correct Answer: A. E[Y(1) − Y(0)]

Question 2

What is the main purpose of random assignment in an experiment?

A. To guarantee that every participant receives treatment

B. To make treatment assignment independent of relevant characteristics in expectation

C. To eliminate all measurement error

D. To ensure the treatment has a positive effect

Correct Answer: B. To make treatment assignment independent of relevant characteristics in expectation

Question 3

An experiment reports treatment conversion of 8.8% and control conversion of 8.0%. What is the absolute increase?

A. 0.08 percentage points

B. 0.8 percentage points

C. 8 percentage points

D. 10 percentage points

Correct Answer: B. 0.8 percentage points

Question 4

Which assumption is central to a basic difference-in-differences design?

A. Treatment effects must be identical for every individual

B. The treated group must be larger than the control group

C. Treated and comparison groups would have followed parallel trends without treatment

D. All units must receive treatment simultaneously

Correct Answer: C. Treated and comparison groups would have followed parallel trends without treatment

Question 5

What does an intent-to-treat estimate measure?

A. The effect of assignment to treatment, regardless of compliance

B. The effect only among perfect compliers

C. The effect after excluding all nonparticipants

D. The effect of treatment on the analyst

Correct Answer: A. The effect of assignment to treatment, regardless of compliance

Question 6

Which situation is an example of interference?

A. A customer’s treatment changes another customer’s outcome through referrals

B. A customer forgets to complete a survey

C. A treatment effect varies by age

D. A confidence interval is wide

Correct Answer: A. A customer’s treatment changes another customer’s outcome through referrals

Question 7

A regression discontinuity design commonly identifies effects for which units?

A. Units far from the assignment threshold

B. Units near the assignment threshold

C. Only units receiving no treatment

D. Every unit in the economy

Correct Answer: B. Units near the assignment threshold

Question 8

Why can a statistically significant result be commercially unimportant?

A. Large samples can estimate very small effects precisely

B. Statistical tests never measure uncertainty

C. Confidence intervals always exclude zero

D. Randomization guarantees large effects

Correct Answer: A. Large samples can estimate very small effects precisely

Question 9

Which practice best reduces false discoveries from examining many outcomes and subgroups?

A. Select the most favorable result after analysis

B. Pre-specify primary outcomes and analysis rules

C. Stop whenever any subgroup is significant

D. Ignore secondary outcomes completely

Correct Answer: B. Pre-specify primary outcomes and analysis rules

Question 10

What is external validity concerned with?

A. Whether the treatment was randomly assigned

B. Whether a result generalizes to other settings or populations

C. Whether the sample mean equals zero

D. Whether a variable was measured after treatment

Correct Answer: B. Whether a result generalizes to other settings or populations