Causal Inference and Business Experimentation

Introduction

Business analytics frequently identifies relationships among variables, but a relationship is not automatically a cause. Sales may rise when advertising expenditure increases, yet the increase may also reflect seasonal demand, competitor actions, or management’s decision to advertise more precisely when demand is expected to grow. Strategic decisions require stronger evidence: what would happen to the outcome if a firm deliberately changed an input while other relevant conditions were held constant?

Causal inference provides a framework for answering such questions. It combines research design, statistical modelling, domain knowledge, and explicit assumptions. In business, it supports decisions about pricing, promotions, credit policies, customer retention, workforce interventions, product features, and resource allocation.

This chapter distinguishes association, prediction, and causation; explains randomized experimentation; and develops four important approaches for observational data: matching, difference-in-differences, instrumental variables, and regression discontinuity. The emphasis is not merely on estimating coefficients, but on determining whether an estimate has a credible causal interpretation.

Learning Objectives

By the end of this chapter, the reader should be able to:

1. Distinguish descriptive association, prediction, and causal effects.
2. Define potential outcomes, treatment effects, confounding, and selection bias.
3. Design and evaluate randomized controlled experiments.
4. Interpret average treatment effects and uncertainty intervals.
5. Apply matching, difference-in-differences, instrumental variables, and regression discontinuity designs.
6. Diagnose threats to validity, spillovers, noncompliance, attrition, and measurement problems.
7. Communicate causal evidence responsibly for strategic decisions.

1. Association, Prediction, and Causation

Association

Association describes systematic co-movement between variables. For example, customers who receive a loyalty offer may spend more than customers who do not. A difference in average spending is an association, not proof that the offer caused the difference.

A sample correlation between variables X and Y is commonly written as:

Here, Cov(X,Y) is the sample covariance and s_X and s_Y are sample standard deviations. Correlation is unit-free and measures linear association. It does not establish direction, causality, or protection against confounding.

Prediction

Prediction concerns the accuracy of forecasts for new or future observations. A predictive model may use historical correlations, even when those relationships are not causal. For example, browsing behaviour may predict customer churn without being a feasible intervention for preventing churn.

A model can have excellent predictive accuracy but provide poor guidance about the consequences of changing a variable. This occurs when the model exploits a proxy, a non-causal pattern, or a relationship that changes after intervention.

Causation

A causal effect concerns the difference between an outcome under treatment and the outcome that would have occurred for the same unit under an alternative treatment. Because the same unit cannot usually be observed simultaneously in both states, causal inference is a missing-data problem.

Let Y_i(1) denote the potential outcome for unit i if it receives treatment, and Y_i(0) the potential outcome if it does not. The individual treatment effect is:

Only one of these two potential outcomes is observed for each unit. The average treatment effect (ATE) in a population of N units is:

The observed outcome can be represented as:

where D_i = 1 if unit i receives treatment and D_i = 0 otherwise. The fundamental problem of causal inference is that one potential outcome is missing for every unit.

For a binary treatment, the observed difference in means is:

This equals the causal effect only under conditions that make treated and untreated groups comparable. Otherwise, it combines a treatment effect with selection differences.

2. A Causal Framework for Business Questions

A well-formed causal question specifies four elements:

• Unit: customer, store, transaction, employee, account, or market.
• Treatment: the intervention or exposure being changed.
• Outcome: the measurable consequence and its time horizon.
• Estimand: the precise causal quantity of interest.

For example: “What is the effect of a 10% discount on 30-day contribution margin among eligible online customers, compared with no discount?” is more useful than “Does discounting improve performance?”

Confounding and selection bias

A confounder affects both treatment assignment and the outcome. Suppose high-value customers are more likely to receive a premium promotion and also spend more independently of the promotion. Comparing promoted and unpromoted customers will overstate the promotion’s effect unless high value is addressed.

A useful conceptual model is:

Here, Y is the outcome, D is treatment, τ is the causal effect under the stated model, X represents covariates, β is their effect, and ε is unexplained variation. A regression coefficient on D is causal only under appropriate design and assumptions; including X mechanically does not remove unmeasured confounding.

Common sources of bias include:

• Confounding: a common cause of treatment and outcome.
• Selection bias: inclusion or treatment depends on variables related to outcomes.
• Reverse causality: the outcome or its precursor influences treatment.
• Post-treatment bias: controlling for a variable affected by treatment blocks part of the treatment effect or creates bias.
• Measurement error: treatment or outcome is recorded inaccurately.
• Interference: one unit’s treatment affects another unit’s outcome.

Causal diagrams, often called directed acyclic graphs, help analysts distinguish confounders from mediators and colliders. A confounder should generally be considered for adjustment; a mediator is part of the pathway through which treatment operates; conditioning on a collider can create a spurious association.

3. Randomized Experiments

Why randomization works

In a randomized controlled trial, eligible units are assigned to treatment or control using a random mechanism. If implementation is successful, treatment assignment is independent of all potential outcomes in expectation:

The symbol ⟂ denotes statistical independence. Randomization balances both observed and, on average, unobserved characteristics. Therefore, the difference in mean outcomes estimates the ATE, subject to implementation and analysis assumptions.

A simple difference-in-means estimator is:

where Ȳ_1 is the sample mean for treated units and Ȳ_0 is the sample mean for control units. If outcomes have sample variances s_1² and s_0² and group sizes n_1 and n_0, the standard error under independent groups is approximately:

A 95% confidence interval is commonly reported as:

where t^* is the relevant critical value from a t distribution. The interval quantifies sampling uncertainty under the design and model; it does not automatically account for bias, poor measurement, or generalization beyond the study population.

Experiment design choices

Unit of randomization. Treatment may be assigned to individuals, stores, geographic regions, or time periods. Cluster randomization is appropriate when customers within a store interact or when treatment naturally operates at store level. Standard errors must then account for within-cluster dependence.

Parallel-arm design. Units are assigned to treatment or control for the entire study. It is simple and robust when carryover is possible.

Factorial design. Two or more interventions are randomized simultaneously, allowing estimation of main effects and interactions. For interventions A and B, an interaction asks whether the effect of A differs depending on whether B is present.

Blocked or stratified randomization. Units are randomized within important strata, such as region or customer segment. This improves balance and precision when strata predict outcomes.

Power and sample size. Sample size depends on baseline variance, minimum detectable effect, significance level, desired power, treatment allocation, and clustering. A study should specify the primary outcome and minimum practically important effect before examining results.

Threats to experimental validity

Randomization does not solve every problem. Noncompliance occurs when assigned treatment differs from treatment received. Attrition occurs when outcomes are missing differentially across groups. Contamination occurs when control units receive treatment indirectly. Spillovers occur when one unit’s treatment changes another unit’s outcome. Multiple outcome or subgroup analyses increase the probability of false discoveries unless planned or adjusted.

The intention-to-treat (ITT) effect compares units according to assignment, regardless of compliance. It preserves the benefit of randomization and answers the operational question: what is the effect of offering or assigning the intervention?

The treatment-on-the-treated or effect among compliers may be more relevant when management wants the effect of actual receipt, but it requires stronger assumptions and a valid instrument such as randomized assignment. ITT is often the primary decision metric.

4. Matching and Observational Adjustment

When randomization is unavailable, analysts may construct a comparison group whose observed characteristics resemble those of treated units. Matching can reduce bias from measured confounders, but it cannot generally remove bias from unmeasured factors.

Let X denote pre-treatment covariates. Under the conditional independence assumption:

and positivity, meaning:

for relevant covariate values, adjustment for X can identify causal effects. The first condition says that, after conditioning on X, treatment assignment is as good as random. The second requires a nonzero probability of both treatment states across the covariate range.

Propensity scores

The propensity score is the probability of treatment conditional on observed covariates:

It may be estimated using logistic regression or a modern prediction method. Matching treated and untreated units with similar propensity scores can improve comparability. Other approaches include exact matching, coarsened exact matching, nearest-neighbour matching, inverse probability weighting, and doubly robust estimators.

A common inverse probability weighting estimator for the ATE is:

The formula requires a correctly specified propensity model or suitable robustness conditions, stable weights, measured confounding, and positivity. Extreme propensity scores can produce unstable weights, so analysts may trim or stabilize weights, reporting the resulting change in the target population.

Matching workflow

1. Define treatment, outcome, time zero, and estimand.
2. Select covariates measured before treatment.
3. Estimate propensity scores or specify a matching rule.
4. Match or weight observations.
5. Assess covariate balance, not merely model fit.
6. Estimate the treatment effect with appropriate uncertainty.
7. Conduct sensitivity analysis for hidden bias.

A standardized mean difference for covariate X is often used to assess balance:

It is dimensionless. Smaller absolute values indicate greater balance, although no universal threshold substitutes for substantive judgment.

Matching is inappropriate when important confounders are unavailable, when treated and untreated groups have little overlap, or when analysts adjust for variables measured after treatment without a clearly defined mediation objective.

5. Difference-in-Differences

Difference-in-differences (DiD) compares changes over time in a treated group with changes in a comparison group that did not receive treatment. It is useful when treatment begins at a known time for some stores, regions, or customer groups.

With groups g ∈ {T,C} and periods t ∈ {pre,post}, the basic estimator is:

The identifying assumption is parallel trends: absent treatment, the treated group would have experienced the same average change as the comparison group, after any systematic differences in levels. Parallel trends concerns unobserved counterfactual changes, so it cannot be proven directly. Pre-treatment trends, institutional knowledge, placebo tests, and alternative control groups can make the assumption more credible.

A regression representation is:

Here, Treat_i identifies the treated group, Post_t identifies the post-treatment period, and δ is the DiD effect under parallel trends. With panel data, unit and time fixed effects are often used:

where α_i are unit fixed effects, λ_t are time fixed effects, and D_it indicates treatment exposure. Standard errors should usually be clustered at the level at which treatment varies, provided the number of clusters is adequate. With few clusters, specialized small-sample or randomization-based methods may be needed.

Extensions and cautions

When treatment begins at different times, a basic two-way fixed-effects model can produce misleading weighted averages, particularly when treatment effects vary over time. Event-study or cohort-specific estimators are often preferable. Analysts should examine anticipation, dynamic effects, treatment reversals, changing composition, concurrent interventions, and spillovers.

6. Instrumental Variables

Instrumental variables (IV) address unmeasured confounding when a variable Z changes treatment exposure but affects the outcome only through treatment. The instrument must satisfy three main conditions:

1. Relevance: Z affects D.
2. Exclusion restriction: Z affects Y only through D.
3. Independence: Z is independent of unobserved determinants of potential outcomes, often conditional on covariates.

A weak first-stage relationship makes IV estimates unstable. A first-stage model may be written:

The second stage substitutes predicted treatment into the outcome model:

The coefficient β_1 is interpreted cautiously. Under additional monotonicity—meaning the instrument does not cause some units to move treatment in the opposite direction—the IV estimator identifies the local average treatment effect (LATE) for compliers, not necessarily the ATE for everyone.

A simple Wald estimator for a binary instrument and treatment is:

The denominator is the instrument’s effect on treatment take-up. The numerator is its reduced-form effect on the outcome. The ratio is meaningful only if the IV assumptions are credible and the denominator is not close to zero.

Examples may include randomized encouragement to use a service, eligibility rules that shift access, or geographic availability when location itself does not directly affect the outcome after adjustment. Business analysts must be especially cautious: geography, sales-representative assignment, or platform exposure often affects outcomes through many channels, violating exclusion.

7. Regression Discontinuity Design

Regression discontinuity (RD) exploits a treatment rule based on whether a running variable crosses a known cutoff. For example, customers with a credit score above a threshold may receive a different offer.

Let R_i be the running variable and c the cutoff. Treatment is assigned when R_i ≥ c. The RD estimand is a local effect at the threshold:

The limits represent expected outcomes immediately above and below the cutoff. The key assumption is continuity: in the absence of treatment, potential outcomes would vary smoothly through c. Units cannot precisely manipulate the running variable around the threshold, or such manipulation must be addressed.

A local linear model is often estimated separately on each side or with an interaction:

The coefficient τ estimates the discontinuity at the cutoff when the specification, bandwidth, and assumptions are appropriate. A narrow bandwidth increases local credibility but reduces precision; a wide bandwidth increases precision but may introduce functional-form bias. Results should be tested across bandwidths and polynomial specifications, with preference for transparent local methods over unnecessarily high-order polynomials.

RD diagnostics include checking the density of observations near the cutoff, continuity of pre-treatment covariates, absence of other policy changes at the same threshold, and sensitivity to bandwidth. The estimate is local and may not generalize to customers far from the cutoff.

8. Worked Example: A Store Promotion

A retailer introduces a digital coupon in 20 stores and selects 20 similar stores as controls. Average weekly contribution margin per store is measured before and after the launch.

The DiD estimate is:

Under the parallel-trends assumption, the coupon increased weekly contribution margin by an estimated $3,000 per treated store. If the 20 treated stores are representative of the intended rollout population, the estimated weekly incremental margin is approximately:

This conclusion requires caution. If treated stores were selected because managers expected stronger growth, if a competitor closed nearby, or if the treatment stores received additional staff, the estimate may not be causal. The analyst should inspect several pre-treatment periods, compare store composition, check concurrent campaigns, and assess whether coupon costs have already been deducted from contribution margin.

9. Illustrative Case: Retention Intervention

Background

A subscription company wants to determine whether proactive customer-service calls reduce cancellations. High-risk customers are selected by a machine-learning model, and service agents call a portion of them.

Problem

The company initially compares called and uncalled high-risk customers. Called customers show a lower cancellation rate, but managers worry that agents may selectively call customers who are more responsive or easier to reach.

Analysis

The analytics team defines the unit as the customer, treatment as an attempted call within seven days of risk identification, and outcome as cancellation within 30 days. The team first proposes randomizing eligible customers to call or no-call groups. It specifies ITT as the primary estimand because management decides whether to offer the intervention, not whether every customer answers.

The design includes stratification by risk band and customer tenure, preregistration of the primary outcome, minimum detectable effect calculations, call-cost tracking, and rules for handling customers who cannot be reached. It also monitors spillovers: a customer’s household or account team may be influenced by conversations with another customer.

If randomization is not feasible, the team considers matching on pre-treatment risk, tenure, plan type, prior service usage, and payment history. Balance and overlap are evaluated before outcome analysis. If a scheduling system randomly allocates call capacity on some days, the allocation may provide an IV, but only if scheduling has no direct effect on cancellation and the exclusion restriction is credible.

Outcome and decision

Suppose the experiment finds that assigned calls reduce cancellation by 1.5 percentage points, with a confidence interval from 0.4 to 2.6 percentage points. The company should combine this estimate with incremental margin, call costs, agent capacity, and possible customer-experience effects. Statistical significance alone does not establish economic value.

Learning points

The case illustrates that causal design begins before data collection. A precisely defined estimand, random assignment, operational compliance measures, transparent outcome definitions, and economic evaluation are more valuable than a sophisticated model applied to a biased comparison.

10. Managerial, Ethical, and Governance Implications

Causal analysis informs high-stakes decisions, including credit, employment, insurance, pricing, and access to services. Analysts should examine heterogeneous effects across protected or vulnerable groups rather than reporting only an average. An intervention that improves average profit may impose disproportionate costs on a subgroup.

Important governance practices include:

• documenting treatment assignment and eligibility rules;
• separating pre-treatment variables from post-treatment variables;
• minimizing collection of unnecessary personal data;
• protecting confidentiality and access controls;
• reporting uncertainty, exclusions, missing data, and deviations from protocol;
• testing fairness and disparate impact where relevant;
• retaining an audit trail for code, data versions, and decisions;
• avoiding claims that exceed the study population or design.

Machine-learning models can improve propensity estimation, heterogeneity analysis, and outcome prediction, but predictive performance does not establish causal validity. Flexible algorithms may overfit, obscure assumptions, or amplify historical inequities. Cross-fitting, sample splitting, causal forests, and doubly robust methods can be useful, but they do not eliminate the need for a credible identification strategy.

11. Best-Practice Causal Workflow

A disciplined workflow is:

1. Translate the business decision into a counterfactual question.
2. Specify unit, treatment, comparison, outcome, timing, and estimand.
3. Draw a causal diagram and identify plausible confounders and mediators.
4. Prefer randomization when ethical and operationally feasible.
5. If observational, choose the design whose assumptions best fit the institutional setting.
6. Check overlap, pre-trends, discontinuities, compliance, attrition, and spillovers.
7. Use robust uncertainty estimation appropriate to clustering and design.
8. Conduct placebo, sensitivity, and alternative-specification analyses.
9. Report practical effect size, uncertainty, costs, and applicability.
10. Monitor post-deployment effects because causal relationships can change.

Chapter Summary

Association describes co-movement, prediction concerns forecast accuracy, and causation concerns counterfactual changes produced by an intervention. Potential outcomes formalize causal effects but reveal the fundamental problem that each unit has only one observed outcome. Randomization creates comparability and supports estimation of average treatment effects, although noncompliance, attrition, interference, and limited generalizability remain important.

Matching and propensity-based methods adjust for measured pre-treatment differences under conditional independence and positivity. Difference-in-differences relies primarily on parallel trends and is useful for policy changes observed over time. Instrumental variables address some unmeasured confounding but require relevance, independence, exclusion, and often monotonicity; they generally identify a local effect for compliers. Regression discontinuity identifies a local treatment effect at a threshold under continuity and limited manipulation.

Credible causal analysis depends more on design and assumptions than on statistical complexity. Business decisions should integrate causal estimates with uncertainty, operational feasibility, economic value, fairness, privacy, and the limits of generalization.

Key Terms

Association: A systematic relationship between variables without necessarily implying causation.

Average treatment effect (ATE): The average difference between treated and untreated potential outcomes.

Causal effect: The change in an outcome attributable to changing treatment for a defined unit or population.

Confounder: A variable related to both treatment assignment and the outcome.

Control group: Units used to represent the counterfactual outcome under no treatment or an alternative treatment.

Difference-in-differences: An estimator comparing outcome changes across treated and comparison groups.

Exclusion restriction: The IV condition that the instrument affects the outcome only through treatment.

Instrumental variable: A variable that shifts treatment and satisfies conditions supporting causal identification.

Intention-to-treat: The effect of assignment to treatment, regardless of treatment receipt.

Local average treatment effect: The IV effect for units whose treatment status responds to the instrument.

Matching: Constructing comparable treated and untreated groups using observed characteristics.

Parallel trends: The DiD assumption that treated and comparison groups would have followed similar untreated trends.

Positivity: A requirement that each relevant covariate profile has a positive probability of each treatment state.

Potential outcome: The outcome a unit would experience under a specified treatment condition.

Propensity score: The conditional probability of receiving treatment given observed covariates.

Regression discontinuity: A design exploiting a treatment rule based on a cutoff in a running variable.

Selection bias: Distortion caused by systematic differences in treatment or inclusion.

Spillover: An effect of one unit’s treatment on another unit’s outcome.

Treatment effect heterogeneity: Variation in causal effects across units or subgroups.

Validity: The credibility and scope of a causal conclusion, including internal and external validity.

Review Questions

Short Answer Questions

1. How do association, prediction, and causation differ?
2. What is the fundamental problem of causal inference?
3. State the parallel-trends assumption in difference-in-differences.
4. What is the purpose of a propensity score?
5. Why does an IV estimate often identify a local average treatment effect?

Long Answer Questions

1. Explain how randomization supports causal inference and discuss four threats that can remain in a randomized experiment.
2. Describe a complete matching workflow, including conditional independence, positivity, balance assessment, and sensitivity analysis.
3. Explain the assumptions and interpretation of a difference-in-differences design.
4. Compare instrumental variables and regression discontinuity in terms of identification, assumptions, and generalizability.
5. Discuss why predictive accuracy from a machine-learning model does not establish a causal effect.

Analytical / Application Questions

1. A retailer’s treated stores increase margin from $30,000 to $35,000, while control stores increase from $28,000 to $31,000. Calculate the DiD estimate and identify two threats to its causal interpretation.
2. A loan offer is available to customers whose score is at least 700. Propose an RD analysis and list the diagnostics required to justify a local causal interpretation.
3. A firm randomly encourages customers to adopt a service, but not everyone complies. Explain which effect ITT estimates, when IV may be used, and what additional assumptions are required for a complier effect.

Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes prediction from causation?

A. Prediction always requires randomization, whereas causation never does.

B. A predictive relationship can forecast outcomes without describing the effect of an intervention.

C. Causation is only concerned with correlation coefficients.

D. Prediction and causation are identical when the sample is large.

Correct Answer: B. A predictive relationship can forecast outcomes without describing the effect of an intervention.

Question 2

In a randomized experiment, the primary purpose of random assignment is to:

A. Guarantee that every observation has the same outcome.

B. Eliminate all measurement error.

C. Make treatment assignment independent of potential outcomes in expectation.

D. Ensure that the treatment effect is identical for every unit.

Correct Answer: C. Make treatment assignment independent of potential outcomes in expectation.

Question 3

A treated group changes from 100 to 130, while a control group changes from 90 to 110. What is the difference-in-differences estimate?

A. 10

B. 20

C. 30

D. 40

Correct Answer: A. 10

Question 4

Which assumption is central to a basic difference-in-differences design?

A. Perfect prediction of every outcome

B. Parallel untreated trends

C. Treatment assignment must be based on a cutoff

D. The instrument must be strongly correlated with the outcome

Correct Answer: B. Parallel untreated trends

Question 5

The propensity score is defined as:

A. The probability of treatment conditional on observed covariates

B. The probability of the outcome conditional only on treatment

C. The average treatment effect in the population

D. The correlation between treatment and outcome

Correct Answer: A. The probability of treatment conditional on observed covariates

Question 6

Which is an essential condition for an instrumental variable?

A. It must directly cause the outcome independently of treatment.

B. It must be unrelated to treatment.

C. It must affect treatment and satisfy a credible exclusion restriction.

D. It must be measured after the outcome occurs.

Correct Answer: C. It must affect treatment and satisfy a credible exclusion restriction.

Question 7

Regression discontinuity primarily identifies:

A. A universal treatment effect for all populations

B. A local treatment effect near the assignment cutoff

C. A prediction error without assumptions

D. An effect only when treatment is randomized globally

Correct Answer: B. A local treatment effect near the assignment cutoff

Question 8

In the formula τ_i = Y_i(1) − Y_i(0), what do Y_i(1) and Y_i(0) represent?

A. Two observed outcomes for the same unit at different times

B. Potential outcomes under treatment and control

C. Two sample variances

D. Predicted and residual outcomes

Correct Answer: B. Potential outcomes under treatment and control

Question 9

Which issue occurs when control units are indirectly affected by treatment assigned to other units?

A. Positivity

B. Spillover or interference

C. Parallel trends

D. Instrument relevance

Correct Answer: B. Spillover or interference

Question 10

The intention-to-treat effect compares units according to:

A. Whether they actually complied with treatment

B. Their treatment assignment

C. Their predicted propensity score only

D. Their outcome after deleting noncompliers

Correct Answer: B. Their treatment assignment