Experimentation, Inference, and Causal Reasoning

Introduction

Business analytics is often used to answer three different questions:

1. What happened? — descriptive analysis.
2. What is likely to happen? — predictive analysis.
3. What will happen if we intervene? — causal analysis.

These questions are related but not interchangeable. A strong correlation may support prediction without establishing that changing one variable will change another. Causal reasoning requires a defensible comparison between what actually happened and what would have happened under an alternative action. Because the alternative outcome is unobservable for the same unit at the same time, analysts use experiments, quasi-experiments, statistical models, and carefully designed comparisons.

This chapter develops the principles required to design and interpret business experiments, especially A/B tests. It also explains hypothesis testing, confidence intervals, regression-based inference, confounding, selection bias, and the validity of analytical conclusions.

Learning Objectives

By the end of this chapter, students should be able to:

• distinguish correlation, prediction, and causation;
• formulate hypotheses and design an appropriate A/B test;
• calculate and interpret differences in means, proportions, confidence intervals, and test statistics;
• explain statistical significance, practical significance, statistical power, and sample-size considerations;
• use regression models for estimation and inference;
• identify confounding, selection bias, interference, and threats to validity;
• evaluate an experiment ethically and communicate its results to decision-makers.

1. Correlation, Prediction, and Causation

Correlation

Correlation describes the strength and direction of association between two variables. The Pearson correlation coefficient is:

Here, Cov(X,Y) is the covariance between variables X and Y, while σ_X and σ_Y are their standard deviations. The value of r lies between −1 and +1. A positive value indicates that larger values of one variable tend to accompany larger values of the other; a negative value indicates an inverse association.

Correlation is symmetric: the correlation of advertising expenditure with sales is the same as the correlation of sales with advertising expenditure. It does not, by itself, establish direction or causality.

Prediction

Prediction concerns the accuracy with which an outcome can be estimated for new or future observations. A model may predict customer churn effectively using variables that are not causes of churn. For example, a customer-service complaint may predict cancellation because it is a symptom of dissatisfaction, even though eliminating the recorded complaint would not necessarily prevent cancellation.

Prediction is evaluated using out-of-sample performance measures such as mean absolute error, root mean squared error, accuracy, precision, recall, or area under the ROC curve. Predictive usefulness and causal interpretation are separate criteria.

Causation

A causal claim states that an intervention changes an outcome. In potential-outcomes notation, let Y_i(1) denote the outcome for unit i if it receives treatment and Y_i(0) the outcome if it does not. The individual causal effect is:

Only one of these two potential outcomes is observed for a given unit. This is the fundamental problem of causal inference. Experiments solve it approximately at the group level by creating comparable treatment and control groups through random assignment.

The average treatment effect (ATE) is:

where E[·] denotes the population average. The ATE is meaningful only with a clearly defined treatment, outcome, population, time period, and assignment mechanism.

2. The Logic of a Business Experiment

An experiment deliberately assigns an intervention and compares outcomes with those of a control condition. A typical digital A/B test assigns users to:

• Control group: receives the existing experience;
• Treatment group: receives the proposed change.

Random assignment makes treatment status independent, in expectation, of both observed and unobserved characteristics. Therefore, systematic differences in average outcomes after assignment can be attributed to the treatment, provided the experiment is implemented correctly.

A basic experimental workflow is:

1. define the decision and estimand;
2. specify the population and unit of randomization;
3. select primary and secondary outcomes;
4. calculate a sample-size requirement;
5. randomize units to treatment and control;
6. monitor implementation and data quality;
7. estimate the treatment effect and uncertainty;
8. interpret business significance and validity;
9. make, document, and review the decision.

Treatment, unit, and outcome

The unit may be a customer, transaction, store, employee, household, or geographic area. The unit of randomization should generally match the unit that could be affected by treatment. Randomizing individual users is inappropriate if users influence one another or if treatment must be delivered at store level.

The treatment must be operationally precise. “Improved customer experience” is not a sufficiently defined treatment; “a checkout page displaying one recommendation instead of three” is more reproducible.

The outcome should be measurable and linked to the decision. Examples include conversion rate, average order value, retention after 30 days, delivery time, defect rate, or profit per eligible customer.

Randomization and balance

Suppose n_T treated units have average outcome Ȳ_T and n_C control units have average outcome Ȳ_C. The difference in means is:

For a binary outcome, such as purchase or no purchase, the estimated effect is the difference in conversion rates:

where p̂_T and p̂_C are the sample proportions in treatment and control. Randomization does not guarantee identical groups in every sample; it makes systematic imbalance unlikely as sample size increases. Analysts should inspect pre-treatment variables and investigate major imbalances, but should not discard randomization merely because every characteristic is not numerically equal.

Control groups and treatment contamination

A control group provides the counterfactual benchmark. A valid control group should experience the same measurement process, calendar conditions, and eligibility rules as treatment, except for the intervention.

Contamination occurs when control units receive some treatment exposure. Conversely, treatment units may fail to receive the assigned treatment. These situations are often called noncompliance. An intention-to-treat analysis preserves the original assignment and estimates the effect of being offered or assigned treatment, which is usually the most credible operational effect.

3. Hypothesis Testing and Uncertainty

Null and alternative hypotheses

A test begins with a null hypothesis, usually representing no effect:

• H₀: μ_T − μ_C = 0
• H₁: μ_T − μ_C ≠ 0

A one-sided alternative may be appropriate when only improvement matters and the direction was specified before observing the data, for example H₁: μ_T − μ_C > 0. Choosing a one-sided test after seeing an unfavorable result is a form of analytical flexibility that inflates false-positive risk.

Standard error and test statistic

For two independent sample means, a commonly used large-sample standard error is:

where s_T² and s_C² are sample variances and n_T and n_C are sample sizes. The corresponding test statistic is:

This approximation is appropriate when observations are independent and sample sizes are sufficient for the sampling distribution to be approximately normal. For clustered, repeated, or highly skewed data, the standard error must be adapted.

For two independent proportions, an approximate standard error under the null hypothesis is:

where the pooled proportion is:

and x_T and x_C are the numbers of successes. For estimation and confidence intervals, an unpooled standard error is generally used:

p-values and significance

A p-value is the probability, under the null model and assumptions of the test, of obtaining a result at least as extreme as the observed result. It is not the probability that the null hypothesis is true, nor is it the probability that the finding will replicate.

A conventional significance level is α = 0.05. If the p-value is below α, analysts reject the null hypothesis according to the prespecified decision rule. This does not prove the alternative hypothesis, and failure to reject does not prove that the effect is zero.

Confidence intervals

A two-sided approximate 95% confidence interval for an estimated difference is:

The multiplier 1.96 applies under a normal approximation. A confidence interval gives a range of parameter values compatible with the data and model at the stated confidence level. It conveys both direction and precision and is usually more informative than a p-value alone.

Worked example: conversion experiment

An online retailer randomly assigns 10,000 visitors to a control page and 10,000 to a redesigned page. The control page produces 800 purchases; the treatment page produces 880.

The conversion rates are:

• Control: p̂_C = 800 ÷ 10,000 = 0.080 or 8.0%.
• Treatment: p̂_T = 880 ÷ 10,000 = 0.088 or 8.8%.

The absolute effect is:

Thus, conversion increased by 0.8 percentage points. The relative lift is:

Using the unpooled standard error:

An approximate 95% confidence interval is:

Therefore, the interval is approximately 0.00028 to 0.01572, or 0.028 to 1.572 percentage points. Because the interval barely excludes zero, the result is statistically significant at approximately the 5% level. Whether it is commercially worthwhile depends on implementation cost, margin, customer experience, and possible effects on later outcomes.

4. Power, Sample Size, and Multiple Testing

Statistical power

Power is the probability that a test rejects the null hypothesis when a specified nonzero effect truly exists. It depends on effect size, sample size, outcome variability, significance level, and the alternative hypothesis. Low-powered studies may produce inconclusive results even when an important effect exists.

Before collecting data, analysts should specify:

• baseline conversion or mean;
• minimum practically important effect;
• desired power, often 80% or 90%;
• significance level;
• allocation ratio;
• expected attrition or noncompliance.

For a two-group comparison of proportions, exact sample-size formulas depend on the selected test and assumptions. Software should be used, but analysts must understand the inputs and verify that the resulting sample is feasible.

Practical significance

An effect can be statistically significant but too small to justify implementation. Conversely, an important effect may fail to reach significance because the experiment is underpowered. Decisions should consider confidence intervals, expected financial value, strategic effects, risks, and implementation costs.

Multiple comparisons and peeking

Testing many outcomes, segments, or treatment variants increases the chance of finding at least one apparently significant result by chance. If each of m independent tests uses significance level α, the probability of at least one false positive is:

This expression assumes independence and is illustrative rather than universally exact. Analysts can control error using prespecified primary outcomes, correction procedures such as Bonferroni or false-discovery-rate methods, or hierarchical testing.

Repeatedly checking results and stopping when significance appears can also inflate false-positive risk. Sequential testing requires a prespecified sequential design or appropriate statistical adjustment.

5. Regression-Based Inference

Regression describes the conditional relationship between an outcome and explanatory variables. A linear model is:

Here, Y_i is the outcome, T_i is treatment assignment, X_ij are additional covariates, β₀ is the intercept, β₁ is the treatment coefficient, and ε_i is the error term. In a randomized experiment, β₁ estimates the average treatment effect under suitable conditions.

Covariate adjustment can improve precision when pre-treatment variables predict the outcome. It should not ordinarily include variables caused by treatment, such as post-treatment satisfaction, because conditioning on mediators can distort the total treatment effect.

Regression assumptions

Important assumptions include:

• correct definition and timing of variables;
• independent or appropriately modelled errors;
• no perfect multicollinearity;
• a reasonable functional form;
• valid treatment assignment for causal interpretation;
• an appropriate variance estimator.

Randomized treatment protects against confounding, but it does not automatically solve clustering, attrition, missing data, interference, or measurement error. Robust or cluster-robust standard errors may be required when observations within stores, regions, classrooms, or users are correlated.

Observational regression

In nonexperimental data, regression adjustment alone does not guarantee causality. A model can control only for measured variables included appropriately. If an unmeasured factor affects both treatment and outcome, the treatment coefficient may remain confounded.

For example, customers who voluntarily use a premium support channel may have more serious problems and may also be more likely to churn. A negative association between support use and retention does not show that support causes churn.

6. Confounding, Selection Bias, and Validity

Confounding

A confounder is a variable associated with both the treatment and outcome that creates a noncausal association. Consider a retailer studying training and employee sales. High-performing stores may be more likely to receive training and may already have better sales. Store performance is a potential confounder.

Randomization balances confounders in expectation. In observational studies, analysts may use matching, stratification, regression adjustment, inverse-probability weighting, difference-in-differences, instrumental variables, or regression discontinuity. Each method requires assumptions that must be explained rather than treated as automatic solutions.

Selection bias

Selection bias occurs when inclusion in the analyzed sample is related to treatment, outcome, or both. Examples include:

• surveying only customers who respond;
• analyzing only users who remain active after treatment;
• comparing voluntary adopters with non-adopters;
• excluding observations with inconvenient outcomes.

Randomized assignment does not protect against biased analysis after assignment. Attrition should be reported by group, and analysts should compare intention-to-treat results with appropriately justified secondary analyses.

Internal and external validity

Internal validity concerns whether the estimated effect is credible for the studied population, treatment, and period. Threats include poor randomization, contamination, differential attrition, inaccurate measurement, interference, and selective reporting.

External validity concerns whether results generalize to other populations, times, settings, channels, or versions of the intervention. A test conducted among frequent mobile shoppers during a holiday period may not generalize to occasional desktop users in another season.

Interference and spillovers

The stable unit treatment value assumption requires that one unit’s outcome is not affected by another unit’s treatment. This may fail when customers share referrals, employees work in teams, or prices affect market-wide demand. Cluster randomization or geographic experimentation may be more appropriate than individual randomization.

7. Designing a High-Quality A/B Test

A credible experiment begins with a decision statement: “Should the firm deploy treatment X for population Y because it changes outcome Z over period W by at least the value required to justify its cost?”

The experimental protocol should document:

• eligibility criteria;
• randomization unit and method;
• allocation ratio;
• treatment details;
• primary outcome and measurement window;
• guardrail metrics;
• minimum detectable effect;
• sample-size and stopping rule;
• exclusion rules defined before analysis;
• handling of missing data and noncompliance;
• privacy, safety, and approval requirements.

Guardrail metrics detect harmful side effects. A checkout redesign may increase conversion but also increase refunds, customer complaints, fraud, or delivery failures. A decision should therefore consider a vector of outcomes, not a single metric in isolation.

Analysis versus interpretation

An analyst should report:

1. the estimated absolute effect;
2. the relative effect where useful;
3. uncertainty, preferably a confidence interval;
4. sample size and exposure;
5. the analysis population;
6. deviations from the protocol;
7. business costs and benefits;
8. limitations and generalizability.

“Treatment won” is insufficient communication. A responsible conclusion might be: “The redesign increased conversion by an estimated 0.8 percentage points, with a 95% confidence interval from 0.03 to 1.57 points. The evidence supports deployment for the tested population, subject to monitoring refund rates and performance in lower-frequency customers.”

8. Illustrative Case: A Subscription Retention Experiment

Background

A streaming company observes that many new subscribers cancel during their first month. The product team proposes a personalized onboarding sequence containing reminders, content recommendations, and a short tutorial.

Problem

The company must determine whether the sequence improves 30-day retention without reducing customer satisfaction or increasing customer-service contacts.

Analysis and approach

Eligible new subscribers are randomly assigned at the account level to control or treatment. Account-level assignment prevents multiple household devices from receiving different experiences. The primary outcome is whether the subscription remains active 30 days after signup. Secondary outcomes include viewing hours, support contacts, refunds, and satisfaction.

The team preregisters a minimum practically important effect of 1 percentage point in retention. It estimates the required sample size using the baseline retention rate, target power, significance level, and expected attrition. Results are analyzed using intention-to-treat assignment. A logistic regression is used as a precision-enhancing secondary analysis with country and acquisition channel measured before treatment.

Outcome

Suppose retention is 64.0% in control and 65.3% in treatment. The estimated absolute effect is 1.3 percentage points. The team reports a confidence interval and finds that support contacts did not materially increase, while viewing hours rose modestly. However, the effect is concentrated among users acquired through paid search and is close to zero among users acquired through partnerships.

Decision

The company deploys the treatment first for paid-search users, continues monitoring retention and support contacts, and conducts a follow-up experiment for partnership users rather than assuming that the effect generalizes.

Learning points

The case demonstrates that causal evidence depends on randomization, a prespecified outcome, an appropriate unit, and transparent uncertainty. It also shows why a statistically positive overall effect may conceal important heterogeneity and why deployment decisions should include guardrails and implementation economics.

9. Ethics, Governance, and Contemporary Practice

Experiments involving customers or employees must respect privacy, autonomy, fairness, and applicable law. Organizations should minimize data collection, restrict access, protect identifiers, and document retention periods. Sensitive attributes should not be used to create discriminatory experiences or conceal unequal harms.

Ethical review is especially important when treatment may affect health, financial access, employment, safety, or essential services. Deception, differential pricing, dark patterns, and experiments that exploit vulnerable users require heightened scrutiny.

Modern analytics platforms support feature flags, automated randomization, real-time dashboards, sequential monitoring, and machine-learning-based personalization. These tools improve speed but can also increase risks: repeated experimentation, algorithmic feedback loops, metric gaming, and opaque treatment assignment. AI-generated hypotheses or segments should not replace prespecified design, human review, or statistical validation.

Organizations should maintain an experiment registry containing the hypothesis, owner, population, treatment, metrics, dates, analysis plan, result, and decision. This reduces publication bias and preserves institutional learning from inconclusive and negative experiments.

Best Practices

• Define the causal question before examining results.
• Randomize at the level at which treatment is delivered and interference is limited.
• Prespecify primary outcomes, stopping rules, and important subgroups.
• Report absolute effects, relative effects, confidence intervals, and sample sizes.
• Distinguish statistical significance from practical value.
• Use intention-to-treat analysis as the primary operational estimate.
• Monitor guardrail metrics and treatment implementation.
• Avoid controlling for post-treatment variables in total-effect models.
• Use cluster-robust methods or cluster randomization when observations interact.
• Treat observational regression as causal only when its identifying assumptions are credible.
• Investigate missing data, attrition, noncompliance, and selection.
• Document limitations and avoid generalizing beyond the tested context.
• Protect privacy and assess fairness before deployment.

Chapter Summary

Experimentation provides a structured way to estimate causal effects by comparing treatment and control conditions. Random assignment is powerful because it balances confounding factors in expectation, but valid conclusions also require correct implementation, reliable measurement, adequate sample size, and transparent analysis.

The difference in treatment and control outcomes estimates an effect, while standard errors and confidence intervals quantify uncertainty. Hypothesis tests support decisions but do not measure practical value or prove that a hypothesis is true. Power, multiple testing, and stopping rules must be considered during design rather than after results are observed.

Regression can improve precision in randomized experiments and can support causal inference in observational studies only under defensible assumptions. Confounding, selection bias, interference, attrition, and poor generalizability threaten validity. Effective analysts therefore distinguish correlation, prediction, and causation; report limitations; include guardrail outcomes; and integrate statistical evidence with economic, ethical, and operational judgment.

Key Terms

Causal effect: Change in an outcome attributable to an intervention.

Potential outcome: Outcome that would occur for a unit under a specified treatment condition.

Average treatment effect: Population-average difference between treated and untreated potential outcomes.

A/B test: Randomized comparison of a control condition and a treatment condition.

Randomization: Assignment mechanism giving units a specified chance of receiving each condition.

Control group: Units receiving the baseline or comparison condition.

Treatment group: Units receiving the intervention being evaluated.

Counterfactual: Outcome that would have occurred under the alternative condition.

Correlation: Statistical association between variables.

Prediction: Estimation of unknown or future outcomes.

Confounding: Distortion caused by a factor related to both treatment and outcome.

Selection bias: Bias caused by nonrepresentative inclusion or analysis of units.

P-value: Probability of data at least as extreme as observed under the null model.

Confidence interval: Interval expressing uncertainty about an estimated parameter.

Statistical power: Probability of detecting a specified effect when it exists.

Practical significance: Importance of an effect for decisions or stakeholders.

Intention-to-treat: Analysis based on original treatment assignment.

Interference: Situation in which one unit’s treatment affects another unit’s outcome.

Internal validity: Credibility of a causal estimate within the studied setting.

External validity: Transferability of findings to other settings or populations.

Review Questions

Short Answer Questions

1. How do correlation, prediction, and causation differ?
2. Why does random assignment help address confounding?
3. What does a p-value measure, and what does it not measure?
4. Why should experiments report guardrail metrics?
5. What is the purpose of intention-to-treat analysis?

Long Answer Questions

1. Explain how to design an A/B test from the business question through final decision-making.
2. Discuss the assumptions required for regression-based causal inference.
3. Explain statistical significance, practical significance, confidence intervals, and power.
4. Compare internal validity and external validity, including threats to each.
5. Discuss how selection bias, attrition, and interference can undermine an experiment.

Analytical / Application Questions

1. A treatment increases conversion from 5.0% to 5.4% but increases refunds by 0.8 percentage points. What additional analyses should management request before deployment?
2. A company compares voluntary users of a training program with non-users and finds higher sales among participants. Identify plausible confounders and propose a stronger design.
3. An experiment is significant overall but has a negative effect for one customer segment. How should the analyst investigate heterogeneity and communicate the deployment decision?

Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes prediction from causation?

A. Prediction always requires randomization.

B. A predictive variable must be a cause of the outcome.

C. A model may predict an outcome accurately without showing that changing the predictor causes the outcome.

D. Causal effects cannot be estimated quantitatively.

Correct Answer: C. A model may predict an outcome accurately without showing that changing the predictor causes the outcome.

Question 2

In an A/B test, the primary purpose of random assignment is to:

A. Guarantee identical outcomes in both groups before treatment.

B. Balance observed and unobserved characteristics in expectation.

C. Ensure that every participant receives the treatment.

D. Eliminate the need for a control group.

Correct Answer: B. Balance observed and unobserved characteristics in expectation.

Question 3

A treatment conversion rate is 12% and the control conversion rate is 10%. What is the absolute treatment effect?

A. 2 percentage points

B. 20 percentage points

C. 16.7 percentage points

D. 1.2 percentage points

Correct Answer: A. 2 percentage points

Question 4

Which interpretation of a p-value is correct?

A. It is the probability that the null hypothesis is true.

B. It is the probability that the observed result will replicate.

C. It is the probability of data at least as extreme as observed under the null model.

D. It is the probability that the treatment has practical value.

Correct Answer: C. It is the probability of data at least as extreme as observed under the null model.

Question 5

Which practice most directly helps control false positives from repeatedly examining many outcomes?

A. Choose the most favorable outcome after analysis.

B. Prespecify a primary outcome and analysis plan.

C. Remove all control observations.

D. Increase the number of reported decimal places.

Correct Answer: B. Prespecify a primary outcome and analysis plan.

Question 6

Why is intention-to-treat analysis commonly used as the primary analysis of a randomized experiment?

A. It preserves the causal value of the original assignment.

B. It guarantees a statistically significant result.

C. It excludes participants who did not comply.

D. It estimates only the effect among perfect compliers.

Correct Answer: A. It preserves the causal value of the original assignment.

Question 7

A variable associated with both treatment assignment and the outcome is called a:

A. Mediator only

B. Confounder

C. Guardrail metric

D. Randomization unit

Correct Answer: B. Confounder

Question 8

Which variable should generally not be included as a control when estimating the total effect of treatment?

A. A pre-treatment customer age category

B. A pre-treatment acquisition channel

C. A post-treatment satisfaction score caused by treatment

D. A pre-treatment historical purchase count

Correct Answer: C. A post-treatment satisfaction score caused by treatment

Question 9

If observations within stores are correlated, which approach may be appropriate?

A. Ignore the store structure completely.

B. Use cluster-robust standard errors or cluster randomization.

C. Delete all observations except one per store.

D. Treat every observation as independent without qualification.

Correct Answer: B. Use cluster-robust standard errors or cluster randomization.

Question 10

A confidence interval for an estimated treatment effect includes zero. Which conclusion is most appropriate?

A. The treatment definitely has no effect.

B. The null hypothesis is proven true.

C. The data do not provide clear evidence of a nonzero effect at that confidence level.

D. The experiment was necessarily fraudulent.

Correct Answer: C. The data do not provide clear evidence of a nonzero effect at that confidence level.