Statistical Inference, Experimentation and Causal Analysis

Introduction

Data science rarely concerns description alone. A dashboard may report that two groups differ, but a decision-maker usually asks a harder question: how large is the uncertainty, is the difference compatible with chance, and did a particular intervention cause the observed outcome? Statistical inference addresses uncertainty from samples. Experimentation creates evidence by deliberately assigning interventions. Causal analysis attempts to estimate intervention effects when randomization is unavailable.

These activities are central to AI and ML practice. They support model comparison, product experimentation, clinical evaluation, policy analysis, pricing decisions and responsible deployment. The central discipline is to distinguish a pattern in observed data from a defensible claim about a population or a cause.

Learning Objectives

By the end of this chapter, a learner should be able to:

1. Construct and interpret confidence intervals and hypothesis tests.
2. Explain sampling distributions, errors, power and minimum detectable effects.
3. Design and analyze A/B, factorial and multivariate experiments.
4. Use causal diagrams to identify confounding, mediation and selection bias.
5. Evaluate matching, instrumental variables and difference-in-differences.
6. Communicate uncertainty, assumptions, limitations and ethical implications.

1. Statistical Inference: From Samples to Populations

A population is the complete set of units relevant to a question; a sample is the observed subset. A parameter, such as a population mean μ or treatment effect τ, is generally unknown. A statistic, such as the sample mean x̄, estimates that parameter.

Inference depends on a sampling distribution: the probability distribution of a statistic over repeated samples drawn under a specified design. The standard error measures the typical sampling variability of an estimator. It is not the same as the standard deviation of individual observations.

For independent observations Y₁, …, Yₙ with finite variance σ², the sample mean has

where s is the sample standard deviation. Independence and a suitable sampling process are important. For large n, the central limit theorem often gives an approximately normal sampling distribution even when individual observations are not normal, although heavy tails, dependence and small samples require caution.

1.1 Confidence Intervals

A two-sided confidence interval for a mean with unknown population standard deviation is commonly

Here α is the significance level, t is a Student's t critical value with n−1 degrees of freedom, and the interval is applicable when observations are approximately independent and the mean-based approximation is reasonable. For a large sample, t may be replaced by the standard normal critical value z.

A 95% confidence procedure means that, over repeated samples and under the model, approximately 95% of intervals constructed by this method contain the fixed parameter. It does not strictly mean that there is a 95% probability that one already-computed fixed parameter lies in the interval.

For a population proportion p, with sample proportion p̂ = x/n, a large-sample Wald interval is p̂ ± z_{1−α/2}√[p̂(1−p̂)/n]. However, the Wald method performs poorly with small samples or proportions near zero or one. Wilson or exact binomial intervals are usually preferable in those settings.

For two independent means, an important practical method is Welch's interval for μ₁ − μ₂:

The Welch–Satterthwaite degrees of freedom are

This avoids assuming equal population variances.

Worked Example: Estimating a Conversion Difference

An existing website converts 420 of 10,000 visitors, while a redesigned website converts 462 of 10,000. The estimated difference is 0.0462 − 0.0420 = 0.0042, or 0.42 percentage points. The approximate standard error for two independent proportions is

Using 1.96 for a 95% normal approximation gives a margin of error of about 0.00572. The approximate interval is −0.00152 to 0.00992, or −0.152 to 0.992 percentage points. Because zero lies inside the interval, the data do not establish a statistically detectable difference at the 5% level. The estimated uplift is positive, but its uncertainty includes no effect and modest benefit.

2. Hypothesis Testing and Decision Errors

A hypothesis test begins with a null hypothesis H₀ and an alternative hypothesis H₁. For example, H₀: μ₁ − μ₀ = 0 and H₁: μ₁ − μ₀ ≠ 0. A test statistic measures how far the data are from the null prediction, relative to sampling variability.

For a difference in independent means, a Welch statistic is

where δ₀ is the null difference, commonly zero. The p-value is the probability, assuming H₀ and the analysis model are correct, of obtaining a result at least as extreme as the observed result. It is not the probability that H₀ is true, nor the probability that the result will replicate.

A Type I error rejects a true null hypothesis; its controlled probability is α under the planned procedure. A Type II error fails to reject a false null; its probability is β. Power is 1 − β, the probability of detecting a specified effect under a specified alternative. “Fail to reject” should not be described as proof of no effect.

Statistical significance and practical significance differ. A huge sample may make a trivial effect statistically significant, while a consequential effect may lack significance in a small, noisy study. Report effect sizes and confidence intervals, not p-values alone.

Multiple Comparisons

Testing many hypotheses increases the chance of at least one false positive. If m independent tests each use α = 0.05, the probability of at least one false rejection is approximately 1 − (1 − 0.05)^m. Dependence changes the exact value, but the problem remains.

Pre-specifying primary outcomes, controlling the family-wise error rate, or controlling the false discovery rate can be appropriate. Bonferroni uses α/m for each of m tests and is simple but conservative. The Benjamini–Hochberg procedure controls expected false discovery proportion under stated dependence conditions and is often useful in exploratory high-dimensional analysis.

Sequentially checking results and stopping when a p-value crosses 0.05 invalidates ordinary fixed-sample inference unless a sequential method or alpha-spending plan is used. Peeking is a design problem, not merely a reporting problem.

3. Power Analysis and Sample-Size Planning

Power analysis should occur before data collection. It requires a significance level, target power, outcome variability or baseline rate, allocation ratio, and a minimum detectable effect (MDE). The MDE must be practically meaningful, not merely convenient mathematically.

For a two-arm comparison of independent means with equal group sizes, a useful planning approximation is

where Δ is the absolute effect to detect, σ is the assumed common standard deviation, α is the two-sided Type I error rate, and 1−β is target power. This approximation assumes independent observations, equal allocation and a normal approximation.

For a two-sided α = 0.05 test with 80% power, z_{1−α/2} ≈ 1.96 and z_{1−β} ≈ 0.84. If σ = 10 units and Δ = 2 units,

Thus approximately 392 observations per group are required before allowing for attrition, clustering or noncompliance.

For binary outcomes, sample-size calculations depend on both baseline and treatment rates; software or exact methods are preferable. Cluster-randomized studies require an inflation factor called the design effect:

where m is the average cluster size and ρ is the intracluster correlation coefficient. The effective sample size is approximately the enrolled sample size divided by DE. Ignoring clustering can substantially overstate precision.

Power is not a property of a test alone. It changes with effect size, noise, sample size, allocation, missingness, multiplicity and analysis method. Post hoc power calculations based on the observed estimate add little beyond the confidence interval; planning should use scientifically justified assumptions.

4. Randomized Experiments and A/B Testing

In an A/B test, eligible units are randomly assigned to control or treatment. Randomization makes treatment assignment independent of potential outcomes in expectation, so differences in average outcomes can estimate a causal effect.

Let Yᵢ(1) be unit i's outcome under treatment and Yᵢ(0) under control. The individual treatment effect is Yᵢ(1) − Yᵢ(0), but both potential outcomes cannot be observed for the same unit. The average treatment effect (ATE) is

Under random assignment, consistency, no interference between units, and adequate adherence, the difference in observed group means estimates the ATE. These assumptions are often summarized as SUTVA plus randomization. Interference can arise in social networks, marketplaces and shared-resource systems.

A sound experiment specifies the unit of randomization, eligibility, treatment, control, primary outcome, analysis window, minimum detectable effect and stopping rule. Randomize at the level at which contamination is unlikely. If users share accounts, devices or teams, user-level randomization may create interference; cluster assignment may be safer but reduces effective sample size.

Covariate adjustment can improve precision. In randomized studies, pre-treatment covariates may be included in regression or ANCOVA, but post-treatment variables should not be adjusted for when estimating the total treatment effect because they may be mediators.

Factorial and Multivariate Experiments

A 2×2 factorial experiment tests two factors, A and B, at two levels each. It estimates the main effect of A, the main effect of B and their interaction. An interaction means the effect of A depends on the level of B. With balanced cell means μ₀₀, μ₁₀, μ₀₁ and μ₁₁, the additive interaction contrast is

A zero contrast indicates additivity on the chosen outcome scale, not necessarily on another scale such as odds or log risk. Multivariate experiments test several treatments or features simultaneously, but they require multiplicity control, sufficient allocation to each arm and careful interpretation of treatment-by-segment interactions.

Experiment Monitoring and Validity

Randomization balance should be checked descriptively, not used as a reason to discard randomization after seeing outcomes. Primary outcomes and analysis methods should be registered before treatment exposure. Guardrail metrics—such as latency, complaints, cancellations or safety events—prevent optimization of one metric at unacceptable cost.

Common threats include novelty effects, seasonality, treatment leakage, noncompliance, attrition, bot traffic, repeated users and metric manipulation. Intention-to-treat analysis preserves the benefit of random assignment by analyzing units according to assigned group. Per-protocol analysis may estimate the effect among adherers but is vulnerable to post-randomization selection.

5. Causal Questions and Potential Outcomes

Causal inference begins by defining a treatment, outcome, population, time horizon and estimand. “Does the recommendation system improve engagement?” is incomplete until treatment, engagement measure, eligible users and observation period are specified.

The ATE is a population average. The average treatment effect on the treated (ATT) is

where T indicates treatment. A conditional average treatment effect varies across covariates X:

Heterogeneity can guide personalization, but searching many subgroups can produce false discoveries. Subgroup analyses should be pre-specified or validated on new data.

6. Causal Diagrams and Identification

A directed acyclic graph (DAG) represents assumed causal relationships. Arrows encode direct causal influence; a common cause of treatment and outcome is a confounder. A backdoor path begins with an arrow into treatment. Conditioning on an appropriate set of pre-treatment variables can block such paths.

Figure: A confounder creates a noncausal backdoor path between treatment and outcome.

A valid adjustment set must not include descendants of treatment that are mediators or colliders. A collider is a variable caused by two variables; conditioning on it can create a spurious association. For example, conditioning on “accepted into a program” may induce association between prior ability and financial need if both influence acceptance.

DAGs encode assumptions; they do not prove them. Their value is making assumptions explicit and guiding which variables should be measured and adjusted for. Standard regression can estimate an adjusted association, but causal interpretation requires identification assumptions, correct temporal ordering and adequate measurement.

7. Matching and Propensity Scores

In observational data, treated and untreated units may differ systematically. Matching attempts to create comparable groups using observed pre-treatment covariates. Exact matching is often infeasible with many continuous variables. The propensity score is

It is commonly estimated using logistic regression or flexible classifiers. Matching, stratification or weighting on e(X) can balance measured covariates, but it cannot balance unmeasured confounders merely because the propensity model predicts treatment well.

Important diagnostics include overlap or positivity: each relevant covariate profile must have a nonzero probability of receiving each treatment. Covariate balance should be assessed after matching, often using standardized mean differences:

where x̄ and s are group means and standard deviations. A small SMD is a diagnostic convention, not a universal proof of balance. Poor overlap means the data cannot credibly estimate effects for some units without extrapolation.

Matching changes the target population depending on the algorithm. Standard errors must account for the design and estimated weights. Doubly robust estimators combine an outcome model and a treatment model; under suitable conditions, consistency can be retained if at least one of the two models is correctly specified, alongside standard identification assumptions. “Doubly robust” does not protect against hidden confounding, interference or severe positivity violations.

8. Instrumental Variables

An instrumental variable Z changes treatment T but affects outcome Y only through T. A common diagram is Z → T → Y, with no direct Z → Y path and no unblocked confounding between Z and Y. The core assumptions are relevance, exclusion, independence and often monotonicity.

With a binary instrument and treatment, the Wald estimator is

The numerator is the intention-to-treat effect on the outcome; the denominator is the instrument's effect on treatment uptake. Units must be compatible: if Y is revenue and T is a treatment probability, the ratio is revenue per additional treated unit.

Under assumptions including monotonicity, the IV estimate identifies a local average treatment effect (LATE) for compliers—units whose treatment status is changed by the instrument—not necessarily the ATE for everyone. Weak instruments produce unstable estimates and large uncertainty. Exclusion is often the most difficult assumption: an instrument such as distance, eligibility threshold or encouragement may affect outcomes through other routes.

9. Difference-in-Differences

Difference-in-differences (DiD) compares outcome changes over time for a treated group with outcome changes for a comparison group:

The key assumption is parallel trends: absent treatment, the treated and comparison groups would have experienced the same average change. This is not proven by a single pre-treatment comparison. Multiple pre-periods, event-study plots and placebo tests can provide evidence, although they cannot guarantee future parallel trends.

A regression representation is

where δ is the DiD effect under the design assumptions. Standard errors should account for serial correlation and clustering at the treatment-assignment level. With staggered treatment timing and heterogeneous effects, conventional two-way fixed-effects estimators can be misleading; modern event-study or group-time methods are preferable.

10. Illustrative Case: Evaluating a Fraud-Detection Intervention

Background and Problem

A financial platform introduces an ML fraud-warning model. Analysts want to know whether showing warnings reduces confirmed fraud without excessively increasing legitimate-payment declines. Historical users who saw warnings differ from users who did not because the model targeted high-risk transactions.

Analysis

A naive comparison finds 4.8% confirmed fraud among warned transactions and 1.2% among unwarned transactions. This does not imply that warnings cause fraud: risk caused both model targeting and fraud. The comparison is confounded by baseline risk.

The platform first conducts a randomized experiment among transactions eligible for the warning. Within risk strata, transactions are randomly assigned to warning or no warning. Primary outcomes are confirmed fraud within 30 days and legitimate-payment completion; guardrails include customer complaints and latency. The analysis is intention-to-treat, with a pre-specified primary endpoint and correction for multiple secondary outcomes.

Suppose 50,000 transactions per arm are observed. Fraud is 1,000/50,000 = 2.0% in control and 850/50,000 = 1.7% in treatment. The estimated absolute reduction is 0.003, or 0.3 percentage points. The approximate number needed to treat is 1/0.003 ≈ 333, interpreted cautiously because the effect is an estimate with uncertainty. If legitimate completion declines by 1.1 percentage points, the business must evaluate whether preventing fraud justifies that cost; a statistically significant fraud reduction is not automatically a desirable product decision.

For users who cannot be randomized because warnings were deployed universally, analysts consider DiD using a rollout by region. They inspect pre-treatment trends and account for region-level clustering. They do not adjust for post-warning customer complaints when estimating the total effect, because complaints may be downstream of treatment.

Outcome and Learning Points

The case demonstrates that targeting creates confounding, randomization directly addresses it, guardrails formalize multi-objective decisions, and observational methods require explicit assumptions. It also illustrates that an ML model can improve prediction while a product intervention still causes undesirable behavioral or equity effects.

11. Ethics, Governance and Reproducible Practice

Inference and causal analysis affect access to credit, healthcare, employment and public services. Ethical practice requires informed governance of sensitive data, proportionality, privacy protection and assessment of disparate impact. Randomization is not automatically ethical: withholding a known effective treatment or exposing participants to material risk may be unacceptable. Institutional review, consent or waiver procedures and safety monitoring may be required.

Analysts should document the estimand, population, treatment versions, exclusions, missing-data rules, stopping rules, code, data provenance and deviations from the protocol. Sensitivity analysis should examine plausible unmeasured confounding, alternative outcome definitions, attrition and model specifications. Negative controls, placebo interventions and falsification tests can reveal some design failures.

AI systems introduce additional concerns: adaptive experimentation can change treatment assignment over time; automated targeting can amplify historical bias; feedback loops can make outcomes depend on earlier model decisions. Causal effects should be evaluated across relevant subgroups, not only in aggregate. Privacy-preserving computation and secure access are important when linking longitudinal or sensitive records.

12. Contemporary Developments and Best Practice

Modern causal ML combines causal identification with flexible prediction. Bayesian causal models represent uncertainty about parameters and missing potential outcomes, while heterogeneous-effect learners estimate treatment-effect variation. These tools can improve adjustment and targeting but do not replace domain knowledge or identification assumptions.

Pre-analysis plans, registered reports and open analytical specifications reduce researcher degrees of freedom. Reproducible pipelines should separate training, validation and causal evaluation data where appropriate. Sensitivity analysis should be routine rather than an afterthought. When an experiment is infeasible, analysts should prefer a transparent quasi-experimental design over an opaque predictive association.

A practical workflow is:

1. Define the causal question, estimand, population and time horizon.
2. Draw a DAG and state assumptions.
3. Choose randomization if feasible and ethical.
4. Pre-specify outcomes, MDE, power, stopping and multiplicity procedures.
5. Diagnose missingness, overlap, interference and measurement quality.
6. Estimate effects with uncertainty intervals and robust standard errors.
7. Test sensitivity to plausible violations.
8. Report practical, subgroup, ethical and operational implications.

Chapter Summary

Statistical inference uses sampling distributions to quantify uncertainty about population quantities. Confidence intervals describe the performance of an estimation procedure, while hypothesis tests assess compatibility with a null model; neither substitutes for effect sizes or substantive judgment. Power analysis links sample size, variability, significance, desired power and practically meaningful effects.

Randomized experiments identify causal effects under appropriate assignment, consistency and interference assumptions. A/B tests require disciplined outcome definition, allocation, monitoring, multiplicity control and intention-to-treat analysis. Factorial designs estimate main effects and interactions efficiently.

Observational causal analysis begins with a clearly defined estimand and explicit assumptions. DAGs help distinguish confounders, mediators and colliders. Matching and propensity scores address measured differences but depend on overlap and cannot remove hidden confounding. Instrumental variables identify local effects under strong assumptions. Difference-in-differences relies primarily on credible parallel trends and careful treatment-timing analysis.

Responsible practice combines statistical validity with transparency, privacy, fairness, safety and reproducibility. The strongest analysis is not the one with the most sophisticated model, but the one whose question, design, assumptions and evidence align.

Key Terms

Estimand: Precisely defined quantity a study seeks to estimate.

Sampling distribution: Distribution of a statistic across repeated samples under a sampling design.

Standard error: Estimated standard deviation of an estimator's sampling distribution.

Confidence interval: Interval produced by a procedure with specified long-run coverage.

Null hypothesis: Reference claim evaluated by a statistical test.

P-value: Probability, under the null model, of data at least as extreme as observed.

Type I error: Rejection of a true null hypothesis.

Power: Probability of detecting a specified effect under a specified alternative.

Minimum detectable effect: Smallest target effect a planned design is intended to detect.

Multiple comparisons: Simultaneous testing of several hypotheses, increasing error risk.

Average treatment effect: Population average of individual causal treatment effects.

Potential outcome: Outcome a unit would have under a particular treatment level.

Confounder: Common cause of treatment and outcome that creates a noncausal association.

Collider: Variable caused by two variables; conditioning on it may induce bias.

Directed acyclic graph: Graphical representation of assumed causal relationships without directed cycles.

Propensity score: Probability of treatment conditional on observed covariates.

Instrumental variable: Variable that shifts treatment while satisfying causal exclusion assumptions.

Difference-in-differences: Comparison of changes over time between treated and comparison groups.

Parallel trends: Assumption that groups would have changed similarly absent treatment.

Intention-to-treat: Analysis based on assigned treatment rather than actual adherence.

Review Questions

5 Short Answer Questions

1. Distinguish a standard deviation from a standard error.
2. What does a 95% confidence procedure mean in repeated sampling?
3. Define Type I error, Type II error and statistical power.
4. Why can conditioning on a collider create bias?
5. State the central identifying assumption of difference-in-differences.

5 Long Answer Questions

1. Explain how confidence intervals and hypothesis tests should be interpreted together, including the difference between statistical and practical significance.
2. Describe the design of a robust A/B experiment, including randomization, outcomes, power, stopping rules and guardrail metrics.
3. Compare matching, regression adjustment and propensity-score methods, emphasizing assumptions and diagnostics.
4. Explain instrumental variables, including relevance, exclusion, monotonicity and the interpretation of LATE.
5. Critically evaluate the use of causal ML for estimating heterogeneous treatment effects.

3 Analytical / Application Questions

1. A treatment increases conversion from 4.0% to 4.4% in two equally sized groups of 20,000. Specify an appropriate effect measure, identify the assumptions needed for a causal interpretation and describe what additional uncertainty calculation is required before deployment.
2. A policy is introduced in one region but not another. The treated region had a faster outcome increase during the two years before implementation. Explain why a basic DiD estimate may be biased and propose diagnostic or alternative strategies.
3. A hospital compares patients receiving an AI triage tool with those not receiving it, but clinicians choose who receives the tool. Draw a plausible DAG in words, identify at least two sources of bias and propose a defensible evaluation design.

Multiple Choice Questions with Answers

Question 1

Which statement correctly interprets a p-value?

A. It is the probability that the null hypothesis is true.

B. It is the probability of observing data at least as extreme as the result, assuming the null model is true.

C. It is the probability that the study will replicate.

D. It is the probability that the alternative hypothesis is true.

Correct Answer: B. It is the probability of observing data at least as extreme as the result, assuming the null model is true.

Question 2

For a two-arm comparison of independent means with equal group sizes, which quantity increases the required sample size when the target effect is held fixed?

A. Lower outcome variability

B. Higher target power

C. Larger absolute effect

D. One-sided reporting after the experiment

Correct Answer: B. Higher target power

Question 3

What is the principal purpose of randomization in an A/B experiment?

A. To guarantee equal outcomes in both groups

B. To make treatment assignment independent of potential outcomes in expectation

C. To eliminate measurement error

D. To ensure every participant complies with treatment

Correct Answer: B. To make treatment assignment independent of potential outcomes in expectation

Question 4

Which variable should generally not be adjusted for when estimating the total treatment effect?

A. A pre-treatment confounder

B. A baseline age measure

C. A post-treatment mediator

D. A pre-treatment stratification variable

Correct Answer: C. A post-treatment mediator

Question 5

A backdoor path from treatment T to outcome Y is best described as:

A. A path beginning with an arrow leaving T

B. A path beginning with an arrow entering T

C. A path containing only mediators

D. A path that exists only in randomized experiments

Correct Answer: B. A path beginning with an arrow entering T

Question 6

What limitation remains even after achieving excellent balance on observed covariates using propensity scores?

A. Randomization has necessarily failed

B. Unmeasured confounding may remain

C. The treatment effect must be zero

D. The outcome cannot be continuous

Correct Answer: B. Unmeasured confounding may remain

Question 7

Under standard instrumental-variable assumptions, a binary-instrument Wald estimator generally identifies:

A. The average effect for always-treated units only

B. The local average treatment effect for compliers

C. The effect of the instrument on the outcome only

D. A purely descriptive correlation

Correct Answer: B. The local average treatment effect for compliers

Question 8

The key identifying assumption in a basic difference-in-differences design is:

A. Equal sample sizes at every time point

B. Parallel trends in the absence of treatment

C. Normally distributed outcomes

D. No change in the comparison group after treatment

Correct Answer: B. Parallel trends in the absence of treatment

Question 9

If a cluster-randomized study has average cluster size m and intracluster correlation ρ, which expression is the approximate design effect?

A. 1 − (m−1)ρ

B. mρ

C. 1 + (m−1)ρ

D. ρ/m

Correct Answer: C. 1 + (m−1)ρ

Question 10

Which practice best addresses the risk of false positives when many outcomes are tested?

A. Report only the smallest p-value

B. Pre-specify primary outcomes and use an appropriate multiplicity procedure

C. Remove all nonsignificant outcomes after analysis

D. Increase the number of subgroup analyses

Correct Answer: B. Pre-specify primary outcomes and use an appropriate multiplicity procedure