Statistical Inference, Experimentation and Causal Reasoning

Introduction and context

Data analysis rarely ends with describing the observations in hand. A data scientist is usually asked to generalise from a sample to a wider population, decide whether an observed difference is compatible with chance, estimate the size and uncertainty of an effect, or assess whether an intervention caused an outcome. These tasks require statistical inference and causal reasoning.

Statistical inference is not a single procedure. It is a chain of reasoning involving a target population, a sampling or assignment process, a statistical model, an estimand, uncertainty quantification, and a decision rule. A mathematically precise answer can still be misleading if the sample is biased, the outcome was selected after inspection, or the causal question is confused with a predictive question.

This chapter develops a unified framework for estimation, confidence intervals, hypothesis testing, effect sizes, power, multiple testing, A/B experimentation, observational studies, causal diagrams, and confounding. The emphasis is on interpretation and study design rather than mechanical execution of statistical software.

Learning objectives

By the end of this chapter, you should be able to:

1. Define populations, samples, parameters, estimands, estimators, and standard errors.
2. Construct and interpret confidence intervals and hypothesis tests.
3. Distinguish statistical significance from practical importance.
4. Plan experiments using effect sizes, power, sample-size reasoning, and stopping rules.
5. Evaluate randomised A/B tests and identify threats to validity.
6. Explain association, prediction, and causation as different analytical goals.
7. Use causal diagrams to reason about confounding, mediation, and collider bias.
8. Select appropriate adjustments and communicate uncertainty responsibly.

1. The inferential framework

A population is the set of units about which a claim is intended to apply. A sample is the observed subset. A parameter is a fixed but usually unknown population quantity, such as a mean μ or a treatment effect τ. An estimator is a rule calculated from data; its realised value is an estimate.

The sampling distribution of an estimator describes how it would vary over repeated samples under a specified design. Its standard deviation is the standard error (SE). For the sample mean of independent observations with population standard deviation σ,

where ȳ is the sample mean and n is the sample size. In practice σ is usually unknown and is replaced by the sample standard deviation s. This formula assumes independent observations with a common finite variance; dependence, clustering, weighting, and unequal variances require modified methods.

A useful distinction is between sampling error and systematic error. Sampling error decreases, in many settings, as n increases. Selection bias, measurement error, nonresponse, coverage error, and confounding are not automatically removed by a larger sample.

Estimands before estimators

An estimand is the precise quantity a study seeks to learn. Examples include:

• the population mean outcome under the current policy;
• the difference in average outcomes under treatment and control;
• the ratio of two risks;
• the average treatment effect for a defined population;
• the treatment effect among those who actually comply with an intervention.

The same dataset may support several estimands, but they are not interchangeable. Clearly specifying the population, treatment, outcome, time horizon, and summary scale prevents later ambiguity.

2. Estimation and confidence intervals

A point estimate summarises the most plausible value under a chosen method, but it does not communicate precision. An interval estimate gives a range of values generated by a procedure with a stated coverage property.

For a mean with a large sample or known σ, an approximate 100(1 − α)% confidence interval is

Here z_{1−α/2} is the corresponding standard-normal critical value. For example, z₀.₉₇₅ ≈ 1.96 for a 95% interval. When σ is estimated and the sample is small under normality assumptions, the t distribution is used:

The statement that a 95% confidence procedure has 95% coverage means that, over repeated samples and under its assumptions, approximately 95% of intervals constructed by that procedure contain the fixed parameter. It is not technically correct to say that there is a 95% probability that a particular fixed parameter lies in the already calculated interval, unless a Bayesian model and posterior probability are being used.

For a difference of two independent means, a common large-sample interval is

where s₁ and s₂ are sample standard deviations and n₁ and n₂ are sample sizes. Welch’s method is generally preferable to the equal-variance method when group variances or sample sizes differ.

For a proportion p̂, the simple Wald interval p̂ ± z × √[p̂(1−p̂)/n] can perform poorly for small samples or proportions near 0 or 1. Wilson or exact methods are often preferable. A confidence interval is therefore a property of both the data and the method used to construct it.

Worked example: estimating a conversion difference

An online service randomly assigns 10,000 visitors to a new page and 10,000 to the existing page. The new page produces 620 purchases and the existing page 560. The observed conversion rates are 0.062 and 0.056, so the difference is 0.006, or 0.6 percentage points.

For two independent proportions, an approximate standard error for the difference is

Substitution gives SE ≈ 0.00325. A 95% interval is therefore approximately 0.006 ± 1.96(0.00325), or (−0.0004, 0.0124). The data are compatible with a very small decrease as well as an increase of about 1.24 percentage points. The point estimate is positive, but the interval indicates substantial uncertainty relative to the effect.

3. Hypothesis testing and effect sizes

A hypothesis test compares data with a null hypothesis H₀, usually a specified value such as no difference, against an alternative H₁. The p-value is the probability, assuming H₀ and the test assumptions are true, of obtaining a result at least as incompatible with H₀ as the observed result. It is not the probability that H₀ is true, nor the probability that the result occurred by chance in an unrestricted sense.

A test has four linked elements:

1. a pre-specified estimand and null value;
2. a test statistic and reference distribution;
3. a significance level α, often 0.05;
4. a decision and substantive interpretation.

A two-sided test of a mean with known standard error uses

The normal reference distribution is justified by design and distributional assumptions, or by an appropriate large-sample approximation. Small p-values indicate incompatibility with the null model; they do not establish causality, importance, or absence of bias.

A Type I error rejects a true null hypothesis, with long-run probability α under the test procedure. A Type II error fails to reject a false null; its probability is β. Power is 1 − β: the probability of detecting a specified effect under a specified alternative.

Effect size and practical significance

An effect should be reported on a meaningful scale, with an interval. Examples include:

• risk difference: p₁ − p₀;
• risk ratio: p₁/p₀;
• odds ratio: [p₁/(1−p₁)] ÷ [p₀/(1−p₀)];
• mean difference: μ₁ − μ₀;
• standardised mean difference: (μ₁ − μ₀)/σ;
• relative or absolute change in revenue, latency, error, or retention.

Standardisation can assist comparison across outcomes, but it can obscure business or clinical meaning. A statistically detectable increase of 0.01 percentage points may be irrelevant operationally, whereas a non-significant estimate of a costly safety improvement may merit further investigation.

4. Power analysis and study planning

Power analysis should occur before data collection. It requires a minimum practically important effect, outcome variability or baseline rate, α, desired power, allocation ratio, and analysis method. Planning around a vague desire to “find significance” encourages exaggerated claims and underpowered studies.

For two independent groups with equal sizes, a commonly used approximate sample-size expression for detecting a mean difference Δ is

This assumes independent observations, equal allocation, a common standard deviation σ, a two-sided test, and a sufficiently accurate normal approximation. For a one-sided test, z_{1−α/2} is replaced by z_{1−α}. Proportion outcomes require a formula based on the null and alternative rates, or simulation.

Power is not a fixed property of a test alone. It changes with effect size, sample size, noise, α, allocation, missingness, clustering, and analysis choices. If units are clustered, the effective sample size is reduced. A common approximation uses the design effect

where m is average cluster size and ρ is the intracluster correlation. The required individual-level sample is then approximately multiplied by DE. This formula assumes roughly equal cluster sizes and a simple exchangeable correlation structure.

Power calculations should include attrition, eligibility failures, noncompliance, and multiplicity where relevant. Post hoc power calculated after observing a non-significant result is generally a poor substitute for an interval and a discussion of detectable effects.

5. Multiple testing and selective analysis

If many true null hypotheses are tested at α = 0.05, some will appear significant by chance. With m independent tests, the probability of at least one false positive is

For m = 20 and α = 0.05, this is about 0.642 under independence. Dependence changes the exact value, but the basic problem remains.

The family-wise error rate (FWER) is the probability of at least one false rejection in a family. Bonferroni controls FWER by testing each hypothesis at α/m, or by multiplying each p-value by m, though it can be conservative. Holm’s step-down procedure also controls FWER and is at least as powerful as ordinary Bonferroni.

The false discovery rate (FDR) is the expected proportion of false discoveries among rejected hypotheses. Benjamini–Hochberg controls FDR under standard independence or certain positive-dependence conditions and is often useful in exploratory high-dimensional analysis.

Multiplicity arises not only from many variables but also from many outcomes, subgroups, time points, model specifications, transformations, and stopping opportunities. Pre-registration, a primary outcome, a defined analysis population, and transparent reporting help control selective inference. Exploratory analyses are valuable, but they should be labelled exploratory and ideally confirmed in new data.

6. Randomised experimentation and A/B testing

A randomised controlled experiment assigns units to treatment conditions using a random mechanism. Under proper implementation, randomisation balances both measured and unmeasured causes in expectation. It therefore identifies a causal effect for the population represented by the experiment, subject to compliance, interference, attrition, and measurement assumptions.

In a simple parallel A/B test, the primary estimand is often the intention-to-treat effect:

where Z is assignment and Y is the outcome. This effect includes noncompliance and is usually the most policy-relevant effect of being assigned to the variant. If treatment is perfectly received, it corresponds closely to the treatment effect of exposure.

Sound A/B testing requires:

• random assignment and allocation integrity;
• a unit of randomisation appropriate to interference and repeated use;
• a pre-specified primary outcome and time window;
• stable treatment exposure;
• a sample-size and stopping plan;
• instrumentation checks and treatment-exposure monitoring;
• appropriate handling of missing outcomes;
• analysis by assignment, not only by users who complied;
• correction or clear accounting for multiple variants and outcomes.

Repeatedly checking a p-value and stopping when it crosses 0.05 inflates the false-positive rate. Sequential designs can be valid, but they require an alpha-spending rule, group-sequential method, or a Bayesian decision framework specified in advance. “Novelty effects,” seasonality, carryover, network interference, and contamination can make short experiments unrepresentative.

Randomisation does not guarantee a useful experiment. A statistically significant result can be operationally harmful if it increases complaints, latency, or inequity. Guardrail metrics and adverse-event monitoring are therefore part of experimental design.

7. Association, prediction, and causation

Association describes statistical dependence: variables vary together. Prediction concerns the accuracy of forecasts for new cases. Causation concerns how an outcome would change under an intervention or counterfactual condition. A variable may be highly predictive without being a useful intervention target. For example, a late payment may predict account closure, but preventing late payment may not eliminate all underlying causes of closure.

A causal effect compares potential outcomes. Let Y(1) denote the outcome a unit would have under treatment and Y(0) under control. The individual effect is Y(1) − Y(0), but only one potential outcome is observed for each unit. The average treatment effect is

The equality follows from linearity of expectation, not from an assumption that both potential outcomes are observed. Randomisation identifies the ATE under consistency, appropriate treatment definition, no problematic interference, and random assignment.

Observational adjustment can identify a causal effect only under stronger assumptions. If all relevant confounders X are measured and sufficiently controlled, conditional exchangeability is expressed as

where ⟂ denotes conditional independence. This assumption cannot generally be verified from observed data alone.

8. Causal diagrams and confounding analysis

A directed acyclic graph (DAG) represents proposed causal directions. A backdoor path from treatment Z to outcome Y begins with an arrow entering Z. A confounder is a common cause of Z and Y, creating a non-causal association. Conditioning on an adequate set of confounders blocks these paths.

Consider socioeconomic status S affecting both training participation T and employment outcome Y. The path T ← S → Y is a backdoor path. Adjusting for S can block it, assuming S is measured adequately and no other relevant unblocked paths remain.

A DAG also distinguishes:

• Mediator: lies on the causal pathway, such as T → skill → Y. Adjusting for it estimates a direct effect rather than the total effect and may block part of the intervention effect.
• Collider: is caused by two variables, such as T → selection ← Y. Conditioning on selection can create a spurious association between T and Y.
• Effect modifier: changes the size or direction of an effect across levels of another variable. It should often be reported through interaction or stratified effects, not automatically “controlled away.”

A practical adjustment workflow is to state the intervention and outcome, draw a DAG from subject-matter knowledge, identify backdoor paths, choose a minimally sufficient pre-treatment adjustment set, and avoid adjusting for post-treatment mediators or colliders. Automated variable-selection algorithms cannot replace this reasoning.

Figure: A confounder creates a backdoor path, while a mediator lies on part of the treatment-to-outcome pathway.

9. Observational evidence and sensitivity

Observational studies are indispensable when randomisation is unethical, infeasible, expensive, or too short to capture long-term outcomes. Common designs include cohort, case–control, cross-sectional, interrupted time-series, difference-in-differences, regression discontinuity, instrumental-variable, and propensity-score analyses.

Each design depends on distinctive assumptions. Difference-in-differences requires that, absent treatment, treated and comparison groups would have followed parallel trends. Regression discontinuity requires that units cannot precisely manipulate the assignment threshold and that potential outcomes are smooth around the cutoff. Instrumental variables require relevance, exclusion, and assumptions concerning independence and monotonicity; these are strong and often difficult to defend.

Propensity scores can balance measured covariates but cannot repair unmeasured confounding. Matching, weighting, and regression adjustment are design and analysis tools, not guarantees of causal validity. Sensitivity analysis asks how strong an unmeasured confounder would need to be to explain the observed result, and negative controls can reveal certain forms of residual bias.

Researchers should report the target population, inclusion rules, missing-data strategy, covariate definitions, model specification, balance diagnostics, overlap, and robustness analyses. A narrow confidence interval around a biased estimate is not strong evidence of causality.

10. Illustrative case: a benefits reminder experiment

Background. A public-service agency wants to increase completion of an online benefits application. It proposes sending a personalised reminder to a random sample of eligible applicants.

Problem. The agency must estimate whether reminders increase completed applications without materially increasing duplicate submissions, support workload, or unequal access.

Analysis and approach. The unit of randomisation is the applicant, not each website visit, because one applicant may revisit the site many times. The primary outcome is completion within 30 days. Secondary outcomes include time to completion, duplicate submissions, and support contacts. Before launch, the team specifies an intention-to-treat contrast, a minimum practically important increase of 1 percentage point, α = 0.05, desired power of 0.80, and a stopping rule. It blocks randomisation by language and region to improve balance, while retaining the overall causal interpretation.

The team validates delivery logs, excludes no assigned applicants merely because a message failed to arrive, and reports assignment-level effects. It examines missing outcome records and uses administrative completion data rather than self-reported success. It also monitors a guardrail: support contacts per 1,000 applicants. A subgroup analysis by language is labelled secondary and is interpreted with multiplicity in mind.

Outcome. Suppose the reminder increases completion by 1.3 percentage points, with a 95% confidence interval of 0.4 to 2.2 points, but increases support contacts by 0.8 per 1,000, with an interval including zero. The result supports deployment if the agency considers the minimum benefit worthwhile and the operational cost acceptable. It does not prove that every subgroup benefits equally; subgroup estimates require their own uncertainty and may be underpowered.

Learning points. The causal claim comes from random assignment, not from the p-value. The estimand, unit of randomisation, primary outcome, guardrails, power, and stopping rule were chosen before observing results. The decision combines statistical evidence, practical effect size, equity, and implementation risk.

11. Ethics, governance, and contemporary practice

Inference affects access to services, credit, employment, healthcare, and public trust. Ethical analysis therefore includes more than statistical validity. Participants should receive appropriate information and protection; sensitive data require lawful collection, minimisation, access control, retention limits, and secure processing. Experiments should not knowingly expose people to disproportionate risk, and vulnerable groups should not be excluded without justification.

Fairness questions include whether treatment effects differ across protected or underserved groups, whether measurement error is unequal, and whether an apparently beneficial intervention imposes burdens on some populations. Privacy-preserving experimentation may require aggregation, differential privacy, or restricted reporting, while recognising that privacy methods can affect precision.

AI systems intensify the importance of separating prediction from intervention. A model trained on historical decisions may accurately predict those decisions while reproducing their biases. Predictive features can be descendants of prior discrimination, mediators, or proxies for protected attributes. Causal evaluation, documentation of data provenance, preregistered analyses, versioned code, and independent replication are essential governance practices.

12. Best-practice checklist

Before accepting an inferential result, ask:

1. What population and estimand are being studied?
2. How were units sampled, assigned, retained, and measured?
3. What assumptions justify the estimator and interval?
4. Is the effect reported in an interpretable absolute scale?
5. Does the interval include effects that would change the decision?
6. Were power, multiplicity, missingness, and stopping addressed?
7. If causal language is used, what identifies the causal contrast?
8. Could confounding, mediation, collider bias, interference, or selection explain the result?
9. Are subgroup and equity implications examined responsibly?
10. Can an independent analyst reproduce the reported result?

Chapter Summary

Statistical inference uses samples, models, and study designs to quantify uncertainty about population quantities. Point estimates should be accompanied by confidence intervals whose coverage interpretation depends on repeated sampling and model assumptions. Hypothesis tests and p-values assess compatibility with a null model; they do not measure practical importance or prove causation. Effect sizes, minimum important differences, power, and pre-specified stopping rules make experimentation decision-oriented. Multiple testing must be controlled when many hypotheses, outcomes, subgroups, or analysis choices are considered.

Randomised experiments identify causal effects under appropriate assignment, compliance, measurement, and interference conditions. Observational studies can be informative but require stronger assumptions, careful design, and sensitivity analysis. Association, prediction, and causation answer different questions. Potential outcomes define causal contrasts, while DAGs help identify confounding paths and warn against inappropriate adjustment for mediators and colliders. Responsible data science combines technical validity with transparency, privacy, fairness, and practical judgment.

Key Terms

Estimand: The precisely defined population quantity a study aims to learn.

Estimator: A rule that uses sample data to estimate a parameter.

Standard error: The standard deviation of an estimator’s sampling distribution.

Confidence interval: An interval produced by a procedure with a stated long-run coverage property.

Null hypothesis: A specified reference claim assessed by a statistical test.

p-value: Probability, under the null model, of data at least as incompatible as those observed.

Type I error: Rejection of a true null hypothesis.

Type II error: Failure to reject a false null hypothesis.

Power: Probability of detecting a specified effect under specified assumptions.

Effect size: A quantitative measure of the magnitude of a difference or relationship.

Multiple testing: Conducting several statistical tests or analyses that create multiplicity concerns.

FWER: Probability of at least one false rejection within a family of tests.

FDR: Expected proportion of false discoveries among rejected hypotheses.

Randomisation: Assignment by a chance mechanism to support causal identification.

Intention-to-treat effect: Effect of assignment to treatment rather than necessarily receiving it.

Confounder: A variable that creates a non-causal association by affecting treatment and outcome.

Mediator: A variable on the causal pathway from treatment to outcome.

Collider: A common effect of two variables; conditioning on it can induce bias.

DAG: Directed acyclic graph representing proposed causal relationships.

Average treatment effect: Population average of the difference between potential outcomes under treatment and control.

Review Questions

Short Answer Questions

1. What is the difference between an estimand and an estimator?
2. Give the correct repeated-sampling interpretation of a 95% confidence interval.
3. Why is statistical significance not equivalent to practical importance?
4. What is the purpose of a pre-specified primary outcome in an A/B test?
5. Define a collider and explain why conditioning on one can be harmful.

Long Answer Questions

1. Explain how effect size, confidence interval, p-value, and power provide different information about an intervention.
2. Discuss the assumptions and operational requirements of a valid randomised A/B experiment.
3. Compare FWER and FDR control, including situations in which each is appropriate.
4. Explain how potential outcomes and DAGs provide complementary approaches to causal reasoning.
5. Evaluate the limitations of propensity-score adjustment in an observational study.

Analytical / Application Questions

1. An experiment reports a treatment difference of 0.8 percentage points with a 95% interval from −0.1 to 1.7 points. Interpret the result statistically and operationally, and identify what additional information a decision-maker needs.
2. Draw a DAG for training participation, prior income, job-search effort, and employment. Identify a plausible confounder and a mediator, then propose an adjustment strategy for estimating the total effect of training.
3. A product team tests 12 variants and 5 outcomes, repeatedly checking results each day. Diagnose the inferential risks and design a pre-analysis and multiplicity-control plan.

Multiple Choice Questions with Answers

Question 1

Which statement correctly interprets a 95% confidence interval?

A. There is a 95% probability that the fixed parameter lies in this already calculated interval.

B. The interval contains 95% of the observed sample values.

C. Across repeated samples, the procedure produces intervals containing the fixed parameter approximately 95% of the time.

D. The null hypothesis is true with probability 95%.

Correct Answer: C. Across repeated samples, the procedure produces intervals containing the fixed parameter approximately 95% of the time.

Question 2

A p-value is best described as:

A. The probability that the null hypothesis is true.

B. The probability, under the null model, of data at least as incompatible as the observed data.

C. The probability that the result is practically important.

D. The probability that the study has adequate power.

Correct Answer: B. The probability, under the null model, of data at least as incompatible as the observed data.

Question 3

Which quantity is most directly reduced by increasing sample size, assuming the design and measurement process remain sound?

A. Selection bias

B. Confounding bias

C. Sampling variability

D. Measurement validity

Correct Answer: C. Sampling variability

Question 4

In a randomised A/B test, the intention-to-treat effect compares participants according to:

A. Whether they actually used the treatment.

B. Their assigned treatment condition.

C. Their predicted probability of conversion.

D. Their post-treatment subgroup.

Correct Answer: B. Their assigned treatment condition.

Question 5

A variable caused by both treatment and outcome is a:

A. Mediator

B. Confounder

C. Collider

D. Moderator necessarily

Correct Answer: C. Collider

Question 6

If 20 independent true null hypotheses are tested at α = 0.05, the probability of at least one false positive is approximately:

A. 0.05

B. 0.10

C. 0.36

D. 0.64

Correct Answer: D. 0.64

Question 7

Which method is designed to control the family-wise error rate and is generally less conservative than ordinary Bonferroni?

A. Holm’s step-down procedure

B. Unadjusted p-values

C. Benjamini–Hochberg only

D. Post hoc subgroup selection

Correct Answer: A. Holm’s step-down procedure

Question 8

The average treatment effect is defined as:

A. E[Y(1)] + E[Y(0)]

B. E[Y(1) − Y(0)]

C. E[Y | Z = 1] only

D. The correlation between treatment and outcome

Correct Answer: B. E[Y(1) − Y(0)]

Question 9

Which adjustment is usually inappropriate when estimating the total effect of a treatment?

A. A measured pre-treatment confounder

B. A baseline stratification variable

C. A post-treatment mediator

D. A variable used to improve covariate balance before treatment

Correct Answer: C. A post-treatment mediator

Question 10

For two equal-sized groups, the approximate mean-difference sample-size formula assumes, among other conditions:

A. Perfect compliance is unnecessary and irrelevant to all estimands.

B. Independent observations and a common standard deviation.

C. The outcome must be binary.

D. Multiple testing has no effect on required sample size.

Correct Answer: B. Independent observations and a common standard deviation.