Statistical Inference, Experimentation and Causal Reasoning

Introduction

Data description tells us what was observed; statistical inference helps us reason beyond the observed data. A data scientist may use a sample of customers to estimate the satisfaction of an entire population, test whether a product change improves conversion, or assess whether an intervention caused an observed outcome. These tasks require more than computing averages. They require explicit assumptions, uncertainty measurement, careful study design, and disciplined interpretation.

Statistical inference is especially important because data are rarely complete or perfectly representative. A dataset may be affected by sampling variation, measurement error, non-response, selection bias, confounding, or changes in the environment. A statistically significant relationship may not be practically important, and a strong prediction may not establish causation.

This chapter develops estimation, confidence intervals, hypothesis testing, A/B experimentation, regression analysis, and causal reasoning. It also distinguishes three related but different questions:

1. Description: What patterns are present in the observed data?
2. Prediction: What outcome is likely for a new case?
3. Causation: What would happen if an action or condition were changed?

Learning Objectives

After studying this chapter, you should be able to:

• distinguish populations, samples, parameters, and statistics;
• calculate and interpret point estimates and confidence intervals;
• formulate and evaluate null and alternative hypotheses;
• explain p-values, statistical power, Type I error, and Type II error;
• design and interpret an A/B test;
• use correlation and regression appropriately;
• distinguish association, prediction, and causation;
• recognize selection bias, confounding, measurement bias, and other threats to validity;
• communicate statistical findings responsibly in professional settings.

1. Foundations of Statistical Inference

A population is the complete set of units about which a conclusion is desired. Examples include all users of an application, all transactions in a year, or all registered voters. A sample is the subset actually observed.

A numerical characteristic of a population is a parameter. Common parameters include the population mean μ, population proportion p, and population variance σ². A numerical quantity calculated from a sample is a statistic, such as the sample mean x̄, sample proportion p̂, and sample standard deviation s.

The distinction matters because a sample statistic is used to estimate an unknown population parameter. Sampling variation means that different random samples generally produce different estimates.

Sampling distributions and standard error

A sampling distribution describes the values a statistic would take over repeated samples of the same size from the same population. The standard deviation of this distribution is the standard error.

For a sample mean based on n independent observations with population standard deviation σ:

When σ is unknown, it is commonly estimated by s:

For a sample proportion p̂ based on n independent observations:

In practice, p is unknown. It may be replaced by p̂ for an estimated standard error, or by 0.5 when planning a conservative sample size because p(1 − p) is largest at p = 0.5.

These formulas assume independent observations and an appropriate sampling process. For a mean, the sampling distribution is approximately normal when the population is normal or when n is sufficiently large under the central limit theorem. Strong dependence, extreme outliers, or a biased sample can invalidate simple interpretations.

2. Estimation and Confidence Intervals

A point estimate is a single best estimate of a parameter. For example, x̄ estimates μ and p̂ estimates p. A point estimate alone does not communicate precision. A confidence interval combines an estimate with a margin of error.

The general form is:

Confidence interval for a mean

If σ is known, or in theoretical settings where a normal approximation is appropriate, a 100(1 − α)% confidence interval for μ is:

Here, α is the probability assigned to the two tails, and z_{1−α/2} is the relevant standard-normal critical value. For a 95% interval, α = 0.05 and z_{0.975} ≈ 1.96.

When σ is unknown, the t distribution is generally used:

The subscript n − 1 denotes the degrees of freedom. The t critical value is larger than the corresponding normal value for small samples, reflecting additional uncertainty from estimating σ.

Confidence interval for a proportion

For a sufficiently large sample, an approximate interval is:

The normal approximation should not be used mechanically when the number of successes or failures is very small. Exact or improved intervals, such as Wilson intervals, may perform better.

Worked example: estimating conversion

An online service observes 400 randomly selected visitors, of whom 96 subscribe. The estimated conversion proportion is:

The estimated standard error is:

Using a 95% normal approximation:

Thus, the interval is approximately 0.198 to 0.282, or 19.8% to 28.2%. Under the sampling assumptions, a procedure that constructs intervals this way will capture the true population proportion in approximately 95% of repeated samples.

A common error is to say that there is a 95% probability that this fixed parameter lies between 19.8% and 28.2%. In the conventional frequentist interpretation, the parameter is fixed and the interval-producing procedure has 95% long-run coverage. The interval expresses uncertainty arising from the sampling process.

3. Hypothesis Testing

Hypothesis testing provides a structured method for comparing observed evidence with a reference claim.

• The null hypothesis, H₀, represents a benchmark such as no difference or no association.
• The alternative hypothesis, H₁ or Hₐ, represents the effect or relationship of interest.
• The test statistic summarizes how far the sample result is from what H₀ predicts.
• The p-value is the probability, assuming H₀ is true and the analysis model is appropriate, of observing a result at least as extreme as the one obtained.

A small p-value indicates incompatibility with H₀; it does not indicate the probability that H₀ is true, the size of the effect, or its practical importance.

One-sample mean test

Suppose a process is claimed to have mean μ₀. If the population standard deviation is known, a z statistic is:

If σ is unknown, a one-sample t statistic is:

The t test assumes independent observations and, for small samples, an approximately normal population. The alternative may be two-sided, μ ≠ μ₀, or one-sided, such as μ > μ₀. The direction must be selected before examining the results.

Errors and power

A Type I error rejects a true H₀. Its planned probability is α, often 0.05. A Type II error fails to reject a false H₀. Its probability is β. Power is 1 − β: the probability of detecting a specified effect under specified assumptions.

Power increases with larger sample size, larger true effect, lower variability, and a larger significance level. It decreases when the design is noisy or the effect is small. Failure to reject H₀ is not proof that there is no effect; it may reflect low power or imprecise measurement.

Statistical and practical significance

A tiny effect can be statistically significant with a very large sample, while a meaningful effect can be non-significant with a small sample. Reports should therefore include effect sizes and confidence intervals. For a difference in proportions, an effect may be reported as:

• absolute difference: p̂₁ − p̂₂;
• relative risk: p̂₁/p̂₂, when meaningful and defined;
• relative change: (p̂₁ − p̂₂)/p̂₂;
• odds ratio, particularly in logistic regression or case-control designs.

4. A/B Testing and Experiment Design

An A/B test is a randomized experiment comparing a control condition A with a treatment condition B. Participants are assigned to conditions, an outcome is measured, and the difference is estimated.

Randomization is central because, in expectation, it balances both measured and unmeasured characteristics between groups. It supports a causal interpretation of the treatment difference, provided implementation, measurement, and follow-up are sound.

A/B test workflow

1. Define the decision and primary outcome.
2. State the estimand, such as the difference in conversion rates.
3. Specify H₀ and H₁ before looking at results.
4. Determine the target effect, significance level, power, and sample size.
5. Randomize eligible units and maintain allocation integrity.
6. Monitor implementation and data quality without repeatedly stopping for favorable results.
7. Analyze according to the pre-specified plan.
8. Report effect size, interval, p-value, practical consequences, and limitations.

For two independent conversion proportions, the estimated absolute effect is:

An approximate standard error for the difference is:

where n_A and n_B are group sizes. This formula is appropriate for independent users and a sufficiently large normal approximation. A hypothesis test under H₀: p_A = p_B may use a pooled proportion in its standard error; the confidence interval for the observed difference generally uses the unpooled estimate.

Experiment threats

Randomization does not solve every problem. Important threats include:

• Noncompliance: users do not receive the assigned treatment.
• Interference: one user’s treatment affects another user.
• Attrition: outcomes are unavailable for some assigned units.
• Novelty effects: behavior changes temporarily because a feature is new.
• Peeking and optional stopping: repeated testing inflates false-positive risk.
• Multiple comparisons: testing many outcomes or segments increases the chance of at least one apparently significant result.
• Spillover: treatment exposure crosses group boundaries.

Pre-registration, a primary metric, treatment logging, stable randomization, and transparent reporting reduce these risks.

5. Correlation, Regression and Prediction

Correlation measures the strength and direction of linear association. Pearson’s correlation coefficient is:

Here xᵢ and yᵢ are paired observations, and x̄ and ȳ are sample means. The value lies between −1 and 1. Correlation is sensitive to outliers and detects linear, not all possible, relationships.

A simple linear regression model is:

where yᵢ is the outcome, xᵢ is a predictor, β₀ is the intercept, β₁ is the expected change in y for a one-unit increase in x, and εᵢ is an error term. The fitted value is:

Ordinary least squares chooses b₀ and b₁ to minimize:

For inference in the usual linear model, assumptions include linearity of the conditional mean, independent errors, constant error variance, and approximately normal errors for small-sample tests and intervals. In observational data, additional concerns include omitted variables, reverse causality, and nonrepresentative observations.

The coefficient of determination is:

where SST = Σ(yᵢ − ȳ)². R² is the proportion of variation around the sample mean explained by the fitted model in the sample. It is not proof of causation and does not guarantee good out-of-sample prediction.

For prediction, data should be separated into training and test sets, or evaluated using cross-validation. Predictive performance should use suitable metrics, such as RMSE for continuous outcomes and log loss, precision, recall, or calibration measures for classification. A model can predict accurately because it exploits proxies or stable associations without identifying an intervention effect.

6. Causal Reasoning

Causal reasoning asks counterfactual questions: What would the outcome have been for the same unit under a different treatment? The unobserved alternative outcome is often called a potential outcome. For treatment T, let Y(1) denote the outcome under treatment and Y(0) the outcome without it. The individual causal effect is Y(1) − Y(0), but both potential outcomes cannot normally be observed for the same unit at the same time.

A population-level average treatment effect is:

where E denotes the population expectation. Randomized experiments estimate the ATE by comparing average outcomes across randomized groups, under assumptions such as consistency, appropriate treatment assignment, and no problematic interference.

Confounding

A confounder influences both the exposure and the outcome. For example, training participation may be associated with higher productivity because highly motivated employees are more likely to participate and also more productive. Motivation confounds the relationship if it is not adequately controlled.

A causal diagram can clarify relationships:

• X → Y: X may cause Y.
• X ← Z → Y: Z is a common cause and may confound the X–Y association.
• X → M → Y: M is a mediator on the causal pathway.
• X → C ← Y: C is a collider; conditioning on C can create bias.

Adjustment should be based on a defensible causal model, not on automatically including every available variable. Controlling for a mediator may remove part of the effect of interest; controlling for a collider may introduce bias.

Observational strategies

When randomization is impossible, researchers may use matching, stratification, regression adjustment, inverse probability weighting, instrumental variables, regression discontinuity, or difference-in-differences. Each requires substantive assumptions.

Difference-in-differences compares changes over time between treatment and comparison groups. Its key identifying assumption is parallel trends: absent treatment, the groups would have followed similar trends. A plausible comparison group and examination of pre-treatment trends are essential.

No statistical method automatically converts biased data into causal evidence. Design, domain knowledge, measurement quality, sensitivity analysis, and transparent assumptions remain crucial.

7. Bias, Validity and Ethical Practice

Selection bias occurs when inclusion or retention depends on factors related to the outcome. Survivorship bias examines only entities that remain observable. Measurement bias results from systematic inaccuracies in variables. Recall bias occurs when people report past events inaccurately. Non-response bias arises when respondents differ from nonrespondents.

Internal validity concerns whether the estimated relationship is credible for the studied units. External validity concerns whether it generalizes to another population, setting, or time. A controlled experiment may have strong internal validity but limited generalizability if participants or conditions are unusual.

Ethical statistical practice includes informed consent where applicable, privacy protection, data minimization, secure storage, fairness assessment, and honest communication of uncertainty. Analysts should not suppress inconvenient outcomes, selectively report favorable subgroups, or present exploratory findings as confirmatory. Sensitive decisions involving employment, credit, health, or education require governance, explainability, bias testing, and human oversight.

AI systems intensify these issues. Automated experimentation can optimize a metric while harming unmeasured outcomes. Predictive models trained on historical decisions may reproduce institutional bias. Causal claims generated by language models or automated analytics must be checked against study design and domain evidence; fluent text is not evidence of identification.

8. Illustrative Case: A Mobile Payment Experiment

Background

A mobile payment company believes that a shorter checkout form will increase completed payments without increasing fraud.

Problem

The company must decide whether to deploy the new form. The primary outcome is completed payment within 24 hours. Secondary outcomes are fraud rate, customer support contacts, and average transaction value.

Analysis and approach

Eligible users are randomly assigned to control A, the existing form, or treatment B, the shorter form. Assignment is performed at the user level, and the same user remains in the assigned condition during the experiment. The primary hypothesis is:

• H₀: p_B − p_A = 0
• H₁: p_B − p_A ≠ 0

Suppose 20,000 users are assigned to each group. Control produces 3,000 completed payments, so p̂_A = 0.150. Treatment produces 3,200, so p̂_B = 0.160. The estimated absolute improvement is:

The estimated standard error is:

An approximate 95% interval is:

The interval is approximately 0.0028 to 0.0172, or a 0.28 to 1.72 percentage-point increase. The point estimate corresponds to a relative increase of approximately 6.7%, calculated as 0.010/0.150.

The company must not conclude success from conversion alone. It should examine fraud, support contacts, and transaction value, account for multiple outcomes, and verify that randomization and tracking worked. If fraud increases materially, the conversion gain may not justify deployment.

Outcome and learning points

Assume the fraud rate is unchanged, support contacts decline slightly, and the confidence interval for conversion excludes zero. The company may deploy gradually while monitoring longer-term effects. The result supports a causal claim for users and conditions represented in the experiment, not necessarily for all markets or future designs.

This case demonstrates that an experiment estimates an effect through randomization, a confidence interval expresses precision, and a business decision requires practical and safety outcomes in addition to a p-value.

Best Practices for Analysts

• Define the target population and estimand before analysis.
• Inspect sampling, missingness, outliers, and measurement processes.
• Use randomization whenever feasible for causal questions.
• Pre-specify primary outcomes and analysis rules.
• Report effect sizes with confidence intervals, not p-values alone.
• Separate exploratory analysis from confirmatory testing.
• Check model assumptions using residuals and domain knowledge.
• Avoid causal language for observational associations unless assumptions are justified.
• Evaluate practical significance, equity, and unintended consequences.
• Preserve reproducible code, data dictionaries, and an audit trail.

Chapter Summary

Statistical inference uses sample data to estimate population quantities while representing uncertainty. Point estimates provide concise summaries, whereas confidence intervals combine estimates with margins of error. Hypothesis tests compare data with a null model; p-values measure incompatibility with that model, not the probability that a hypothesis is true. Type I error, Type II error, power, effect size, and practical significance must be considered together.

A/B tests use random assignment to support causal comparisons, but implementation failures, attrition, interference, multiple testing, and optional stopping can compromise conclusions. Correlation and regression describe association and support prediction under suitable assumptions; neither alone establishes causation. Causal reasoning requires counterfactual thinking, attention to confounding, and explicit identification assumptions. Bias, privacy, fairness, and transparent reporting are essential parts of professional data science.

Key Terms

Population: The complete set of units about which inference is desired.

Sample: The observed subset drawn from a population.

Parameter: A numerical characteristic of a population.

Statistic: A numerical quantity calculated from a sample.

Sampling distribution: The distribution of a statistic across repeated samples.

Standard error: The standard deviation of a sampling distribution.

Point estimate: A single sample-based estimate of a population parameter.

Confidence interval: An interval produced by a procedure designed to capture a parameter at a stated long-run rate.

Null hypothesis: A benchmark claim evaluated by a statistical test.

p-value: The probability, under the null model, of a result at least as extreme as the observed result.

Type I error: Rejection of a true null hypothesis.

Power: The probability of detecting a specified effect when it exists.

A/B test: A randomized experiment comparing two conditions.

Correlation: A measure of linear association between two variables.

Regression: A model relating an outcome to one or more predictors.

Confounding: Distortion of an exposure–outcome relationship by a common cause.

Selection bias: Systematic distortion caused by how units enter or remain in a study.

Average treatment effect: The population average of the difference between potential outcomes under treatment and control.

External validity: The extent to which findings generalize beyond the studied setting.

Review Questions

Short Answer Questions

1. Distinguish a parameter from a statistic and provide one example of each.
2. What does a 95% confidence interval communicate in the frequentist framework?
3. Define a p-value without interpreting it as the probability that the null hypothesis is true.
4. Why does randomization help support causal inference?
5. Explain the difference between statistical significance and practical significance.

Long Answer Questions

1. Explain the construction and interpretation of a confidence interval for a population mean. State the assumptions behind the t-based interval.
2. Describe the complete workflow for designing and analyzing an A/B test, including threats to validity.
3. Compare correlation, regression, prediction, and causation. Explain why a predictive model need not be causal.
4. Discuss Type I error, Type II error, power, sample size, and effect size in hypothesis testing.
5. Explain confounding, mediation, and collider bias using a suitable applied example.

Analytical / Application Questions

1. A website records 180 purchases among 1,000 visitors in group A and 210 purchases among 1,000 visitors in group B. Calculate the estimated absolute difference in conversion and explain what additional information is needed before recommending deployment.
2. A training program is associated with higher employee productivity, but motivated employees are more likely to enroll. Identify the likely confounder and propose two strategies for improving causal inference.
3. An analyst tests 30 product metrics and reports the one with p < 0.05 as the experiment’s success. Critique this conclusion and recommend a more defensible analysis plan.

Multiple Choice Questions with Answers

Question 1

Which quantity is a population parameter?

A. The mean of a sample of 200 users

B. The conversion rate of all users in the target population

C. The median calculated from a dataset extract

D. The standard deviation of an experimental group

Correct Answer: B. The conversion rate of all users in the target population

Question 2

What does a p-value represent?

A. The probability that the alternative hypothesis is true

B. The probability that the sample contains no errors

C. The probability, under the null model, of a result at least as extreme as observed

D. The probability that the observed effect is practically important

Correct Answer: C. The probability, under the null model, of a result at least as extreme as observed

Question 3

For a 95% confidence interval using a normal approximation, which critical value is commonly used?

A. 0.05

B. 0.50

C. 1.00

D. 1.96

Correct Answer: D. 1.96

Question 4

Which feature most directly supports a causal interpretation in an A/B test?

A. A very large correlation

B. Random assignment to treatment conditions

C. A high R² value

D. A statistically significant observational relationship

Correct Answer: B. Random assignment to treatment conditions

Question 5

A Type I error occurs when an analyst:

A. Fails to reject a false null hypothesis

B. Rejects a true null hypothesis

C. Uses a sample that is larger than necessary

D. Reports an effect size with a confidence interval

Correct Answer: B. Rejects a true null hypothesis

Question 6

If a treatment conversion rate is 0.16 and a control conversion rate is 0.15, the absolute improvement is:

A. 0.01 percentage points

B. 0.10 percentage points

C. 1 percentage point

D. 15 percentage points

Correct Answer: C. 1 percentage point

Question 7

Which relationship is most clearly a confounding structure?

A. X causes Y directly

B. X and Y have identical units

C. Z influences both X and Y

D. X is measured after Y

Correct Answer: C. Z influences both X and Y

Question 8

In the simple regression model yᵢ = β₀ + β₁xᵢ + εᵢ, β₁ represents:

A. The predicted value when x equals zero

B. The expected change in y for a one-unit increase in x

C. The unexplained error for observation i

D. The total variation in y

Correct Answer: B. The expected change in y for a one-unit increase in x

Question 9

Which practice helps reduce false-positive findings when many outcomes are examined?

A. Testing more outcomes after seeing the data

B. Pre-specifying a primary outcome

C. Stopping whenever one result is significant

D. Ignoring confidence intervals

Correct Answer: B. Pre-specifying a primary outcome

Question 10

Why can a high predictive accuracy fail to establish causation?

A. Prediction always requires randomization

B. Predictive associations may reflect confounding or proxies

C. Accurate models cannot use observational data

D. Causal effects never improve prediction

Correct Answer: B. Predictive associations may reflect confounding or proxies