Statistical Inference and Experimentation

Introduction

Data science rarely has access to an entire population. Instead, analysts observe a sample and use it to learn about a wider population, make predictions, or evaluate whether an intervention has produced a meaningful change. Statistical inference provides the formal tools for doing this while acknowledging uncertainty. Experimentation provides a disciplined way to investigate causal relationships.

This chapter develops confidence intervals, hypothesis tests, effect sizes, regression inference, A/B testing, and experimental design. It also distinguishes statistical significance from practical significance and explains how bias, confounding, multiplicity, and poor study design can undermine apparently persuasive results.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain sampling variability, standard errors, confidence intervals, and sampling distributions.
2. Construct and interpret confidence intervals for common parameters.
3. Formulate and evaluate null and alternative hypotheses.
4. Distinguish p-values, statistical significance, effect size, and practical significance.
5. Apply t-tests, tests for proportions, and regression inference appropriately.
6. Design and interpret A/B tests and controlled experiments.
7. Identify bias, confounding, threats to validity, and ethical issues.
8. Communicate inferential results accurately to technical and non-technical audiences.

1. From Samples to Populations

A population is the complete group about which a researcher wants to draw conclusions. A sample is the observed subset. A population parameter, such as the population mean μ or proportion p, is usually unknown. A sample statistic, such as the sample mean x̄ or sample proportion p̂, is calculated from observed data and used to estimate the parameter.

Different random samples produce different statistics. This variation is called sampling variability. The probability distribution of a statistic over repeated samples of the same size is its sampling distribution.

For a sample mean based on independent observations with population mean μ, variance σ², and sample size n:

Here, E(x̄) is the expected sample mean and SE(x̄) is its standard error. When σ is unknown, it is commonly estimated by s, the sample standard deviation:

The standard error measures the typical sampling variation of an estimator; it is not the standard deviation of individual observations. Larger samples generally reduce standard error at a rate proportional to 1/√n. Consequently, quadrupling the sample size approximately halves the standard error, assuming the sampling design and data-generating process remain comparable.

The Central Limit Theorem states, under broad conditions, that the sampling distribution of the sample mean becomes approximately normal as sample size increases, even when the population distribution is not normal. The approximation is less reliable for very small samples, highly skewed populations, or data with extreme outliers.

2. Confidence Intervals

A confidence interval gives a range of plausible values for an unknown population parameter, together with a procedure that has a specified long-run coverage rate. A 95% confidence interval does not mean that there is a 95% probability that the fixed parameter lies inside this particular interval. Rather, if the same valid procedure were repeated many times, approximately 95% of the resulting intervals would contain the parameter.

A general interval has the form:

The critical value depends on the reference distribution and the desired confidence level.

2.1 Confidence interval for a mean

If σ is known, or if a sufficiently large sample justifies a normal approximation, a confidence interval for μ is:

where α = 1 − confidence level and z_{α/2} is the standard-normal critical value. For a 95% interval, z_{α/2} = 1.96.

When σ is unknown and the population is reasonably normal, especially for smaller samples, use the t distribution:

The subscript n − 1 denotes the degrees of freedom. The t distribution has heavier tails than the normal distribution, reflecting additional uncertainty from estimating σ.

Worked example: mean delivery time

A random sample of 25 deliveries has mean time 42 minutes and standard deviation 8 minutes. Assuming independent observations and an approximately normal population, a 95% confidence interval uses t_{0.025,24} ≈ 2.064.

Thus, the interval is approximately 38.70 to 45.30 minutes. The correct interpretation is that this method produces intervals containing the population mean in approximately 95% of repeated samples. It is not correct to say that 95% of individual delivery times lie in this interval.

2.2 Confidence interval for a proportion

For a sufficiently large sample, an approximate interval for a population proportion p is:

where p̂ = x/n, x is the number of successes, and n is the sample size. The normal approximation is generally considered reasonable when both np̂ and n(1 − p̂) are sufficiently large. For small counts or proportions near 0 or 1, Wilson or exact intervals are preferable.

2.3 Difference between two means or proportions

For independent groups, a confidence interval for μ₁ − μ₂ is based on the difference in sample means and its standard error. Welch’s method is often preferred because it does not assume equal population variances:

For two independent proportions, an approximate interval for p₁ − p₂ is:

These formulas assume independent observations and an appropriate approximation. Paired data require a different analysis based on within-pair differences.

3. Hypothesis Testing

Hypothesis testing evaluates whether observed data are inconsistent with a specified null model. The null hypothesis, H₀, commonly represents no difference, no association, or a parameter value of interest. The alternative hypothesis, H₁ or Hₐ, represents a difference or effect.

A test statistic measures how far the observed result is from what H₀ predicts, usually in standard-error units. For a one-sample mean with hypothesized value μ₀:

Under the usual assumptions, this statistic follows a t distribution with n − 1 degrees of freedom when H₀ is true.

A p-value is the probability, assuming H₀ is true and the analysis model is appropriate, of obtaining a result at least as extreme as the observed result in the direction specified by the alternative. It is not the probability that H₀ is true, nor is it the probability that the result occurred by chance in an unrestricted sense.

A significance level α is selected before analysis. If p ≤ α, the analyst rejects H₀; otherwise, the analyst fails to reject H₀. “Failing to reject” does not prove that H₀ is true. It may reflect a genuinely small effect, insufficient sample size, high variability, or poor measurement.

3.1 Type I and Type II errors

A Type I error occurs when a true null hypothesis is rejected. Its long-run probability is controlled approximately by α under the assumptions of the test. A Type II error occurs when a false null hypothesis is not rejected. Statistical power is:

where β is the probability of a Type II error for a specified alternative. Power increases with larger samples, larger true effects, lower variability, and a larger significance level. Sample-size planning should be based on a meaningful minimum effect, not merely on the desire to obtain p < 0.05.

3.2 One-sided and two-sided tests

A two-sided test considers departures in either direction, such as Hₐ: μ ≠ μ₀. A one-sided test considers a specified direction, such as Hₐ: μ > μ₀. A one-sided test should be chosen before seeing the data and only when effects in the opposite direction would not address the decision question. Switching from two-sided to one-sided after observing the result inflates the false-positive risk.

4. Effect Size and Practical Significance

Statistical significance depends strongly on sample size. A very small effect can achieve a small p-value in a huge sample, while a substantively important effect may fail to reach significance in a small sample. Therefore, inferential reporting should include an effect estimate and confidence interval.

For two independent means, Cohen’s d is commonly defined as:

where the pooled standard deviation is:

Cohen’s d is dimensionless. Conventional labels such as small, medium, and large are rough guides, not universal laws. Domain-specific thresholds are more useful. For business or operational decisions, compare the estimated effect and its uncertainty with a pre-specified minimum practically important difference (MPID), such as a reduction of at least 2 minutes in service time or an increase of at least 1 percentage point in conversion.

A result may be statistically significant but practically trivial, or practically important but statistically uncertain. Both dimensions must be considered.

5. Regression Inference

Regression models describe relationships between an outcome and one or more predictors. In simple linear regression:

where yᵢ is the outcome, xᵢ is a predictor, β₀ is the intercept, β₁ is the expected change in y for a one-unit increase in x, and εᵢ is an error term. The fitted model is:

The slope estimate b₁ can be tested with:

Usually β_{1,0} = 0, corresponding to no linear association after accounting for the model variables.

For multiple regression:

βⱼ represents the expected change in y associated with a one-unit increase in xⱼ, holding the other included predictors constant. This conditional interpretation does not automatically establish causation. Omitted variables, reverse causality, measurement error, selection bias, and inappropriate functional form can all produce misleading conclusions.

Common assumptions for ordinary least squares inference include linearity in the conditional mean, independent errors, constant error variance for conventional standard errors, and approximately normal errors for small-sample t and F inference. Normality of the predictors is not required. Residual plots should be examined for curvature, unequal spread, outliers, and dependence. Heteroscedasticity-robust standard errors can help with unequal variance, but they do not correct confounding or dependence.

The coefficient of determination is:

where SSE is the residual sum of squares and SST is the total sum of squares. R² describes in-sample variation explained relative to a mean-only model; it is not proof of causation, model correctness, or useful out-of-sample prediction.

6. A/B Testing

An A/B test is a randomized experiment comparing a control condition A with a treatment condition B. Participants, sessions, users, or other experimental units are randomly assigned to conditions. The primary outcome and analysis plan should be defined before examining results.

For a binary outcome, such as purchase, the absolute treatment effect is:

The relative lift is:

The absolute difference is usually more stable for decision-making because relative lift can look large when the baseline is small. For example, increasing conversion from 2% to 3% is a 1-percentage-point increase but a 50% relative lift.

For a continuous outcome, compare group means, preferably with a confidence interval. For skewed monetary outcomes, analysts may use robust summaries, transformations, quantile methods, or models suited to the distribution.

A valid A/B test requires:

• random assignment;
• a clearly defined experimental unit;
• stable treatment exposure;
• no interference between units where possible;
• a pre-specified primary outcome;
• adequate sample size and duration;
• monitoring of data quality and assignment balance;
• analysis consistent with the randomization unit.

Multiple comparisons and peeking

Testing many outcomes, segments, or variants increases the chance of at least one apparently significant result. Remedies include specifying one primary outcome, correcting for multiplicity, using false-discovery-rate procedures, or applying sequential methods designed for repeated monitoring. Repeatedly checking a conventional fixed-horizon p-value and stopping when it becomes significant changes its error properties.

7. Experimental Design and Causal Reasoning

Randomization makes treatment assignment independent of both observed and unobserved characteristics in expectation. This supports causal interpretation, provided implementation is sound. A randomized experiment estimates the causal effect of treatment for the relevant experimental population; it does not necessarily generalize to every population or setting.

A useful design workflow is:

1. Define the causal question and target population.
2. Specify the experimental unit, treatment, control, outcome, and time horizon.
3. Choose the estimand, such as average treatment effect.
4. Predefine inclusion rules, exclusions, primary analysis, and stopping criteria.
5. Randomize assignment using a reproducible procedure.
6. Determine sample size using a meaningful effect, variability, α, and desired power.
7. Monitor implementation without changing the inferential rules opportunistically.
8. Report estimates, confidence intervals, assumptions, attrition, and deviations from the plan.

Blocking and stratification

If important baseline variables are associated with the outcome, block randomization or stratified randomization can improve balance and precision. For example, users may be randomized separately within device type or geographic region. Blocking does not replace randomization; it organizes it.

Confounding

A confounder is a variable related to both the exposure and the outcome that creates a misleading association if not handled. For instance, an association between exercise-app use and health may be confounded by motivation. Randomization helps prevent confounding by exposure assignment. In observational studies, matching, stratification, regression adjustment, inverse-probability weighting, and causal graphical reasoning may help, but none guarantees removal of unmeasured confounding.

Internal and external validity

Internal validity concerns whether the estimated effect is credible for the studied units under the study conditions. External validity concerns generalization to other people, times, settings, and implementations. A tightly controlled experiment may have high internal validity but limited generalizability.

8. Bias, Data Quality, and Ethical Governance

Important sources of bias include:

• Selection bias: inclusion or retention depends on characteristics related to the outcome.
• Measurement bias: instruments or procedures systematically mismeasure variables.
• Nonresponse bias: respondents differ from nonrespondents.
• Attrition bias: dropout differs by treatment or outcome risk.
• Survivorship bias: analysis includes only units that remain observable.
• Publication or reporting bias: favorable findings are more likely to be shared.
• P-hacking: analytical choices are changed to obtain significance.
• HARKing: hypotheses are presented as prior predictions after results are known.

Ethical experimentation requires informed consent where appropriate, minimization of risk, privacy protection, secure data handling, fairness assessment, and independent review for higher-risk research. A/B testing should not knowingly expose users to unreasonable harm. Sensitive outcomes require special safeguards, and automated decisions should be audited for disparate effects.

Researchers should distinguish exploratory analyses from confirmatory analyses, retain an audit trail, document exclusions, and report null or unfavorable findings. Reproducible code and versioned datasets improve accountability but do not compensate for flawed design.

9. Illustrative Case: An Online Learning Reminder Experiment

Background

An online learning platform wants to know whether a weekly reminder email increases the proportion of enrolled learners who complete a lesson within seven days.

Problem

The platform historically sent reminders mainly to learners who had previously been active. Comparing recipients with non-recipients would therefore be confounded by prior engagement.

Analysis and approach

The platform defines the experimental unit as the learner, the treatment as a reminder email, and the primary outcome as lesson completion within seven days. Among 20,000 eligible learners, 10,000 are randomly assigned to treatment and 10,000 to control. The treatment group has 1,260 completions; the control group has 1,100.

The observed rates are:

The absolute effect is:

Thus, the estimated increase is 1.6 percentage points. The relative lift is approximately 14.5% because 0.016/0.110 ≈ 0.145.

An approximate standard error for the difference is:

A 95% confidence interval is approximately:

or approximately 0.67 to 2.53 percentage points. Since the interval excludes zero, the result is statistically significant at the 5% level under the approximation. However, management must still assess whether a 0.67-percentage-point lower plausible effect justifies email costs, learner experience risks, and possible notification fatigue.

The analysis should also check assignment integrity, duplicate accounts, delivery failures, unequal exposure, unsubscribes, and whether the effect differs by accessibility needs or region. Segment findings should be treated cautiously unless they were pre-specified or multiplicity was addressed.

Outcome and learning points

Random assignment avoids the historical engagement confounding problem. The confidence interval communicates uncertainty more usefully than a p-value alone. The absolute effect supports operational planning, while the relative lift provides a complementary business description. A statistically positive result is not automatically a universal recommendation: cost, user welfare, fairness, long-term retention, and replication must also be considered.

10. Best Practices for Inferential Data Science

1. Begin with a clear estimand: define whose outcome, under which treatment, over what period, is being estimated.
2. Use probability sampling or randomization when feasible.
3. Report estimates with confidence intervals, not p-values alone.
4. State assumptions and inspect data visually and diagnostically.
5. Separate exploratory discovery from confirmatory testing.
6. Pre-register important hypotheses, outcomes, sample sizes, and stopping rules.
7. Use domain-relevant thresholds for practical importance.
8. Account for clustering, repeated measures, missingness, and multiple comparisons.
9. Avoid causal language for observational associations unless a defensible causal design supports it.
10. Document data cleaning, exclusions, code, model versions, and deviations.
11. Replicate important findings and test robustness to reasonable analytical choices.
12. Communicate uncertainty plainly, including what the study cannot establish.

Chapter Summary

Statistical inference uses samples to estimate population quantities while accounting for sampling variability. Confidence intervals combine an estimate, standard error, and critical value; their interpretation concerns the long-run performance of the interval procedure. Hypothesis tests evaluate data against a null model, but p-values do not measure the probability that a hypothesis is true. Type I error, Type II error, power, and sample size are central to responsible testing.

Effect sizes and confidence intervals are essential because statistical significance does not establish practical importance. Regression inference quantifies conditional relationships but does not by itself prove causation. A/B tests use random assignment to support causal conclusions, provided that treatment, outcomes, units, exposure, and analysis are properly defined. Bias, confounding, attrition, multiple testing, optional stopping, and poor measurement can invalidate conclusions. Ethical, transparent, and reproducible practice requires pre-specification, appropriate governance, careful communication, and attention to both statistical and practical significance.

Key Terms

Population: The complete group about which conclusions are intended.

Sample: The observed subset drawn from a population.

Parameter: A numerical characteristic of a population.

Statistic: A numerical quantity calculated from sample data.

Sampling variability: Variation in a statistic across repeated samples.

Standard error: The standard deviation of an estimator’s sampling distribution.

Confidence interval: An interval produced by a procedure designed to capture a parameter at a stated long-run rate.

Null hypothesis: A model or parameter claim evaluated by a statistical test.

Alternative hypothesis: A claim representing an effect or departure from the null.

p-value: The probability, under the null model, of data at least as extreme as observed.

Type I error: Rejection of a true null hypothesis.

Type II error: Failure to reject a false null hypothesis.

Power: The probability of rejecting a false null hypothesis for a specified alternative.

Effect size: A numerical measure of the magnitude of an effect.

Practical significance: Importance of an effect for real-world decisions.

Confounding: Distortion of an exposure–outcome relationship by another variable.

Randomization: Allocation of experimental units to conditions by chance.

A/B test: A randomized comparison of two conditions.

Internal validity: Credibility of conclusions for the studied conditions.

External validity: Generalizability of findings beyond the study setting.

Review Questions

Short Answer Questions

1. What is the difference between a population parameter and a sample statistic?
2. Explain the meaning of a 95% confidence interval.
3. What does a p-value measure, and what does it not measure?
4. Distinguish statistical significance from practical significance.
5. Why does randomization help with confounding?

Long Answer Questions

1. Explain the construction and interpretation of confidence intervals for a mean when the population standard deviation is unknown.
2. Discuss Type I error, Type II error, power, sample size, and their relationships in hypothesis testing.
3. Explain the assumptions and interpretation of inference in multiple linear regression.
4. Describe how an organization should plan, conduct, and analyze an A/B test.
5. Discuss major sources of bias in data science experiments and methods for reducing them.

Analytical / Application Questions

1. A treatment increases conversion from 4% to 5%. Explain the absolute difference, relative lift, and why both should be reported. What further information is needed before recommending the treatment?
2. An observational analysis finds that users who receive coaching have higher retention, but coaching is offered mainly to highly engaged users. Identify the likely confounder and propose a stronger design.
3. An experiment produces p = 0.03, but the estimated effect is far below the organization’s minimum practically important difference. How should the result be communicated and used in decision-making?

Multiple Choice Questions with Answers

Question 1

Which statement correctly describes a 95% confidence interval?

A. There is a 95% probability that the fixed parameter lies in this particular interval.

B. Approximately 95% of intervals from repeated valid samples would contain the parameter.

C. 95% of individual observations must lie inside the interval.

D. The null hypothesis has a 95% probability of being true.

Correct Answer: B. Approximately 95% of intervals from repeated valid samples would contain the parameter.

Question 2

A sample mean is 50, the sample standard deviation is 10, and n = 25. What is the estimated standard error?

A. 0.4

B. 2

C. 5

D. 50

Correct Answer: B. 2

Question 3

A p-value is best interpreted as:

A. The probability that the null hypothesis is true.

B. The probability that the result was caused by chance in every possible sense.

C. The probability, under the null model, of observing a result at least as extreme as the one obtained.

D. The probability that the alternative hypothesis is true.

Correct Answer: C. The probability, under the null model, of observing a result at least as extreme as the one obtained.

Question 4

Which change generally increases statistical power, holding other factors constant?

A. Reducing the sample size

B. Increasing measurement variability

C. Using a smaller true effect

D. Increasing the sample size

Correct Answer: D. Increasing the sample size

Question 5

What is the main purpose of random assignment in an A/B test?

A. To guarantee a large effect

B. To balance observed and unobserved characteristics in expectation

C. To eliminate sampling variability

D. To ensure every participant receives both treatments

Correct Answer: B. To balance observed and unobserved characteristics in expectation

Question 6

An estimated treatment effect is 0.016, meaning the treatment conversion rate exceeds control by:

A. 0.016 percentage points

B. 0.16 percentage points

C. 1.6 percentage points

D. 16 percentage points

Correct Answer: C. 1.6 percentage points

Question 7

In multiple regression, a coefficient is usually interpreted as the expected change in the outcome for a one-unit predictor increase, assuming:

A. All other included predictors are held constant

B. The outcome is held constant

C. The p-value equals zero

D. The sample contains the entire population

Correct Answer: A. All other included predictors are held constant

Question 8

Which is an example of a Type I error?

A. Failing to detect a real treatment effect

B. Rejecting a true null hypothesis

C. Obtaining a wide confidence interval

D. Using a small sample

Correct Answer: B. Rejecting a true null hypothesis

Question 9

Why can statistical significance fail to imply practical significance?

A. A very large sample can make a tiny effect statistically detectable

B. Confidence intervals never measure effect size

C. Randomization always creates large effects

D. p-values are measured in business units

Correct Answer: A. A very large sample can make a tiny effect statistically detectable

Question 10

Which practice best reduces the risk of false-positive findings in an experiment?

A. Changing the primary outcome after seeing results

B. Stopping whenever any subgroup is significant

C. Pre-specifying outcomes and accounting for multiple comparisons

D. Reporting only favorable analyses

Correct Answer: C. Pre-specifying outcomes and accounting for multiple comparisons