Statistical Inference and Bayesian Reasoning

Introduction

Data science rarely concerns an entire population directly. Instead, analysts observe a sample and use it to learn about an underlying population, process, or data-generating mechanism. Statistical inference provides the formal language for making such generalizations while quantifying uncertainty. Bayesian reasoning adds a coherent framework for combining previous knowledge with new evidence.

Inference is central to experimentation, model evaluation, clinical research, policy analysis, financial risk assessment, and machine learning. A data scientist must distinguish a numerical result from a defensible conclusion: an estimated effect may be statistically detectable but too small to matter operationally; a non-significant result may reflect inadequate data rather than evidence of no effect.

This chapter develops sampling theory, estimation, confidence intervals, hypothesis testing, likelihood, maximum likelihood estimation, Bayesian updating, posterior inference, and resampling. It also examines assumptions, practical significance, reproducibility, and ethical interpretation.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain population, sample, parameter, statistic, sampling distribution, bias, and variance.
2. Derive and interpret standard errors and confidence intervals.
3. Formulate and evaluate null and alternative hypotheses.
4. Distinguish p-values, confidence intervals, statistical significance, and practical significance.
5. Construct likelihoods and obtain maximum likelihood estimates.
6. Apply Bayes’ theorem and interpret prior, likelihood, posterior, and predictive distributions.
7. Use bootstrap and permutation methods appropriately.
8. Diagnose assumptions, communicate uncertainty, and recognize ethical risks in inferential analysis.

1. Populations, Samples, and the Logic of Inference

A population is the full set of units about which a question is posed. It may be finite, such as all customers in a year, or conceptual, such as all future transactions generated by a system. A sample is the observed subset. A parameter is a fixed but usually unknown population quantity, such as a mean μ, variance σ², or proportion p. A statistic is calculated from sample data, such as the sample mean x̄ or sample proportion p̂.

The inferential process can be represented as:

Figure: Statistical inference uses sample information to learn about a population.

Sampling and representativeness

Under simple random sampling, every sample of a specified size has an equal probability of selection. Other designs include stratified sampling, cluster sampling, systematic sampling, and convenience sampling. The design affects both precision and validity.

A large sample cannot automatically correct selection bias. If an online survey systematically excludes people with limited internet access, increasing its size may produce a more precise estimate of the wrong target. Dependence also matters: repeated observations from the same individual or time series observations are not generally independent.

For independent and identically distributed observations X₁, …, Xₙ with finite mean μ and variance σ², the sample mean is

and its expected value and variance are

The standard error of x̄ is therefore σ/√n when σ is known, or approximately s/√n when σ is estimated by the sample standard deviation s. The reduction by √n explains why quadrupling the sample size approximately halves standard error.

The central limit theorem states, under broad regularity conditions, that the standardized sample mean approaches a standard normal distribution as n increases:

This approximation is not a license to ignore skewness, dependence, heavy tails, or a poorly designed sample. The required sample size depends on the population distribution, estimand, precision, and design.

2. Estimation and Confidence Intervals

Point estimation

A point estimator is a rule that produces an estimate from data. Desirable properties include:

• Unbiasedness: E[θ̂] = θ.
• Consistency: θ̂ approaches θ as sample size increases.
• Efficiency: among competing estimators, one has smaller variance under stated conditions.
• Robustness: performance is not excessively sensitive to violations or outliers.

Unbiasedness is not always optimal. An estimator with small bias and substantially lower variance can have lower mean squared error:

Here θ is the target parameter and θ̂ is its estimator. The decomposition applies when the relevant expectations exist.

Confidence intervals

A 95% confidence interval for a population mean, when a normal approximation is appropriate, is

where t₀.₉₇₅,n−1 is the 97.5th percentile of the t distribution with n − 1 degrees of freedom. The t interval is appropriate for independent observations when the population is approximately normal or the sample is sufficiently large; severe skewness, outliers, or dependence require additional care.

The frequentist interpretation is precise: if the same sampling procedure were repeated many times, approximately 95% of intervals constructed in this way would contain μ. It is not correct to say that μ has a 95% probability of lying in one already-computed interval under the frequentist framework.

For a proportion p, a simple large-sample interval is p̂ ± z₀.₉₇₅√[p̂(1 − p̂)/n]. This approximation can perform poorly for small samples or proportions near 0 or 1; Wilson or exact methods are often preferable.

Worked example: mean delivery time

Suppose n = 64 independent deliveries have mean x̄ = 42 minutes and standard deviation s = 8 minutes. A 95% interval using a normal critical value of approximately 1.96 is

SE = 8/√64 = 1 minute.

The interval estimates the population mean delivery time, not the time of an individual future delivery. The latter requires a prediction interval, which is wider because it includes individual-level variability.

3. Hypothesis Testing and Decision Errors

A hypothesis test begins with a null hypothesis H₀ and an alternative hypothesis H₁. For example, H₀: μ = 40 versus H₁: μ ≠ 40. The test statistic measures how inconsistent the observed data are with H₀.

A two-sided one-sample t statistic is

where μ₀ is the null value. Under H₀ and normal-population assumptions, t follows a t distribution with n − 1 degrees of freedom.

A p-value is the probability, assuming H₀ and the analysis model are correct, of obtaining a result at least as extreme as the observed result in the direction specified by H₁. It is not the probability that H₀ is true, nor the probability that the result occurred by chance in an informal sense.

A significance level α controls the long-run probability of rejecting H₀ when H₀ is true. A Type I error is a false rejection; a Type II error is failure to reject a false null. Statistical power is 1 − β, where β is the Type II error probability under a specified alternative.

The phrase “fail to reject” is preferable to “accept H₀.” A non-significant result may reflect a genuinely small effect, large uncertainty, low power, measurement error, or an inappropriate model.

Statistical versus practical significance

Statistical significance depends strongly on sample size. With millions of observations, a trivial effect can have a very small p-value. Practical significance asks whether the magnitude is important in context.

Suppose a recommendation system increases conversion from 10.00% to 10.08%. The relative increase is 0.08 percentage points, or 0.8% relative to the original rate. Whether this matters depends on traffic, implementation cost, revenue per conversion, and possible harms. A confidence interval for the effect should accompany the p-value.

Multiple testing creates another problem. If 20 independent null hypotheses are tested at α = 0.05, the probability of at least one false positive is

when all nulls are true and tests are independent. Pre-registration, correction procedures such as Holm’s method, and false-discovery-rate control can reduce misleading discoveries. Exploratory findings should be labeled as exploratory rather than presented as confirmatory evidence.

4. Likelihood and Maximum Likelihood Estimation

The likelihood treats observed data as fixed and parameters as variable. For observations x₁, …, xₙ with probability density or mass function f(xᵢ | θ), the likelihood is

The log-likelihood is usually easier to optimize:

The maximum likelihood estimator (MLE) is the parameter value that maximizes L or, equivalently, ℓ. Maximizing the likelihood does not mean that θ is random; it means selecting the parameter under which the observed data are most plausible relative to the model.

Bernoulli example

Let Xᵢ be a binary observation with P(Xᵢ = 1) = p, and let y = ∑Xᵢ be the number of successes. The likelihood is

The log-likelihood is y log p + (n − y) log(1 − p). Differentiating and setting the derivative to zero gives

Thus the MLE of a Bernoulli probability is the observed success proportion. MLEs are often consistent and asymptotically normal, but finite-sample bias, boundary estimates, model misspecification, and separation in logistic regression require attention.

Likelihood-ratio methods compare maximized likelihoods. For nested models, the statistic 2[ℓ(θ̂_full) − ℓ(θ̂_restricted)] is often approximately chi-square distributed under regularity conditions. Those conditions can fail for parameters on boundaries, mixture models, or small samples.

5. Bayesian Reasoning and Posterior Inference

Bayesian inference represents uncertainty about an unknown parameter with a probability distribution. Bayes’ theorem is

where p(θ) is the prior distribution, p(x | θ) is the likelihood, p(θ | x) is the posterior distribution, and p(x) = ∫p(x | θ)p(θ)dθ is the marginal likelihood or evidence.

The prior should express defensible knowledge or a deliberate weakly informative choice. It is not a device for hiding subjective assumptions. Sensitivity analysis should examine whether conclusions change under plausible priors.

Beta–binomial updating

For y successes in n Bernoulli trials, choose a Beta(a,b) prior:

The posterior is

The posterior mean is (a + y)/(a + b + n). The prior contributes a + b pseudo-observations, an interpretation that is useful but should not be treated as literal data in every context.

Worked Bayesian example

Assume a prior p ~ Beta(2, 8), whose prior mean is 0.20. In n = 20 trials, y = 7 successes are observed. The posterior is Beta(9, 21), with mean

The data move the estimate from 0.20 toward the sample proportion 0.35, but the posterior does not equal the sample proportion because prior information remains relevant. A 95% credible interval is obtained from the 2.5th and 97.5th percentiles of Beta(9,21). Unlike a confidence interval, it can be interpreted as a probability statement about p conditional on the model and prior: there is 95% posterior probability that p lies in that interval.

A posterior predictive distribution describes future observations:

This integrates parameter uncertainty and is often more directly useful for decisions than a parameter estimate alone.

Bayesian computation may use conjugate analysis, numerical integration, Monte Carlo, Markov chain Monte Carlo, sequential Monte Carlo, or variational inference. MCMC diagnostics include trace behavior, effective sample size, and convergence measures such as R̂. Approximate methods can be fast but may underestimate uncertainty; computational convenience does not guarantee inferential validity.

6. Resampling Methods

Resampling methods estimate sampling behavior using the observed data.

Bootstrap

The nonparametric bootstrap repeatedly samples n observations with replacement from the empirical sample. For each resample, calculate the statistic θ̂. The empirical distribution of θ̂ approximates the sampling distribution of θ̂. A percentile interval uses selected quantiles of the bootstrap statistics; bias-corrected and accelerated intervals can improve performance in some settings.

Bootstrap methods are useful for medians, quantiles, complex prediction metrics, and statistics without simple standard errors. They can fail with very small samples, extreme dependence, boundary parameters, or a sample that does not represent the target population. Time series require block bootstrap or other dependence-aware procedures.

Permutation tests

A permutation test evaluates a null hypothesis of exchangeability by rearranging labels or observations. For a two-group mean difference, labels are shuffled and the difference is recalculated. The fraction of permuted statistics at least as extreme as the observed statistic gives a randomization-based p-value. Validity requires the exchangeability implied by the null; clustered or paired data need restricted permutations.

Bootstrap and permutation are not interchangeable: bootstrap approximates sampling variability, whereas permutation primarily constructs a null distribution under exchangeability.

7. Integrated Case: Evaluating a Personalised Reminder System

Background and problem

An online learning platform introduces personalised reminders intended to increase weekly assignment completion. The product team conducts an experiment with 12,000 eligible learners randomly assigned to treatment or control. The estimand is the intention-to-treat difference in completion probability after four weeks.

Analysis

The treatment group has 6,000 learners, of whom 2,520 complete the assignment; the control group has 6,000, of whom 2,400 complete it. The estimated rates are 0.420 and 0.400, so the estimated absolute effect is

or 2 percentage points. The approximate standard error for the difference between independent proportions is

Substitution gives approximately √[0.42(0.58)/6000 + 0.40(0.60)/6000] ≈ 0.00895. A 95% approximate interval is 0.020 ± 1.96(0.00895), or approximately (0.002, 0.038). The effect is statistically distinguishable from zero at the 5% level, but its lower bound is close to zero.

Decision and practical interpretation

If each additional completion produces substantial educational value and the reminder has low cost, a 2-point increase may justify deployment. If reminders create distraction, privacy concerns, or notification fatigue, the same effect may not justify implementation. The team should also examine heterogeneity by learner group, attrition, repeated exposure, and longer-term outcomes. Subgroup analyses should be pre-specified or adjusted for multiplicity.

A Bayesian analysis could combine a weakly informative prior on Δ with the experimental likelihood and report the posterior probability that Δ exceeds a decision threshold, such as 1 percentage point. This directly addresses the operational question, but conclusions remain conditional on the prior, model, randomization, and outcome quality.

Learning points

Randomization supports causal interpretation, not automatic generalization to other populations. The estimand must be defined before analysis. Reporting only a p-value conceals effect magnitude and uncertainty. Product decisions should combine statistical evidence with costs, benefits, fairness, privacy, and potential unintended effects.

8. Assumptions, Limitations, and Best Practices

Important assumptions include random or defensible sampling, independence or correctly modeled dependence, valid measurement, appropriate distributional approximations, and a clearly defined estimand. Missing data can introduce bias when missingness depends on unobserved outcomes. Measurement error can attenuate associations or produce misleading estimates. Model misspecification can invalidate standard errors and p-values even when point estimates appear plausible.

Best practices include:

• Define the population, sampling frame, estimand, and decision threshold.
• Inspect data quality, missingness, outliers, dependence, and distributional shape.
• Use design-based or robust methods when simple assumptions fail.
• Report estimates, intervals, sample sizes, and assumptions—not p-values alone.
• Distinguish confirmatory from exploratory analyses.
• Pre-specify primary outcomes and analysis plans where feasible.
• Conduct sensitivity analyses for priors, missing data, model choices, and unmeasured confounding.
• Use simulation or power analysis before collecting data.
• Preserve reproducible code, versions, random seeds, and data provenance.
• Avoid overstating causal conclusions from observational data.

9. Ethical, Governance, and Contemporary Issues

Inference can amplify inequity when samples exclude vulnerable groups or when historical data encode discrimination. A model may produce a precise estimate for a biased target. Analysts should report coverage limitations, subgroup performance, and possible harms. Privacy-preserving aggregation, access controls, and data minimization are relevant when individual observations are sensitive.

Automated experimentation and AI systems make multiple testing, adaptive allocation, and feedback loops increasingly common. Sequential monitoring can inflate false-positive rates if stopping rules are not designed in advance. Adaptive experiments require methods that account for changing assignment probabilities. Bayesian decision rules can support continuous updating, but they do not remove the need for transparency, governance, and human accountability.

Chapter Summary

Statistical inference uses samples to learn about populations, with uncertainty quantified through sampling distributions, standard errors, confidence intervals, and hypothesis tests. The central limit theorem supports normal approximations under appropriate conditions, while sampling design and dependence determine whether those approximations are credible. Estimators should be evaluated through bias, variance, mean squared error, consistency, and robustness.

Likelihood provides a general route to estimation, and maximum likelihood selects parameters that make observed data most plausible under a specified model. Bayesian inference combines a prior with a likelihood to obtain a posterior and can produce posterior predictive distributions. Confidence intervals and credible intervals answer different questions and must not be conflated.

Bootstrap methods approximate sampling distributions; permutation tests construct null distributions under exchangeability. Across all methods, statistical significance must be distinguished from practical significance. Reliable analysis requires explicit estimands, transparent assumptions, sensitivity analysis, reproducibility, and attention to fairness, privacy, and consequences.

Key Terms

Population: The full set of units or potential observations of interest.

Sample: Observed subset used to learn about a population.

Parameter: Fixed population quantity, usually unknown.

Statistic: Quantity calculated from sample data.

Sampling distribution: Distribution of a statistic over repeated samples.

Standard error: Estimated standard deviation of an estimator’s sampling distribution.

Estimator: Rule for calculating an estimate of a parameter.

Bias: Difference between an estimator’s expected value and the target parameter.

Consistency: Convergence of an estimator to the target as sample size increases.

Confidence interval: Interval-producing procedure with long-run coverage probability.

p-value: Probability, under the null model, of data at least as extreme as observed.

Type I error: Rejection of a true null hypothesis.

Power: Probability of rejecting a false null under a specified alternative.

Likelihood: Function measuring data compatibility across parameter values.

Maximum likelihood estimator: Parameter value maximizing the likelihood.

Prior distribution: Probability distribution representing uncertainty before current data.

Posterior distribution: Updated parameter distribution after observing data.

Credible interval: Posterior-probability interval for a parameter.

Bootstrap: Resampling observations with replacement to approximate sampling behavior.

Permutation test: Null-based rearrangement procedure relying on exchangeability.

Review Questions

Short Answer Questions

1. Distinguish a parameter from a statistic and give one example of each.
2. What does a 95% frequentist confidence interval mean?
3. Why does quadrupling sample size approximately halve the standard error of a mean?
4. Explain the difference between a p-value and the probability that the null hypothesis is true.
5. Distinguish a bootstrap procedure from a permutation test.

Long Answer Questions

1. Explain how sampling design, dependence, and the central limit theorem affect the validity of statistical inference.
2. Derive the maximum likelihood estimator for a Bernoulli success probability.
3. Compare confidence intervals and Bayesian credible intervals, including their interpretations and assumptions.
4. Discuss statistical significance, practical significance, power, and sample size using a data-science example.
5. Explain how prior information, likelihood, posterior inference, and posterior prediction are connected in Bayesian analysis.

Analytical / Application Questions

1. An experiment estimates an effect of 0.5 percentage points with a 95% interval from −0.1 to 1.1 percentage points. Interpret the result statistically and practically, and identify additional information needed for a decision.
2. A survey has a very large sample but excludes people without internet access. Analyze why its estimate may remain biased and propose a more defensible design.
3. An analyst tests 30 customer segments and reports the one segment with p < 0.05. Explain the inferential problem and recommend an appropriate analysis and reporting strategy.

Multiple Choice Questions with Answers

Question 1

For independent observations with finite variance, what is the variance of the sample mean?

A. σ²n

B. σ²/n

C. σ/n²

D. n/σ²

Correct Answer: B. σ²/n

Question 2

Which statement correctly interprets a frequentist 95% confidence interval?

A. There is a 95% probability that the fixed parameter lies in this computed interval.

B. The null hypothesis has a 95% probability of being true.

C. Across repeated samples, 95% of intervals produced by the procedure would contain the parameter.

D. 95% of observations in the population lie inside the interval.

Correct Answer: C. Across repeated samples, 95% of intervals produced by the procedure would contain the parameter.

Question 3

A p-value is best described as the probability of:

A. The null hypothesis being true.

B. The observed result, or a more extreme result, assuming the null model.

C. Making a Type II error.

D. Obtaining the alternative hypothesis.

Correct Answer: B. The observed result, or a more extreme result, assuming the null model.

Question 4

If the sample size is increased from n to 4n, the standard error of an independent sample mean approximately:

A. Doubles

B. Quadruples

C. Halves

D. Remains unchanged

Correct Answer: C. Halves

Question 5

For y successes in n Bernoulli trials and a Beta(a,b) prior, the posterior distribution is:

A. Beta(a+y, b+n−y)

B. Beta(a+n, b+y)

C. Beta(y, n−y)

D. Normal(a+y, b+n−y)

Correct Answer: A. Beta(a+y, b+n−y)

Question 6

The maximum likelihood estimate of a Bernoulli probability p is:

A. The number of failures

B. The observed proportion of successes

C. The prior mean

D. The sample variance

Correct Answer: B. The observed proportion of successes

Question 7

Which issue is most directly associated with testing many hypotheses without adjustment?

A. Guaranteed reduction in Type I error

B. Inflated probability of at least one false positive

C. Elimination of sampling bias

D. Automatic increase in practical significance

Correct Answer: B. Inflated probability of at least one false positive

Question 8

What distinguishes a permutation test from a bootstrap procedure?

A. A permutation test commonly constructs a null distribution by rearranging labels.

B. A permutation test always requires a Bayesian prior.

C. A bootstrap never resamples observations.

D. A bootstrap is valid only for normally distributed data.

Correct Answer: A. A permutation test commonly constructs a null distribution by rearranging labels.

Question 9

An effect can be statistically significant but practically unimportant primarily because:

A. Large samples can detect very small effects.

B. Confidence intervals never measure uncertainty.

C. p-values measure monetary value directly.

D. Randomization prevents small effects.

Correct Answer: A. Large samples can detect very small effects.

Question 10

In Bayes’ theorem, p(θ | x) represents the:

A. Prior distribution

B. Likelihood

C. Posterior distribution

D. Sampling frame

Correct Answer: C. Posterior distribution