Probability, Statistical Inference and Experimentation

Introduction

Data science is concerned not only with describing observed data but also with reasoning under uncertainty. A model may be trained on a sample, an experiment may produce an estimated treatment effect, or a prediction may be accompanied by a probability. In each case, the analyst must distinguish what is directly observed from what is inferred about a wider population or a future outcome.

Probability provides the mathematical language for uncertainty. Statistical inference uses sample data to estimate unknown population quantities and evaluate claims. Experimentation provides a disciplined method for identifying causal effects. In professional data science, these areas are inseparable: a technically sophisticated analysis can still be misleading if sampling is biased, uncertainty is ignored, multiple comparisons are uncontrolled, or a statistically significant effect is operationally unimportant.

This chapter develops these ideas and implements them using R. The emphasis is on assumptions, interpretation, reproducibility and decision-making rather than on mechanical computation alone.

Learning Objectives

By the end of this chapter, students should be able to:

1. Explain conditional probability, Bayes’ theorem and common probability distributions.

2. Distinguish populations, samples, parameters, statistics and sampling distributions.

3. Construct and interpret point estimates and confidence intervals.

4. Formulate and evaluate hypotheses using parametric, non-parametric and resampling methods.

5. Design and analyse A/B experiments, including power and sample-size considerations.

6. Control false discoveries in multiple-testing settings.

7. Distinguish statistical significance from practical and business significance.

8. Implement sound inference workflows in R and communicate their limitations.

1. Probability as a Foundation for Data Science

A random experiment has possible outcomes in a sample space, Ω. An event A is a subset of Ω, and its probability P(A) lies between 0 and 1. For complementary events, P(Aᶜ) = 1 − P(A). If A and B are events, the addition rule is

P(A ∪ B) = P(A) + P(B) − P(A ∩ B).

The subtraction avoids counting the intersection twice. When A and B are mutually exclusive, P(A ∩ B) = 0.

Conditional probability is defined, when P(B) > 0, as

P(A | B) = P(A ∩ B) / P(B).

Events are independent when P(A | B) = P(A), equivalently P(A ∩ B) = P(A)P(B). Independence is a substantive or design assumption; it should not be inferred merely because two variables appear unrelated in a small data set.

Bayes’ theorem

For events A and B with positive probabilities,

P(A | B) = P(B | A)P(A) / P(B).

For mutually exclusive and exhaustive hypotheses H₁, ..., Hₖ,

P(Hᵢ | D) = P(D | Hᵢ)P(Hᵢ) / Σⱼ P(D | Hⱼ)P(Hⱼ).

Here P(Hᵢ) is a prior probability, P(D | Hᵢ) is the likelihood, and P(Hᵢ | D) is the posterior probability. Bayes’ theorem explains why a highly accurate diagnostic can have a modest positive predictive value when the condition is rare. It also underlies Bayesian modelling, spam filtering and probabilistic risk assessment.

Random variables and expectation

A random variable assigns a numerical value to each outcome. A discrete variable has a probability mass function; a continuous variable has a density. For a discrete X,

E(X) = Σₓ xP(X = x),

and for a continuous X, E(X) = ∫ x f(x) dx. Variance is

Var(X) = E[(X − μ)²],

where μ = E(X). Standard deviation is the square root of variance and has the same units as X. For independent random variables X and Y, Var(X + Y) = Var(X) + Var(Y). More generally, covariance terms must be included.

2. Probability Distributions Relevant to Data Science

A distribution is a model for uncertainty, not a guarantee that data follow a theoretical curve exactly. Selection should reflect the measurement process and the purpose of analysis.

Bernoulli and binomial distributions

A Bernoulli variable records one trial with success probability p: X = 1 for success and X = 0 otherwise. Thus E(X) = p and Var(X) = p(1 − p).

If n trials are independent and have the same success probability p, the number of successes Y follows a binomial distribution:

P(Y = y) = C(n,y)pʸ(1 − p)ⁿ⁻ʸ, y = 0, ..., n.

Its mean is np and variance is np(1 − p). Independence, a fixed number of trials and a common probability are required. Repeated observations from the same user may violate independence.

Poisson distribution

The Poisson model describes counts in a fixed exposure interval when events occur independently at a constant rate λ:

P(X = x) = e⁻λ λˣ / x!, x = 0,1,2,...

Both mean and variance equal λ. In applications such as incident counts, the exposure time or population size may differ; Poisson regression then uses an offset to model rates rather than raw counts. Overdispersion, clustering and changing rates may require a negative binomial or another model.

Normal distribution

A normal variable has density

f(x) = [1/(σ√(2π))] exp[−(x − μ)²/(2σ²)],

where μ is the mean and σ > 0 is the standard deviation. The normal model is continuous and symmetric. It is often useful for measurement errors and sample means, but not automatically appropriate for bounded, strongly skewed or count data.

The central limit theorem states, under suitable regularity conditions, that standardized sums or sample means tend toward a normal distribution as sample size increases. It does not say that the original observations become normal. Dependence, extreme heavy tails and infinite variance can undermine simple applications.

Other useful distributions

The t distribution has heavier tails than the standard normal and is used when estimating a mean with unknown variance, particularly in small samples. The chi-square distribution is used in variance-related inference and contingency-table tests. The F distribution appears in variance ratios and analysis of variance. The exponential distribution models waiting times under a Poisson process. Beta distributions are useful for probabilities and proportions in Bayesian models.

In R, distribution functions follow a consistent naming scheme: `d` gives a density or mass, `p` a cumulative probability, `q` a quantile and `r` random generation. For example, `pbinom(7, size=10, prob=.6)` computes P(Y ≤ 7), while `qnorm(.975)` gives the 97.5th normal quantile.

3. Sampling Theory and Sampling Distributions

A population is the complete set about which a conclusion is intended; a sample is the observed subset. A parameter, such as μ or p, describes a population. A statistic, such as the sample mean x̄ or sample proportion p̂, is calculated from a sample.

A probability sample gives units a known, non-zero chance of selection. Simple random sampling, stratified sampling and cluster sampling have different variance and cost properties. Convenience and opt-in samples may be useful for exploration, but their sampling mechanism often prevents defensible population generalisation.

Sampling distributions

A sampling distribution is the distribution of a statistic over repeated samples from the same population and design. If observations X₁, ..., Xₙ are independent with mean μ and variance σ², then

E(x̄) = μ and Var(x̄) = σ²/n.

The standard error of x̄ is therefore σ/√n; in practice σ is usually replaced by the sample standard deviation s. Increasing n reduces standard error at a square-root rate: quadrupling sample size approximately halves it. This relationship does not remove systematic bias.

The standard error of a sample proportion under independent Bernoulli sampling is approximately √[p(1 − p)/n], estimated by substituting p̂ for p. For a difference between independent means, the estimated standard error is

SE(x̄₁ − x̄₂) = √(s₁²/n₁ + s₂²/n₂).

This formula assumes independent groups; Welch’s method does not require equal population variances and is generally preferable to the pooled-variance t test unless equal variances are substantively justified.

4. Estimation and Confidence Intervals

A point estimate is a single value used to estimate a parameter. An interval estimate expresses uncertainty arising from sampling. A 95% confidence interval is constructed by a procedure that would cover the fixed parameter in approximately 95% of repeated samples under the model and sampling design. It is not strictly correct to say that there is a 95% probability that a fixed parameter lies in one already-computed frequentist interval.

For a mean with unknown variance, a conventional interval is

x̄ ± t₁₋α/₂, n₋₁ (s/√n),

where α is the error rate, t is a t-distribution quantile, n is sample size and s is sample standard deviation. This is exact under independent normal observations and often reasonably robust for moderate samples without severe skewness or outliers.

For a difference in independent means, use

(x̄₁ − x̄₂) ± t* SE(x̄₁ − x̄₂),

with degrees of freedom obtained by the Welch approximation. A confidence interval for a proportion based on a normal approximation can perform poorly for small n or proportions near 0 or 1; Wilson or exact methods are often better.

A confidence interval communicates both direction and precision. Narrow intervals can result from large samples, while wide intervals may reflect noisy outcomes or limited data. A confidence interval does not account for bias introduced by nonresponse, confounding or measurement error.

R example: estimating a mean

```r

x <- c(71, 68, 75, 74, 69, 77, 72, 70)

t.test(x, conf.level = 0.95)

```

`t.test` reports the sample mean, a t-based interval and a test of a null mean if one is supplied. Analysts should inspect the data and design assumptions rather than treating the function output as an automatic conclusion.

5. Hypothesis Testing

A hypothesis test compares data with a null model. The null hypothesis H₀ usually represents no difference or a specified value; H₁ represents an alternative. A p-value is the probability, assuming H₀ and the test assumptions are true, of obtaining a result at least as extreme as the observed result in the prespecified direction or in both directions, depending on the test.

A p-value is not the probability that H₀ is true, nor is it the probability that results will replicate. A small p-value indicates incompatibility with the null model, not necessarily a large or useful effect.

A Type I error rejects a true H₀ and has probability α under the test procedure. A Type II error fails to reject a false H₀ and has probability β. Power is 1 − β: the probability of detecting a specified effect under a specified alternative, sample size, variability and significance threshold. “Fail to reject” should not be reported as proof of no effect.

Common tests

A one-sample t test evaluates a mean against a reference value; a two-sample Welch t test compares independent means; a paired t test analyses within-unit differences. The paired design can substantially reduce noise when the same unit is measured twice.

A chi-square test can assess independence in a contingency table, although sparse expected counts may require exact or model-based methods. For two proportions, a model or confidence interval for the difference is often more informative than a test alone. Rank-based tests such as Mann–Whitney compare distributional ranks, but they are not universally tests of means or medians.

Regression provides a flexible inference framework. In a linear model, a coefficient represents the expected change in the outcome associated with a one-unit predictor change, conditional on other included predictors and the model specification. Causal interpretation requires design or credible assumptions such as exchangeability, correct temporal order and adequate control of confounding.

6. Resampling and Robust Inference

Resampling methods use the observed data to approximate repeated sampling. The non-parametric bootstrap repeatedly samples n observations with replacement from the empirical data distribution, calculates a statistic, and uses the resulting bootstrap distribution to form standard errors or intervals.

```r

set.seed(42)

boot_mean <- replicate(5000, mean(sample(x, replace = TRUE)))

quantile(boot_mean, c(.025, .975))

```

The bootstrap is useful for medians, ratios and complex statistics for which analytical standard errors are difficult. It may perform poorly with very small samples, dependent time series, extreme boundaries or data that do not represent the target population. Block or stationary bootstrap methods are more appropriate for serial dependence.

Permutation tests generate a null distribution by rearranging labels under an exchangeability assumption. For a randomized two-group experiment, treatment labels are exchangeable under the null of no treatment effect, making permutation inference attractive. Resampling cannot repair confounding or selection bias; it quantifies uncertainty conditional on the observed design.

7. Multiple Testing

Testing many hypotheses increases the chance of false positives. If m independent tests each use α = .05, the probability of at least one false rejection under all true nulls is 1 − (1 − .05)^m. For ten tests this is approximately 0.401, not .05.

The family-wise error rate (FWER) is the probability of one or more false rejections in a family. Bonferroni controls FWER by testing each hypothesis at α/m or multiplying p-values by m. Holm’s step-down procedure is usually at least as powerful while controlling FWER.

The false discovery rate (FDR) is the expected proportion of false discoveries among rejected hypotheses. The Benjamini–Hochberg procedure ranks p-values p(1) ≤ ... ≤ p(m), identifies the largest k satisfying p(k) ≤ (k/m)q, and rejects hypotheses 1 through k, where q is the target FDR. FDR is often appropriate in exploratory genomics, feature screening and large-scale experimentation, whereas safety-critical decisions may require FWER control.

The family must be defined in advance. Selective reporting, optional stopping and trying many outcomes until one is significant invalidate nominal error rates. Registering hypotheses, outcomes and analysis rules helps preserve credibility.

8. Power Analysis and Sample Size

Power analysis should occur before data collection. For a two-group comparison, power depends on the minimum effect considered important, outcome variance, allocation ratio, α, sidedness and the assumed analysis. A statistically detectable effect may be too small to matter; a practically important effect may be missed by an underpowered study.

For a rough one-sample mean calculation with known standard deviation σ, two-sided α and target difference δ, the required sample size is approximately

n ≈ [(z₁₋α/₂ + z₁₋β)σ/δ]².

This approximation assumes independent observations, a continuous outcome and a normal-based analysis. Round upward and allow for attrition or noncompliance. For proportions and clustered data, use formulas or software appropriate to those designs.

In R, prospective calculations can use packages such as `pwr`, but the analyst must supply defensible effect and variance assumptions. Sensitivity analyses over several plausible effects are preferable to presenting one falsely precise number.

9. A/B Experimentation and Causal Interpretation

An A/B experiment randomly assigns eligible units to a control or treatment condition. Randomisation balances measured and, in expectation, unmeasured confounders, so the difference in group means or proportions estimates an average treatment effect under stable implementation and appropriate compliance.

A credible experiment specifies the unit of randomisation, population, treatment, primary outcome, analysis window, exclusion rules and stopping policy. Randomise at the user level when interference between users is limited. If users can appear repeatedly, analyse at the randomisation unit or use methods that account for clustering. Guard against contamination, treatment switching, novelty effects and unequal exposure.

For a binary conversion outcome, the absolute effect is p̂T − p̂C. Relative lift is (p̂T − p̂C)/p̂C when p̂C > 0. Number needed to treat, for beneficial outcomes, is approximately 1/|pT − pC|, subject to uncertainty and interpretation. Absolute differences are generally more stable for operational decisions than relative percentages alone.

A good experiment reports an effect estimate, confidence interval, prespecified primary analysis and practical decision threshold. It does not stop merely because a p-value crosses .05. Sequential monitoring requires alpha-spending, group-sequential methods or Bayesian decision rules designed for repeated looks.

Illustrative A/B case: checkout redesign

Background. An online retailer considers a simplified checkout page. The product team expects more completed purchases but is concerned about refunds and average order value.

Problem. A historical conversion rate is 8%. The organisation defines a minimum practically important improvement as 0.8 percentage points, from 8.0% to 8.8%, and requires no material increase in refunds.

Analysis and approach. Eligible users are randomly assigned once to control or treatment. The primary outcome is purchase within seven days; refund rate and revenue per eligible user are secondary outcomes. The team calculates sample size using the baseline rate, target difference, α = .05, desired power and expected attrition. Assignment and exposure are logged, and the analysis is intention-to-treat: users are analysed according to assignment. The team does not repeatedly inspect the primary p-value without a prespecified sequential procedure.

Suppose the final data contain 50,000 users per group. Control conversion is 4,000/50,000 = 0.080; treatment conversion is 4,450/50,000 = 0.089. The observed absolute difference is 0.009, or 0.9 percentage points. An approximate standard error for the independent difference in proportions is

SE = √[0.08(0.92)/50,000 + 0.089(0.911)/50,000] ≈ 0.00175.

An approximate 95% interval is 0.009 ± 1.96(0.00175), or approximately (0.0056, 0.0124). The interval excludes zero and lies above the 0.008-point practical threshold only partly: its lower endpoint is below 0.008. Thus the evidence supports improvement, but not high confidence that the minimum important effect has been achieved. The decision should also incorporate refund and revenue results, implementation cost and longer-term effects.

Outcome. If refunds remain stable and revenue is not harmed, deployment may be justified with continued monitoring. If refunds rise, a statistically positive conversion result may not be commercially beneficial.

Learning points. Randomisation supports causal interpretation, but it does not eliminate operational bias. A primary metric, practical threshold, power plan and guardrail outcomes should be established before the experiment.

10. Statistical Versus Practical Significance

Statistical significance is a statement about compatibility with a null model at a chosen error threshold. Practical significance concerns whether the magnitude is valuable, harmful or relevant to a decision. With millions of observations, a negligible latency reduction may have p < .001. With a small sample, a clinically important improvement may have p > .05.

Interpretation should therefore combine effect size, confidence interval, baseline risk, cost, benefit, harm, heterogeneity and decision thresholds. Standardised effects can aid comparison across scales, but they should not replace domain units. Analysts should communicate both absolute and relative effects and explain what uncertainty remains.

11. Limitations, Ethics and Governance

Inference is threatened by selection bias, nonresponse, confounding, measurement error, missing data, dependence, model misspecification and outcome switching. Missingness mechanisms matter: methods suitable for missing completely at random are not automatically valid for missing not at random. Sensitivity analyses should examine plausible departures from assumptions.

Ethically, experiments require respect for autonomy, privacy, fairness and safety. Users should not be exposed to unreasonable risk merely to increase conversion. Sensitive attributes require access controls, minimisation and lawful processing. Randomisation may be inappropriate where withholding a known effective intervention causes harm. Data protection, institutional review and sector-specific regulation should be considered.

AI systems intensify these issues. Automated experimentation can generate thousands of comparisons, adapt treatments and affect vulnerable groups. Governance should require versioned data and code, audit logs, prespecified metrics, subgroup checks, bias assessment and human review of consequential decisions. Reproducibility includes recording random seeds, software versions, exclusions and analysis changes.

12. Best-Practice Workflow in R

A defensible workflow is:

1. Define the estimand: population, treatment contrast, outcome and time horizon.

2. Specify the sampling or assignment mechanism and unit of analysis.

3. Inspect data quality, missingness, dependence and outliers without outcome-driven exclusions.

4. State hypotheses, practical thresholds, primary outcomes and stopping rules.

5. Choose a method whose assumptions match the design and measurement scale.

6. Estimate effects with uncertainty, preferably using confidence intervals or compatible intervals.

7. Adjust for multiplicity when a family of hypotheses is examined.

8. Perform sensitivity, subgroup and robustness analyses, distinguishing confirmatory from exploratory work.

9. Reproduce results using scripts, seeds and documented software environments.

10. Report limitations and translate statistical results into an operational decision.

R functions such as `t.test`, `prop.test`, `chisq.test`, `lm`, `glm`, `wilcox.test`, `p.adjust` and simulation tools support these tasks, but no function can determine whether a sampling design is representative or whether an effect is practically worthwhile.

Chapter Summary

Probability formalises uncertainty through events, conditional probability, independence, random variables and distributions. Bayes’ theorem updates prior beliefs using evidence. Sampling theory distinguishes populations and samples and explains why standard errors generally decline with the square root of sample size. Estimation combines point estimates with confidence intervals, while hypothesis testing evaluates data against a null model and must not be confused with proof or probability of a hypothesis.

Resampling provides flexible uncertainty estimates but cannot correct biased designs. Power depends on effect size, variability, sample size and the error threshold. Multiple testing requires explicit control of FWER or FDR. Randomised A/B experiments can support causal conclusions when assignment, exposure, outcomes and analysis are carefully designed. Finally, statistical significance must be interpreted alongside practical importance, cost, risk, fairness and governance.

Key Terms

Probability: Numerical representation of uncertainty about an event.

Conditional probability: Probability of an event given that another event has occurred.

Bayes’ theorem: Rule for updating a probability using prior information and evidence.

Random variable: Numerical function of outcomes of a random experiment.

Expectation: Probability-weighted long-run average of a random variable.

Variance: Expected squared deviation from the mean.

Population: Complete set to which an inference is intended to apply.

Sample: Observed subset of a population.

Parameter: Numerical characteristic of a population.

Statistic: Quantity calculated from sample data.

Sampling distribution: Distribution of a statistic across repeated samples.

Standard error: Estimated standard deviation of a sampling distribution.

Confidence interval: Interval produced by a procedure with stated long-run coverage.

Null hypothesis: Claim used as a reference model in hypothesis testing.

P-value: Probability, under the null model, of data at least as extreme as observed.

Power: Probability of detecting a specified effect under specified conditions.

Bootstrap: Resampling observations with replacement to approximate sampling uncertainty.

False discovery rate: Expected proportion of false rejections among all rejections.

A/B experiment: Randomised comparison of a treatment and control condition.

Practical significance: Decision relevance of an effect’s magnitude and consequences.

Review Questions

Short Answer Questions

1. Distinguish a parameter from a statistic and give one example of each.

2. What does a 95% confidence interval mean in repeated sampling?

3. Why does increasing sample size not necessarily remove bias?

4. Define statistical power and name two factors that affect it.

5. Why is a p-value not the probability that the null hypothesis is true?

Long Answer Questions

1. Explain Bayes’ theorem and discuss how base rates affect interpretation of diagnostic results.

2. Compare the binomial, Poisson and normal distributions, including their assumptions and applications.

3. Explain how sampling design, dependence and missing data affect statistical inference.

4. Compare FWER and FDR control and identify contexts in which each is appropriate.

5. Describe a complete workflow for designing, analysing and reporting a credible A/B experiment.

Analytical / Application Questions

1. An experiment observes conversion of 800/10,000 control users and 860/10,000 treatment users. Calculate the absolute difference and relative lift, then explain what additional information is needed before deployment.

2. A researcher tests 40 independent hypotheses at α = .05. Explain why unadjusted testing is problematic and recommend an error-control strategy for an exploratory study.

3. A statistically significant model shows a latency reduction of 0.2 milliseconds with a very large sample. Evaluate whether the result is practically significant and identify the evidence required for a decision.

Multiple Choice Questions with Answers

Question 1

Which statement correctly describes a p-value?

A. The probability that the null hypothesis is true

B. The probability of data at least as extreme as observed, assuming the null model

C. The probability that the alternative hypothesis is true

D. The probability that a result will replicate

Correct Answer: B. The probability of data at least as extreme as observed, assuming the null model

Question 2

If the variance of independent observations is σ², what is the variance of their sample mean for sample size n?

A. σ²n

B. σ²/n

C. σ/n²

D. σ²

Correct Answer: B. σ²/n

Question 3

Which distribution is most directly associated with a count of successes in n independent trials with common success probability p?

A. Poisson

B. Normal

C. Binomial

D. Exponential

Correct Answer: C. Binomial

Question 4

What is the principal purpose of random assignment in an A/B experiment?

A. To guarantee a large sample

B. To balance confounding factors in expectation

C. To eliminate measurement error

D. To ensure every outcome is normally distributed

Correct Answer: B. To balance confounding factors in expectation

Question 5

Which procedure controls the family-wise error rate while generally being less conservative than Bonferroni?

A. Holm’s step-down procedure

B. Benjamini–Hochberg

C. Ordinary least squares

D. Bootstrap percentile sampling

Correct Answer: A. Holm’s step-down procedure

Question 6

A fourfold increase in sample size generally changes the standard error by what factor, assuming other conditions remain constant?

A. It doubles it

B. It halves it

C. It quarters it

D. It leaves it unchanged

Correct Answer: B. It halves it

Question 7

Which resampling method samples observations with replacement from the observed data?

A. Permutation test

B. Bootstrap

C. Stratified randomisation

D. Cross-validation only

Correct Answer: B. Bootstrap

Question 8

Which statement best distinguishes practical from statistical significance?

A. Practical significance depends only on the p-value

B. Statistical significance measures business value directly

C. Practical significance concerns whether the effect is consequential for a decision

D. They are always identical in large samples

Correct Answer: C. Practical significance concerns whether the effect is consequential for a decision

Question 9

For a Poisson distribution with rate λ, which relationship holds?

A. Mean equals variance, both λ

B. Mean equals λ² and variance equals λ

C. Mean equals variance, both √λ

D. Variance is always zero

Correct Answer: A. Mean equals variance, both λ

Question 10

What is the primary reason to define a minimum practically important effect before an experiment?

A. To guarantee statistical significance

B. To connect sample-size planning and interpretation to a meaningful decision threshold

C. To avoid collecting any control data

D. To make all confidence intervals narrower

Correct Answer: B. To connect sample-size planning and interpretation to a meaningful decision threshold