Limit Theorems and Sampling Distributions

Introduction

Probability models describe uncertainty before data are observed; statistics uses observed data to learn about unknown features of those models. Limit theorems provide the mathematical bridge between these two activities. They explain why averages stabilize as sample size increases, why the normal distribution appears so frequently in statistical work, and how the uncertainty of estimators can be quantified.

These ideas are central to artificial intelligence and machine learning. Training data are samples from a broader population or data-generating process. Sample means, proportions, loss estimates, validation scores, simulation outputs, and estimated model parameters all exhibit sampling variation. Understanding that variation helps practitioners distinguish genuine patterns from random fluctuations.

This chapter develops the laws of large numbers, the central limit theorem, order statistics, and important sampling distributions. It also examines Monte Carlo computation, practical assumptions, limitations, and an applied AI case.

Learning Objectives

By the end of this chapter, students should be able to:

1. Explain convergence in probability and almost sure convergence.

2. State and interpret the weak and strong laws of large numbers.

3. Apply the central limit theorem to sums, means, proportions, and Monte Carlo estimates.

4. Distinguish a population distribution from a sampling distribution.

5. Derive and use sampling distributions for the normal mean, variance, and proportion.

6. Explain order statistics and use them to analyze minima, maxima, quantiles, and nearest-neighbor behavior.

7. Identify assumptions, limitations, and dependence problems in asymptotic approximations.

8. Interpret sampling uncertainty in AI and data-science applications.

1. Convergence and Repeated Sampling

Let X_1, X_2, ..., X_n be random variables representing observations. The sample mean is

\[

\bar X_n=\frac{1}{n}\sum_{i=1}^{n}X_i.

\]

If the observations are independent and identically distributed (i.i.d.) with mean μ and variance σ², then

\[

E(\bar X_n)=\mu, \qquad \operatorname{Var}(\bar X_n)=\frac{\sigma^2}{n}.

\]

The expected value of the sample mean remains μ, while its variance decreases as n increases. Thus, larger samples generally produce more stable averages.

1.1 Modes of convergence

A sequence X_n converges in probability to a constant c if, for every ε>0,

\[

P(|X_n-c|>\varepsilon)\to 0 \quad \text{as } n\to\infty.

\]

This is written X_n →p c. It means that the probability of a substantial discrepancy from c becomes negligible.

Almost sure convergence, written X_n →as c, means

\[

P\left(\lim_{n\to\infty}X_n=c\right)=1.

\]

Almost sure convergence is stronger than convergence in probability. In practical statistical language, convergence in probability is often sufficient for consistency: an estimator becomes increasingly likely to be close to the parameter it estimates.

Convergence in distribution concerns the limiting behavior of probability distributions. If the distribution function of X_n approaches that of X at every continuity point of X's distribution function, then X_n →d X. Convergence in distribution does not necessarily imply that the numerical values of X_n become close to X on each outcome.

2. Laws of Large Numbers

2.1 Weak law of large numbers

The weak law of large numbers states that if X_1, X_2, ... are i.i.d. with finite mean μ, then

\[

\bar X_n \xrightarrow{p} \mu.

\]

In words, the sample mean converges in probability to the population mean. A common proof uses Chebyshev's inequality. Since Var(\bar X_n)=σ²/n,

\[

P(|\bar X_n-\mu|\ge \varepsilon)

\le \frac{\operatorname{Var}(\bar X_n)}{\varepsilon^2}

=\frac{\sigma^2}{n\varepsilon^2}\to 0.

\]

The finite-variance assumption is sufficient but not necessary in the most general versions; a finite first absolute moment is often enough for i.i.d. observations. At the bachelor level, the finite-variance form is the most useful operational version.

2.2 Strong law of large numbers

Under standard conditions, particularly for i.i.d. observations with E(|X_1|)<∞, the strong law states

\[

\bar X_n \xrightarrow{as} \mu.

\]

The strong law says that, with probability one, the entire sequence of sample means eventually settles toward μ. The weak law describes probabilities at each sample size; the strong law describes almost every infinite sequence of observations.

2.3 Interpretation and limitations

The law of large numbers does not say that every finite sample mean is close to μ. It says that large samples reduce the probability of a large error. It also does not guarantee fast convergence. Heavy-tailed data may require very large samples before averages become reliable. Dependence can also invalidate the simple i.i.d. argument. For example, repeated measurements from the same patient or highly correlated time-series observations contain less information than the same number of independent observations.

3. The Central Limit Theorem

The law of large numbers concerns the location of a sample mean. The central limit theorem (CLT) concerns its distribution after appropriate scaling.

3.1 Classical CLT

Suppose X_1, ..., X_n are i.i.d. with finite mean μ and finite, positive variance σ². Then

\[

Z_n=\frac{\sqrt n(\bar X_n-\mu)}{\sigma}

=\frac{\sum_{i=1}^{n}X_i-n\mu}{\sigma\sqrt n}

\xrightarrow{d}N(0,1).

\]

Equivalently, for sufficiently large n,

\[

\bar X_n \approx N\left(\mu,\frac{\sigma^2}{n}\right).

\]

The symbol N(a,b) here denotes a normal distribution with mean a and variance b. Its standard deviation is σ/√n, often called the standard error of the mean when σ is known.

The CLT does not claim that the original observations are normal. It says that the standardized sum or mean is approximately normal under broad conditions. The approximation is often good for moderate sample sizes when the population is not extremely skewed or heavy-tailed, but no universal sample-size rule exists.

3.2 From sums to means

For a sum S_n=ΣX_i,

\[

E(S_n)=n\mu, \qquad \operatorname{Var}(S_n)=n\sigma^2,

\]

and the CLT gives

\[

\frac{S_n-n\mu}{\sigma\sqrt n}\xrightarrow{d}N(0,1).

\]

For a sample mean, the center is μ and the variance is σ²/n. This distinction is important: sums become more variable as n grows, whereas averages become more concentrated.

3.3 Standardizing an observed mean

If σ is known, an observed sample mean x̄ can be standardized using

\[

z=\frac{\bar x-\mu_0}{\sigma/\sqrt n},

\]

where μ_0 is a hypothesized population mean. If σ is unknown and the population is normal, the exact t distribution is used instead; for large samples, replacing σ with the sample standard deviation s often leads to an approximate normal procedure.

3.4 Berry–Esseen perspective

The quality of a normal approximation depends on skewness and tail behavior. A Berry–Esseen result provides a bound of the general form

\[

\sup_x\left|P(Z_n\le x)-\Phi(x)\right|

\le \frac{C\,E|X-\mu|^3}{\sigma^3\sqrt n},

\]

when the third absolute central moment is finite. Here Φ is the standard normal distribution function and C is a universal constant. The expression shows the usual 1/√n rate and explains why strongly skewed or heavy-tailed populations converge slowly.

4. The CLT for Proportions and Monte Carlo Estimation

4.1 Sample proportions

Let X_i be Bernoulli variables, where X_i=1 indicates success and X_i=0 indicates failure. If P(X_i=1)=p, then

\[

E(X_i)=p, \qquad \operatorname{Var}(X_i)=p(1-p).

\]

The sample proportion is

\[

\hat p=\frac{1}{n}\sum_{i=1}^{n}X_i.

\]

For large n,

\[

\hat p\approx N\left(p,\frac{p(1-p)}{n}\right).

\]

A common approximation is reasonable when both np and n(1−p) are sufficiently large, often at least about 5 to 10 depending on the required accuracy. For small counts or p near 0 or 1, exact binomial or improved interval methods are preferable.

4.2 Monte Carlo estimation

Suppose the goal is to estimate θ=E[g(X)] by simulation. Generate independent X_1,...,X_n and calculate

\[

\hat\theta_n=\frac{1}{n}\sum_{i=1}^{n}g(X_i).

\]

If E[g(X)] exists, the law of large numbers gives consistency. If Var(g(X))=τ²<∞, then

\[

\sqrt n(\hat\theta_n-\theta)\xrightarrow{d}N(0,\tau^2),

\]

so the Monte Carlo standard error is approximately τ/√n. In practice τ is unknown and is estimated by the sample standard deviation of g(X_i), divided by √n.

The 1/√n rate has an important practical implication: reducing Monte Carlo error by a factor of two requires approximately four times as many independent simulations. Better random-number generation, variance reduction, quasi-Monte Carlo methods, antithetic variables, and importance sampling can be more efficient than simply increasing n.

Worked Example 1: Monte Carlo uncertainty

A simulation estimates a failure probability using n=10,000 independent trials. The estimated probability is p̂=0.04. The estimated standard error is

\[

\sqrt{\frac{\hat p(1-\hat p)}{n}}

=\sqrt{\frac{0.04(0.96)}{10000}}

\approx 0.00196.

\]

Using the approximate 95% normal margin 1.96×0.00196≈0.00384, an approximate interval is

\[

0.04\pm0.00384=(0.03616,0.04384).

\]

This interval represents simulation uncertainty under independent trials and the adequacy of the normal approximation; it does not account for model error or bias in the simulated system.

5. Sampling Distributions

A sampling distribution is the probability distribution of a statistic over repeated samples of a fixed size from a population. It is not the distribution of individual observations.

For example:

• The population distribution describes individual customer waiting times.
• The distribution of X̄ describes sample-average waiting times.
• The distribution of p̂ describes sample proportions.
• The distribution of S² describes sample variance estimates.

Sampling distributions provide standard errors, confidence intervals, hypothesis tests, and power calculations.

5.1 Sampling distribution of the normal sample mean

If X_1,...,X_n are i.i.d. N(μ,σ²), then exactly

\[

\bar X\sim N\left(\mu,\frac{\sigma^2}{n}\right).

\]

No large-sample approximation is needed. If the population is not normal, the CLT provides an approximate result under suitable conditions.

5.2 The chi-square distribution and sample variance

For a normal sample, define

\[

S^2=\frac{1}{n-1}\sum_{i=1}^{n}(X_i-\bar X)^2.

\]

Then

\[

\frac{(n-1)S^2}{\sigma^2}\sim \chi^2_{n-1},

\]

where χ² with ν degrees of freedom is the distribution of the sum of squares of ν independent standard normal variables. The n−1 degrees of freedom arise because deviations from the sample mean sum to zero, leaving one linear constraint.

For normal data, \bar X and S² are independent. This special independence supports exact small-sample inference for a normal mean.

5.3 The Student t distribution

If X_1,...,X_n are i.i.d. normal with unknown μ and σ², then

\[

T=\frac{\bar X-\mu}{S/\sqrt n}\sim t_{n-1}.

\]

The t distribution has heavier tails than N(0,1), reflecting the extra uncertainty from estimating σ. As degrees of freedom increase, the t distribution approaches the standard normal distribution.

A two-sided 100(1−α)% confidence interval for μ is

\[

\bar x\pm t_{n-1,1-\alpha/2}\frac{s}{\sqrt n},

\]

where t_{n−1,1−α/2} is the corresponding t quantile. The exact interpretation is that a procedure constructed this way captures μ in 100(1−α)% of repeated samples under the model; it is not that μ has a frequentist probability of lying in the already computed interval.

Worked Example 2: A confidence interval for a mean

A normally distributed measurement has n=16, sample mean x̄=72, and sample standard deviation s=8. For a 95% confidence interval, the critical value with 15 degrees of freedom is approximately 2.131. The standard error is

\[

\frac{s}{\sqrt n}=\frac{8}{4}=2.

\]

Therefore,

\[

72\pm2.131(2)=72\pm4.262,

\]

giving approximately (67.74, 76.26) measurement units.

6. Order Statistics

Given observations X_1,...,X_n, arrange them as

\[

X_{(1)}\le X_{(2)}\le\cdots\le X_{(n)}.

\]

The value X_(j) is the jth order statistic. The minimum is X_(1), the maximum is X_(n), and sample medians and quantiles are based on central order statistics.

Order statistics are important in anomaly detection, robust statistics, reliability engineering, ranking systems, nearest-neighbor methods, and extreme-value analysis.

6.1 Distribution of an order statistic

Assume X_1,...,X_n are i.i.d. from a continuous distribution with cumulative distribution function F and density f. The density of X_(k) is

\[

f_{X_{(k)}}(x)=\frac{n!}{(k-1)!(n-k)!}[F(x)]^{k-1}[1-F(x)]^{n-k}f(x).

\]

The factors represent: k−1 observations below x, one observation near x, and n−k observations above x.

For the minimum,

\[

P(X_{(1)}>x)=[1-F(x)]^n,

\]

so

\[

F_{X_{(1)}}(x)=1-[1-F(x)]^n.

\]

For the maximum,

\[

P(X_{(n)}\le x)=[F(x)]^n.

\]

6.2 Uniform example

If X_i are i.i.d. Uniform(0,1), then F(x)=x for 0≤x≤1. The kth order statistic has a Beta(k,n+1−k) distribution. Consequently,

\[

E[X_{(k)}]=\frac{k}{n+1}.

\]

For n=9, the expected maximum is 9/10=0.9, not 1. The sample maximum approaches 1 as n increases, but for every finite n it is typically below 1.

6.3 Quantiles and medians

Order statistics provide estimates of population quantiles. The sample median is robust to extreme values because it depends on rank rather than magnitude. However, quantile estimates can be unstable in small samples, and their sampling distributions may be asymmetric, particularly near the tails.

7. Beyond Independent Identically Distributed Data

The classical results are foundations, not universal substitutes for modeling.

7.1 Dependence

If observations are correlated, then

\[

\operatorname{Var}(\bar X_n)=\frac{1}{n^2}\sum_{i=1}^{n}\sum_{j=1}^{n}\operatorname{Cov}(X_i,X_j).

\]

Positive covariance increases the variance of the mean relative to the i.i.d. formula. In time series, the effective sample size can be much smaller than n. Randomly splitting correlated observations into training and test sets can therefore produce overly optimistic uncertainty estimates.

7.2 Non-identical observations

Independent but non-identically distributed observations may still satisfy generalized laws of large numbers and CLTs, but conditions must control the contribution of any single observation. The Lindeberg and Lyapunov conditions are examples of sufficient CLT conditions. Their practical message is that no small group of observations should dominate the total variability.

7.3 Heavy tails and robust alternatives

If variance is infinite, the classical CLT may fail, and sums may converge to stable non-normal distributions. Even with finite variance, extreme observations can make normal approximations inaccurate at realistic sample sizes. Transformations, trimmed means, winsorization, bootstrap methods, or robust estimators may be appropriate, but each changes the target or assumptions and must be justified.

8. Applied Case: Evaluating an AI-Based Defect Detector

The following is an illustrative case.

Background

A manufacturer deploys an image-classification model to identify defective components. A validation set contains n=2,000 independently selected components, and the model correctly classifies 1,860 of them. The observed accuracy is

\[

\hat p=1860/2000=0.93.

\]

Problem

Managers want to know whether the model's true accuracy is reliably above 0.90, rather than whether 0.93 is merely a favorable result from one validation sample.

Analysis

Treating each validation outcome as Bernoulli with probability p, the estimated standard error is

\[

\sqrt{\frac{0.93(0.07)}{2000}}\approx0.00570.

\]

An approximate 95% interval is

\[

0.93\pm1.96(0.00570)

\approx(0.9188,0.9412).

\]

This interval lies above 0.90, suggesting strong sampling evidence that accuracy exceeds 90% under the validation-data assumptions.

Approach and governance checks

The statistical calculation is not sufficient by itself. The team must verify that:

• validation images are independent enough;
• the validation distribution represents future operating conditions;
• images from the same physical component or production batch have not been split across training and validation sets;
• class imbalance has not hidden poor performance on rare defects;
• confidence intervals are also reported for recall, precision, and subgroup performance;
• the threshold and evaluation metric were specified before inspecting results.

If many candidate models were evaluated and only the best result was reported, the displayed interval understates selection uncertainty. If deployment data drift over time, the historical interval may not describe current accuracy.

Outcome

The model appears to exceed 90% accuracy on the sampled population, but the deployment decision should combine sampling uncertainty with uncertainty about representativeness, label quality, subgroup fairness, and temporal drift.

Learning points

Sampling distributions quantify only the uncertainty represented by the statistical model. They do not automatically correct biased sampling, data leakage, adaptive experimentation, or measurement error. In AI governance, statistical validity and data governance must be considered together.

9. Best Practices and Common Errors

1. Separate population, sample, parameter, and statistic. Confusing μ with x̄ or p with p̂ leads to incorrect interpretations.

2. Check dependence. Repeated records, clustered users, batches, and time series often violate independence.

3. Use the correct standard error. The standard deviation of individual observations is not the standard error of the mean; for i.i.d. data, the latter is s/√n.

4. Do not equate large n with unbiasedness. A very large biased sample can produce a very precise estimate of the wrong target.

5. Inspect skewness and tails. The CLT is asymptotic, not a guarantee of accurate finite-sample normality.

6. Use exact or specialized methods for small samples and rare events. Normal approximations can be poor when expected successes or failures are small.

7. Report uncertainty with estimates. A point estimate without a standard error or interval can create false precision.

8. Account for multiple comparisons and model selection. Repeated searching changes the distribution of the selected result.

9. Distinguish simulation error from model error. More Monte Carlo draws reduce the former but not the latter.

10. Use reproducible computation. Record random seeds, sampling procedures, software versions, and inclusion criteria.

10. Contemporary Relevance to AI and Machine Learning

Limit theorems appear throughout modern ML workflows. Empirical risk is an average loss,

\[

\hat R_n=\frac{1}{n}\sum_{i=1}^{n}L(f(X_i),Y_i),

\]

and, under suitable sampling assumptions, the law of large numbers links empirical risk to expected population risk. The CLT can approximate uncertainty in average loss or performance metrics, although dependence, adaptive training, and hyperparameter selection complicate this interpretation.

Bootstrap procedures repeatedly resample observed data to approximate sampling distributions when analytic formulas are difficult. The bootstrap is powerful but relies on the sample representing the population and may fail for dependent data, extreme statistics, or non-smooth procedures unless modified methods are used.

For large-scale and distributed systems, asymptotic reasoning supports approximate confidence intervals, sequential monitoring, online experimentation, and simulation-based decision making. Nevertheless, statistical guarantees must match the data-collection process. A mathematically elegant limit theorem cannot repair label leakage, selection bias, distribution shift, or unfair sampling.

Chapter Summary

The laws of large numbers explain why sample averages stabilize around population means. The weak law gives convergence in probability, while the strong law gives almost sure convergence under suitable conditions. The central limit theorem explains why standardized sums and means are often approximately normal, with uncertainty decreasing at the rate 1/√n. This result supports inference for means, proportions, and Monte Carlo estimates.

Sampling distributions describe the behavior of statistics across repeated samples. For normal populations, the sample mean is exactly normal, the scaled sample variance follows a chi-square distribution, and the standardized mean with an estimated standard deviation follows Student's t distribution. Order statistics describe ranked observations and support analysis of extremes and quantiles.

These results depend on assumptions such as independence, representative sampling, finite moments, and appropriate sample size. Dependence, heavy tails, selection, data drift, and model search can invalidate naïve uncertainty estimates. In AI and ML, limit theorems are valuable foundations for evaluating models and simulations, but statistical uncertainty must be considered alongside data and governance uncertainty.

Key Terms

Convergence in probability: A sequence becomes increasingly likely to lie arbitrarily close to a limiting value.

Almost sure convergence: A sequence converges to a limit with probability one.

Convergence in distribution: Convergence of the distributions of random variables to a limiting distribution.

Weak law of large numbers: The sample mean converges in probability to the population mean.

Strong law of large numbers: The sample mean converges almost surely to the population mean.

Central limit theorem: A theorem stating that standardized sums approach a standard normal distribution under suitable conditions.

Sampling distribution: The probability distribution of a statistic over repeated samples.

Standard error: The standard deviation of a statistic's sampling distribution.

Monte Carlo estimator: An estimate obtained by averaging results from random simulation trials.

Order statistic: An observation's rank-ordered value in a sample.

Quantile: A population value below which a specified proportion of observations falls.

Chi-square distribution: The distribution of a sum of squared independent standard normal variables.

Student t distribution: A symmetric distribution with heavier tails than the standard normal, used when estimating a normal mean with unknown variance.

Degrees of freedom: The number of independent pieces of information available for estimating variability.

Consistency: The property that an estimator converges to its target as sample size increases.

Independence: A condition in which the occurrence of one random outcome does not alter the probability of another.

Heavy-tailed distribution: A distribution with relatively high probability of extreme observations.

Effective sample size: The amount of independent information represented by potentially dependent observations.

Sampling bias: Systematic deviation caused by a sample not representing its target population.

Distribution shift: A change between the data-generating distribution used for development and that encountered in deployment.

Review Questions

Short Answer Questions

1. What is the difference between convergence in probability and convergence almost surely?

2. State the classical central limit theorem for i.i.d. observations.

3. Why is the standard error of an i.i.d. sample mean equal to σ/√n?

4. What is a sampling distribution?

5. Why does the t distribution have heavier tails than the standard normal distribution?

Long Answer Questions

1. Explain the weak and strong laws of large numbers, including their assumptions and practical interpretation.

2. Derive the approximate sampling distribution of a sample proportion and discuss when the normal approximation is appropriate.

3. Explain the roles of the normal, chi-square, and Student t distributions in inference for a normal population.

4. Describe order statistics and derive the distribution function of the sample maximum for i.i.d. observations.

5. Discuss how dependence, heavy tails, and biased sampling can affect the use of limit theorems in machine learning evaluation.

Analytical / Application Questions

1. A simulation estimates an expected loss using 25,000 independent draws. Explain how you would estimate its Monte Carlo standard error and determine how many additional simulations are needed to halve the error.

2. A classifier achieves 475 correct predictions out of 500 validation cases. Construct an approximate 95% confidence interval for accuracy and identify at least three reasons why the interval might be misleading.

3. Nine independent Uniform(0,1) measurements are collected. Find the expected value of the maximum and explain how this result changes as the sample size increases.

Multiple Choice Questions with Answers

Question 1

Under the classical weak law of large numbers, the sample mean converges to the population mean in which sense?

A. Almost surely only

B. In probability

C. In distribution to a chi-square variable

D. In mean square only

Correct Answer: B. In probability

Question 2

If X_i are i.i.d. with mean μ and variance σ², what is Var(X̄_n)?

A. σ²n

B. σ²

C. σ²/n

D. σ/√n

Correct Answer: C. σ²/n

Question 3

The classical central limit theorem states that which quantity converges in distribution to N(0,1)?

A. (X̄−μ)/σ

B. √n(X̄−μ)/σ

C. n(X̄−μ)/σ²

D. σ(X̄−μ)/√n

Correct Answer: B. √n(X̄−μ)/σ

Question 4

For a Bernoulli sample with estimated proportion p̂=0.04 and n=10,000, the approximate standard error is closest to:

A. 0.000196

B. 0.00196

C. 0.0196

D. 0.196

Correct Answer: B. 0.00196

Question 5

If X_1,...,X_n are i.i.d. Uniform(0,1), what is E[X_(n)]?

A. 1/n

B. 1/2

C. n/(n+1)

D. (n+1)/n

Correct Answer: C. n/(n+1)

Question 6

For a normal population with unknown variance, the statistic (X̄−μ)/(S/√n) follows which distribution?

A. Standard normal

B. Chi-square with n degrees of freedom

C. Student t with n−1 degrees of freedom

D. F distribution with n degrees of freedom

Correct Answer: C. Student t with n−1 degrees of freedom

Question 7

If Monte Carlo estimates are based on independent draws, reducing the standard error by a factor of two generally requires:

A. Half as many simulations

B. Twice as many simulations

C. Three times as many simulations

D. Four times as many simulations

Correct Answer: D. Four times as many simulations

Question 8

Which condition can increase the variance of a sample mean relative to the i.i.d. formula?

A. Positive dependence among observations

B. Using a larger sample size

C. Centering observations at their mean

D. Using a random-number seed

Correct Answer: A. Positive dependence among observations

Question 9

For a normal sample, (n−1)S²/σ² has which distribution?

A. χ² with n−1 degrees of freedom

B. t with n−1 degrees of freedom

C. Normal with mean zero

D. Beta with parameters n and n−1

Correct Answer: A. χ² with n−1 degrees of freedom

Question 10

Which statement about a 95% frequentist confidence interval is correct?

A. There is a 95% probability that the fixed parameter lies in this completed interval.

B. 95% of the observed data lie inside the interval.

C. The procedure captures the parameter in about 95% of repeated samples under its assumptions.

D. The interval is valid even when the sample is systematically biased.

Correct Answer: C. The procedure captures the parameter in about 95% of repeated samples under its assumptions.