5. Sampling Theory and Asymptotic Foundations

Introduction and Context

Data science rarely has access to an entire population in a directly observable form. A population may consist of all customers of a bank, all transactions processed by a platform, all manufactured items produced during a year, or all possible future users of a digital service. Although a census of the population is conceptually attractive, it is often too expensive, slow, technically difficult, or impossible. Sampling provides a principled way to learn about a population from a manageable subset of observations.

Sampling theory supplies the bridge between data and inference. It explains how a sample should be selected, how sample statistics vary from one sample to another, and under what conditions those statistics provide reliable information about population parameters. The subject is therefore central to estimation, hypothesis testing, survey research, experimentation, quality control, and machine learning evaluation.

Two ideas are especially important. First, a sample statistic is itself a random quantity because it depends on the observations selected. Its probability distribution is called a sampling distribution. Second, as sample size increases, suitably defined estimators often become more stable and approximately normal. The law of large numbers and the central limit theorem formalize these ideas and provide the asymptotic foundations of scalable analytics.

The word “asymptotic” refers to the behavior of a procedure as the sample size becomes very large. Asymptotic reasoning does not claim that real datasets are infinite. Instead, it provides approximations that become increasingly accurate under stated assumptions. Understanding those assumptions is essential: a large dataset does not automatically eliminate bias, dependence, measurement error, or poor sampling design.

Learning Objectives

By the end of this chapter, students should be able to:

1. Distinguish populations, samples, parameters, statistics, and sampling frames.

2. Compare probability and non-probability sampling methods.

3. Explain sampling distributions and derive the standard error of common statistics.

4. Analyze the bias–variance trade-off in sampling and estimation.

5. State and interpret the law of large numbers and the central limit theorem.

6. Recognize the effects of dependence, clustering, nonresponse, and selection bias.

7. Apply sampling concepts to surveys, experiments, quality control, and large-scale analytics.

8. Evaluate the ethical, technical, and organizational consequences of sampling decisions.

The Population–Sample Relationship

A population is the complete set of units about which a researcher wishes to make a statement. The units may be people, firms, events, images, web sessions, or physical products. A target population is the conceptual population of interest, whereas the accessible population is the portion that can actually be reached. The difference matters. If a study intends to understand all citizens but samples only registered online users, its accessible population is narrower than its target population.

A sampling frame is the operational list or mechanism from which observations are selected. Examples include a customer database, a register of households, a list of manufactured components, or a stream of logged transactions. Coverage error arises when the frame excludes, duplicates, or incorrectly represents members of the target population.

A population characteristic is called a parameter. Common parameters include the population mean μ, variance σ², proportion p, and regression coefficient β. A sample statistic is computed from observed data, such as the sample mean X̄, sample variance S², sample proportion p̂, or estimated regression coefficient b. Parameters are usually fixed but unknown; statistics are observable but vary across samples.

This distinction is foundational. If repeated samples of equal size were drawn from the same population, their sample means would generally differ. Therefore, the uncertainty of an estimate cannot be assessed merely by inspecting one observed value. It must be understood through the distribution that the statistic would have across repeated samples.

Probability Sampling: Known Chances of Selection

In probability sampling, every population unit has a known, non-zero probability of selection, although the probabilities need not be equal. This requirement makes it possible to use probability theory to quantify sampling uncertainty and, with suitable weighting, to produce population-representative estimates.

Simple Random Sampling

In simple random sampling, every sample of a specified size has the same probability of being selected. If n units are drawn without replacement from a population of N units, each unit has inclusion probability n/N. Simple random sampling is conceptually clean and often serves as the benchmark for other designs.

For a sample mean under simple random sampling without replacement,

SE(X̄) = (σ/√n)√[(N − n)/(N − 1)],

where the second factor is the finite population correction. When the sampling fraction n/N is small, the correction is close to one and is often ignored. Increasing n reduces standard error at a rate proportional to 1/√n, not 1/n. Thus, obtaining four times as much precision generally requires approximately sixteen times the sample size only if precision is measured by halving the standard error; more exactly, quadrupling n halves the standard error.

Systematic Sampling

Systematic sampling selects units at a regular interval from an ordered frame. If the sampling interval is k, a random starting point is selected among the first k units, after which every kth unit is included. The method is operationally convenient, particularly for production lines or ordered records. However, periodic patterns in the frame can create serious bias if the interval aligns with an underlying cycle.

Stratified Sampling

In stratified sampling, the population is divided into relatively homogeneous subgroups called strata, and a sample is drawn from each stratum. Strata may be defined by region, age group, customer type, or risk category. Stratification can improve precision when units within strata are similar and strata differ from one another. It also guarantees representation of important subgroups.

Samples may be allocated proportionally or disproportionately. Disproportionate allocation may deliberately oversample a small but important group. Population estimates must then use appropriate weights; otherwise, the sample composition can distort the overall result.

Cluster and Multistage Sampling

Cluster sampling selects groups, or clusters, such as schools, hospitals, villages, or retail branches, and then surveys units within selected clusters. It is cost-effective when travel, administration, or data extraction is organized around groups. Its disadvantage is that observations within a cluster tend to be correlated. Consequently, the effective sample size may be much smaller than the nominal number of observations.

Multistage sampling combines stages, for example selecting regions, then branches, then customers. It is common in large surveys and distributed operational systems. Variance estimation must reflect all stages of selection rather than treating the final observations as a simple random sample.

Non-Probability Sampling and Its Consequences

Non-probability sampling does not provide a known selection probability for every population unit. Common forms include convenience sampling, purposive sampling, quota sampling, volunteer sampling, and respondent-driven or snowball recruitment.

Such methods may be appropriate when a sampling frame is unavailable, when research is exploratory, or when access to a specialized population is difficult. They are also common in product research and online experimentation. Their central limitation is that conventional sampling error cannot generally be calculated from the sample alone. A large convenience sample can be less representative than a small probability sample if the selection mechanism is related to the outcome.

For example, an online poll about public transport may attract people with unusually strong opinions or unusually frequent internet access. Increasing the number of respondents reduces random variation within the recruited group, but it does not necessarily correct systematic differences between respondents and nonrespondents. This distinction is often summarized as the difference between precision and accuracy.

Sampling Distributions and Standard Error

A sampling distribution is the probability distribution of a statistic over repeated samples drawn under a specified design. It is not the distribution of individual observations. The distribution of incomes in a population, for example, differs from the sampling distribution of the sample mean of income.

If observations X₁, …, Xn are independent and identically distributed with mean μ and variance σ², then

E(X̄) = μ,

Var(X̄) = σ²/n,

and

SE(X̄) = σ/√n.

Because σ is usually unknown, it is estimated by the sample standard deviation S, giving the estimated standard error S/√n. For a sample proportion p̂ based on independent Bernoulli observations,

E(p̂) = p,

Var(p̂) = p(1 − p)/n,

and the estimated standard error is √[p̂(1 − p̂)/n].

The standard error measures the expected variation of an estimator across repeated samples. It is not the standard deviation of the raw observations and not a guarantee that the estimate is close to the parameter. A biased procedure can have a small standard error while consistently missing the target.

Under appropriate regularity conditions, an approximate 95 percent interval for a parameter may be formed as estimate ± 1.96 × standard error. This familiar expression relies on an approximately normal sampling distribution and on a valid variance estimate. For complex survey designs, clustering and weighting require design-based variance estimators rather than the simple formula for independent observations.

Bias, Variance, and the Quality of an Estimator

Bias is the difference between the expected value of an estimator and the parameter it is intended to estimate:

Bias(θ̂) = E(θ̂) − θ.

Variance describes the estimator’s variability across repeated samples. Mean squared error combines both:

MSE(θ̂) = Var(θ̂) + [Bias(θ̂)]².

The bias–variance trade-off occurs because procedures designed to reduce variability may introduce systematic distortion, while highly flexible or poorly controlled procedures may reduce bias in one setting but increase variance. In sampling, a carefully designed stratified sample may lower variance, whereas undercoverage may introduce bias. In prediction, a simple model may be stable but systematically inadequate, while a complex model may fit the sample closely but generalize poorly.

It is useful to distinguish sampling bias from non-sampling errors. Sampling bias results from the selection mechanism. Nonresponse, inaccurate measurement, data processing errors, and changes in the population can also affect estimates. A census eliminates sampling variability but does not eliminate these other errors.

Design Effect and Effective Sample Size

When observations are correlated, the usual independent-observation standard error is too small. A common summary is the design effect:

DEFF = Var(actual estimator) / Var(estimator under simple random sampling).

For equal-sized clusters, a simplified expression is

DEFF ≈ 1 + (m − 1)ρ,

where m is the average cluster size and ρ is the intracluster correlation. The corresponding effective sample size is approximately n/DEFF. If DEFF equals four, a nominal sample of 1,000 observations may contain information comparable to only about 250 independent observations.

The Law of Large Numbers

The law of large numbers explains why averages stabilize as the amount of information increases. In its weak form, if X₁, X₂, … are independent and identically distributed with finite mean μ, then for every ε > 0,

P(|X̄n − μ| > ε) → 0 as n → ∞.

In words, the sample mean converges in probability to the population mean. The law justifies using averages to estimate long-run expectations, provided the data-generating conditions are suitable.

The law does not say that every large sample is representative of every population. It applies to observations generated according to the relevant probabilistic mechanism. If a sample is selected through a biased process, increasing its size may make the biased estimate more stable rather than more correct. Similarly, if observations are strongly dependent, heavy-tailed, or generated by a changing process, the usual version may not apply without modification.

In data engineering, the law of large numbers supports the monitoring of aggregate metrics such as conversion rates, average latency, defect rates, and expected revenue. Yet analysts must check whether the stream is stationary, whether repeated events from the same user dominate the data, and whether the metric is being measured consistently.

The Central Limit Theorem

The central limit theorem (CLT) provides a remarkably general approximation for the distribution of standardized sums or means. Under common conditions, if X₁, …, Xn are independent and identically distributed with finite mean μ and finite variance σ², then

√n(X̄n − μ)/σ → N(0, 1)

in distribution as n becomes large.

Equivalently, for sufficiently large n,

X̄n ≈ N(μ, σ²/n).

The CLT explains why normal approximations arise even when individual observations are not normally distributed. A population may be skewed or discrete, but averages of sufficiently many independent observations can have an approximately bell-shaped sampling distribution.

The required sample size is not universal. It depends on skewness, tail behavior, dependence, the statistic being studied, and the accuracy required. For a highly skewed distribution, a larger sample may be necessary. For a heavy-tailed distribution with infinite or poorly behaved variance, the classical CLT may be inaccurate or inapplicable. Resampling, robust methods, transformations, or specialized limit theorems may then be preferable.

The CLT is often misinterpreted. It does not state that the raw data become normal. It states that a suitably standardized aggregate statistic approaches a normal distribution. Nor does it imply that a large sample corrects selection bias. Normality of the sampling distribution concerns uncertainty; representativeness concerns validity of the target being estimated.

Resampling as an Empirical Alternative

When analytical sampling distributions are difficult to derive, resampling methods provide practical approximations. In the nonparametric bootstrap, observations are sampled with replacement from the observed dataset, and the statistic is recomputed for each resample. The empirical distribution of the repeated statistics approximates the sampling distribution.

The bootstrap can estimate standard errors, confidence intervals, and bias for means, medians, regression coefficients, and many machine-learning metrics. It is especially useful when the statistic has a complicated form. However, the ordinary bootstrap assumes that the observed data adequately represent the population and that the resampling units are appropriate. For time series, clustered data, or survey samples, block, cluster, or design-based bootstrap methods may be needed.

The bootstrap is not a remedy for a defective sampling frame. Resampling a biased sample reproduces its selection structure. It estimates uncertainty conditional on the observed data, not necessarily total error relative to the target population.

Applications in Data Science and Organizations

Sampling appears throughout the data lifecycle. During data collection, it reduces the cost of surveys, labeling, experiments, and quality inspection. During model development, a sample may be used to train or validate an algorithm when the full data repository is too large. During monitoring, sampled events can estimate service-level metrics while reducing storage and processing requirements.

In A/B testing, random assignment creates comparable treatment groups and permits causal interpretation under appropriate assumptions. The number of users required depends on baseline variability, minimum detectable effect, desired power, and significance level. Repeatedly checking results and stopping when a favorable value appears can inflate false-positive risk unless sequential methods are used.

In fraud detection, rare-event sampling may be necessary because fraud cases are uncommon. Deliberate oversampling can improve model training, but predicted probabilities and business-level estimates may require recalibration to reflect the real prevalence. In manufacturing, stratification by machine, shift, or raw-material batch may reveal localized defects hidden by an overall average.

For scalable analytics, sampling can reduce computational cost in exploratory queries, approximate dashboards, and distributed model training. A uniform sample may be acceptable for estimating a global mean but unsuitable for rare subgroups or tail quantiles. Sampling design should therefore follow the decision objective. If the organization must estimate a 99.9th-percentile response time, sampling only typical requests can be seriously misleading.

Illustrative Case Study: Estimating Customer Churn from a Large Platform

Background

A subscription platform has 20 million active accounts and wants to estimate the proportion of customers likely to cancel during the next quarter. A full customer survey is expensive, and the analytics team proposes an online questionnaire distributed through an in-app banner.

Situation and Problem

The banner is shown to a simple operational sample of 200,000 active accounts, but only 18,000 customers respond. The resulting estimated churn intention is 14 percent. Management asks whether this figure can be reported as the churn rate for all customers.

Analysis

The first issue is nonresponse. Respondents may differ from nonrespondents in engagement, satisfaction, language, device access, or likelihood of cancellation. The response rate alone does not determine bias, but it signals that the observed sample is not automatically representative.

The second issue is construct validity. Stated intention to cancel is not identical to actual cancellation. The organization must link responses to subsequent behavior while respecting privacy and consent requirements.

The third issue is subgroup structure. Churn may vary substantially by subscription plan, region, tenure, and customer-support history. A simple respondent average can overrepresent groups more likely to see or complete the questionnaire.

Approach and Solution

The team first defines the target population and constructs strata based on plan, region, tenure, and recent activity. It compares response rates across strata and calculates nonresponse-adjustment weights using known population totals. The weighted estimate is then calibrated against observed historical churn where comparable cohorts exist.

For uncertainty estimation, the team uses a survey-weighted variance procedure rather than treating the 18,000 respondents as a simple random sample. It also reports subgroup estimates and sensitivity analyses under alternative assumptions about nonrespondents. A separate probability sample of customers is contacted through multiple channels to assess whether the in-app survey has systematic coverage problems.

To predict actual churn, the analysts use the survey responses as features in a validated model, evaluate calibration on a later cohort, and distinguish predictive performance from population prevalence estimation. The model is monitored for differential error across customer groups.

Outcome

The organization does not publish the unadjusted 14 percent figure. Instead, it reports a weighted estimate with an uncertainty interval, describes the assumptions behind adjustment, and presents separate estimates for important segments. The predictive model is used to prioritize retention interventions, while causal testing is used to determine whether those interventions reduce cancellations.

Learning Points

A large respondent count cannot compensate for a selection mechanism related to the outcome. Weighting can reduce known composition differences but cannot guarantee removal of nonresponse bias. Sampling, measurement, prediction, and causal evaluation are distinct tasks. Sound decision-making requires identifying which quantity is being estimated and designing the data process around that quantity.

Limitations, Risks, and Common Mistakes

Several errors recur in professional sampling practice. Treating a convenient dataset as a random sample is perhaps the most consequential. Analysts also confuse the standard deviation of observations with the standard error of an estimate, ignore clustering, use inappropriate weights, and report confidence intervals without explaining the sampling design.

Another mistake is equating statistical significance with practical importance. A very large sample can make a trivial difference statistically significant. Conversely, a meaningful effect may be estimated imprecisely in a small or highly variable sample.

Analysts should be cautious with extremely large datasets. Big data can contain duplicate records, automated activity, feedback loops, and systematic missingness. It may represent platform behavior rather than the broader population. Data volume improves precision only for the process actually observed.

Heavy-tailed outcomes create additional difficulties. A few extreme observations can dominate means and produce unstable variance estimates. Median-based summaries, trimmed means, robust standard errors, transformations, or explicit tail modeling may be more appropriate.

Ethical, Governance, and Professional Considerations

Sampling decisions affect whose experiences become visible in organizational decisions. Excluding people without reliable internet access, underrepresenting minority groups, or oversampling easily reachable customers can produce unequal service quality and unfair resource allocation. Ethical practice therefore requires attention to coverage, inclusion, privacy, consent, and the consequences of error.

Data minimization should guide collection: analysts should sample and retain only information necessary for the stated purpose. Sensitive attributes may be needed to audit representativeness and fairness, but access must be governed carefully. Documentation should record the target population, frame, selection method, inclusion probabilities, weights, response rates, missing-data treatment, and uncertainty calculations.

In regulated settings, estimates used for credit, employment, healthcare, insurance, or public services may require reproducible procedures and evidence that affected groups were not systematically excluded. Transparency about uncertainty is a professional obligation, not merely a presentation preference.

Contemporary Developments and Best Practice

Modern data platforms increasingly support approximate query processing, reservoir sampling, distributed sketching, and streaming summaries. These techniques make it possible to estimate large-scale metrics with bounded memory or reduced computation. Their statistical guarantees depend on the sampling design and on whether the stream is changing over time.

Machine-learning systems also use active sampling, where observations are selected because they are informative for labeling or model improvement. Active sampling can be highly efficient, but the resulting data are not automatically representative. Importance weighting, exploration policies, and evaluation on an unbiased holdout set help separate learning efficiency from population estimation.

A sound sampling workflow should:

• define the target population and estimand before collecting data;
• inspect the sampling frame for coverage and duplication;
• choose a probability design when population inference is required;
• document inclusion probabilities and apply justified weights;
• account for clustering, stratification, finite populations, and nonresponse;
• distinguish bias, variance, measurement error, and model error;
• use simulation or resampling to assess finite-sample behavior;
• report uncertainty, sensitivity analyses, and subgroup performance;
• monitor whether the data-generating process changes over time.

Chapter Summary

Sampling theory explains how information from a subset can support conclusions about a population. Probability sampling provides known selection probabilities and a foundation for design-based uncertainty, while non-probability sampling may be useful operationally but is vulnerable to unknown selection bias. Sampling distributions describe the repeated-sample behavior of statistics, and standard errors quantify their variability.

Bias and variance are distinct components of estimation quality, jointly summarized by mean squared error. The law of large numbers explains the stabilization of averages under suitable conditions, whereas the central limit theorem supports normal approximations for many standardized aggregate statistics. Neither theorem removes selection bias or dependence. Clustered designs, nonresponse, heavy tails, and changing data-generating processes require additional care.

In data science, sampling affects surveys, experiments, model training, monitoring, fraud analytics, quality control, and scalable computation. Responsible practice requires a clearly defined estimand, a defensible sampling process, appropriate variance estimation, transparent documentation, and attention to fairness and privacy.

Key Terms

Population: The complete set of units about which inference is intended.

Target population: The conceptual population to which conclusions are meant to apply.

Sampling frame: The operational list or mechanism from which a sample is selected.

Sample: A subset of units observed from a population or accessible frame.

Parameter: A fixed numerical characteristic of a population.

Statistic: A numerical quantity calculated from sample data.

Probability sampling: Sampling in which each unit has a known, non-zero selection probability.

Non-probability sampling: Sampling in which selection probabilities are unknown or not defined by design.

Sampling distribution: The probability distribution of a statistic over repeated samples.

Standard error: The standard deviation of an estimator’s sampling distribution.

Sampling bias: Systematic deviation caused by the selection mechanism.

Nonresponse bias: Bias arising when participation is related to the outcome or relevant characteristics.

Finite population correction: An adjustment reducing variance when a substantial fraction of a finite population is sampled.

Design effect: The ratio of actual estimator variance to variance under a reference simple-random design.

Effective sample size: The size of an independent random sample with approximately equivalent information.

Law of large numbers: A result stating that sample averages converge to the population mean under suitable conditions.

Central limit theorem: A result describing the approximate normality of standardized sums or means under suitable conditions.

Bootstrap: A resampling method used to approximate a statistic’s sampling distribution.

Estimand: The precise population quantity a study seeks to estimate.

Mean squared error: The sum of an estimator’s variance and squared bias.

Review Questions

Short Answer Questions

1. Distinguish a target population from a sampling frame.

2. What is a sampling distribution, and why is it different from the distribution of raw observations?

3. Define standard error and explain how it differs from standard deviation.

4. State the law of large numbers in practical terms.

5. Why can a large convenience sample remain biased?

Long Answer Questions

1. Compare simple random, stratified, cluster, and systematic sampling, including an appropriate use case for each.

2. Explain the bias–variance trade-off and relate it to mean squared error.

3. Present the central limit theorem, discuss its assumptions, and explain two common misinterpretations.

4. Explain how clustering affects standard errors and effective sample size.

5. Discuss how sampling theory supports scalable analytics while preserving statistical validity.

Analytical / Application Questions

1. A company samples 10,000 transactions from a stream in which 80 percent come from five large customers. Explain why the sample may not support inference about all customers and propose a better design.

2. A survey estimate has a small standard error but is based on an online volunteer sample. Analyze what can and cannot be concluded from the reported precision.

3. An experiment reports a statistically significant improvement after repeated interim analyses. Identify the inferential risk and recommend a more defensible analysis strategy.

22. Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes a parameter from a statistic?

A. A parameter is calculated from a sample, whereas a statistic describes a population.

B. A parameter describes a population, whereas a statistic is calculated from sample data.

C. A parameter always has greater variance than a statistic.

D. A parameter is necessarily unknown, whereas a statistic is necessarily known before sampling.

Correct Answer: B. A parameter describes a population, whereas a statistic is calculated from sample data.

Question 2

A researcher wants every major customer segment represented and expects customers within each segment to be relatively similar. Which design is most appropriate?

A. Stratified sampling

B. Convenience sampling

C. Simple systematic sampling without checking the frame

D. Volunteer sampling

Correct Answer: A. Stratified sampling

Question 3

What does the standard error of a sample mean primarily measure?

A. The spread of individual observations around the sample mean

B. The expected variation of the sample mean across repeated samples

C. The bias caused by an incomplete sampling frame

D. The proportion of missing observations

Correct Answer: B. The expected variation of the sample mean across repeated samples

Question 4

If independent observations have finite variance and the sample size is quadrupled, the standard error of the sample mean is approximately:

A. Doubled

B. Unchanged

C. Halved

D. Reduced to one-fourth

Correct Answer: C. Halved

Question 5

Which conclusion follows from the law of large numbers under its usual assumptions?

A. Every large sample is representative of every target population.

B. The sample average converges in probability to the population mean.

C. Raw observations become normally distributed as sample size increases.

D. Sampling bias disappears when the sample is sufficiently large.

Correct Answer: B. The sample average converges in probability to the population mean.

Question 6

The central limit theorem primarily concerns the behavior of:

A. Individual observations in the population

B. The sampling distribution of suitably standardized sums or means

C. The sampling frame before data collection

D. The response rate of a survey

Correct Answer: B. The sampling distribution of suitably standardized sums or means

Question 7

A survey samples many people from the same school, producing correlated responses. If the analyst treats all responses as independent, the reported standard error will most likely be:

A. Too large

B. Too small

C. Exactly unbiased in every case

D. Equal to the population standard deviation

Correct Answer: B. Too small

Question 8

Which statement about a large non-probability sample is most accurate?

A. Its large size guarantees unbiased population estimates.

B. Its standard error automatically captures selection bias.

C. It may have high precision for the recruited group while remaining biased for the target population.

D. It is equivalent to a stratified random sample.

Correct Answer: C. It may have high precision for the recruited group while remaining biased for the target population.

Question 9

In the illustrative churn case, why was weighting alone not treated as a complete solution?

A. Weights always increase sampling bias.

B. Weighting cannot address any differences between subgroups.

C. Weighting can adjust known composition differences but cannot guarantee removal of nonresponse bias.

D. Weights prevent the calculation of uncertainty intervals.

Correct Answer: C. Weighting can adjust known composition differences but cannot guarantee removal of nonresponse bias.

Question 10

If a clustered sampling design has a design effect of four and a nominal sample size of 1,000, the approximate effective sample size is:

A. 40

B. 250

C. 996

D. 4,000

Correct Answer: B. 250