Sampling and Statistical Power

Introduction

Business analytics rarely permits researchers to observe every member of a target market, employee population, customer base, transaction, or household. Sampling provides a defensible way to learn about a population from a manageable subset. However, a large sample is not automatically a good sample. A poorly selected sample may produce precise but systematically wrong results, while a smaller probability sample may produce more credible population estimates.

This chapter explains how to define populations and sampling frames, select probability and non-probability samples, determine sample size, understand statistical power, and evaluate sampling error, selection bias, non-response, and representativeness. These issues affect surveys, experiments, predictive-model development, customer research, human-resources analytics, and market analysis.

Learning objectives

By the end of this chapter, you should be able to:

1. Distinguish a target population, study population, sampling frame, sample, and observation.
2. Select an appropriate probability or non-probability sampling design.
3. Calculate basic sample-size requirements for proportions and means.
4. Explain statistical power, significance level, effect size, and minimum detectable effect.
5. Diagnose sampling error, coverage error, selection bias, non-response, and weighting problems.
6. Design and evaluate a sampling plan ethically and professionally.

1. Populations, frames, and samples

A target population is the complete group to which a researcher intends to generalize. For example, a study may target all active retail customers in a country during the last twelve months. The study population is the portion of the target population that can actually be accessed under the study’s operational and legal constraints.

A sampling frame is the operational list, database, register, or procedure used to identify members of the study population. Examples include a customer relationship management database, an employee payroll list, a business directory, or a list of registered voters. A frame may be incomplete, duplicated, outdated, or contain ineligible units. These defects create coverage error.

A sampling unit is the entity selected at a particular stage: an individual, household, store, firm, account, or geographic area. The sample is the set of units selected for analysis. An element is the ultimate unit about which information is collected, such as a customer or employee. In cluster designs, the selected unit may be a store while the elements are customers within stores.

The distinction between a population parameter and a sample statistic is fundamental. A parameter, such as the population mean μ or population proportion P, is generally unknown. A statistic, such as the sample mean x̄ or sample proportion p̂, is calculated from observed data and used to estimate the parameter.

A useful sampling plan specifies:

2. Probability sampling

In probability sampling, every eligible unit has a known, non-zero probability of selection. This permits design-based estimation and, when the design is implemented correctly, supports defensible estimates of sampling uncertainty.

Simple random sampling

Under simple random sampling without replacement, every sample of a specified size has an equal chance of selection. A random-number generator selects n units from a frame of N units. This design is conceptually simple, but it requires a reasonably complete frame and can be costly when units are geographically dispersed.

For a population proportion, an approximate 95% confidence interval is:

Here p̂ is the sample proportion, n is the sample size, α is the probability outside the confidence interval, and z_{α/2} is the standard-normal critical value. The formula assumes a probability sample, independent or appropriately sampled observations, and a sufficiently large sample for the normal approximation. For a finite population, a finite population correction may be applied when sampling a substantial fraction of the population:

In planning, P is often replaced by 0.5 because this produces the largest variance for a proportion.

Systematic sampling

Systematic sampling selects every kth unit after a random starting point, where approximately k = N/n. It is efficient and easy to administer. It can, however, be biased if the frame has periodic ordering that corresponds to the interval—for example, every tenth record belongs to the same branch.

Stratified sampling

Stratified sampling divides the population into relatively homogeneous subgroups called strata, then samples within each stratum. Strata may be region, customer type, firm size, or age group. Proportionate allocation selects from each stratum in proportion to its population size. Disproportionate allocation deliberately oversamples small or strategically important groups; weights are then normally required for population estimates.

Stratification can improve precision when strata differ from one another but are internally similar. It also ensures representation of important subgroups. The researcher must define mutually exclusive and collectively exhaustive strata and know, or estimate, their sizes.

Cluster and multistage sampling

Cluster sampling selects naturally occurring groups, such as branches, schools, or postal areas, and then studies all or some units within selected clusters. It reduces travel and administrative costs but often increases sampling variance because units in the same cluster tend to resemble one another.

A multistage design combines methods: regions may be selected first, stores second, and customers third. Analysis must account for unequal selection probabilities and clustering. Treating a complex sample as a simple random sample can produce standard errors that are too small.

Sampling weights

A basic design weight is the inverse of a unit’s selection probability:

where π_i is the probability that unit i is included. Weights may subsequently be adjusted for non-response and calibrated to known population totals. Weighted estimates can reduce bias but may increase variance, especially when weights vary greatly. Weighting does not repair a severely deficient frame or unmeasured selection mechanism automatically.

3. Non-probability sampling

Non-probability sampling does not give every eligible unit a known, non-zero selection probability. It is common in exploratory research, rapid product testing, qualitative research, and situations where no adequate frame exists. Its limitation is that conventional margins of sampling error do not generally describe the full uncertainty, because selection probabilities are unknown.

Convenience sampling recruits readily available units, such as website visitors or students. It is inexpensive but vulnerable to strong selection bias. Purposive sampling deliberately selects cases with relevant characteristics, such as experienced data managers. Quota sampling fills preset subgroup counts without random selection within each quota. Snowball sampling asks participants to refer others and can help reach hidden populations, but participants may belong to tightly connected networks. Volunteer panels may be efficient but often contain people with unusually high interest, time, or digital access.

Non-probability samples are not always unacceptable. They can be appropriate when the objective is theory development, instrument testing, or understanding a range of experiences rather than estimating a population percentage. Researchers must state the sampling limitation clearly and avoid presenting a convenience sample as representative without evidence.

4. Sample size for estimation

Sample size depends on the objective, design, precision, confidence level, population variability, and available resources. It cannot be chosen responsibly from population size alone.

Estimating a proportion

For a large population and a desired margin of error E, the planning formula is:

Here n₀ is the initial sample size, z_{α/2} is the critical value, P is the anticipated population proportion, and E is the maximum tolerable absolute error. All quantities are on a proportion scale: 5 percentage points means E = 0.05. If P is unknown, use P = 0.5 for the conservative maximum-variance choice.

If the population is finite and n₀ is not negligible relative to N, use:

Worked example. A retailer wants a 95% estimate of the proportion of 20,000 customers who would recommend its service, with a margin of error of ±5 percentage points. Using z = 1.96 and P = 0.5:

n₀ = (1.96² × 0.5 × 0.5)/(0.05²) = 384.16

Thus, round upward to 385. Applying the finite-population correction gives approximately 378. If the expected response rate is 60%, the number invited should be approximately 378/0.60 = 630. This calculation addresses precision under the stated design; it does not eliminate non-response bias.

Estimating a mean

For a mean, the approximate sample-size formula is:

where σ is the anticipated population standard deviation in the measurement units and E is the desired margin of error in those same units. The value of σ may come from prior studies or a pilot sample. The formula assumes a probability sample, an approximately normal sampling distribution, and an appropriate measure of variability.

Design effect

Cluster sampling and some other complex designs require inflation. The design effect for equal-sized clusters is often approximated by:

where m is the average number of sampled elements per cluster and ρ is the intraclass correlation coefficient. The effective sample size is approximately n_eff = n/deff. Even a small ρ can matter when m is large. If deff = 2, a nominal sample of 800 provides information roughly comparable to 400 independent observations, subject to the assumptions of the approximation.

5. Statistical power and hypothesis testing

Statistical power is the probability of rejecting a null hypothesis when a specified alternative is true. It is commonly written as 1 − β, where β is the Type II error probability. The significance level α is the probability of rejecting a true null hypothesis under the test’s assumptions; it is not the probability that the null hypothesis is true.

Power depends on sample size, effect size, outcome variability, significance level, and whether the test is one- or two-sided. Larger samples generally increase power; larger true effects are easier to detect; lower variability increases power; and a more stringent α decreases power unless sample size increases.

A power analysis should be conducted before data collection. It may be:

• A priori: determine the sample needed for a chosen effect, α, and power.
• Sensitivity: determine the smallest effect detectable with a fixed sample.
• Precision-based: determine the sample needed for a confidence-interval width.
• Post hoc or retrospective: evaluate detectable effects after data collection; this is usually less informative than reporting the confidence interval and observed estimate.

For a two-group comparison of means with equal group sizes, a simplified planning expression is:

Here n_per_group is the number required in each group, σ is the common standard deviation, ∆ is the practically important difference, z_{α/2} is the two-sided significance critical value, and z_β corresponds to the desired power. This approximation assumes independent observations, equal variances, equal allocation, and an approximately normal outcome.

Worked planning example. Suppose an A/B test seeks to detect a mean improvement of ∆ = 4 minutes, the standard deviation is estimated as σ = 12 minutes, α = 0.05 two-sided, and desired power is 0.80. Using z_{α/2} = 1.96 and z_β = 0.84:

n_per_group = 2(1.96 + 0.84)²(12²)/(4²) = 141.12

Round upward to 142 per group, or 284 completed observations overall. If only 80% of assigned users are expected to provide usable outcomes, approximately 355 users should be assigned. This is a planning approximation; software should be used for complex outcomes, unequal allocation, clustering, covariates, or sequential designs.

Statistical significance and practical importance are different. A tiny effect may be statistically significant in a very large sample but irrelevant to a business decision. Conversely, a meaningful effect may fail to reach significance in a small study. Analysts should report effect estimates, confidence intervals, assumptions, and practical consequences—not p-values alone.

6. Sources of sampling error and bias

Sampling error is random difference between a sample estimate and the population parameter arising because only part of the population is observed. It generally decreases as sample size increases, approximately at a rate proportional to 1/√n under simple random sampling. Sampling error is quantified by a standard error or confidence interval.

Coverage error occurs when the frame omits eligible units, includes ineligible units, or duplicates units. An online-only frame may omit customers with limited internet access. Selection bias occurs when inclusion probabilities are related to the outcome or important predictors and are not adequately corrected. Non-response bias occurs when those who do not respond differ systematically from respondents. Increasing the number of invitations does not necessarily solve either form of bias.

Measurement error arises when the recorded value differs from the true value because of ambiguous questions, recall problems, interviewer effects, instrument defects, or data-processing mistakes. It is not sampling error, although it can be equally damaging.

Representativeness is not a single visual property. A sample can match population demographics yet differ on unmeasured attitudes, usage intensity, or motivation. Researchers should compare known frame variables, report response rates and recruitment flow, examine subgroup estimates, and conduct sensitivity analyses.

A useful non-response rate is:

Exact response-rate definitions vary by professional standard, especially when eligibility is unknown. The calculation must therefore state its denominator and treatment of partial interviews, refusals, and unreachable units.

7. A practical sampling workflow

1. Define the estimand. Specify whether the goal is a population proportion, mean, treatment effect, prediction performance, or qualitative understanding.
2. Define eligibility and time. State who qualifies, when, and under what exclusions.
3. Assess the frame. Identify omissions, duplicates, stale records, linkage errors, and privacy constraints.
4. Choose the design. Use probability sampling for population estimation when feasible; justify non-probability methods when appropriate.
5. Determine sample size. Set precision or minimum detectable effect, α, desired power, expected variability, design effect, and anticipated non-response.
6. Randomize where required. Use auditable random selection, allocation, and replacement rules. Do not replace refusals with convenient alternatives.
7. Monitor fieldwork. Track recruitment by stratum, channel, geography, and time. Monitor unusual response patterns without manipulating inclusion.
8. Document deviations. Record frame changes, ineligible cases, missing data, protocol deviations, and weighting decisions.
9. Analyze according to design. Use survey weights, clustering corrections, stratification variables, and appropriate variance estimators.
10. Report limitations. Explain who was excluded, who did not respond, what the results estimate, and how far they can reasonably be generalized.

8. Illustrative case: a customer-retention survey

Background

A fictional subscription company wants to estimate why customers cancel and test whether a redesigned support process reduces cancellation. Its customer database contains 120,000 accounts, but 8% are duplicated, 5% lack a valid contact channel, and customers vary substantially by plan and region.

Problem

A manager proposes emailing the first 1,000 customers in the database. This would be convenient but could overrepresent recently added accounts and underrepresent customers with outdated email addresses. The company also needs a reliable comparison of the redesigned and existing support processes.

Analysis and approach

The research team first removes duplicates and defines an eligible account as one active for at least 30 days during the study period. It stratifies the frame by plan type and region, then draws a random sample within each stratum. For the survey, it targets 385 completed responses for a 5-point margin of error under a simple-random approximation, inflating invitations for the expected response rate and design effect. It oversamples the smallest plan category to obtain a useful subgroup estimate, then applies design and non-response weights for overall estimates.

For the intervention, eligible support cases are randomly assigned to redesigned or existing service, with allocation blocked by region and plan. Before the study, the team specifies a practically important reduction in cancellation, selects α = 0.05 and 80% power, and calculates the required number of cases using a two-proportion power procedure. It monitors assignment integrity and reports both the estimated difference and its confidence interval.

Outcome

The final report distinguishes the weighted survey estimate from the randomized intervention estimate. It reports frame exclusions, response rates by stratum, weighting variability, missing outcomes, and deviations from protocol. The redesigned service shows a reduction in cancellation, but the confidence interval includes effects ranging from modest benefit to near-zero benefit. Management therefore continues a larger pilot rather than claiming certainty.

Learning points

A defensible sample begins with a defensible estimand and frame. Stratification can protect subgroup representation, while random assignment supports causal inference in the intervention. Weighting and larger samples cannot fully rescue systematic coverage problems. Transparent uncertainty is more valuable than an apparently precise but biased answer.

9. Ethics, governance, and technology

Sampling must respect informed consent, privacy, fairness, and data-protection requirements. Researchers should collect only necessary contact information, restrict access, protect identifiers, and explain how data will be used. Sensitive groups should not be excluded merely because they are difficult to recruit, nor should they be burdened or exposed to disproportionate risk.

Automated sampling from customer or social-media data can reproduce historical exclusion. Machine-learning systems may identify likely responders, but response propensity models should not silently determine who is eligible. AI-assisted recruitment, synthetic data, and panel providers require documentation of provenance, consent, inclusion rules, and validation. Synthetic records do not substitute for probability sampling of the real population when population inference is the objective.

Contemporary practice increasingly combines administrative data, digital panels, adaptive allocation, and real-time response monitoring. These tools improve efficiency but make governance more important. A transparent audit trail should show the frame version, selection algorithm, random seed or equivalent reproducibility information, exclusions, response dispositions, weights, and analysis code.

Challenges and limitations

No sampling design simultaneously maximizes precision, speed, low cost, and universal coverage. Probability sampling may be expensive or impossible when no frame exists. Non-probability panels may be practical but require cautious claims. Power calculations depend on assumptions that may be wrong, especially the anticipated variance, response rate, intraclass correlation, or baseline outcome rate. Multiple outcomes, interim analyses, attrition, and subgroup comparisons can further alter effective power.

Researchers should avoid arbitrary rules such as “30 observations is always enough” or “a sample must be 10% of the population.” Adequacy depends on the estimand, design, variability, precision, power, and risk of bias. Small populations may require a census, while very large populations may require only a few hundred observations for a proportion estimate under simple random sampling.

Chapter Summary

Sampling connects a research question to a population claim. The target population defines the intended scope, while the sampling frame determines who can actually be selected. Probability methods—simple random, systematic, stratified, cluster, and multistage sampling—provide known selection probabilities and support design-based uncertainty estimates. Non-probability methods can be useful for exploration and hard-to-frame populations but require restrained generalization.

Sample-size planning should be based on desired precision or minimum detectable effect, confidence level, variability, statistical power, design effect, and expected non-response. Statistical power is the probability of detecting a specified effect under defined assumptions; it does not measure the truth of a hypothesis. Sampling error is random, whereas coverage error, selection bias, non-response, and measurement error can create systematic distortion. Good practice combines careful definitions, transparent randomization, fieldwork monitoring, appropriate weighting and variance estimation, ethical governance, and explicit limitations.

Key Terms

Target population: The full group to which a study intends to generalize.

Study population: The accessible portion of the target population.

Sampling frame: The operational list or process from which units are selected.

Sampling unit: The entity selected at a stage of sampling.

Element: The ultimate unit about which data are collected.

Parameter: A numerical characteristic of a population.

Statistic: A numerical characteristic calculated from a sample.

Simple random sampling: Selection in which every eligible sample of a given size is equally likely.

Stratified sampling: Sampling separately within defined population subgroups.

Cluster sampling: Sampling naturally occurring groups rather than isolated elements.

Multistage sampling: Sampling carried out through two or more sequential stages.

Design weight: The inverse of a unit’s selection probability.

Sampling error: Random sample-to-population difference caused by observing only part of a population.

Coverage error: Error caused by mismatch between the frame and the eligible population.

Non-response bias: Systematic distortion caused by differences between respondents and non-respondents.

Selection bias: Distortion caused by outcome-related inclusion mechanisms.

Statistical power: Probability of rejecting a false null hypothesis under a specified alternative.

Effect size: The magnitude of a difference or relationship of interest.

Design effect: Inflation or deflation of variance caused by a complex sampling design.

Representativeness: The extent to which a sample supports appropriate inferences about the target population.

Review Questions

Short Answer Questions

1. Distinguish a target population from a sampling frame.
2. Why can stratified sampling improve precision?
3. What is the relationship between statistical power and Type II error?
4. Explain the difference between sampling error and selection bias.
5. Why may weights be required after disproportionate sampling?

Long Answer Questions

1. Compare simple random, systematic, stratified, cluster, and multistage sampling, including one advantage and limitation of each.
2. Explain how a researcher should plan sample size for estimating a population proportion.
3. Discuss why a high response rate does not guarantee an unbiased survey.
4. Explain the difference between statistical significance and practical importance.
5. Develop a complete sampling plan for a national customer-satisfaction study.

Analytical / Application Questions

1. A firm has 50,000 customers and expects a proportion near 0.50. Determine the approximate sample size for 95% confidence and ±4 percentage-point precision, then explain how non-response changes the number invited.
2. A survey samples 25 customers per store from 40 stores. Explain why the effective sample size may be below 1,000 and identify information needed to assess the design effect.
3. A company recruits volunteers through a mobile-app advertisement and obtains a very large sample. Evaluate whether it can claim that its estimate represents all customers, and propose two improvements.

Multiple Choice Questions with Answers

Question 1

Which statement best defines a sampling frame?

A. The complete theoretical population of interest

B. The operational list or process from which units are selected

C. The final set of observations used in analysis

D. The statistical estimate of a population parameter

Correct Answer: B. The operational list or process from which units are selected

Question 2

Which sampling method deliberately divides a population into subgroups and samples within each subgroup?

A. Convenience sampling

B. Snowball sampling

C. Stratified sampling

D. Cluster sampling

Correct Answer: C. Stratified sampling

Question 3

Using P = 0.5, z = 1.96, and E = 0.05, the approximate large-population sample size for a proportion is:

A. 96

B. 196

C. 385

D. 1,000

Correct Answer: C. 385

Question 4

Statistical power is the probability of:

A. Accepting a true null hypothesis

B. Rejecting a false null hypothesis under a specified alternative

C. Obtaining a representative sample by convenience

D. Making a Type I error

Correct Answer: B. Rejecting a false null hypothesis under a specified alternative

Question 5

If the intraclass correlation is positive, sampling many individuals from the same cluster generally:

A. Increases the effective sample size without limit

B. Reduces the design effect to zero

C. Makes observations more redundant and can reduce effective sample size

D. Eliminates non-response bias

Correct Answer: C. Makes observations more redundant and can reduce effective sample size

Question 6

Which problem occurs when eligible members are missing from the sampling frame?

A. Coverage error

B. Type I error

C. Measurement reliability

D. Randomization balance

Correct Answer: A. Coverage error

Question 7

A researcher oversamples a small subgroup to obtain reliable subgroup estimates. What is commonly needed for overall population estimates?

A. Removing all observations from the subgroup

B. Design or analysis weights

C. Changing the confidence level to zero

D. Replacing probability selection with convenience selection

Correct Answer: B. Design or analysis weights

Question 8

Which statement about a larger sample is most accurate?

A. It always removes selection bias

B. It generally reduces random sampling error but may not remove systematic bias

C. It guarantees statistical significance

D. It makes a deficient sampling frame complete

Correct Answer: B. It generally reduces random sampling error but may not remove systematic bias

Question 9

In an A/B experiment, random assignment primarily supports:

A. Causal comparison between treatment conditions

B. Removal of all measurement error

C. A guarantee of universal generalizability

D. A higher response rate

Correct Answer: A. Causal comparison between treatment conditions

Question 10

A statistically significant result may still be unsuitable for a business decision when:

A. The effect is smaller than the practically important threshold

B. The sample was selected randomly

C. The confidence interval is reported

D. The outcome has units

Correct Answer: A. The effect is smaller than the practically important threshold