Data Collection, Sampling and Experimental Design

Introduction

Statistics for analytics begins before calculation. An analyst must first determine what information is needed, which units should provide it, how those units will be selected, and whether the resulting evidence can support a reliable conclusion. Poor data collection cannot be repaired fully by sophisticated software or machine-learning algorithms. A biased sample, unclear measurement, or confounded experiment can produce precise but misleading results.

This chapter examines the design of data collection for business and operational decisions. It distinguishes populations from samples, explains probability and non-probability sampling, introduces practical sample-size calculations, identifies sampling and non-sampling errors, and compares observational studies, surveys, and experiments. Particular attention is given to control groups, randomization, confounding, ethics, and the interpretation of evidence.

Learning objectives

By the end of this chapter, you should be able to:

• define populations, sampling frames, samples, parameters, and statistics;
• distinguish probability from non-probability sampling;
• select an appropriate sampling method for an analytics problem;
• estimate a practical sample size for a proportion or mean;
• recognize sampling error and important non-sampling errors;
• distinguish observational studies, surveys, and experiments;
• explain the roles of control groups, randomization, blinding, and replication;
• identify confounding and threats to causal interpretation; and
• evaluate data-collection designs ethically and professionally.

1. The foundation: population, sample and sampling frame

A population is the complete set of units about which a study seeks information. Units may be customers, transactions, employees, households, stores, machines, or loans. The population must be defined in terms of content, geography, time, and eligibility. “Customers” is incomplete unless it specifies, for example, customers who purchased from the online store between January and March.

A sample is a subset of the population selected for measurement. A census measures every eligible unit. A census eliminates sampling error, but it may be expensive, slow, impossible, or subject to substantial non-sampling error.

A sampling frame is the operational list or mechanism from which the sample is selected. Examples include a customer database, employee register, list of stores, or randomly generated telephone numbers. The frame may differ from the target population. A customer database may omit anonymous cash purchasers; an employee list may exclude contractors. This difference creates coverage error.

A parameter is a numerical characteristic of a population, such as the population mean μ or population proportion p. A statistic is the corresponding value calculated from a sample, such as the sample mean x̄ or sample proportion p̂. Sampling uses statistics to estimate unknown parameters.

For a variable X measured on n sampled units, the sample mean is:

Here xᵢ is the observed value for unit i and n is the sample size. The formula assumes that the observations are measured on a meaningful numerical scale. For a binary outcome, the sample proportion is:

where x is the number of sampled units with the characteristic of interest and n is the number successfully observed.

2. Probability and non-probability sampling

Probability sampling

In probability sampling, every eligible population unit has a known, non-zero probability of selection. Selection is governed by a random mechanism rather than the investigator’s preference. This permits estimation of sampling uncertainty and supports statistical generalization, provided coverage and response problems are controlled.

Simple random sampling

In simple random sampling, every sample of a specified size has an equal chance of selection. A random-number generator can select customer IDs from a complete frame. The method is conceptually simple and statistically defensible, but it requires a suitable frame and may produce a geographically dispersed sample.

Systematic sampling

Systematic sampling selects every k-th unit after a random starting point. If the frame contains N units and the required sample is n, the approximate interval is:

For example, if N = 10,000 and n = 500, k = 20. After randomly selecting a starting position from 1 to 20, every twentieth unit is selected. The method is efficient, but periodic patterns in the frame can create bias. A delivery list ordered by a repeating route, for example, may make every twentieth unit unrepresentative.

Stratified random sampling

Stratified sampling divides the population into relatively homogeneous subgroups called strata, such as region, customer type, or company size, and then randomly samples within each stratum. It ensures that important subgroups are represented. Proportional allocation selects from each stratum in proportion to its population size. Disproportionate allocation deliberately oversamples a small but important group; analysis must then use appropriate weights when estimating population totals or averages.

Cluster sampling

Cluster sampling divides the population into natural groups, such as branches, schools, or delivery areas. The analyst randomly selects clusters and measures all or some units within selected clusters. It reduces travel and administrative cost, but units within a cluster are often similar. Consequently, a cluster sample may contain less information than a simple random sample of the same nominal size.

Multistage sampling

Multistage sampling combines methods. An organization might randomly select regions, then stores within regions, then customers within stores. It is practical for large populations but requires careful planning at each stage and attention to unequal selection probabilities.

Non-probability sampling

Non-probability sampling does not provide a known selection probability for every unit. It is often useful for exploratory research, rapid feedback, qualitative work, or populations without a usable frame. However, conventional margins of error do not have their usual probability interpretation when selection is uncontrolled.

Common forms include:

Non-probability methods should not automatically be described as invalid. Their appropriate claim is usually narrower: they may reveal themes, generate hypotheses, or provide operational feedback, but they do not automatically represent the target population.

3. Choosing a sampling design

A professional sampling plan should answer five questions:

1. What precisely is the target population?
2. What is the sampling unit: person, transaction, account, store, or time period?
3. What frame is available, and who is missing from it?
4. Which characteristics must be represented?
5. What resources, precision, response rate, and deadline constrain the design?

Stratification is particularly useful when subgroup comparisons are central. Cluster or multistage sampling is attractive when fieldwork costs dominate. Simple random or systematic sampling is suitable when a reliable, well-ordered frame exists. Convenience sampling should be labelled clearly and its limitations reported.

A sample may be statistically large yet substantively poor. For example, 100,000 responses from a voluntary online poll can be less representative than 1,000 randomly selected customers. Sample size addresses random variation; it does not remove systematic bias.

4. Practical sample-size requirements

Sample-size planning should be based on the research question, desired precision, confidence level, variability, population structure, and expected response rate. The following formulas apply most directly to probability samples or designs that reasonably approximate them.

Estimating a population proportion

For a large population, the approximate sample size for estimating a proportion is:

where n₀ is the initial required sample size, z is the standard-normal critical value for the selected confidence level, p is the anticipated population proportion, and E is the desired margin of error expressed as a proportion. Common z values are 1.645 for 90%, 1.96 for 95%, and 2.576 for 99% confidence.

If p is unknown, p = 0.5 is conservative because p(1 − p) is then largest. The formula assumes an approximately independent probability sample, a binary or proportion outcome, and a sufficiently large population relative to n.

Worked example. A retailer wants a 95% confidence estimate of the proportion of customers satisfied with a service, within ±5 percentage points. No prior estimate is available. Therefore z = 1.96, p = 0.5, and E = 0.05:

The result is rounded upward: n = 385 completed responses. If the expected response rate is 70%, the number invited should be:

where nᵢ is the number invited and r is the expected response rate. This adjustment protects the number of completed observations; it does not correct non-response bias.

Finite population correction

When the sample is a substantial fraction of a finite population of size N, the corrected sample size is:

For N = 2,000 and n₀ = 385:

Thus approximately 323 completed observations are required under the stated assumptions.

Estimating a population mean

If the analyst has a planning estimate of the population standard deviation σ and wants margin of error E for a mean, the large-population formula is:

Here σ and E must use the same units as the measured variable. For example, if σ = 12 minutes, E = 2 minutes, and z = 1.96, then:

The required sample is rounded upward to 139 completed observations. If σ is estimated from a pilot study, that estimate should be documented. These formulas do not automatically account for clustering, stratification, design effects, multiple comparisons, or a planned hypothesis test. In cluster samples, the effective sample size may be smaller than the nominal size.

5. Sampling error and non-sampling error

Sampling error is random difference between a sample result and the population value caused by observing only part of the population. It generally decreases as an appropriate random sample becomes larger. It cannot be eliminated entirely unless a census is conducted.

The standard error of a sample proportion under a simple random-sampling approximation is:

The expression assumes independent observations and an appropriate probability design. Complex designs may require a design-effect adjustment. A confidence interval is commonly expressed as estimate ± critical value × standard error, but the interpretation depends on the sampling design and assumptions.

Non-sampling errors can occur in a census or sample and are often more damaging than sampling error:

• Coverage error: eligible units are omitted or ineligible units are included in the frame.
• Non-response error: selected units do not participate, and respondents differ systematically from non-respondents.
• Measurement error: the recorded value differs from the true value because of ambiguous questions, faulty instruments, recall problems, or social desirability.
• Processing error: coding, data entry, merging, weighting, or programming introduces mistakes.
• Interviewer error: interviewer behaviour influences responses.
• Response error: participants misunderstand, conceal, or inaccurately report information.

Quality control includes piloting instruments, using clear definitions, validating ranges, documenting missingness, monitoring response rates by subgroup, checking duplicate records, auditing data transformations, and comparing respondents with known population benchmarks. Weighting may reduce imbalance, but weights cannot guarantee removal of bias when important variables are unobserved.

6. Observational studies, surveys and experiments

An observational study records variables without assigning an intervention. An analyst might compare employee performance with and without remote working as employees choose their own arrangements. Observational studies are often feasible and ethically appropriate, but associations may reflect pre-existing differences.

A survey is a structured method of collecting self-reported information from selected respondents. Surveys may be cross-sectional, collecting information at one time, or longitudinal, following units over time. A well-designed survey defines the population, uses neutral and specific questions, controls question order, provides appropriate response options, and protects confidentiality.

An experiment deliberately assigns an intervention or treatment and observes outcomes. Randomized experiments are especially valuable for causal inference because random assignment tends to balance both measured and unmeasured characteristics between groups, on average.

The distinction is important:

• An association describes how variables vary together.
• A causal effect concerns what would happen to the same unit under alternative interventions.
• Randomized assignment supports causal interpretation when implementation and analysis are sound.
• Observational evidence can support causal reasoning, but it requires stronger assumptions and careful adjustment.

7. Experimental design: treatment, control and randomization

A treatment is the intervention whose effect is being studied. A control group provides a comparison under a defined alternative condition, such as no change, usual practice, placebo, or an existing process. The control must be appropriate to the decision being made.

Randomization assigns eligible units to treatment conditions using a random mechanism. It is different from random sampling. Random sampling supports generalization from a sample to a population; random assignment supports comparison of treatment outcomes within the study. A study may have random assignment but a narrow volunteer population, limiting generalization.

A basic parallel-group experiment includes:

1. define eligible units and outcomes;
2. recruit or select units ethically;
3. measure relevant baseline variables;
4. randomly assign units to treatment and control;
5. implement the intervention consistently;
6. measure outcomes using the same rules;
7. analyse according to the planned design; and
8. report exclusions, missing data, and deviations.

Replication means applying each condition to multiple units, not repeatedly measuring one unit and calling it replication. Blinding prevents participants, administrators, assessors, or analysts from knowing assignment when such knowledge could influence behaviour or measurement. A placebo or attention control may be important when expectations affect outcomes.

Randomized block designs first divide units into relatively homogeneous blocks, such as region or prior performance, and then randomize within blocks. Blocking can improve precision while preserving random assignment. In business, A/B testing commonly randomizes users, accounts, sessions, or geographic units to alternative versions of a product or message. The randomization unit must match the risk of spillover: if users influence one another, randomizing individuals may contaminate the comparison.

8. Confounding and internal validity

A confounder is a variable associated with both the presumed cause and the outcome that can create a misleading association. Suppose customers who receive a premium support service have higher retention. If high-value customers are preferentially given premium support, customer value may confound the relationship. Retention may reflect customer value rather than the support service.

Confounding differs from mediation. A mediator lies on the pathway through which a treatment affects an outcome; controlling for it may hide part of the treatment effect. A confounder is a pre-existing alternative explanation. Analysts should draw a causal diagram or reason explicitly about time order and plausible mechanisms before selecting adjustment variables.

Threats to experimental validity include poor randomization, noncompliance, attrition, contamination between groups, unequal exposure, selective outcome measurement, and changing external conditions. An intention-to-treat analysis compares units according to their assigned groups and preserves the benefit of randomization, although it estimates the effect of assignment rather than necessarily the effect among compliers.

External validity concerns whether findings apply to other populations, settings, times, or interventions. Internal validity comes first: a highly generalizable but biased estimate is not useful.

9. Ethics, governance and responsible data collection

Data collection must respect informed consent, privacy, fairness, and applicable law. Participants should understand the purpose, procedures, foreseeable risks, expected use, and voluntary nature of participation. Consent should not be coerced. Sensitive information should be minimized, access-controlled, retained only as long as necessary, and protected through appropriate security.

Organizations should distinguish operational necessity from convenient data accumulation. De-identification reduces risk but is not an absolute guarantee, especially when datasets can be linked. Analysts should document lawful purpose, data provenance, retention rules, access permissions, and deletion procedures. Where automated systems use collected data, fairness assessments should examine whether sampling or measurement excludes groups or produces systematically different errors.

Ethical review is particularly important when experiments affect prices, employment, health, credit, safety, or access to essential services. A statistically significant result does not justify an intervention that causes unreasonable harm.

10. Technology and contemporary practice

Digital platforms enable continuous event collection, online panels, app-based experiments, and near-real-time dashboards. These technologies increase scale but also create new risks: bot responses, duplicate identities, algorithmic selection, cookie loss, platform changes, and unclear consent. Automated sampling can reproduce historical coverage bias.

AI tools can assist with frame deduplication, anomaly detection, questionnaire translation, adaptive survey routing, and sample monitoring. They should support—not replace—design judgement. Analysts must validate automated classifications, document model versions, test subgroup performance, and avoid treating generated data as evidence about a real population unless its provenance and limitations are explicit.

Applied case: evaluating a customer-retention intervention

Background

This illustrative case concerns a subscription company whose monthly cancellation rate has increased. Management proposes a proactive support email for customers showing signs of disengagement.

Problem

The company wants to know whether the email causes improved retention, not merely whether recipients are more likely to remain subscribed. Sending the email to every customer would provide no contemporaneous comparison.

Analysis and approach

The analytics team defines the eligible population as active customers who meet the disengagement rule during one calendar month. It excludes accounts already scheduled for closure and verifies that each customer appears once in the frame. Customers are randomly assigned at the account level to either the support email or usual communication. Randomizing accounts, rather than individual messages, prevents a customer from receiving contradictory treatments. The primary outcome is active subscription 30 days after assignment. The team records delivery, opening, and actual exposure separately from assignment.

A pre-specified analysis compares the retention proportion in the assigned treatment and control groups. Baseline region and tenure are checked for major imbalance, but randomization—not post hoc matching—is the main protection against confounding. The team monitors unsubscribe complaints and permits customers to opt out of marketing communication.

Outcome and interpretation

Suppose retention is higher in the treatment group. The result supports a causal effect of assignment if randomization was implemented correctly, outcome measurement was comparable, and loss to follow-up was limited. It does not prove that every customer would benefit, nor does it establish long-term profitability. The company should examine costs, heterogeneous effects, customer experience, and whether the intervention remains effective after repeated use.

Learning points

The case illustrates the difference between a targeted observational comparison and a randomized experiment; the importance of defining the unit of randomization; the need for a control group; and the distinction between treatment assignment, treatment receipt, short-term retention, and business value.

Best-practice checklist

Before collecting data, an analyst should:

• define the population, unit, time period, variables, and outcome;
• assess frame coverage and likely sources of exclusion;
• justify the sampling or assignment method;
• estimate required completed observations and adjust for non-response;
• pilot questions, instruments, data pipelines, and randomization;
• predefine primary outcomes and key analyses where feasible;
• protect confidentiality and obtain appropriate consent;
• monitor response, missingness, attrition, and subgroup representation; and
• report limitations, deviations, and uncertainty honestly.

Chapter Summary

Reliable analytics depends on a defensible data-collection design. A population is the complete target set, a sample is the observed subset, and a sampling frame is the operational source used to select units. Probability sampling enables known selection probabilities and supports statistical generalization, while non-probability sampling is useful for exploration but has weaker claims to representativeness.

Simple random, systematic, stratified, cluster, and multistage designs make different trade-offs among precision, cost, coverage, and administration. Sample-size formulas for proportions and means require assumptions about confidence, variability, precision, independence, and population size. Increasing sample size reduces random sampling error but does not solve coverage, non-response, measurement, processing, or other non-sampling errors.

Observational studies and surveys measure existing conditions; experiments assign interventions. Control groups provide comparisons, random assignment reduces confounding, and blinding, replication, blocking, and pre-specified analysis strengthen validity. Analysts must distinguish internal validity from generalizability and must address ethics, privacy, fairness, and data governance throughout the project.

Key Terms

Population: Complete set of units about which conclusions are intended.

Sample: Subset of the population measured in a study.

Sampling frame: Operational list or mechanism from which a sample is selected.

Parameter: Numerical characteristic of a population.

Statistic: Numerical characteristic calculated from a sample.

Probability sampling: Sampling in which each eligible unit has a known, non-zero selection probability.

Non-probability sampling: Sampling without known selection probabilities for all units.

Stratified sampling: Random sampling within defined population subgroups.

Cluster sampling: Sampling natural groups and measuring units within selected groups.

Sampling error: Random difference caused by observing a sample rather than the entire population.

Coverage error: Error caused by mismatch between the target population and sampling frame.

Non-response error: Bias arising when non-respondents differ from respondents.

Measurement error: Difference between the recorded and intended value.

Survey: Structured collection of information from respondents.

Observational study: Study in which variables are observed without assigned intervention.

Experiment: Study in which an intervention is deliberately assigned.

Control group: Group receiving a comparison condition.

Randomization: Assignment to conditions using a random mechanism.

Confounder: Variable that provides an alternative explanation for an observed relationship.

External validity: Extent to which findings apply beyond the study setting.

Review Questions

Short Answer Questions

1. Distinguish a population, sample, sampling frame, parameter, and statistic.
2. What is the principal difference between probability and non-probability sampling?
3. Why can a large convenience sample still be biased?
4. State two examples of non-sampling error.
5. What is the purpose of a control group in an experiment?

Long Answer Questions

1. Compare simple random, stratified, cluster, and systematic sampling, including an appropriate business application for each.
2. Explain how coverage, non-response, measurement, and processing errors can affect an analytics project.
3. Describe the steps required to design and report a randomized experiment.
4. Explain the difference between association and causation, using confounding in your explanation.
5. Discuss the ethical and governance responsibilities involved in collecting customer data.

Analytical / Application Questions

1. A company has 5,000 customers and wants a 95% confidence estimate of a proportion with a ±5% margin of error and no prior estimate. Calculate the initial and finite-population-corrected sample sizes, showing assumptions.
2. An analyst observes that customers receiving premium support have higher retention. Identify at least two possible confounders and propose an improved study design.
3. A survey receives many more responses from urban customers than rural customers. Explain how this may affect conclusions and recommend two corrective actions.

Multiple Choice Questions with Answers

Question 1

Which feature defines probability sampling?

A. Every unit is selected by an interviewer

B. Each eligible unit has a known, non-zero selection probability

C. Only volunteers may participate

D. The sample must contain exactly 1,000 units

Correct Answer: B. Each eligible unit has a known, non-zero selection probability

Question 2

A population contains 10,000 units and the required sample is 500. What is the systematic sampling interval?

A. 5

B. 10

C. 20

D. 50

Correct Answer: C. 20

Question 3

For estimating a proportion with no prior estimate, which value of p is conventionally conservative?

A. 0.01

B. 0.25

C. 0.50

D. 0.95

Correct Answer: C. 0.50

Question 4

A study obtains 385 completed responses and expects a 70% response rate. Approximately how many people should be invited?

A. 270

B. 385

C. 500

D. 550

Correct Answer: D. 550

Question 5

Which is a non-sampling error?

A. Random variation from selecting a sample

B. Coverage error caused by an incomplete frame

C. Variation reduced by increasing sample size

D. Sampling fluctuation around a population parameter

Correct Answer: B. Coverage error caused by an incomplete frame

Question 6

What is the main purpose of random assignment in an experiment?

A. To guarantee a representative national sample

B. To eliminate the need for a control group

C. To balance potential confounding factors across conditions on average

D. To ensure every participant receives the treatment

Correct Answer: C. To balance potential confounding factors across conditions on average

Question 7

Which statement best describes an observational study?

A. The researcher assigns every intervention

B. The researcher records variables without assigning an intervention

C. The researcher always uses simple random sampling

D. The study cannot include numerical outcomes

Correct Answer: B. The researcher records variables without assigning an intervention

Question 8

A customer’s prior loyalty affects both receipt of premium support and later retention. Prior loyalty is a likely:

A. Sampling interval

B. Control group

C. Confounder

D. Response rate

Correct Answer: C. Confounder

Question 9

If the expected response rate falls, what should normally happen to the number invited to achieve a fixed number of completed responses?

A. It should increase

B. It should decrease

C. It should remain exactly unchanged

D. It should become zero

Correct Answer: A. It should increase

Question 10

Which statement about a control group is most accurate?

A. It provides a comparison condition for evaluating the treatment

B. It must always receive a placebo

C. It is selected only after results are known

D. It guarantees that the sample represents the population

Correct Answer: A. It provides a comparison condition for evaluating the treatment