Data Collection, Sampling and Study Design

Introduction

Statistical analysis is only as trustworthy as the data on which it is based. A sophisticated model cannot correct a sample that systematically excludes important people, a questionnaire that encourages inaccurate answers, or measurements collected with an unreliable instrument. Data collection and study design therefore form the methodological foundation of data science.

This chapter examines how to translate a practical question into a defensible empirical study. It covers surveys, observational studies and experiments; probability and non-probability sampling; selection, response and measurement bias; sample-size planning; data-collection protocols; and ethical and governance requirements. The central principle is that design decisions must be made before analysis begins.

Learning Objectives

By the end of this chapter, you should be able to:

• distinguish a population, sampling frame, sample and target parameter;
• select an appropriate survey, observational or experimental design;
• compare probability and non-probability sampling methods;
• identify selection, non-response, response and measurement bias;
• calculate approximate sample sizes for proportions and means;
• explain the effects of confidence level, precision, variability and finite populations;
• prepare a practical data-collection protocol;
• evaluate validity, reliability, ethics and data governance in a study.

1. From a Business Question to a Statistical Study

A study begins with a clearly defined question. “Are customers satisfied?” is too vague to guide sampling or measurement. A stronger question is: “What proportion of customers who completed an online purchase in March would recommend the service, and how does this differ between new and returning customers?” This version identifies a population, time period, outcome and comparison groups.

Important terms include:

• Population: the complete set of units about which conclusions are intended.
• Element or unit: an individual person, household, transaction, device or organization in the population.
• Target population: the conceptual population relevant to the research question.
• Sampling frame: the operational list or mechanism from which units are selected.
• Sample: the units actually included in the study.
• Parameter: a numerical characteristic of the population, such as its mean or proportion.
• Statistic: the corresponding quantity calculated from a sample.
• Variable: a characteristic recorded for each unit.
• Sampling error: random difference between a sample statistic and its population parameter.
• Nonsampling error: error caused by coverage, non-response, measurement, processing or other factors not removed merely by increasing sample size.

A useful design sequence is:

1. define the decision or research question;
2. identify the target population and time frame;
3. specify variables and operational definitions;
4. select the study design;
5. construct or assess the sampling frame;
6. choose the sampling method and sample size;
7. develop instruments and procedures;
8. pilot-test the process;
9. collect, document and secure the data;
10. evaluate data quality before analysis.

An operational definition converts an abstract concept into an observable measure. For example, “customer retention” might mean an account with at least one purchase in each of two consecutive 90-day periods. Different operational definitions can produce different results, so the definition must be justified and documented.

2. Major Study Designs

2.1 Surveys

A survey collects standardized information from respondents, usually through questionnaires or structured interviews. Surveys are appropriate for attitudes, self-reported behaviours, experiences and demographic characteristics that may not exist in administrative records.

A cross-sectional survey measures units at one point or period. It estimates prevalence and describes associations, but it normally cannot establish temporal order. A longitudinal survey follows the same units over time, allowing change and temporal sequence to be examined. Panel attrition—the loss of participants between waves—can nevertheless introduce bias.

Question wording must be specific, neutral and understandable. A double-barrelled question such as “How satisfied are you with the price and delivery speed?” combines two constructs. Leading wording, such as “How helpful was our excellent support team?”, can encourage a favourable answer. Response categories should be mutually exclusive and collectively exhaustive where appropriate. “Prefer not to say” and “Don’t know” are often necessary, but they should not be used to avoid designing meaningful categories.

A well-designed questionnaire generally proceeds from easy, non-threatening questions to more sensitive topics, uses logical skip patterns, and provides a consistent reference period. Pretesting can reveal ambiguous wording, excessive length, missing response options and technical problems.

2.2 Observational studies

In an observational study, researchers measure exposures and outcomes without assigning the exposure. Examples include analysing hospital records, comparing purchasing patterns and monitoring machine failures. Observational studies are often necessary when experimentation would be unethical, impractical or too expensive.

They can establish association but may not justify a causal claim. Confounding occurs when a third variable is related to both the exposure and outcome. For example, a relationship between training attendance and job performance may partly reflect prior experience: experienced employees may be more likely to attend training and also perform better.

Common observational designs include:

2.3 Experiments

An experiment assigns an intervention or treatment to units and compares outcomes. Randomized controlled experiments use random assignment so that, on average, treatment groups are comparable in both measured and unmeasured characteristics. The difference in average outcomes can then estimate a causal effect under appropriate compliance and measurement conditions.

A basic parallel-group experiment contains a treatment group and a control or comparison group. Randomization is not the same as random sampling: random sampling supports generalization to a population, whereas random assignment supports causal comparison within the study context. A study may have one, both or neither.

Blinding reduces the influence of expectations. In a single-blind design, participants or assessors may be unaware of assignment; in a double-blind design, both participants and relevant investigators are unaware. Placebos, standardized instructions, allocation concealment and preregistered outcomes can reduce bias. In digital product testing, an A/B test is a randomized experiment in which users are assigned to alternative versions.

Ethical experiments require that risks be minimized, participation be voluntary and informed consent be obtained where applicable. A control group must not be denied established effective treatment without compelling ethical justification.

3. Sampling Designs

3.1 Probability sampling

In probability sampling, every element has a known, non-zero probability of selection. This permits design-based estimation and quantification of sampling uncertainty, provided implementation is faithful to the design.

Simple random sampling selects units so that every sample of a given size has equal probability. It is conceptually straightforward but requires a complete sampling frame.

Systematic sampling selects every kth unit after a random start. If the frame contains N units and n are required, the interval is approximately k = N/n. It is efficient, but hidden periodicity in the frame can distort results.

Stratified sampling divides the population into relatively homogeneous strata, such as regions or customer types, and samples within each. It ensures representation and can improve precision. Proportional allocation uses approximately n_h = n(N_h/N), where N_h is the size of stratum h. Disproportionate allocation may deliberately oversample a small but important group; analysis then requires appropriate weights.

Cluster sampling selects naturally occurring groups, such as schools, branches or villages, and surveys units within selected clusters. It reduces travel and administration costs. However, units in the same cluster tend to resemble one another, so the effective information may be less than that of the same number of independently selected units.

Multistage sampling combines stages, for example selecting regions, then branches, then customers. It is common in large-scale surveys but requires careful calculation of selection probabilities and weights.

3.2 Non-probability sampling

Non-probability samples do not provide known selection probabilities for all population elements. They may be useful for exploratory research, rapid feedback or hard-to-reach populations, but conventional margins of error should not automatically be reported as if the sample were a simple random sample.

• Convenience sampling: recruits readily available units.
• Purposive or judgment sampling: deliberately selects information-rich cases.
• Quota sampling: fills specified subgroup counts without random selection within groups.
• Snowball or respondent-driven recruitment: existing participants help recruit others.
• Voluntary response sampling: people self-select in response to an invitation.

Weighting, calibration and post-stratification may reduce imbalance, but they cannot guarantee removal of unknown selection bias. A large non-probability sample can be less representative than a smaller, carefully designed probability sample.

4. Bias, Error and Validity

4.1 Selection and coverage bias

Selection bias occurs when inclusion or participation is related to the outcome or its determinants. Coverage error occurs when the sampling frame omits parts of the target population or includes ineligible units. For example, an online-only survey excludes people without reliable internet access. A customer database may omit former customers—the very group relevant to churn.

Possible remedies include combining frames, screening for eligibility, using multiple modes, oversampling under-covered groups and comparing the achieved sample with known population benchmarks. These remedies should be documented rather than presented as proof that bias has disappeared.

4.2 Non-response and response bias

Non-response occurs when selected units do not provide usable data. Unit non-response affects the whole questionnaire; item non-response affects particular questions. Non-response is especially concerning when respondents and non-respondents differ in relevant ways.

Response bias arises from inaccurate or distorted answers. Causes include social desirability, recall difficulty, acquiescence, fear of consequences and interviewer effects. Anonymous administration, neutral wording, shorter recall periods, assurance of confidentiality and carefully trained interviewers can help.

Response rate is often reported as:

The exact denominator must be defined. Unknown eligibility, partial interviews and duplicate contacts require a documented reporting standard. A high response rate does not prove absence of bias, and a low response rate does not prove severe bias; the pattern of non-response matters.

4.3 Measurement error and reliability

Measurement error is the difference between the recorded value and the underlying value intended to be measured. It may arise from an unsuitable instrument, poor calibration, inconsistent coding, interviewer effects or respondent misunderstanding.

Reliability concerns consistency. Validity concerns whether the measure represents the intended construct. A scale can be reliable but invalid—for example, a consistently miscalibrated thermometer. Pilot testing, validated scales, interviewer training, instrument calibration, duplicate measurements and audit checks improve quality.

For a quantitative measurement, a useful conceptual model is:

This is a model rather than a claim that a “true value” is always directly observable. Random measurement error commonly increases variability; systematic error shifts measurements in a consistent direction and threatens validity.

4.4 Internal and external validity

Internal validity concerns whether the study supports the claimed relationship for the units studied. Threats include confounding, selection, attrition, history, maturation, contamination and inaccurate measurement. External validity concerns whether findings generalize to other populations, settings, times or treatment versions. Generalization requires attention to both sampling and contextual similarity.

5. Sample-Size Planning

Sample size should be determined from the primary outcome, desired precision, confidence level, variability, design and available resources—not from an arbitrary rule such as “ten percent of the population.” A larger sample reduces random error but does not automatically remove bias.

5.1 Estimating a population proportion

For a simple random sample estimating a proportion p, an approximate two-sided confidence interval has margin of error E when:

Here n₀ is the initial sample size, z is the standard-normal critical value for the selected confidence level, p is the anticipated population proportion, and E is the desired maximum margin of error expressed as a proportion. The formula assumes independent observations, a probability sample, a sufficiently large sample for the normal approximation, and a simple analysis of a proportion.

If p is unknown, p = 0.5 is conservative because p(1 − p) is then maximized. For 95% confidence, z ≈ 1.96. With E = 0.05 and p = 0.5:

The result is rounded upward to 385 completed observations. If a 20% non-response rate is anticipated, the number invited should be approximately:

Thus, 385 ÷ 0.80 = 481.25, so at least 482 units should be invited, subject to operational rounding.

For a finite population of size N sampled without replacement, a finite-population correction can be applied:

This is useful when the sample is a substantial fraction of the population. It does not correct coverage or non-response bias.

5.2 Estimating a population mean

For estimating a mean with anticipated standard deviation σ and margin of error E:

The units are consistent: σ and E must be expressed in the same measurement units, so their ratio is dimensionless. The formula assumes a probability sample, approximately independent observations and a defensible estimate of σ from prior data or a pilot study.

5.3 Comparing groups and complex designs

A sample-size calculation for comparing groups must specify the minimum practically important difference, variability, significance level, desired power and allocation ratio. Statistical power is the probability of detecting a specified effect when that effect truly exists. Small effects require larger samples than large effects.

Cluster sampling often requires a design effect. A common approximation for equal-sized clusters is:

where m is the average number of sampled units per cluster and ρ is the intracluster correlation coefficient. The required simple-random-sample size is multiplied by DEFF. This approximation assumes roughly equal cluster sizes and a common intracluster correlation; real designs may require specialist survey methods.

6. A Valid Data-Collection Process

A professional protocol should specify:

• study purpose, population, eligibility and exclusion criteria;
• sampling frame, selection algorithm and replacement rules;
• recruitment messages, contact attempts and refusal recording;
• instrument version, question order, coding and skip logic;
• interviewer training, supervision and escalation procedures;
• consent language and participant information;
• data formats, identifiers, timestamps and version control;
• quality checks, corrections, audit trails and missing-data codes;
• storage, access control, retention and destruction schedules;
• responsibilities for data protection and incident reporting.

A pilot should test the entire workflow, not merely ask whether respondents like the questionnaire. Examine completion time, item non-response, unusual distributions, duplicate records, device compatibility and staff adherence to the protocol. A data dictionary should define variable names, labels, types, units, permitted values and missing-value meanings.

Data should be separated into identifying and analytical components wherever feasible. Access should follow least-privilege principles. Personal data require lawful collection, purpose limitation, data minimization, security and transparent retention, subject to the applicable legal and institutional framework. Sensitive research may require ethics review, informed consent, encryption and a formal data-management plan.

7. Applied Case: Illustrative Customer-Support Study

Background

A software company wants to estimate the proportion of business customers satisfied with its support service and determine whether a new priority-support process improves satisfaction.

Problem

The company initially proposes an open website poll. Because dissatisfied users may be more motivated to complain and many customers never visit the website, the poll would not provide a defensible estimate for all business customers.

Analysis

The target population is active business accounts that opened at least one support ticket during the quarter. The sampling frame is the deduplicated ticketing database. The primary outcome is a five-point satisfaction item asked within seven days of ticket closure. The company also records customer size, region, issue category and resolution time.

For estimation, the team selects a stratified probability sample by region and customer size. Small but strategically important enterprise accounts are oversampled, and their selection probabilities are retained for weighting. For causal evaluation of the new process, eligible tickets—not merely willing customers—are randomly assigned to standard or priority handling, subject to service and ethical constraints. Satisfaction is measured with the same instrument in both groups.

The questionnaire is pilot-tested with 40 accounts. The pilot identifies an ambiguous phrase, a missing “not applicable” option and a mobile display error. Two reminder contacts are scheduled, and refusals and unreachable cases are recorded separately. A 20% non-response allowance is included in recruitment targets.

Outcome and interpretation

The company reports a weighted satisfaction estimate with its confidence interval, response patterns by stratum and a comparison of randomized groups. It does not describe the website poll as representative and does not infer causality from the observational relationship between resolution time and satisfaction, because issue severity may confound that relationship.

Learning points

The case illustrates that a sampling frame, an unbiased recruitment process, valid measurement and a suitable design are more important than the apparent sophistication of later analytics. It also demonstrates the different purposes of random sampling and random assignment.

8. Contemporary Tools and Professional Practice

Digital forms, computer-assisted interviews, sensor platforms and administrative data can reduce transcription error and permit real-time validation. However, automation can create new problems: platform users may not represent the population, adaptive interfaces may alter responses, and algorithmic eligibility rules may exclude groups without detection.

AI can assist with translation, questionnaire testing, coding and anomaly detection, but it should not silently rewrite responses, infer sensitive attributes or replace methodological judgment. Researchers should preserve original data, record model versions, audit performance across relevant groups and disclose AI-assisted processing. Synthetic data are useful for development and training, but they do not substitute for representative observations when estimating real-world parameters.

Challenges and Limitations

No design eliminates all uncertainty. Probability sampling can be expensive and depends on a credible frame. Non-probability recruitment may be the only feasible approach for hidden populations but limits formal generalization. Randomized experiments may have limited realism or ethical constraints. Self-reports can be affected by memory and social desirability, while administrative data may contain inconsistent definitions and missing fields. Weighting can increase variance, and multiple modes can introduce mode effects. These limitations should be reported explicitly rather than hidden behind a single quality score.

Best-Practice Checklist

Before collecting data, ask:

1. Is the target population explicit and does the frame cover it?
2. Is the design capable of answering the causal or descriptive question?
3. Are selection probabilities known or are limitations clearly stated?
4. Are key variables operationally defined and measured validly?
5. Has the instrument and workflow been piloted?
6. Is sample size based on precision or power, with allowance for attrition?
7. Are consent, privacy, security and retention addressed?
8. Are non-response, deviations and data corrections documented?
9. Can another analyst reproduce the sampling and cleaning process?
10. Are conclusions restricted to what the design supports?

Chapter Summary

Sound data science begins with sound study design. A researcher must define the target population, operationalize concepts, identify a suitable sampling frame and choose among surveys, observational studies and experiments. Probability sampling supports defensible population inference when selection probabilities are known; non-probability sampling can be useful but requires cautious claims. Random sampling and random assignment serve different purposes: the former supports generalization, while the latter supports causal comparison.

Selection, coverage, non-response, response and measurement errors can threaten validity even in large studies. Sample-size formulas help control random uncertainty under stated assumptions, but they cannot repair bias. Finite populations, non-response, clustering and weighting must be considered. Finally, valid data collection requires piloting, standardized procedures, documentation, ethical oversight, privacy protection and transparent reporting. The quality of an analytical conclusion is inseparable from the quality of the process that produced its data.

Key Terms

Population: Complete set of units about which conclusions are intended.

Sampling frame: Operational list or mechanism used to select units.

Sample: Units actually included in a study.

Parameter: Numerical characteristic of a population.

Statistic: Numerical characteristic calculated from a sample.

Survey: Structured collection of information from respondents.

Observational study: Study in which exposures are measured rather than assigned.

Experiment: Study in which an intervention is assigned to units.

Random sampling: Selection using known probabilities.

Random assignment: Allocation of treatments using chance.

Stratified sampling: Sampling separately within defined subgroups.

Cluster sampling: Sampling groups of naturally associated units.

Selection bias: Systematic distortion caused by how units enter a study.

Non-response bias: Distortion associated with failure to provide data.

Response bias: Distortion in answers provided by respondents.

Measurement error: Difference between recorded and intended values.

Reliability: Consistency of a measurement.

Validity: Degree to which a measure or inference represents what it should.

Margin of error: Planned maximum distance between an estimate and parameter under stated assumptions.

Design effect: Inflation of variance caused by features such as clustering.

Review Questions

Short Answer Questions

1. Distinguish a target population from a sampling frame.
2. Why are random sampling and random assignment not equivalent?
3. Define non-response bias and give one possible remedy.
4. What is the purpose of a pilot study?
5. Distinguish reliability from validity.

Long Answer Questions

1. Compare cross-sectional, cohort and case-control observational studies, including an appropriate use and a major limitation for each.
2. Explain how stratified, cluster and multistage sampling differ in purpose and implementation.
3. Discuss how questionnaire wording and administration can create response and measurement bias.
4. Explain the assumptions behind the sample-size formula for estimating a population proportion.
5. Develop a data-collection protocol for a longitudinal customer survey, including ethical and governance controls.

Analytical / Application Questions

1. A university wants to estimate the proportion of enrolled students who use its learning platform weekly. Propose a sampling frame, sampling method, sample-size assumptions and procedures for handling non-response. Justify each decision.
2. A retailer observes that customers receiving promotional emails purchase more often. Explain at least two plausible confounders and design an experiment or observational analysis that would improve causal interpretation.
3. A health survey uses an online voluntary poll and obtains 10,000 responses. Critically assess whether it can report a 95% margin of error of ±1 percentage point for the national adult population. Explain what additional evidence would be required.

Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes random sampling from random assignment?

A. Random sampling supports population generalization, while random assignment supports causal comparison.

B. Random sampling is used only in experiments, while random assignment is used only in surveys.

C. Random sampling eliminates measurement error, while random assignment eliminates non-response.

D. They are two names for the same procedure.

Correct Answer: A. Random sampling supports population generalization, while random assignment supports causal comparison.

Question 2

A researcher samples every 20th record after choosing a random starting record. Which method is being used?

A. Cluster sampling

B. Systematic sampling

C. Quota sampling

D. Purposive sampling

Correct Answer: B. Systematic sampling

Question 3

Using p = 0.5, z = 1.96 and E = 0.05, the approximate initial sample size for a population proportion is:

A. 96

B. 196

C. 385

D. 1,960

Correct Answer: C. 385

Question 4

Which problem is most directly associated with an online voluntary poll?

A. Random assignment bias

B. Self-selection and coverage bias

C. Finite-population correction

D. Instrument calibration error only

Correct Answer: B. Self-selection and coverage bias

Question 5

In an experiment, the main purpose of random assignment is to:

A. Guarantee a high response rate

B. Ensure every population member is sampled

C. Balance treatment groups probabilistically on relevant characteristics

D. Remove all measurement error

Correct Answer: C. Balance treatment groups probabilistically on relevant characteristics

Question 6

If the anticipated non-response rate is 20% and 385 completed responses are required, approximately how many units should be invited?

A. 308

B. 385

C. 462

D. 482

Correct Answer: D. 482

Question 7

A question asks, “How satisfied are you with our price and delivery speed?” The principal design problem is that it is:

A. Double-barrelled

B. Randomized

C. Stratified

D. Clustered

Correct Answer: A. Double-barrelled

Question 8

Which condition most strongly supports a causal interpretation of a treatment difference?

A. A large convenience sample

B. Random assignment with appropriate implementation

C. A high response rate in a cross-sectional survey

D. A strong correlation in administrative data

Correct Answer: B. Random assignment with appropriate implementation

Question 9

In the approximation DEFF ≈ 1 + (m − 1)ρ, ρ represents:

A. The response rate

B. The sampling fraction

C. The intracluster correlation coefficient

D. The confidence level

Correct Answer: C. The intracluster correlation coefficient

Question 10

A measure that gives highly consistent readings but is systematically 5 units too high is best described as:

A. Reliable but potentially invalid

B. Valid but unreliable

C. Neither reliable nor measurable

D. Unbiased by definition

Correct Answer: A. Reliable but potentially invalid