6. Sampling, Fieldwork and Data Quality

Introduction and Context

Advertising decisions are often made with incomplete information. A brand may wish to estimate awareness among a national market, compare responses to two creative executions, evaluate media reach, or understand why customers abandon a digital platform. In most such situations, studying every member of the relevant population is impossible, unnecessarily expensive, or too slow. Market researchers therefore select a sample and use the resulting evidence to make carefully qualified statements about a wider population.

Sampling is not merely a technical step between questionnaire design and data analysis. It determines who has a chance to be represented, whose experiences remain invisible, and how confidently results can be generalized. Fieldwork then converts a research design into actual observations through interviews, online panels, mobile surveys, focus recruitment, or other data-collection modes. At every stage, errors can enter the process. A sample may exclude important groups; selected participants may refuse; interviewers may influence responses; a questionnaire may be misunderstood; or automated systems may allow low-quality responses into the dataset.

For advertising and media research, these risks are especially consequential. Media audiences are fragmented across platforms, devices, languages, and locations. Consumers are exposed to different advertising environments, and their behaviour may change rapidly. A survey claiming that a campaign reached 60 per cent of adults is useful only if the sampling frame, recruitment process, weighting, response quality, and measurement procedures support that claim. This chapter examines how researchers design samples, manage fieldwork, adjust data, and diagnose threats to validity.

Learning Objectives

By the end of this chapter, students should be able to:

1. Distinguish between target populations, sampling frames, samples, and study populations.

2. Select appropriate probability and non-probability sampling methods for advertising and media studies.

3. Estimate sample size while considering precision, variability, design effect, subgroup analysis, and statistical power.

4. Plan and monitor fieldwork using panels, quotas, quality controls, and interviewer management.

5. Explain coverage error, sampling error, nonresponse error, selection bias, and measurement error.

6. Apply weighting, calibration, and nonresponse adjustments responsibly.

7. Evaluate the ethical, technological, and managerial implications of data-quality decisions.

The Logic of Sampling

The target population is the complete group about which the researcher intends to draw conclusions. It might be all adults in a country, users of a particular streaming service, or people aged 18–34 who saw a campaign during a specified period. The study population is the portion of the target population that can actually be reached under the research design. The sampling frame is the operational list, database, platform, or procedure used to identify potential participants.

These three concepts should not be treated as interchangeable. Suppose a study aims to understand television viewing among all adults, but recruitment occurs through an online access panel. The target population includes all adults, whereas the sampling frame consists of panel members with internet access who have agreed to participate in research. If internet access and panel membership are associated with media habits, the gap between the target population and frame can produce coverage error.

A sample is the set of units selected for participation. The unit may be an individual, household, business, device, social-media account, advertisement impression, or geographical area. The element is the basic entity about which information is collected, while the sampling unit is the entity selected at a particular stage. In a multistage design, for example, districts may be selected first, households second, and individuals third.

Sampling seeks a defensible relationship between the observed sample and the population. It does not promise that every sample will look exactly like the population. Rather, the design should make the selection process understandable and allow the researcher to assess uncertainty and possible bias.

Probability Sampling: Known Chances of Selection

In probability sampling, each eligible unit has a known, non-zero probability of selection, although those probabilities need not be equal. This feature supports the calculation of sampling error and the use of design-based statistical inference.

Simple Random Sampling

Under simple random sampling, every unit in the frame has an equal chance of selection. A random-number generator selects the required number of cases. The method is conceptually straightforward and statistically efficient when a complete, accurate frame exists. In advertising research, however, complete lists of all members of a target audience are uncommon. Privacy restrictions, outdated records, duplicate entries, and inaccessible contact information often make simple random sampling impractical.

Systematic Sampling

Systematic sampling selects every kth unit after a random starting point. If a list contains 10,000 units and 500 are needed, the interval is 20. The researcher chooses a random start between 1 and 20 and then selects every twentieth unit. This method is operationally convenient, but the list must be examined for periodic patterns. If the list is ordered in a way that corresponds to the selection interval, systematic bias can arise.

Stratified Sampling

Stratified sampling divides the population into relatively homogeneous subgroups, or strata, and draws samples within each stratum. Relevant strata may include region, age, language, income, or media-market size. Stratification can improve precision and ensure that strategically important groups are represented. Proportionate allocation samples each stratum in relation to its population size. Disproportionate allocation deliberately oversamples smaller or analytically important groups, such as heavy streaming users or a minority language community. Weights are then generally required to restore population representation.

Stratification should be based on information available before selection. It should also serve a substantive purpose. Creating many small strata without reliable population information can make fieldwork and weighting unstable.

Cluster and Multistage Sampling

Cluster sampling selects groups rather than individual elements, such as households, postal areas, retail outlets, or schools. It reduces travel and administrative costs, especially in geographically dispersed populations. Its disadvantage is that people within a cluster may resemble one another, reducing the amount of independent information generated by a given sample size.

Multistage sampling combines procedures. A national media survey might select regions, then census areas, then households, and finally one adult within each household. The design is often practical and economical, but its analysis must reflect unequal probabilities and clustering. Treating such data as if it came from a simple random sample can produce standard errors that are too small and claims that are too confident.

Non-Probability Sampling and Its Uses

In non-probability sampling, the probability of selection is unknown. Participants may be recruited because they are accessible, selected to meet quotas, referred by other participants, or identified by a researcher’s judgement. These methods do not automatically produce invalid research. They are often appropriate for exploratory work, qualitative research, hard-to-reach populations, rapid concept testing, and situations where no adequate sampling frame exists. Their limitations must, however, be acknowledged rather than concealed through misleading margins of error.

Convenience sampling recruits readily available participants, such as students, website visitors, or social-media followers. It is inexpensive but vulnerable to strong selection effects. Purposive sampling selects participants because they possess relevant characteristics, such as advertising planners, creative directors, or recent purchasers of a specialist product. Quota sampling sets target numbers for categories such as age and gender, then allows fieldworkers or platforms to recruit cases within those categories. Quotas can improve demographic balance but do not guarantee representativeness within each category.

Snowball sampling uses participant referrals and may help access hidden or networked populations. Expert sampling deliberately seeks people with specialized knowledge. Online access panels occupy an important middle ground in industry practice: they may use stratified recruitment, profiling, quality controls, and calibration, but panel membership remains a form of voluntary participation and does not by itself create a probability sample.

Researchers should match the claim to the design. A non-probability sample can estimate patterns within the respondents and can support comparisons under carefully controlled conditions, but population estimates require stronger assumptions than those made under a well-designed probability sample.

Sample Size, Precision and Statistical Power

Sample size is a design decision, not a number selected by habit. The appropriate size depends on the main estimate, desired precision, confidence level, expected variability, design, subgroup requirements, expected response or completion rate, and available resources.

For a proportion estimated from a simple random sample, a commonly used initial formula is:

n = Z²p(1 − p) / e²

where n is the required sample size, Z is the critical value associated with the desired confidence level, p is the anticipated proportion, and e is the tolerated margin of error. When no reasonable estimate of p is available, 0.50 is often used because it produces the largest variance and therefore a conservative sample size. For finite populations, a finite population correction may reduce the required number when the sample represents a substantial share of the population.

The familiar margin of error applies only under assumptions such as probability sampling and an appropriate variance calculation. It does not correct for coverage bias, nonresponse bias, poor measurement, or an online volunteer sample. A very large biased sample can remain biased.

Advertising studies frequently compare groups or test creative alternatives. In these situations, power analysis is more informative than a general rule of thumb. Statistical power is the probability of detecting an effect of a specified size when that effect truly exists. Power depends on sample size, effect size, significance level, variability, and whether the test is one-sided or two-sided. Small expected differences require larger samples. If a campaign test needs reliable results separately for four audience segments, the total sample must be designed around those subgroup comparisons rather than around the overall average.

Researchers should also account for design effect. Clustering, unequal weights, repeated observations, and other features can increase variance. A practical planning expression is:

n adjusted = n required × design effect / anticipated completion rate.

If a study requires 800 completed interviews, expects a design effect of 1.4, and anticipates a 40 per cent completion rate among invited eligible participants, the operational recruitment requirement is substantially larger than 800. The exact calculation depends on how the design effect and completion rate are defined, but the principle is essential: the number of completed cases is not the same as the number of invitations.

Fieldwork as a Managed Research Process

Fieldwork is the controlled execution of the sample and measurement plan. Before launch, researchers should document eligibility criteria, recruitment sources, quotas, contact rules, incentive arrangements, language versions, interviewer training, escalation procedures, data-security requirements, and quality thresholds.

A pilot or soft launch is valuable because it reveals problems that are invisible in a spreadsheet. It can show that respondents misunderstand a media-use question, that a mobile survey loads slowly, that a quota fills too quickly, or that a particular translation changes the meaning of an item. Pilot results should lead to documented revisions rather than informal changes that make cases incomparable.

Fieldwork monitoring should examine both quantity and quality. Quantity includes completes by region, demographic group, device, date, and recruitment source. Quality includes interview duration, straight-lining, inconsistent answers, excessive open-text copying, duplicate identifiers, speeding, suspicious geolocation, and patterns associated with bots or professional respondents. A low completion rate may indicate a problem in recruitment, eligibility, questionnaire burden, incentive design, or the perceived sensitivity of the subject.

Interviewer-administered research requires additional controls. Interviewers should receive standardized instructions, practice difficult probes, understand neutral wording, and know when a respondent is ineligible. Supervisors can review recordings or paradata where consent and law permit. Excessively fast interviews, unusually high agreement rates, or a concentration of cases from one interviewer may indicate fabrication, coaching, or procedural drift.

Panel management introduces its own responsibilities. Panels need transparent recruitment, current profile information, duplicate prevention, fair incentives, limits on survey frequency, and procedures for inactive or fraudulent accounts. “Professional respondents” may recognize common questions, rush through studies, or participate mainly for rewards. Removing such cases is not a substitute for designing a panel and questionnaire that reduce these incentives.

Data Quality: Error as a System, Not a Single Defect

The total survey error perspective treats quality as the combined result of several error sources. The most relevant for market research are coverage error, sampling error, nonresponse error, selection error, measurement error, processing error, and specification error.

Coverage error occurs when the sampling frame excludes eligible members, includes ineligible members, or represents some groups differently from others. A mobile-only frame may miss people without mobile access; a platform-based recruitment source may overrepresent frequent users. Coverage error is often systematic and cannot be solved simply by increasing sample size.

Sampling error is the random difference between a sample estimate and the population value that would arise from selecting another sample under the same design. It is quantifiable under probability sampling and is reflected in confidence intervals or standard errors. It is distinct from bias.

Nonresponse error occurs when selected or eligible participants do not participate, and nonrespondents differ from respondents in relevant ways. A response rate is useful for monitoring fieldwork but is not, by itself, a complete measure of bias. A low response rate can sometimes produce limited bias if respondents and nonrespondents are similar; a high response rate can still be biased if recruitment excludes an important group.

Selection error is particularly important in opt-in digital research. It arises when participation is related to the topic, media behaviour, motivation, or access conditions. For example, people with unusually strong opinions about an advertisement may be more likely to click a recruitment invitation.

Measurement error is the difference between the value recorded and the construct the researcher intended to measure. It may arise from ambiguous wording, poor recall, leading questions, social desirability, interviewer effects, device constraints, or inappropriate response scales. Asking respondents to recall every advertisement seen during a long period is likely to generate substantial memory error. Passive measurement may improve exposure data but raises privacy and consent questions.

Processing error includes coding mistakes, duplicate records, incorrect skips, faulty merges, and transcription problems. Specification error begins earlier, when the research question or construct is defined incorrectly. If “campaign success” is specified only as immediate liking, the study may measure an incomplete outcome regardless of how carefully fieldwork is conducted.

Weighting and Nonresponse Adjustment

Weighting assigns a numerical contribution to each case. A basic design weight is often the inverse of the unit’s probability of selection:

wᵢ = 1 / πᵢ

where πᵢ is the selection probability for case i. If some groups are oversampled, their weights are reduced during analysis so that estimates represent the intended population structure.

Weights may then be adjusted for nonresponse or calibrated to known external totals. Post-stratification aligns the sample with population distributions across variables such as age, sex, region, or education. Raking, or iterative proportional fitting, adjusts weights across several marginal distributions. These methods can reduce imbalance on observed characteristics, but they cannot guarantee correction for unobserved differences. If heavy media users and light media users have the same age and region distributions but different propensities to join a panel, demographic raking will not fully solve the problem.

Extreme weights can inflate variance. Weight trimming may improve stability, but it involves a trade-off between reducing variance and preserving representation. Every weighting procedure should be documented, including the variables used, source of population benchmarks, trimming rules, final weight distribution, and effect on estimates.

Weighted percentages should be calculated with the correct survey procedures. Standard errors and significance tests must reflect weights, strata, clusters, and finite population features where relevant. Presenting weighted point estimates with unweighted statistical tests can create misleading conclusions.

An Illustrative Case: Measuring a Streaming Campaign

Background

This illustrative case concerns “Northstar,” a fictional subscription video service launching a campaign among adults aged 18–44 in three metropolitan regions. The company wants to estimate aided awareness, compare awareness among exposed and unexposed audiences, and understand whether the campaign is reaching its intended segments.

Situation and Problem

The initial research proposal recommends 1,200 online interviews recruited from a general access panel. After two days, the sample shows a large surplus of highly educated urban respondents and an unusually high proportion of people who stream video daily. The campaign team argues that the sample is acceptable because the quotas for age and gender are full. The research manager is concerned that the findings will overstate campaign reach and awareness.

Analysis

Age and gender quotas control only two dimensions of composition. They do not ensure balance by region, education, household connectivity, streaming intensity, or panel tenure. The frame also excludes adults who rarely use the internet, even though they belong to the target population. In addition, people who stream frequently may be more likely to join a panel and more likely to notice the campaign.

The researcher reviews recruitment source, completion time, device, response consistency, and exposure questions. Several cases complete the survey in less than one-third of the median duration and give identical responses across long rating grids. These cases are flagged for review rather than automatically removed. The team also examines whether the exposed group is defined through self-reported recognition alone, which could be contaminated by false recognition.

Approach and Solution

The team revises the design in four ways. First, it introduces region and education controls and sets a soft target for streaming intensity. Second, it uses a campaign delivery log, where available, to construct a more defensible exposure classification instead of relying exclusively on recall. Third, it applies documented calibration weights to align the completed sample with population benchmarks on age, gender, region, education, and internet access. Fourth, it conducts sensitivity analyses: results are compared before and after weighting, with and without flagged cases, and under alternative exposure definitions.

The questionnaire is also shortened on mobile devices, and a neutral explanation is added before the awareness questions. A small recontact study checks whether reported exposure is reasonably stable after a short interval.

Outcome

The unweighted data initially suggest high campaign reach and a strong awareness difference between exposed and unexposed respondents. After quality review and calibration, the estimated reach decreases and the awareness difference becomes smaller, though it remains directionally positive. The final report does not claim that the campaign caused the entire difference, because exposure was not randomly assigned. It reports the sampling source, weighting variables, exclusions, uncertainty, and alternative estimates.

Learning Points

The case demonstrates that quotas are not equivalent to representativeness, weighting cannot repair every selection problem, and data-quality review must occur before interpretation. It also shows why exposure measurement, sample composition, and causal claims must be considered together. A polished dashboard cannot compensate for a weak design.

Ethical, Regulatory and Professional Considerations

Sampling and fieldwork involve people, personal data, and potentially sensitive behavioural information. Participants should receive clear information about the purpose of the study, expected duration, voluntary nature of participation, incentives, data use, and withdrawal procedures. Consent must be appropriate to the activity, especially when passive metering, device identifiers, location data, or linkage to advertising exposure logs is involved.

Researchers should collect only data necessary for the stated purpose, protect identifiers, restrict access, and establish retention and deletion policies. Children and other vulnerable groups require additional safeguards. Incentives should encourage participation without becoming coercive. Panel members should not be over-surveyed or penalized unfairly for declining a study.

Professional standards require honest reporting of sample sources, eligibility, field dates, completion criteria, weighting, exclusions, and limitations. Researchers should not label an opt-in sample as nationally representative merely because its weighted demographics resemble national totals. They should also avoid suppressing inconvenient quality findings or presenting exploratory subgroup differences as established effects.

Technology, Automation and Emerging Practice

Digital fieldwork platforms can automate invitations, quota management, fraud detection, duplicate checking, and dashboard reporting. Machine-learning tools may identify unusual response patterns, bot activity, or likely professional respondents. Passive data collection and advertising-platform logs can supplement self-report, while experience-sampling tools can capture media use closer to the time of exposure.

Automation should support, not replace, methodological judgement. A fraud score is not proof of fraudulent behaviour, and an algorithm trained on past decisions may reproduce their biases. Researchers should validate automated flags, preserve audit trails, explain decisions where feasible, and ensure that data linkage follows consent and legal requirements. Platform-based measurement also raises questions about proprietary definitions, missing data, changing APIs, and comparability across vendors.

Emerging research increasingly combines survey responses, observed behaviour, media delivery data, and first-party customer records. Such integration can improve measurement but may create new coverage and privacy risks. The central principle remains unchanged: the more complex the data system, the more explicitly researchers must describe who is represented, how records were linked, and where uncertainty remains.

Best Practices for Researchers and Managers

A defensible sampling and fieldwork plan should:

• define the target population and intended estimand before recruitment;
• assess the sampling frame for exclusions, duplication, and outdated information;
• choose probability or non-probability methods in relation to the claim required;
• calculate sample size around precision, power, design effect, and subgroup needs;
• pilot the instrument and recruitment process;
• monitor composition and response quality throughout fieldwork;
• document eligibility, exclusions, incentives, quotas, and interviewer procedures;
• use weighting transparently and assess the influence of extreme weights;
• separate sampling uncertainty from possible bias;
• conduct sensitivity analyses when exposure, eligibility, or quality decisions are uncertain; and
• report limitations in language that decision-makers can understand.

The strongest research organizations treat data quality as a continuous governance process rather than a final cleaning exercise. Quality begins with the research question and continues through design, recruitment, measurement, processing, analysis, and reporting.

Chapter Summary

Sampling determines how evidence is connected to a target population. Probability methods provide known selection probabilities and support design-based inference, while non-probability methods can be useful for exploratory, qualitative, rapid, or hard-to-reach research when their limitations are openly stated. Sample size must be based on precision, statistical power, subgroup objectives, design effect, and expected completion rather than convention alone.

Fieldwork requires structured recruitment, piloting, monitoring, interviewer management, panel governance, and quality controls. Data quality is threatened by coverage, sampling, nonresponse, selection, measurement, processing, and specification errors. Weighting, post-stratification, raking, and nonresponse adjustment can improve alignment with known population characteristics, but they cannot eliminate unobserved selection bias. Ethical practice requires informed participation, privacy protection, transparent reporting, and responsible use of automated tools. In advertising analytics, credible conclusions depend as much on the integrity of the sample and fieldwork as on the sophistication of the final model or dashboard.

Key Terms

Target population: The complete group about which a study intends to make conclusions.

Study population: The portion of the target population that can be reached under the research design.

Sampling frame: The list, database, or operational mechanism used to identify eligible units.

Sampling unit: The entity selected at a particular stage of a sampling design.

Probability sampling: Sampling in which each eligible unit has a known, non-zero chance of selection.

Non-probability sampling: Sampling in which selection probabilities are unknown.

Stratified sampling: Sampling separately within defined population subgroups or strata.

Cluster sampling: Sampling groups of units rather than selecting individual units directly.

Quota sampling: Recruiting cases until specified category targets are reached.

Sampling error: Random difference between a sample estimate and the corresponding population value.

Coverage error: Error caused by omissions, duplications, or ineligible units in the sampling frame.

Nonresponse error: Bias arising when nonparticipants differ from participants in relevant ways.

Measurement error: Difference between the intended construct and the value recorded.

Statistical power: Probability of detecting a specified effect when that effect exists.

Design effect: Inflation or adjustment of variance caused by features such as clustering or unequal weights.

Weighting: Assigning analysis values to cases to reflect selection probabilities or population characteristics.

Post-stratification: Adjusting weights so a sample matches known population distributions.

Raking: Iteratively adjusting weights to match several marginal population totals.

Panel: A group of people recruited and profiled for possible participation in repeated research studies.

Paradata: Operational information about data collection, such as duration, device, timestamp, or routing behaviour.

Review Questions

Short Answer Questions

1. Distinguish between a target population, study population, and sampling frame.

2. What is the principal difference between probability and non-probability sampling?

3. Why can a large sample still produce a biased estimate?

4. Define statistical power in the context of campaign testing.

5. What is the purpose of weighting in survey analysis?

Long Answer Questions

1. Explain the main probability sampling methods and discuss their suitability for advertising and media research.

2. Discuss the factors that should be considered when determining sample size for a campaign evaluation.

3. Explain how coverage, nonresponse, selection, and measurement errors can affect an audience survey.

4. Describe the stages of a rigorous fieldwork management process, from pilot testing to final quality review.

5. Evaluate the benefits and limitations of post-stratification and raking in online panel research.

Analytical / Application Questions

1. A campaign survey meets quotas for age and gender but contains many more heavy streaming users than the population. Diagnose the likely sources of bias and recommend corrective actions.

2. A researcher proposes using a 1,000-person opt-in panel and reporting a conventional margin of error. Assess whether this is methodologically defensible and explain what should be reported instead.

3. During fieldwork, 12 per cent of interviews are completed unusually quickly and show straight-lined answers. Design a decision process for investigating, retaining, or removing these cases without compromising research integrity.

22. Multiple Choice Questions with Answers

Question 1

A study aims to estimate campaign awareness among all adults, but recruits only from an online access panel. Which issue is most directly indicated by this difference?

A. Coverage error

B. Processing error

C. Finite population correction

D. Cluster efficiency

Correct Answer: A. Coverage error

Question 2

A researcher deliberately samples a larger number of a small language group so that its campaign responses can be analysed separately. Which design is being used?

A. Convenience sampling

B. Disproportionate stratified sampling

C. Systematic sampling without strata

D. Snowball sampling

Correct Answer: B. Disproportionate stratified sampling

Question 3

Which statement about a conventional margin of error is most accurate?

A. It corrects for all forms of nonresponse bias.

B. It is meaningful only when the sample is very large.

C. It primarily describes sampling uncertainty under appropriate design assumptions.

D. It proves that a weighted opt-in panel is representative.

Correct Answer: C. It primarily describes sampling uncertainty under appropriate design assumptions.

Question 4

A sample is selected through geographical clusters, and respondents within each cluster tend to have similar media habits. What is the likely statistical consequence?

A. The effective sample size may be smaller than the nominal sample size.

B. Measurement error is automatically eliminated.

C. Every respondent has an equal selection probability.

D. Nonresponse becomes impossible.

Correct Answer: A. The effective sample size may be smaller than the nominal sample size.

Question 5

Which formula expresses the basic design weight for a sampled unit with selection probability πᵢ?

A. πᵢ

B. 1 − πᵢ

C. 1 / πᵢ

D. πᵢ²

Correct Answer: C. 1 / πᵢ

Question 6

A panel sample is adjusted to match population totals for age, region, and education. What is the main purpose of this operation?

A. To create a probability sample after fieldwork

B. To calibrate the sample on observed characteristics

C. To remove all measurement error

D. To increase the response rate retrospectively

Correct Answer: B. To calibrate the sample on observed characteristics

Question 7

In a campaign test, the expected difference between two creative executions is very small. Which planning implication follows?

A. A smaller sample is sufficient because small effects are easier to detect.

B. A larger sample may be needed to achieve adequate statistical power.

C. Weighting is no longer relevant.

D. The significance level has no effect on sample size.

Correct Answer: B. A larger sample may be needed to achieve adequate statistical power.

Question 8

Which fieldwork finding is the strongest reason to investigate possible low-quality online responses?

A. Respondents use different devices.

B. Some interviews are substantially faster than the established completion-time pattern and contain straight-lining.

C. The sample includes both urban and rural respondents.

D. The questionnaire contains demographic questions.

Correct Answer: B. Some interviews are substantially faster than the established completion-time pattern and contain straight-lining.

Question 9

Why is demographic raking unable to guarantee that all selection bias has been removed?

A. Raking can be applied only to probability samples.

B. It adjusts only variables and differences represented in the calibration information.

C. Raking always reduces the sample size to zero.

D. It changes the wording of survey questions.

Correct Answer: B. It adjusts only variables and differences represented in the calibration information.

Question 10

A researcher uses self-reported campaign recognition as the sole measure of advertising exposure and then claims that exposure caused higher awareness. What is the principal methodological concern?

A. The claim may confuse association with causation and may contain recognition measurement error.

B. The sample must automatically be a cluster sample.

C. The study has no need for a target population.

D. Statistical power is irrelevant to the comparison.

Correct Answer: A. The claim may confuse association with causation and may contain recognition measurement error.