Sampling, Measurement, Reliability and Validity

Introduction

Market research decisions depend not only on the questions asked but also on who is studied, how information is recorded, and whether the evidence is trustworthy. A startup may survey potential customers, interview retailers, observe competitors, or analyse digital behaviour. In each situation, the researcher must decide how to select participants and how to measure concepts such as satisfaction, brand awareness, willingness to pay, and purchase intention.

Four linked issues are central. Sampling determines which members of a target population provide data. Measurement determines how abstract concepts are represented as observations. Reliability concerns the consistency of a measurement process, while validity concerns whether the process measures what it is intended to measure. Poor decisions in any one area can mislead a startup, even when the data appear precise.

This chapter develops practical foundations for selecting samples, estimating sample requirements, identifying error, constructing measures, and evaluating the quality and limitations of research evidence.

Learning objectives

By the end of this chapter, you should be able to:

1. Distinguish a population, sampling frame, sample and sampling unit.
2. Compare probability and non-probability sampling methods.
3. Estimate practical sample requirements and explain the assumptions behind common formulas.
4. Distinguish sampling error from non-sampling error.
5. Identify nominal, ordinal, interval and ratio levels of measurement.
6. Assess reliability and validity using appropriate evidence.
7. Recognise ethical obligations and limitations in startup research.
8. Apply these ideas to a practical market-research decision.

1. The sampling logic

A population is the complete set of units relevant to a research question. For a food-delivery startup, the population might be all adults in a city who order prepared meals online. A sample is a subset of that population. A sampling unit is the individual element selected, such as a customer, household, business or store. A sampling frame is the operational list or procedure used to identify population members.

The sampling frame may not perfectly represent the population. A customer database excludes potential customers who have never purchased. A social-media advertisement reaches users of that platform rather than all residents. The difference between the target population and the accessible frame creates potential coverage error.

A useful sampling plan specifies:

• the target population;
• the sampling unit;
• the geographical and time boundaries;
• the sampling frame or recruitment source;
• the selection method;
• the planned sample size; and
• procedures for non-response and quality control.

The purpose of sampling is not simply to obtain many responses. It is to obtain evidence that is sufficiently relevant, diverse and accurate for the decision being made.

2. Probability sampling

In probability sampling, each eligible population member has a known, non-zero probability of selection. This permits statistical inference from the sample to the population, provided the design and implementation are appropriate.

Simple random sampling

Each member of the frame has an equal chance of selection. A random-number generator can select customer identification numbers. The method is conceptually strong but requires a reasonably complete sampling frame and may be expensive when the population is geographically dispersed.

Systematic sampling

The researcher selects every kth unit after a random starting point. If the frame contains N units and the required sample is n, the approximate interval is:

Here, k is the sampling interval, N is the frame size and n is the desired sample size. Systematic sampling is efficient, but the researcher must check for periodic patterns in the list. For example, selecting every tenth customer could be biased if records are arranged in a ten-day cycle.

Stratified random sampling

The population is divided into meaningful, non-overlapping subgroups called strata, such as age groups, districts or customer types. A random sample is then selected within each stratum. Proportionate allocation selects numbers in proportion to stratum size:

where n_h is the sample size for stratum h, n is the total sample size, N_h is the size of stratum h and N is the population size. Stratification can improve representation and precision, especially when subgroups differ substantially. Disproportionate sampling may deliberately oversample a small but strategically important group; appropriate weighting may then be required during analysis.

Cluster sampling

The researcher randomly selects naturally occurring groups, such as schools, neighbourhoods or retail outlets, and studies units within selected clusters. Cluster sampling reduces travel and administrative cost, but units within a cluster often resemble one another. This reduces the amount of independent information obtained from a given number of observations.

A common design-effect approximation is:

where deff is the design effect, m is the average number of sampled units per cluster and ρ is the intracluster correlation coefficient. The approximation assumes roughly equal cluster sizes and a simple clustered design. If ρ is zero, clustering adds little variance; if ρ is positive, the effective information is lower than the raw sample count suggests.

Multistage sampling

Multistage designs combine methods. A researcher might randomly select districts, then stores within districts, then customers within stores. Such designs are practical for large or dispersed populations, but selection probabilities and design effects must be considered carefully.

3. Non-probability sampling

In non-probability sampling, the selection probability is unknown or not governed by random selection. These methods are often appropriate for exploratory research, early-stage customer discovery, limited budgets or hard-to-reach groups. However, the usual margin-of-error calculations for simple random samples do not automatically apply.

Convenience sampling recruits readily accessible participants, such as students or visitors to a website. It is fast but vulnerable to coverage and self-selection bias.

Purposive or judgement sampling deliberately selects people with relevant knowledge, such as restaurant owners or experienced users of a specialist service.

Quota sampling sets target numbers for categories, such as age and gender, and recruits participants until each quota is filled. It can improve visible composition but remains non-random within categories.

Snowball sampling asks participants to refer other eligible participants. It can assist research with hidden or networked populations, but people recruited through the same networks may share characteristics.

Voluntary-response sampling allows people to opt in, often through an online poll. Those with strong opinions may be overrepresented.

The appropriate method depends on the research purpose. A startup validating language and discovering customer problems may use purposive interviews. A startup estimating the percentage of households likely to subscribe should seek a defensible probability design, or state clearly that the results are descriptive of the respondents rather than statistically representative of all households.

4. Determining practical sample requirements

Sample size is a design decision balancing precision, confidence, variability, cost, time and the consequences of error. Larger samples do not correct a biased sampling frame or a defective questionnaire.

For estimating a population proportion under a simple random-sampling model, a commonly used large-population formula is:

where n₀ is the initial sample size, z is the standard-normal critical value associated with the desired confidence level, p is the anticipated population proportion, and e is the maximum tolerated margin of error expressed as a proportion. When no prior estimate of p is available, p = 0.50 is conservative because it produces the largest required sample for a given z and e.

For a 95% confidence level, z is approximately 1.96. If p = 0.50 and e = 0.05:

The result is normally rounded upward to 385 completed responses. This calculation assumes a simple random sample, independent observations, a valid frame, and a proportion estimate. It does not include non-response or design effects.

For a finite population of size N, a finite-population correction can be applied:

For example, if N = 2,000 and n₀ = 384.16:

Thus, approximately 323 completed responses may be sufficient under the stated assumptions. If the expected completion rate is 60%, the number invited should be:

Therefore, 323 ÷ 0.60 ≈ 538 invitations, rounded upward.

For estimating a mean, a planning formula is:

where σ is the anticipated population standard deviation and e is the tolerated margin of error in the same units as the measured variable. This formula requires a defensible estimate of variability, often from a pilot study or prior research.

A sample-size calculation should also consider subgroup analysis. A total of 400 responses may be adequate for an overall estimate but inadequate for comparing four customer segments if each segment contains only a small number of cases. Practical planning should identify the minimum useful number in each critical subgroup.

5. Sampling and non-sampling error

Sampling error is the difference between a sample estimate and the population value caused by observing a sample rather than a census. Under a valid probability design, it can be estimated statistically. Increasing sample size generally reduces random sampling error, approximately at a rate proportional to 1 ÷ √n under simple random-sampling conditions.

Sampling error is not the same as bias. A very large convenience sample can produce a precise description of the wrong population.

Non-sampling error includes all other errors arising during research. Important forms include:

• Coverage error: eligible units are missing from the frame, or ineligible units are included.
• Non-response error: selected participants do not respond, and responders differ systematically from non-responders.
• Response error: participants misunderstand, forget, conceal or inaccurately report information.
• Interviewer error: interviewer wording, tone or recording influences answers.
• Processing error: coding, data entry, cleaning or analysis is incorrect.
• Questionnaire error: ambiguous, leading, double-barrelled or excessively burdensome questions distort responses.

Quality control includes pilot testing, clear eligibility screening, neutral wording, interviewer training, duplicate checks, range checks, monitoring response patterns and documenting exclusions. Weighting can reduce some compositional imbalances, but it cannot fully repair unknown bias or poor measurement.

6. Measurement and levels of measurement

Measurement is the systematic assignment of numbers or labels to attributes according to rules. A construct is an abstract concept, such as trust or perceived value. An indicator is an observable question or behaviour used to represent that construct. Operationalisation converts a construct into measurable variables.

Nominal measurement

Nominal categories identify membership without order. Examples include payment method, city and product category. Numbers used as codes have no arithmetic meaning. Appropriate summaries include frequencies, percentages and the mode.

Ordinal measurement

Ordinal categories have a meaningful order, but the distances between categories are not assumed equal. Examples include satisfaction categories from very dissatisfied to very satisfied and business-size bands. Medians, percentiles and rank-based comparisons are often appropriate.

Interval measurement

Interval scales have ordered values with equal intervals, but zero is not a true absence of the attribute. Temperature in Celsius is a standard example. Means and differences can be meaningful, but ratio statements such as “twice as much” are not necessarily meaningful.

Ratio measurement

Ratio scales have equal intervals and a meaningful zero. Revenue, number of purchases, distance and time duration are examples. All common arithmetic operations, including ratios, are meaningful.

A five-point agreement item is technically ordinal, although a multi-item scale composed of several such items is often analysed approximately as interval when its construction and distribution justify that treatment. The researcher should state the decision and avoid claiming greater precision than the scale supports.

7. Constructing useful measures

A good measure begins with a clear construct definition. “Customer loyalty” might mean repeat purchase behaviour, an intention to repurchase, willingness to recommend, or emotional commitment; these are related but not identical.

Use one idea per question. “The app is fast and reliable” is double-barrelled because a respondent may agree with one part and disagree with the other. Avoid leading wording, unexplained technical terms, excessive recall periods and answer options that omit realistic alternatives. Include “not applicable” or “don’t know” where appropriate. Response scales should be balanced, labelled consistently and ordered logically.

Multi-item measures are useful for broad constructs. Items should cover important dimensions without being redundant. Reverse-worded items may detect acquiescence but can also confuse respondents, especially in low-literacy contexts; they should be used sparingly.

8. Reliability

Reliability is the consistency or dependability of a measurement procedure. It does not by itself establish validity: a measure can consistently capture the wrong concept.

Test–retest reliability examines stability by administering the same measure to the same respondents on two occasions when the underlying construct is expected to remain stable. A correlation or agreement statistic can be used, but a low value may reflect real change rather than poor measurement.

Inter-rater reliability concerns agreement among observers or coders. It is important when interviews, observations or open-ended responses require judgement. Percentage agreement is easy to understand but can be misleading when agreement occurs by chance; statistics such as Cohen’s kappa may be suitable for two categorical raters under relevant assumptions.

Internal consistency concerns whether items intended to measure the same construct produce coherent responses. Cronbach’s alpha is commonly reported:

where α is alpha, k is the number of items, σᵢ² is the variance of item i and σₜ² is the variance of the total score formed from the items. Alpha assumes, approximately, that items reflect a common construct and have suitable coding; it is influenced by the number of items and does not prove unidimensionality or validity. A high alpha can result from redundant questions. Reliability should therefore be evaluated alongside item content, dimensionality and the research context rather than by applying a universal cut-off mechanically.

9. Validity

Validity concerns the degree to which evidence supports the intended interpretation and use of scores. It is better understood as an argument based on multiple sources of evidence than as a permanent property of a questionnaire.

Face validity is the apparent appropriateness of a measure to non-specialists. It can support acceptance but is weak evidence on its own.

Content validity asks whether the measure adequately covers the relevant domain. Subject-matter experts can map items against a construct definition and identify omissions or irrelevant content.

Construct validity examines whether the measure behaves as theory predicts. Evidence may include convergent validity, where related constructs correlate, and discriminant validity, where theoretically different constructs are not excessively correlated.

Criterion-related validity examines association with a relevant outcome or benchmark. Predictive validity concerns future outcomes, such as whether purchase-intention scores relate to later purchases. Concurrent validity compares the measure with a credible measure obtained at the same time.

Internal validity is especially important in experiments and causal claims. It concerns whether an observed effect can reasonably be attributed to the proposed cause rather than confounding, selection, maturation or other explanations.

External validity concerns the extent to which findings can transfer to other people, settings, products or times. Generalisation requires attention to sampling, context and the similarity of the new situation.

Reliability and validity are related but distinct. Random measurement error reduces reliability. Systematic error may produce reliable but invalid results. A scale that always overstates satisfaction by two points could be highly consistent yet inaccurate.

10. An illustrative case: FreshBox meal subscriptions

The following is a fictional illustrative case.

Background

FreshBox, a small startup, planned a weekly meal-subscription service for urban professionals. The founders wanted to estimate demand, identify attractive customer segments and decide whether to launch a low-cost or premium plan.

Problem

The initial proposal was an online poll posted in a fitness group. The team planned to ask 100 members whether they would subscribe. This design risked coverage error because fitness-group members might be more health-conscious than the wider market. It also used a vague question: “Would you definitely buy our healthy meals?”

Analysis

The research adviser defined the target population as adults aged 20–55 living in the launch districts who purchase prepared meals at least twice per month. The team identified three important strata: existing meal-delivery users, occasional users and non-users who expressed interest in convenience food. It also separated exploratory qualitative work from demand estimation.

The team conducted 12 purposive interviews to learn customers’ language, constraints and desired features. It then prepared a structured survey using a screened sample from a local consumer panel, with quotas or probability procedures documented according to the panel’s capabilities. The survey measured recent behaviour, acceptable price ranges, perceived benefits, purchase barriers and purchase intention under a specific product description.

A five-point intention scale was used, but the researchers did not treat a positive intention as equivalent to a purchase. They included a behavioural benchmark question about recent meal-delivery purchases and planned a small pilot launch to compare stated intention with actual conversion.

Approach and outcome

The questionnaire was piloted with 30 people. The pilot revealed that “healthy” meant low-calorie to some respondents and organic to others, so the product description was revised. Two double-barrelled questions were split. A “not applicable” option was added for people who had never used delivery services.

The final report described the sampling frame, response rate, exclusions and limitations. Results suggested stronger interest in a mid-priced plan among frequent delivery users, but the evidence did not justify claiming that all city residents would subscribe. FreshBox launched a limited test in two districts, measured trial, repeat purchase and cancellation, and revised its forecast using observed behaviour.

Learning points

The case illustrates that sample size cannot compensate for poor population definition, that qualitative research can improve measurement before a survey, and that stated intention is not the same as market behaviour. It also shows the value of transparent limitations and staged decision-making.

11. Ethics, governance and contemporary practice

Researchers must obtain informed consent, explain the purpose and expected use of data, minimise collection of sensitive information and protect confidentiality. Participation should be voluntary, with no deceptive pressure. Personal data should be collected, stored and shared according to applicable privacy law and institutional or organisational policy. Identifiers should be separated from responses where possible, access should be restricted, and retention periods should be defined.

Digital recruitment creates additional risks. Online panels, social-media advertising and platform analytics may exclude people with limited internet access and may encourage self-selection. AI tools can assist with translation, coding, questionnaire debugging and pattern detection, but generated text and classifications require human review. Researchers should not upload identifiable participant data to an unapproved system. Automated sentiment analysis may reproduce language, cultural or demographic biases, and its accuracy should be validated against human-coded examples.

Contemporary research increasingly uses mixed methods, mobile surveys, behavioural data and rapid experiments. These tools improve speed but can blur the distinction between observed behaviour, inferred identity and consented research participation. Professional practice requires documentation of data origin, permissions, transformations, exclusions and analytical decisions.

Best-practice checklist

Before fieldwork, ask:

1. Is the target population precisely defined?
2. Does the frame cover that population adequately?
3. Is probability sampling necessary for the decision?
4. Are key subgroups sufficiently represented?
5. Is the proposed sample size linked to precision and cost assumptions?
6. Have likely non-response and design effects been considered?
7. Does each question measure one clearly defined idea?
8. Are the measurement level and planned analysis compatible?
9. Has the instrument been piloted with people like the intended respondents?
10. Is there evidence for reliability and validity?
11. Are privacy, consent, security and data retention addressed?
12. Are limitations reported honestly?

Chapter Summary

Sampling determines which units contribute evidence. Probability methods support statistical inference when selection probabilities are known and implementation is sound; non-probability methods are often valuable for exploration but require cautious claims. Sample size depends on confidence, precision, variability, population size, design and expected response, and formulas apply only under stated assumptions. Sampling error is distinct from non-sampling error, which includes coverage, non-response, response, interviewer, questionnaire and processing problems.

Measurement operationalises constructs through indicators. Nominal, ordinal, interval and ratio levels determine what comparisons and summaries are defensible. Reliability concerns consistency, including stability, inter-rater agreement and internal consistency. Validity concerns the justification of interpretations and uses through content, construct, criterion, internal and external evidence. Reliable measures are not necessarily valid. Ethical research protects participants, documents limitations and uses technology responsibly. For startups, disciplined sampling and measurement improve decisions about customers, positioning, pricing and launch without creating false confidence.

Key Terms

Population: The complete set of units relevant to a research question.

Sample: A subset of the target population selected for study.

Sampling frame: The list or procedure used to identify eligible population members.

Sampling unit: The individual element selected in a sample.

Probability sampling: Sampling in which each eligible unit has a known, non-zero selection probability.

Non-probability sampling: Sampling in which selection probabilities are unknown or non-random.

Stratified sampling: Random sampling within defined population subgroups.

Cluster sampling: Sampling naturally occurring groups and studying units within selected groups.

Coverage error: Error caused by mismatch between the target population and sampling frame.

Non-response error: Bias arising when non-responders differ from responders.

Measurement: Systematic assignment of values or labels according to rules.

Construct: An abstract concept intended for measurement.

Operationalisation: Converting a construct into observable indicators and variables.

Reliability: Consistency or dependability of a measurement procedure.

Validity: Evidence supporting the intended interpretation and use of a measure.

Internal consistency: Coherence among items intended to measure one construct.

Content validity: Adequate coverage of the relevant construct domain.

External validity: The potential for findings to generalise to other contexts.

Design effect: The variance inflation associated with a complex sample design.

Margin of error: The tolerated distance between a sample estimate and a population parameter under stated assumptions.

Review Questions

Short Answer Questions

1. Distinguish a target population from a sampling frame.
2. What is the main difference between probability and non-probability sampling?
3. Name four types of non-sampling error.
4. How does ordinal measurement differ from nominal measurement?
5. Why does reliability not prove validity?

Long Answer Questions

1. Compare simple random, stratified, cluster and systematic sampling, including one advantage and one limitation of each.
2. Explain the assumptions and practical meaning of the formula for estimating a proportion sample size.
3. Discuss how a researcher can improve questionnaire measurement quality before fieldwork.
4. Explain test–retest reliability, inter-rater reliability and internal consistency.
5. Discuss content, construct, criterion-related, internal and external validity in market research.

Analytical / Application Questions

1. A startup wants to estimate the percentage of 2,000 registered users who would renew a subscription. Using 95% confidence, a 5% margin of error and p = 0.50, calculate an approximate finite-population sample size and identify two assumptions that may not hold in practice.
2. A survey receives 1,000 responses from a social-media advertisement, and 72% favour a proposed product. Evaluate whether the startup can claim that 72% of the city’s adults favour it. Identify at least three sources of possible error.
3. Design a sampling and measurement plan for testing demand for a new campus delivery service. Define the population, propose a sampling method, specify two measures, and explain how you would assess reliability and validity.

Multiple Choice Questions with Answers

Question 1

Which feature defines probability sampling?

A. Participants select themselves without eligibility rules

B. Every eligible unit has a known, non-zero probability of selection

C. The researcher selects only expert participants

D. Participants recruit people from their own networks

Correct Answer: B. Every eligible unit has a known, non-zero probability of selection

Question 2

A researcher selects every 20th customer after choosing a random starting point. What method is being used?

A. Systematic sampling

B. Quota sampling

C. Snowball sampling

D. Cluster sampling

Correct Answer: A. Systematic sampling

Question 3

If no prior estimate of a population proportion is available, which value of p is commonly used for conservative planning?

A. 0.05

B. 0.10

C. 0.50

D. 0.95

Correct Answer: C. 0.50

Question 4

Which is an example of coverage error?

A. A respondent misremembers a purchase

B. An interviewer records an answer incorrectly

C. Eligible people without internet access are absent from an online frame

D. A respondent skips a sensitive question

Correct Answer: C. Eligible people without internet access are absent from an online frame

Question 5

Which level of measurement identifies categories without implying order?

A. Nominal

B. Ordinal

C. Interval

D. Ratio

Correct Answer: A. Nominal

Question 6

A measure gives very similar results each time but consistently measures the wrong construct. It is best described as:

A. Valid but unreliable

B. Reliable but invalid

C. Both invalid and unreliable by definition

D. Unmeasurable

Correct Answer: B. Reliable but invalid

Question 7

Which evidence asks whether a measure adequately covers all important parts of a construct?

A. Content validity

B. Predictive validity

C. External validity

D. Inter-rater reliability

Correct Answer: A. Content validity

Question 8

In the design-effect approximation deff ≈ 1 + (m − 1)ρ, what does ρ represent?

A. The confidence level

B. The margin of error

C. The intracluster correlation coefficient

D. The response rate

Correct Answer: C. The intracluster correlation coefficient

Question 9

Which question is double-barrelled?

A. How often did you order delivery last month?

B. Which payment method did you use?

C. How satisfied are you with the app’s speed and reliability?

D. Have you used a delivery app before?

Correct Answer: C. How satisfied are you with the app’s speed and reliability?

Question 10

Which statement about a large convenience sample is most accurate?

A. It automatically eliminates selection bias

B. It guarantees population representativeness

C. It may still describe the wrong population despite having many responses

D. It makes validity irrelevant

Correct Answer: C. It may still describe the wrong population despite having many responses