5. Quantitative Research, Surveys and Measurement

Introduction and Context

Advertising decisions increasingly depend on evidence about audiences, markets, media behaviour and campaign effects. Quantitative research provides a systematic way to convert observations, opinions and behaviours into data that can be compared, analysed and used for decisions. In advertising and media, it helps answer questions such as: How many consumers recall a campaign? Which audience segment has the strongest purchase intention? Does exposure to a digital video advertisement improve brand consideration? What attributes explain satisfaction with a media platform?

A survey may appear to be a simple instrument for asking questions, but a high-quality survey is the visible outcome of a chain of methodological decisions. The researcher must define the problem, identify the construct to be measured, translate it into observable indicators, choose suitable scales, design questions, select respondents, test the instrument and assess the quality of the resulting measures. Errors at any stage can produce misleading findings, even when the sample is large or the analysis is technically sophisticated.

This chapter focuses on the development and evaluation of questionnaires and measurement instruments. It examines constructs, operationalization, levels of measurement, attitude scales, index construction, response effects, pilot testing, reliability and construct validity. These topics are central to advertising analytics because campaign evaluation, brand tracking and media research often depend on measures that cannot be observed directly, such as brand attitude, involvement, trust, perceived relevance or advertising irritation.

Learning Objectives

After studying this chapter, students should be able to:

1. Distinguish between concepts, constructs, variables, indicators and measures.

2. Operationalize abstract advertising and media constructs into measurable variables.

3. Select appropriate nominal, ordinal, interval and ratio scales.

4. Design clear, unbiased and analytically useful questionnaire items.

5. Compare major attitude-scaling approaches and construct multi-item indices.

6. Explain response effects and methods for reducing them.

7. Design and interpret a pilot test.

8. Evaluate reliability and construct validity.

9. Apply measurement principles to campaign and media research decisions.

10. Recognize ethical, technological and managerial risks in survey measurement.

From Research Questions to Measurable Variables

A research question expresses what the study seeks to understand. A construct is an abstract idea used to represent a phenomenon of interest. In advertising research, examples include brand awareness, perceived authenticity, ad involvement, persuasion knowledge, media trust and purchase intention. Constructs are not directly observable in the way that age, number of purchases or clicks may be observable. They must therefore be represented through indicators.

A variable is a characteristic that can take different values across cases. For example, campaign exposure may be measured as the number of times an advertisement was seen, while purchase intention may be measured through responses to a series of statements. An indicator is a specific observable item used to represent a construct. A measurement instrument is the organized set of questions, response options, instructions and procedures used to collect those observations.

The movement from an abstract idea to a measurable form is called operationalization. Consider the construct “perceived advertising relevance.” A researcher might define it as the extent to which an audience member considers an advertisement personally useful, appropriate or connected to current needs. Possible indicators include agreement with statements such as “This advertisement was relevant to me,” “The message addressed a need I currently have,” and “The information in this advertisement was useful.” The operational definition determines what is included and excluded from the construct.

Operationalization requires conceptual precision. If “brand engagement” is defined partly as attention and partly as emotional attachment, the instrument should contain indicators for both dimensions or explicitly treat engagement as multidimensional. An unclear definition may result in an index that combines unrelated ideas and becomes difficult to interpret.

Constructs, Dimensions and Indicators

Some constructs are unidimensional: they represent one relatively coherent attribute. Others are multidimensional. Brand equity, for instance, may include awareness, perceived quality, associations and loyalty. A multidimensional construct should not automatically be measured by simply adding every available item. The researcher must establish whether the dimensions are theoretically related and whether a total score is meaningful.

A useful measurement specification identifies:

• the construct and its conceptual definition;
• its dimensions, if any;
• the population to which it applies;
• the observable indicators;
• the response format and scoring rule; and
• the interpretation of high and low scores.

The distinction between formative and reflective measurement is also important. In a reflective model, the construct is understood to cause the observed indicators. If consumers possess a favourable brand attitude, they may agree with several related statements. In a formative model, the indicators combine to form the construct. For example, media accessibility, content variety and platform convenience may together form a service-value index. Removing a formative indicator may change the meaning of the construct, whereas reflective indicators may be interchangeable only when they capture the same underlying attribute.

Levels of Measurement and Scale Selection

Measurement scales specify how values are assigned to observations and what comparisons are legitimate. Four traditional levels are commonly distinguished.

A nominal scale classifies cases into mutually exclusive categories without implying order. Gender categories, device type, preferred social platform or campaign exposure status may be nominal variables. Numerical codes assigned to categories are labels rather than quantities; a platform coded “3” is not three times another platform coded “1.”

An ordinal scale ranks cases, but the distance between ranks is not assumed to be equal. Examples include agreement rankings, preference order and education categories. If one respondent ranks a brand first and another ranks it third, the difference between those positions cannot automatically be interpreted as a fixed amount of preference.

An interval scale has ordered values with equal intervals, although its zero point is not a true absence of the attribute. Many attitude scales are treated as approximately interval for analytical purposes when they use several balanced response categories and meet suitable assumptions. Researchers should state this decision rather than treating it as automatic.

A ratio scale has equal intervals and a meaningful zero. Advertising expenditure, number of exposures, time spent viewing content, number of purchases and click counts are examples. Ratio comparisons such as “twice as many exposures” are meaningful for these variables.

Scale selection should follow the research purpose. If the objective is classification, nominal measurement may be enough. If ranking is required, ordinal measurement is suitable. If the researcher intends to calculate means, correlations or change scores, a more graduated scale is generally preferable, provided its conceptual and statistical assumptions are defensible.

Designing the Questionnaire

Questionnaire design begins with the information requirement, not with the wording of individual questions. Every item should have a purpose linked to a research objective, construct or classification need. Questions that are interesting but unnecessary increase respondent burden and may reduce data quality.

A well-designed questionnaire normally moves from an introduction and eligibility screening to warm-up questions, core measures, behavioural questions, sensitive questions and demographic classification. This sequence is not universal. In a campaign-exposure study, exposure questions may need to appear before attitude questions to reduce confusion. In a brand-tracking study, unaided awareness should generally be measured before aided awareness because presenting brand names can influence recall.

Questions should be specific, concise and understandable to the target population. Avoid double-barrelled items such as “The advertisement was informative and entertaining,” because a respondent may agree with one attribute but not the other. Avoid double negatives, ambiguous time frames and technical jargon. “How often do you use streaming media?” is incomplete unless a period is specified, such as “during the past seven days.”

A leading question contains language that encourages a particular answer: “How effective was our excellent campaign?” A loaded question embeds a value judgement: “Do you support wasteful advertising practices?” A better formulation is neutral and allows respondents to express their own evaluation.

Researchers must decide whether to use open-ended or closed-ended questions. Open-ended questions can reveal unexpected associations and are useful for unaided recall, but they require coding and may produce brief or incomplete answers. Closed-ended questions facilitate comparison and statistical analysis, but their categories may constrain responses. “Other, please specify” can provide a useful escape option where the list may not be exhaustive.

Response options should be mutually exclusive, collectively exhaustive and appropriate to the question. A frequency question should not combine overlapping categories such as “1–3 times” and “3–5 times.” When respondents may not know or may not wish to answer, “don’t know” and “prefer not to say” should be considered, especially for sensitive questions. However, these options should not be used carelessly to avoid designing a clear question.

Attitude Scales and Index Construction

Attitudes are commonly measured with multi-item scales because a single question may not capture the complexity of an evaluative orientation. A Likert scale asks respondents to indicate their degree of agreement with statements, often from strongly disagree to strongly agree. Items should be written so that agreement has a consistent conceptual meaning. If negatively worded items are included, they must be carefully reverse-coded and tested for comprehension; excessive use of reverse wording can create confusion rather than reduce acquiescence.

A semantic differential scale asks respondents to position an object between bipolar adjectives, such as untrustworthy–trustworthy, old-fashioned–modern or irrelevant–relevant. The two adjectives must be meaningful opposites for the target audience. A numerical rating scale asks respondents to evaluate an object on a defined continuum, such as 0 to 10 for likelihood to recommend.

A ranking scale requires respondents to order alternatives. It is useful when the research question concerns relative preference, but ranking can become burdensome with many alternatives and does not show the size of differences between positions. A constant-sum scale asks respondents to distribute a fixed number of points across attributes, revealing relative importance but requiring more cognitive effort.

An index combines scores from multiple indicators into a summary measure. Before constructing an index, the researcher should verify that the items refer to the same conceptual domain, use compatible response formats and have a defensible scoring direction. If a five-item attitude scale uses responses from 1 to 5, a simple sum or mean may be calculated after reverse-scoring any negatively oriented item. A mean score is often easier to interpret because it remains on the original response scale.

Suppose a campaign attitude index contains four items scored from 1 to 7. If the item scores are 6, 5, 6 and 4, the mean index is 5.25. This value is meaningful only in relation to the scale anchors, the target population and the measurement design. It should not automatically be interpreted as a percentage or as proof that the campaign caused a favourable attitude.

Weighting items equally is convenient, but not always justified. If theory indicates that one dimension is more important, a weighted index may be appropriate. However, weights should be established before examining outcomes where possible, documented clearly and subjected to sensitivity analysis. Data-driven weighting can produce an index that fits one sample but performs poorly elsewhere.

Response Effects and Sources of Measurement Error

Observed responses contain both systematic and random error. Random error produces inconsistency, while systematic error pushes responses in a particular direction. Measurement error may result from the respondent, the questionnaire, the interviewer, the mode of administration or the context of data collection.

Acquiescence bias is the tendency to agree with statements regardless of content. Social desirability bias occurs when respondents give answers they believe are socially acceptable. Recall error affects questions about past exposure or behaviour, particularly when the time period is long or the behaviour is routine. Satisficing occurs when respondents select a convenient answer rather than carefully considering the question. Straight-lining, speeding and patterned responding are observable signs in online surveys, although they should be interpreted alongside substantive evidence.

Order effects arise when earlier questions influence later responses. Priming respondents with brand names may increase recognition. Contrast effects can occur when an item is evaluated differently after an extremely positive or negative item. Mode effects refer to differences caused by online, mobile, telephone, face-to-face or self-administered formats. Interviewer presence may increase social desirability, while a small mobile screen may increase missing data or accidental selection.

Researchers can reduce response effects by using neutral language, clear definitions, short recall periods, appropriate randomization, balanced response options, confidentiality assurances and suitable self-administration. Randomizing answer options can reduce position bias, but it should not be used where a logical order is necessary, such as a frequency or time scale. Attention checks may identify careless responding, but poorly designed checks can exclude thoughtful respondents or create frustration.

Pilot Testing as Instrument Development

A pilot test is not merely a small version of the final survey. It is a diagnostic stage used to identify comprehension problems, missing response options, excessive length, technical failures and weaknesses in the measurement model. Cognitive interviewing can be especially valuable. The researcher asks participants to explain how they interpreted a question, what information they used and how they selected their answer. This reveals misunderstandings that cannot be detected from final response distributions alone.

A pilot should resemble the intended population and mode of administration sufficiently to expose realistic problems. Researchers should examine completion time, item nonresponse, inconsistent routing, concentration at scale endpoints, straight-lining and respondent comments. Preliminary reliability analysis may be conducted for multi-item measures, but a small pilot should not be treated as definitive evidence of validity.

Pilot findings may lead to rewriting, deleting, adding or reordering items. Such changes should be recorded in an instrument-development log. If a substantial change is made, the revised instrument should be tested again. Translation and adaptation require additional review, including back-translation or committee comparison where appropriate, because literal translation may not preserve conceptual meaning.

Reliability: Consistency of Measurement

Reliability concerns the consistency of a measurement procedure. Internal consistency asks whether items intended to measure the same construct produce related responses. Cronbach’s alpha is commonly reported, but it is not a universal quality certificate. Alpha is influenced by the number of items, item covariance and assumptions such as tau-equivalence. A high value may result from redundant items, while a low value may reflect multidimensionality, too few items or poor wording.

Other approaches include test–retest reliability, which examines stability over time, and inter-rater reliability, which is relevant when human coders classify content or responses. McDonald’s omega may provide a more appropriate estimate under some factor structures. Reliability should be interpreted with the construct and context in mind. A measure of a temporary emotional response may reasonably be less stable than a measure of a relatively enduring preference.

Reliability is necessary but not sufficient. A scale can be highly consistent and still measure the wrong construct. For example, respondents may consistently agree with several items that capture advertising entertainment while the researcher claims to measure advertising credibility.

Construct Validity and Evidence for Measurement Quality

Construct validity concerns whether an instrument measures the theoretical construct it is intended to measure. It is supported through multiple forms of evidence rather than established by one statistic.

Content validity asks whether the measure adequately covers the relevant domain. Literature review, expert assessment and consultation with practitioners can help determine whether important dimensions have been omitted. Face validity is the more limited judgement that items appear appropriate to respondents or reviewers; it may be useful diagnostically but is not strong evidence by itself.

Convergent validity is supported when measures that should be related are positively associated. Discriminant validity is supported when a measure is not excessively related to theoretically distinct constructs. Factor analysis can help examine whether items form the expected dimensions. Exploratory factor analysis is useful during scale development, whereas confirmatory factor analysis tests a specified measurement structure using new or suitably separated data.

Criterion-related validity concerns association with an external criterion. A measure of purchase intention may be examined in relation to subsequent purchase, although intention and behaviour are not identical. Predictive evidence should be interpreted cautiously because outcomes may be affected by price, availability, distribution and other factors.

Measurement invariance is important when comparing groups, languages, countries, devices or time periods. An apparent difference in brand attitude may result from respondents interpreting the scale differently. Researchers should examine whether the same construct and scale function comparably before making strong cross-group comparisons.

An Illustrative Case: Evaluating a Streaming-Service Campaign

Background

A fictional streaming platform, StreamPlus, launches a campaign positioning its service as “the home of stories that matter.” The marketing team wants to know whether the campaign improves perceived relevance and subscription intention among urban adults aged 21–40. The team initially proposes a single question: “How much did you like the campaign?”

Situation and Problem

The proposed item does not distinguish liking from relevance, credibility, emotional response or behavioural intention. It also does not establish whether respondents saw the campaign. Management needs a measurement instrument that can compare exposed and unexposed audiences and support decisions about creative refinement and media allocation.

Analysis

The research team defines three focal constructs: perceived campaign relevance, brand attitude and subscription intention. Relevance is treated as a reflective construct represented by usefulness, personal connection and appropriateness. Subscription intention is measured separately because a favourable advertising response does not necessarily produce an intention to subscribe. The team also identifies campaign recognition, unaided recall, recent subscription behaviour and category usage as relevant variables.

Approach and Solution

The questionnaire begins with eligibility and recent streaming-use questions. Unaided campaign recall is asked before showing any brand or campaign cues. Respondents are then asked whether they recognize the campaign, followed by a standardized exposure description. Perceived relevance is measured with four seven-point agreement items, including “The campaign felt relevant to my viewing interests.” Brand attitude is measured with semantic differential pairs such as unfavourable–favourable and untrustworthy–trustworthy. Subscription intention uses a separate five-point likelihood item.

The team conducts cognitive interviews with 12 participants and a pilot with 80 respondents. Interviews reveal that “stories that matter” is interpreted by some participants as socially responsible content and by others as emotionally moving content. The researchers revise the questionnaire to avoid treating the slogan itself as a clear measure of relevance. The pilot identifies a missing “not applicable” option for respondents who do not pay for streaming services and shows that one negatively worded item produces inconsistent answers. That item is removed.

After the main survey, the research team examines missing data, response duration, item distributions and internal consistency. Factor analysis supports separate relevance and brand-attitude dimensions. The relevance index is calculated as the mean of four compatible items after confirming their direction and reliability. Exposed respondents report higher mean relevance than unexposed respondents, but the team avoids claiming that exposure alone caused the difference because the design is observational. Differences in prior interest in StreamPlus and genre preference are considered in the analysis.

Outcome

The final instrument provides management with more actionable evidence than the original liking question. Campaign recognition is adequate, but perceived relevance is high only among respondents already interested in the platform’s content categories. The creative team therefore develops alternative executions for less-engaged segments, while the media team examines whether reach is being delivered to audiences with a realistic category need.

Learning Points

The case demonstrates that measurement begins with construct definition, not with selecting a familiar question. It also shows why exposure, advertising response and behavioural intention should not be collapsed into one score. Pilot testing identified a semantic ambiguity and a problematic item, while validity reasoning prevented the organization from interpreting correlation as campaign causation.

Professional and Ethical Considerations

Survey quality is also a governance issue. Respondents should receive understandable information about the study purpose, data use, voluntary participation and withdrawal where applicable. Personal identifiers should be separated from analytical data whenever possible, and sensitive information should be collected only when necessary. Researchers must follow relevant privacy and data-protection requirements, organizational policies and professional codes.

Advertising researchers should be especially cautious when collecting data from minors, profiling vulnerable audiences or linking survey responses with digital behavioural records. Consent for research participation does not automatically authorize unrelated marketing use. Transparency is required when automated systems, panels, cookies or identity-resolution tools contribute to sampling or measurement.

The sponsor’s commercial interest should not determine the wording of questions, selective reporting of favourable results or suppression of limitations. A reliable and valid instrument can still be used unethically if findings are exaggerated or presented without uncertainty. Reports should document the sample, fieldwork mode, questionnaire wording, exclusions, weighting, missing-data treatment and relevant validity limitations.

Digital Surveys, Automation and Emerging Practice

Digital survey platforms support branching, randomization, multilingual deployment, real-time monitoring and integration with dashboards. These capabilities improve operational efficiency but do not solve conceptual weaknesses. Automated logic can route respondents correctly only when eligibility and construct definitions have been designed correctly.

Machine learning may assist with open-ended response coding, fraud detection and prediction of likely nonresponse. Large language models can help generate alternative wording or identify possible ambiguity, but human researchers must verify conceptual equivalence, cultural appropriateness, privacy implications and bias. Synthetic respondents or simulated data may be useful for testing survey flow, but they are not substitutes for evidence from the target population.

Contemporary measurement increasingly combines survey responses with passive behavioural data, media-exposure logs and experiments. Such integration can improve insight, but the data sources may measure different constructs. A video completion is not necessarily attention, and a click is not necessarily persuasion. Triangulation is valuable only when the meaning, timing and limitations of each measure are understood.

Best Practices for Advertising and Media Researchers

A defensible instrument follows several principles:

• Begin with a precise decision problem and construct definition.
• Map every item to a research objective or analytical requirement.
• Use established scales where their meaning and performance fit the context, while documenting adaptations.
• Keep recall periods short and wording neutral.
• Separate exposure, attitudes, intentions and behaviours.
• Pretest through cognitive interviews and a realistic pilot.
• Examine reliability together with content, factor, convergent and discriminant validity.
• Test comparability before making group or time-based claims.
• Protect confidentiality and report limitations honestly.
• Treat statistical sophistication as complementary to, not a replacement for, sound measurement design.

Chapter Summary

Quantitative survey research transforms theoretical concepts into observable evidence. The central task is measurement: defining constructs, identifying dimensions, selecting indicators and choosing scales that support appropriate comparisons. Questionnaire quality depends on clear wording, logical sequencing, balanced response options and control of response effects. Likert, semantic differential, ranking and constant-sum methods serve different purposes, while indices require justified item selection and scoring.

Pilot testing reveals comprehension, routing and burden problems before full data collection. Reliability assesses consistency, but a reliable instrument may still lack validity. Construct validity requires evidence concerning content coverage, factor structure, convergence, discrimination, external criteria and measurement invariance. In advertising and media, these principles support more credible campaign evaluation, brand tracking and audience analytics. Ethical governance, privacy protection and cautious interpretation remain essential, particularly as surveys are integrated with digital behavioural data and automated technologies.

Key Terms

Construct: An abstract theoretical attribute that cannot usually be observed directly.

Operationalization: The process of translating a construct into observable indicators and measurement procedures.

Variable: A characteristic that can take different values across cases.

Indicator: An observable item used to represent a construct.

Nominal scale: A classification scale with categories but no inherent order.

Ordinal scale: A scale that ranks observations without assuming equal distances between ranks.

Interval scale: A scale with ordered values and equal intervals but no meaningful absolute zero.

Ratio scale: A scale with equal intervals and a meaningful zero point.

Likert scale: An agreement-based attitude scale using ordered response categories.

Semantic differential: A scale positioning an object between bipolar descriptive adjectives.

Index: A composite score formed by combining multiple indicators.

Reflective measure: A measure in which the construct is theorized to cause the observed indicators.

Formative measure: A measure in which indicators combine to create the construct.

Response bias: Systematic distortion in answers caused by wording, context, respondent or administration conditions.

Pilot test: A preliminary administration used to diagnose and improve an instrument.

Reliability: The consistency or stability of a measurement procedure.

Construct validity: Evidence that a measure represents the theoretical construct it is intended to measure.

Convergent validity: Evidence that theoretically related measures are associated.

Discriminant validity: Evidence that a measure is distinct from theoretically different constructs.

Measurement invariance: Comparable functioning of a measure across groups, settings or time periods.

Review Questions

Short Answer Questions

1. What is the difference between a construct and an indicator?

2. Define operationalization in the context of advertising research.

3. Distinguish nominal measurement from ordinal measurement.

4. Why can a high reliability coefficient fail to establish validity?

5. What is the purpose of cognitive interviewing during pilot testing?

Long Answer Questions

1. Explain the process of developing a multi-item measure of perceived advertising relevance.

2. Compare Likert, semantic differential, ranking and constant-sum scales, indicating when each may be appropriate.

3. Discuss major sources of response effects in online advertising surveys and methods for reducing them.

4. Explain reliability and construct validity, including the main forms of evidence used to assess them.

5. Describe how questionnaire design, pilot testing and measurement invariance affect the credibility of a brand-tracking study.

Analytical / Application Questions

1. A campaign survey uses the item “The advertisement was informative, entertaining and trustworthy.” Identify the measurement problems and rewrite the instrument design.

2. A brand-attitude index has six items, but factor analysis shows two distinct dimensions: emotional appeal and credibility. Should the researcher report one total index? Justify your answer.

3. Exposed respondents show stronger subscription intention than unexposed respondents, but they also had higher prior interest in the brand. Explain how the finding should be interpreted and what additional design or analysis could improve the conclusion.

22. Multiple Choice Questions with Answers

Question 1

A researcher defines perceived advertising relevance as personal usefulness, connection to current needs and appropriateness. Which step is this primarily illustrating?

A. Operationalization of a construct

B. Random sampling

C. Inter-rater reliability

D. Criterion weighting

Correct Answer: A. Operationalization of a construct

Question 2

Which variable is measured on a ratio scale?

A. Preferred social platform

B. Brand preference rank

C. Number of advertisement exposures

D. Agreement with a brand statement

Correct Answer: C. Number of advertisement exposures

Question 3

Why is the item “The advertisement was informative and entertaining” methodologically weak?

A. It uses a ratio scale

B. It is double-barrelled

C. It lacks a demographic category

D. It measures behaviour rather than attitude

Correct Answer: B. It is double-barrelled

Question 4

A semantic differential scale is most appropriately represented by which pair?

A. Yes–No

B. First–Second–Third

C. Untrustworthy–Trustworthy

D. Never–Sometimes–Often

Correct Answer: C. Untrustworthy–Trustworthy

Question 5

What is the main purpose of a pilot test?

A. To guarantee causal validity

B. To replace the main sample

C. To diagnose and improve the instrument

D. To eliminate the need for reliability analysis

Correct Answer: C. To diagnose and improve the instrument

Question 6

A scale produces highly consistent scores but its items measure advertising entertainment rather than credibility. What does this demonstrate?

A. Validity always guarantees reliability

B. Reliability is necessary but does not ensure validity

C. The scale has perfect discriminant validity

D. The sample must be representative

Correct Answer: B. Reliability is necessary but does not ensure validity

Question 7

Which finding would support convergent validity for a brand-trust scale?

A. It is unrelated to every other measure

B. It correlates with a theoretically related trust measure

C. It produces different meanings across languages

D. Its items use different response directions

Correct Answer: B. It correlates with a theoretically related trust measure

Question 8

A researcher asks unaided campaign recall only after showing respondents a list of brand names. What problem is most likely?

A. Priming and order effects

B. Inter-rater disagreement

C. Ratio-scale distortion

D. Formative measurement

Correct Answer: A. Priming and order effects

Question 9

When indicators such as content variety, accessibility and convenience together create a service-value score, the measure is most plausibly:

A. Reflective

B. Formative

C. Nominal

D. Unaided

Correct Answer: B. Formative

Question 10

A higher campaign score among exposed respondents may not prove that exposure caused the result because:

A. Attitude scales cannot be analysed

B. Exposure groups may differ in prior brand interest

C. All surveys are necessarily invalid

D. Ratio variables cannot be compared

Correct Answer: B. Exposure groups may differ in prior brand interest