3. Research Designs and Evidence Quality

Introduction: Why Design Determines What Evidence Can Mean

Advertising decisions increasingly depend on evidence: audience profiles, brand-lift estimates, platform dashboards, experiments, social listening, interviews, and customer data. Yet data do not speak for themselves. The credibility of a conclusion depends on how the research question was framed, how observations were produced, which alternatives were considered, and what limitations remain.

Research design is the logical plan connecting a research question to evidence and ultimately to a decision. In advertising and media research, design determines whether a study can discover an unfamiliar consumer motivation, describe the size or distribution of a behaviour, or establish whether a campaign caused a change. It also determines the extent to which findings can be generalised across audiences, markets, devices, and time.

A high-performing campaign may coincide with an increase in sales without causing it. A survey may produce precise percentages while systematically excluding non-users of the survey platform. An interview may reveal a powerful insight without estimating how widespread that insight is. Evidence quality therefore requires more than statistical sophistication. It requires alignment among the research purpose, design, measurement, sampling, analysis, and interpretation.

Learning Objectives

By the end of this chapter, students should be able to:

• distinguish exploratory, descriptive, and causal research designs;
• select appropriate cross-sectional, longitudinal, observational, experimental, and mixed-method approaches;
• explain internal, external, construct, and statistical conclusion validity;
• assess reliability, measurement quality, sampling error, non-sampling error, and bias;
• interpret evidence cautiously in advertising and media decisions;
• identify ethical, technological, and governance issues in contemporary research; and
• design a defensible research approach for a practical advertising problem.

From Business Problem to Research Design

A business problem is usually expressed in managerial language: declining consideration, inefficient media spend, weak campaign resonance, or poor retention. A research problem translates that concern into a question that can be investigated. For example, “Why is the campaign underperforming?” may become: “How do target consumers interpret the campaign promise, and does exposure to the revised creative increase brand consideration relative to a control condition?”

The design should follow the information need, not the researcher’s preferred method. Four decisions are central. First, what is the unit of analysis—individual, household, advertisement, impression, market, or time period? Second, what is the target population? Third, is the purpose discovery, description, explanation, prediction, or evaluation? Fourth, what level of evidence is required for the decision?

A useful distinction is between a construct and its measure. “Brand trust” is a construct; agreement with a set of carefully worded survey items is one possible operationalisation. “Exposure” may mean an opportunity to see an advertisement, a verified impression, self-reported recall, or viewing completion. Different operationalisations can produce different findings, even when they appear to measure the same concept.

Three Purposes of Inquiry

Exploratory Designs: Making the Unfamiliar Understandable

Exploratory research is appropriate when the problem is poorly understood, the relevant variables are uncertain, or existing categories may be inadequate. Its aim is insight generation rather than population estimation or definitive causal proof. Common approaches include depth interviews, focus groups, ethnography, diary studies, social listening, semiotic analysis, and review of secondary sources.

Exploration is especially useful at the beginning of creative development. Interviews may reveal that consumers do not reject a product’s price but distrust its claims. Observation may show that shoppers ignore a supposedly prominent display because they navigate directly to familiar brands. Such findings help researchers develop hypotheses, language, stimuli, and survey measures.

Exploratory samples are usually purposive rather than statistically representative. Researchers seek variation and relevance—for example, heavy users, lapsed users, category rejecters, or influential intermediaries. Saturation is a practical guide: data collection may stop when additional cases produce little new conceptual insight. Saturation, however, is not the same as statistical representativeness.

Exploratory evidence is vulnerable to moderator influence, participant self-presentation, selective interpretation, and overgeneralisation. A vivid quotation is not evidence that a majority of consumers share the same view. The appropriate output is a set of themes, hypotheses, mechanisms, or design implications that can be examined further.

Descriptive Designs: Establishing What Is Happening

Descriptive research measures the incidence, distribution, characteristics, or associations of phenomena. It answers questions such as: What proportion of the target audience recognises the brand? Which segments use short-form video most frequently? What is the distribution of media consumption across dayparts? How does awareness vary by region?

Surveys, audience measurement systems, content audits, database analysis, and structured observation are common descriptive methods. A cross-sectional survey can estimate attitudes at one point in time, while a repeated survey can monitor trends. Descriptive research may identify associations, but association alone does not establish causation.

Good description depends on operational definitions, valid sampling, adequate sample size, and transparent reporting. A percentage without its denominator, fieldwork period, question wording, or eligibility criteria is difficult to interpret. Likewise, a dashboard metric such as clicks may describe platform activity without representing persuasion, purchase intention, or incremental sales.

Causal Designs: Testing What Produces Change

Causal research asks whether a change in one factor produces a change in another. In advertising, examples include whether a new message increases consideration, whether frequency produces incremental reach, or whether a placement generates additional conversions.

A causal claim requires more than temporal sequence. Three conditions are normally important: the presumed cause precedes the outcome; the cause and outcome covary; and plausible alternative explanations are controlled or assessed. Randomised experiments provide the strongest basis for causal inference because random assignment tends to balance observed and unobserved differences between treatment and control groups.

In a simple advertising experiment, eligible participants may be randomly assigned to view either a campaign advertisement or a control stimulus. Brand consideration is then measured under the same conditions. The estimated treatment effect is the difference in average outcomes between groups. Field experiments may randomise users, geographic areas, publishers, or time periods. Quasi-experiments, such as difference-in-differences designs, are used when randomisation is not feasible but a credible comparison group and time structure exist.

Causal interpretation remains conditional. Noncompliance, contamination, selective attrition, weak treatment delivery, inadequate statistical power, and interference between participants can weaken an experiment. A laboratory result may not transfer to real media environments, while a field result may be difficult to reproduce in another market.

Time, Behaviour, and the Choice of Design

Cross-Sectional Research

Cross-sectional research collects data at one point or within a short fieldwork window. It is efficient for profiling audiences, measuring current awareness, and comparing segments. Its main weakness is limited temporal inference. If people who recall an advertisement report higher purchase intention, the design cannot determine whether exposure increased intention or whether people already interested in the brand were more likely to notice and remember the advertisement.

Longitudinal Research

Longitudinal research observes change over time. A panel follows the same units, allowing analysis of individual-level change, while a repeated cross-sectional design surveys different samples from the same population at several time points. Tracking studies are valuable for monitoring awareness, consideration, preference, and brand associations across a campaign.

Longitudinal studies introduce attrition, panel conditioning, maturation, history effects, and measurement changes. Participants who remain in a panel may differ from those who leave. Repeatedly answering brand questions may itself increase awareness. Researchers should document recruitment, retention, weighting, questionnaire consistency, and changes in the target population.

Observational Research

Observational designs measure behaviour without assigning an intervention. They include naturalistic observation, passive digital measurement, social listening, retailer data, and analysis of campaign logs. Their strength is behavioural realism, especially when self-reports are affected by memory or social desirability. Their central challenge is confounding: users who see more advertising may also differ in need, income, prior brand preference, or media intensity.

Statistical adjustment, matching, stratification, instrumental variables, or natural experiments can improve inference, but adjustment cannot automatically eliminate unmeasured confounding. Observational findings should therefore be described precisely—for example, “exposure was associated with higher conversion” rather than “exposure caused conversion,” unless the design supports the stronger claim.

Mixed-Method Designs

Mixed-method research intentionally integrates qualitative and quantitative evidence. A sequential exploratory design may begin with interviews, use the findings to construct a survey, and then test the prevalence of identified themes. A sequential explanatory design may begin with a survey and follow it with interviews to explain unexpected patterns. A convergent design collects qualitative and quantitative data in parallel and compares their interpretations.

Integration is more than placing two methods in the same report. Researchers should specify where integration occurs: in sampling, instrument development, analysis, or interpretation. If survey results show low trust but interviews reveal that respondents interpret “trust” as product safety rather than corporate honesty, the construct itself may require revision. Mixed methods are particularly useful when managerial decisions require both scale and meaning.

A Framework for Evidence Quality

Validity

Construct validity concerns whether a measure represents the intended concept. A campaign may claim to measure engagement but use only video completion, overlooking attention, emotional response, or active processing. Content validity asks whether important dimensions of a construct are included. Convergent and discriminant evidence examine whether a measure relates strongly to similar constructs and weakly to distinct ones.

Internal validity concerns whether a study supports its explanation of observed effects. Threats include selection differences, history, maturation, testing effects, regression to the mean, instrumentation changes, attrition, and concurrent marketing activity. Randomisation, control groups, pre-registration, consistent measurement, and appropriate statistical models strengthen internal validity.

External validity concerns transferability to other people, settings, messages, platforms, and times. A highly controlled experiment may have strong internal validity but limited realism. A field study may be realistic but less controlled. Generalisation should be argued from the population, context, treatment, and outcome—not assumed from sample size alone.

Statistical conclusion validity concerns whether the analysis supports the stated relationship. Low power, unreliable measures, multiple comparisons, model misspecification, missing data, and inappropriate standard errors can produce misleading conclusions. Statistical significance is not equivalent to practical importance; a small effect may be precisely estimated yet strategically irrelevant.

Reliability and Measurement Error

Reliability refers to consistency of measurement. Internal consistency examines whether items intended to measure one construct produce coherent responses. Test–retest reliability assesses stability over time when the construct is expected to remain stable. Inter-rater reliability is important in coding advertisements, comments, or observed behaviours.

Reliability is necessary but not sufficient for validity. A consistently miscalibrated instrument can be reliable without measuring the intended construct. Researchers should pilot instruments, use clear wording, avoid double-barrelled questions, balance response options, define coding rules, and assess whether respondents understand the questions as intended.

Measurement error may be random or systematic. Random error reduces precision. Systematic error shifts results in a consistent direction, as when respondents underreport undesirable media behaviour. Recall error, acquiescence, leading questions, device effects, interviewer effects, and poorly defined exposure measures are common in advertising research.

Sampling, Bias, and Error

Sampling error is the difference between a sample estimate and the population value arising from selecting a sample rather than measuring everyone. It can be quantified under probability sampling assumptions. Non-sampling errors include coverage error, nonresponse error, response error, processing error, and specification error.

Coverage error occurs when the sampling frame excludes relevant units—for example, an online panel that underrepresents people with limited internet access. Nonresponse bias occurs when participation or completion is related to the outcome of interest. Weighting may correct known imbalances, but it cannot guarantee correction for unmeasured differences.

Selection bias can enter at recruitment, exposure, retention, or analysis stages. Survivorship bias appears when only successful campaigns or retained users are examined. Confirmation bias may lead a team to favour findings supporting an existing creative strategy. Publication or reporting bias may arise when null or negative results receive less attention.

Researchers should report eligibility, recruitment, response or completion rates, exclusions, weighting, missing-data treatment, and uncertainty. Transparency allows decision-makers to judge how much confidence is warranted.

Designing a Defensible Advertising Study

A practical design process can be organised into seven connected decisions:

1. Define the decision and research question precisely.

2. Identify constructs, outcomes, units of analysis, and target population.

3. Select the design that matches the required inference—exploratory, descriptive, causal, or integrated.

4. Develop a sampling and recruitment plan, including inclusion and exclusion criteria.

5. Pretest stimuli, questions, coding procedures, and technical delivery.

6. Establish an analysis plan before inspecting results where feasible.

7. Interpret findings alongside uncertainty, limitations, alternative explanations, and managerial relevance.

For example, a brand considering a new sustainability message may first conduct interviews to identify credible and sceptical interpretations. A survey can then estimate which interpretations are prevalent across segments. Finally, a randomised experiment can compare the sustainability message with the existing message on brand trust and purchase intention. Each phase answers a different question; none should be presented as a substitute for the others.

Illustrative Case: Evaluating a Video Campaign for a Subscription Service

Background

A fictional subscription-based learning service launches a six-week video campaign aimed at young professionals. Platform reports show strong completion rates and a rise in website visits. Management wants to know whether the campaign increased subscriptions.

Situation and Problem

A simple before-and-after comparison shows that subscriptions rose during the campaign. However, the period also coincided with a seasonal promotion and a competitor’s service outage. In addition, the platform’s exposed audience is more likely to have visited the website previously. The central problem is therefore not whether subscriptions rose, but whether the campaign generated incremental subscriptions.

Analysis and Approach

The research team begins with exploratory interviews among exposed and unexposed users to examine how the message is interpreted and whether “career advancement” is a credible benefit. A cross-sectional survey then measures awareness, message comprehension, perceived relevance, and subscription intention, while distinguishing self-reported exposure from verified delivery data.

For causal evaluation, eligible geographic areas are randomly assigned to campaign-on or campaign-off conditions for the test period, while the promotion is held constant. Subscription outcomes are measured using the same attribution window in both groups. The team pre-specifies the primary outcome, checks whether delivery differed by area, monitors spillover, and reports confidence intervals rather than relying only on a significance threshold.

Outcome

Suppose the campaign-on areas show a modest increase in subscriptions, but the effect is concentrated among people who had previously visited the website. This supports an incremental retargeting effect more strongly than a broad prospecting effect. Interviews also reveal that the phrase “advance your career” is interpreted as unrealistic by some non-subscribers, suggesting a creative revision.

Learning Points

The platform’s completion metric was useful for describing delivery but insufficient for establishing persuasion. The before-and-after increase was confounded by concurrent events. Mixed methods explained not only whether an effect occurred but also for whom and why. The study’s conclusion should remain bounded: the estimated effect applies to the tested audience, message, period, and implementation conditions.

Ethics, Governance, and Contemporary Practice

Advertising research involves personal data, inferred interests, identity information, and sometimes sensitive attributes. Ethical practice requires informed consent where appropriate, data minimisation, purpose limitation, secure storage, controlled access, and transparent retention policies. Researchers should avoid disguising promotional activity as neutral research and should clearly distinguish research participation from advertising exposure.

Digital measurement creates additional concerns. Cookies, device identifiers, clean rooms, platform APIs, and machine-learning models may enable audience analysis while reducing individual visibility into data use. Automated sentiment or attention measures can encode linguistic, cultural, or demographic bias. Synthetic respondents and generative AI may assist with stimulus development or coding, but they cannot be treated as substitutes for empirical participants without validation.

Governance should include documentation of data provenance, variable definitions, model assumptions, version changes, human review, and escalation procedures. Privacy-preserving aggregation, differential privacy, secure computation, and controlled experimentation can reduce risk, but technical safeguards do not remove the need for lawful and ethically justified research.

Common Mistakes and Better Practice

A frequent mistake is treating correlation as causation. The remedy is to seek randomisation, credible comparison groups, temporal structure, and sensitivity analyses. Another is choosing a large sample with weak measurement. A smaller, well-designed study may provide more useful evidence than a large biased survey.

Researchers also overstate generalisability, ignore missing data, change outcomes after seeing results, and report only favourable segments. Better practice includes pre-specifying key outcomes, distinguishing exploratory from confirmatory analysis, reporting null results, conducting robustness checks, and presenting limitations as part of the finding rather than as an afterthought.

The strongest professional reports separate observation, interpretation, and recommendation. “Recall was higher among exposed respondents” is an observation. “Exposure may have improved salience” is an interpretation. “Increase reach among this segment” is a recommendation that requires consideration of cost, feasibility, and alternative evidence.

Current Directions

Advertising research is moving toward integrated measurement across platforms, privacy-conscious attribution, incrementality testing, attention research, and real-time experimentation. These developments increase analytical capability but also intensify problems of fragmented identity, changing platform definitions, algorithmic allocation, and reproducibility. As automated systems optimise delivery, researchers must distinguish platform-reported performance from independently established causal impact.

The central professional skill is evidence calibration: matching the strength of a claim to the strength of the design. Sophisticated analytics cannot repair an ambiguous question, biased sample, invalid measure, or unexamined alternative explanation.

Chapter Summary

Research design is the plan linking a question to evidence and decision. Exploratory designs generate concepts and hypotheses; descriptive designs estimate distributions and associations; causal designs test whether an intervention produces change. Cross-sectional studies provide snapshots, longitudinal studies examine change, observational studies capture naturally occurring behaviour, and mixed-method designs integrate scale with meaning.

Evidence quality depends on validity, reliability, sampling quality, measurement accuracy, transparent analysis, and appropriate interpretation. Internal validity concerns causal explanation, external validity concerns transferability, construct validity concerns measurement, and statistical conclusion validity concerns analytical support. Bias and error can arise through sampling, nonresponse, measurement, attrition, platform data, and researcher interpretation. Ethical governance, privacy protection, documentation, and responsible use of automation are essential to contemporary advertising analytics.

Key Terms

Research design: The logical plan connecting a research question, data collection, analysis, and inference.

Exploratory research: Inquiry intended to develop understanding, concepts, themes, or hypotheses.

Descriptive research: Research that measures characteristics, distributions, frequencies, or associations.

Causal research: Research designed to assess whether one factor produces a change in another.

Cross-sectional design: A design collecting data at one point or within a limited period.

Longitudinal design: A design observing units or populations across multiple time points.

Panel study: A longitudinal study that follows the same participants or units.

Observational research: Research measuring naturally occurring behaviour without assigning an intervention.

Experiment: A design in which an intervention is deliberately assigned and outcomes are compared.

Mixed-method research: Intentional integration of qualitative and quantitative approaches.

Construct validity: The extent to which a measure represents the intended theoretical construct.

Internal validity: The credibility of a study’s explanation of observed effects.

External validity: The extent to which findings transfer to other contexts or populations.

Reliability: The consistency or stability of a measurement process.

Measurement error: Difference between an observed measure and the intended value.

Sampling error: Difference caused by studying a sample rather than the entire population.

Nonresponse bias: Bias arising when participation or completion differs systematically from nonparticipants.

Confounding: A situation in which an alternative factor influences both presumed cause and outcome.

Incrementality: The outcome attributable to an intervention beyond what would have occurred without it.

Evidence calibration: Matching the strength of a conclusion to the quality and limits of its evidence.

Review Questions

Short Answer Questions

1. What distinguishes exploratory research from descriptive research?

2. Why does a cross-sectional association not usually establish causation?

3. Define internal validity and identify one threat to it.

4. What is the difference between reliability and validity?

5. Why is a platform completion metric insufficient by itself to prove advertising effectiveness?

Long Answer Questions

1. Compare exploratory, descriptive, and causal designs in terms of purpose, methods, outputs, and limitations.

2. Explain how cross-sectional, longitudinal, and observational designs can be used in advertising research.

3. Discuss the four forms of validity and show how each affects managerial interpretation.

4. Explain sampling error, non-sampling error, and bias using advertising research examples.

5. Describe how a mixed-method study could evaluate a new advertising message from insight generation through causal testing.

Analytical / Application Questions

1. A campaign is followed by a 12 per cent increase in sales, but a price promotion began at the same time. Design a study to assess incrementality and justify your choice of comparison.

2. A survey reports high brand trust among respondents recruited from a brand’s social-media followers. Identify likely evidence-quality problems and propose improvements.

3. Design an evidence plan for evaluating a video campaign that combines interviews, survey measurement, digital behavioural data, and an experiment. State what each component can and cannot establish.

22. Multiple Choice Questions with Answers

Question 1

A researcher wants to discover why consumers distrust a sustainability claim before constructing a survey. Which design is most appropriate initially?

A. Exploratory qualitative research

B. Cross-sectional descriptive research

C. Randomised field experiment

D. Longitudinal panel measurement

Correct Answer: A. Exploratory qualitative research

Question 2

Which conclusion is most defensible from a cross-sectional finding that advertisement recall is associated with purchase intention?

A. The advertisement caused the purchase intention

B. Purchase intention caused the advertisement to be shown

C. Recall and purchase intention are related, but causation is not established

D. The finding has no analytical value

Correct Answer: C. Recall and purchase intention are related, but causation is not established

Question 3

Which feature most directly strengthens internal validity in an advertising experiment?

A. A very large convenience sample

B. Random assignment to treatment and control conditions

C. A highly memorable advertisement

D. Reporting only statistically significant outcomes

Correct Answer: B. Random assignment to treatment and control conditions

Question 4

A survey consistently measures brand trust, but its questions capture only product safety and omit honesty. This primarily indicates a problem with:

A. Reliability

B. Sampling error

C. Construct validity

D. Panel attrition

Correct Answer: C. Construct validity

Question 5

Which design follows the same respondents over several campaign waves?

A. Repeated cross-sectional design

B. Panel longitudinal design

C. One-shot experiment

D. Structured content audit

Correct Answer: B. Panel longitudinal design

Question 6

Why might a platform’s high video-completion rate fail to demonstrate campaign effectiveness?

A. Completion rates are always qualitative data

B. It describes delivery behaviour but does not establish incremental persuasion or sales

C. Digital metrics cannot be used in research

D. It automatically excludes all exposed users

Correct Answer: B. It describes delivery behaviour but does not establish incremental persuasion or sales

Question 7

In the illustrative subscription case, geographic randomisation was used mainly to:

A. Increase the emotional appeal of the creative

B. Eliminate the need for outcome measurement

C. Support estimation of incremental campaign impact

D. Replace qualitative research entirely

Correct Answer: C. Support estimation of incremental campaign impact

Question 8

Which is an example of nonresponse bias?

A. A sample estimate differs from the population by chance

B. Respondents misunderstand a double-barrelled question

C. People with negative campaign experiences are less likely to complete the survey

D. An analyst rounds percentages to one decimal place

Correct Answer: C. People with negative campaign experiences are less likely to complete the survey

Question 9

What is the main value of integrating interviews with a survey in a mixed-method design?

A. It guarantees statistical representativeness

B. It combines understanding of meaning with evidence about prevalence or distribution

C. It removes all measurement error

D. It makes causal testing unnecessary

Correct Answer: B. It combines understanding of meaning with evidence about prevalence or distribution

Question 10

Which statement best reflects evidence calibration?

A. A large dataset always supports a strong causal claim

B. A vivid respondent quotation proves population prevalence

C. The strength of a conclusion should match the design’s evidence and limitations

D. Statistical significance makes external validity irrelevant

Correct Answer: C. The strength of a conclusion should match the design’s evidence and limitations