Surveys, Panels and Questionnaire Design

Introduction

Audience research converts patterns of media use, attention, attitudes and behaviour into evidence for editorial, commercial and managerial decisions. Surveys are particularly valuable because they can collect standardized information from many people at relatively low cost. Panels extend this logic through repeated measurement of the same or carefully recruited respondents, allowing researchers to study change over time. Neither method is automatically representative or objective. Their quality depends on the population defined, the sample selected, the questions asked, the mode of administration, the fieldwork process and the way responses are interpreted.

A questionnaire is therefore not merely a list of questions. It is a measurement instrument linking an abstract concept—such as trust in news, programme satisfaction or platform loyalty—to observable answers. This chapter explains how to design, pilot and administer audience surveys and panels across face-to-face, telephone, online and mobile modes while controlling wording, ordering, interviewer and mode effects.

Learning Objectives

By the end of this chapter, students should be able to:

1. Distinguish surveys, cross-sectional studies, longitudinal panels and repeated cross-sectional studies.
2. Define a target population, sampling frame and appropriate sample design.
3. Translate media concepts into measurable variables and questionnaire items.
4. Design clear, unbiased and accessible questions and response scales.
5. Compare face-to-face, telephone, online and mobile administration modes.
6. Pilot a questionnaire and diagnose reliability, validity, nonresponse and mode effects.
7. Apply ethical, legal and data-governance principles to audience research.
8. Interpret survey and panel findings cautiously for managerial decision-making.

1. Surveys and Panels as Research Designs

A survey collects standardized information from a sample using a structured instrument. A cross-sectional survey measures respondents at one point in time. It is suitable for estimating current reach, awareness, satisfaction or attitudes, but it is weak for establishing individual-level change or causal direction.

A panel repeatedly collects information from the same respondents. A panel can reveal whether particular people changed their listening, viewing, reading or subscription behaviour. A repeated cross-sectional design instead draws a new sample from the same population at each wave. It estimates population-level trends without the attrition and conditioning problems associated with following the same people.

Panels are useful for questions such as: “Who stopped subscribing after a price increase?” or “Which users moved from broadcast television to streaming?” They also create operational demands: respondent retention, incentives, contact management, wave scheduling, confidentiality and consistent measurement.

The survey error framework

A useful framework separates total survey error into several interacting sources:

• Coverage error: some members of the target population cannot be reached through the sampling frame.
• Sampling error: a sample differs from the population by chance.
• Nonresponse error: people who do not participate differ systematically from participants.
• Measurement error: answers do not accurately represent the intended construct.
• Processing error: coding, weighting, transcription or analysis introduces mistakes.
• Attrition error: panel members drop out and remaining members become less representative.

These errors cannot be eliminated completely. Good research identifies them, reduces them and reports their likely consequences.

2. From Research Problem to Measurable Variables

Questionnaire design begins before wording. The researcher should specify the decision problem, audience, constructs, indicators and analysis plan. For example, “Does the news app serve young adults?” is too broad. It may be translated into measurable constructs such as weekly use, session frequency, perceived credibility, ease of navigation, perceived relevance and subscription intention.

A construct is an abstract concept. A variable is a characteristic that can take different values. An indicator is an observable measure used to represent a construct. A unit of analysis is the entity being studied, such as an individual audience member, household, account or media title.

A measurement specification should answer five questions:

1. What exactly is being measured?
2. Whose experience is relevant?
3. What recall period is appropriate?
4. What response scale reflects meaningful differences?
5. How will the answer be coded and analysed?

Operationalization example

Suppose the construct is “podcast engagement.” It might be operationalized through:

• aided and unaided awareness;
• listening in the past seven days;
• number of episodes started;
• proportion usually completed;
• sharing or commenting behaviour;
• emotional involvement;
• intention to listen again.

No single indicator captures the whole construct. A frequency question may measure behaviour but not depth of attention; an attitude scale may measure evaluation but not actual use. Combining indicators can improve construct validity, provided the items are theoretically justified and not unnecessarily repetitive.

Levels of measurement

• Nominal: categories without inherent order, such as preferred platform.
• Ordinal: ordered categories without assured equal intervals, such as satisfaction from very dissatisfied to very satisfied.
• Interval-like scale: a numerical scale often treated as having approximately equal intervals, such as a 0–10 recommendation scale. This assumption should be justified rather than taken for granted.
• Ratio: values with a meaningful zero, such as number of viewing sessions.

Measurement level affects coding, summaries and statistical tests. A category such as “daily” is not automatically equivalent to a precise count of seven uses per week.

3. Sampling for Audience Surveys and Panels

The target population is the complete group about which conclusions are intended. The sampling frame is the operational list or mechanism from which units are selected. A frame may be a postal address file, a probability telephone sample, a customer database or a recruited online panel. The frame may omit people, duplicate units or include ineligible cases.

Probability and non-probability samples

In a probability sample, each eligible unit has a known, non-zero selection probability. Simple random, systematic, stratified and cluster samples are examples. Probability sampling supports design-based estimates of population quantities when fieldwork and weighting are properly handled.

In a non-probability sample, inclusion probabilities are unknown or selection is based on convenience, volunteering, quotas or a panel provider’s recruitment process. Such samples can be useful for exploratory research and rapid commercial measurement, but conventional margins of sampling error should not be presented as if they were automatically valid.

Sample size and margin of error

For a simple random sample estimating a population proportion, an approximate sample-size formula is:

Here, n₀ is the required sample size before finite-population adjustment, z is the standard-normal critical value for the chosen confidence level, p is the anticipated proportion, and e is the desired maximum margin of error expressed as a proportion. The formula assumes a probability sample, independent observations, a valid proportion estimate and simple random sampling. It does not compensate for nonresponse, design effects, poor measurement or coverage bias.

When no prior estimate of p is available, p = 0.5 is conservative because it produces the largest required sample for a given z and e. For a 95% confidence level, z ≈ 1.96. With p = 0.5 and e = 0.05:

The result is rounded up to 385 respondents under the stated assumptions. If the sample uses clusters or has unequal weights, the variance may be larger. A design effect can be represented as DEFF, giving an approximate effective sample size:

where n is the achieved sample size and n_eff is the simple-random-sample equivalent. This is a planning relationship, not a universal correction. Researchers must document how the design effect was estimated.

For a finite population of size N, a finite-population correction may be applied when the sampling fraction is substantial:

The formula is appropriate for simple random sampling without replacement and does not solve nonresponse or coverage problems.

Stratification, quotas and weighting

Stratification divides a frame into relevant groups—such as region, age or language—and samples within each group. It can improve precision and ensure coverage of analytically important subgroups. Quotas may produce similar visible distributions, but a quota sample is not necessarily probability-based because selection within quotas may remain uncontrolled.

Weights adjust the contribution of cases to better reflect selection probabilities and known population benchmarks. A basic design weight is the inverse of a unit’s selection probability:

where wᵢ is the design weight for case i and πᵢ is its probability of selection. Nonresponse and calibration adjustments may be applied, but extreme weights can increase variance. Weighted results should identify the benchmark source, variables used and any trimming procedure.

4. Questionnaire Architecture

A well-designed questionnaire has a logical flow and a clear respondent burden. A typical audience questionnaire may include:

1. Introduction, consent and eligibility screening.
2. Warm-up questions about general media behaviour.
3. Core behavioural measures.
4. Evaluation, attitudes and motivations.
5. Sensitive or potentially tiring questions.
6. Demographics and classification variables.
7. Thank-you message and support information where needed.

This order is not universal. Sensitive topics may need earlier placement if eligibility depends on them, and experiments may require randomization. The principle is to make the sequence psychologically and analytically defensible.

Question wording

Effective questions are:

• specific about the object, time period and context;
• written in ordinary language appropriate to the audience;
• limited to one idea;
• neutral rather than persuasive;
• free of unnecessary technical terms;
• answerable by the selected respondent;
• consistent with the response options.

Avoid double-barrelled questions such as “How satisfied are you with the content and navigation?” A respondent may like one and dislike the other. Avoid loaded wording such as “How valuable is our excellent service?” Avoid assumptions: “How often do you share our posts?” presumes sharing. Use screening or branching instead.

Define recall periods carefully. “How often do you use social media?” is less precise than “On how many of the past seven days did you use the news section of this app?” Still, precise recall should not imply accuracy that respondents cannot provide. For high-frequency behaviour, categories may be more reliable than exact counts.

Response options

Response categories should be exhaustive for the intended population and mutually exclusive where only one answer is permitted. Include “Don’t know,” “Not applicable” or “Prefer not to say” when substantively justified. Do not force an opinion where respondents may have no basis for one.

A balanced agreement scale generally provides comparable positive and negative options. If a midpoint is included, its meaning should be clear: neutrality, mixed evaluation or uncertainty are not identical. “Not sure” should not be treated automatically as neutral.

For ratings, label endpoints and explain the direction. A 0–10 scale can be useful, but the question should specify what 0 and 10 mean. Visual scales on mobile devices must be easy to tap and read without horizontal scrolling.

Routing and skip logic

Routing reduces irrelevant questions and burden. For example, non-subscribers should not be asked to rate cancellation processes. Every route must be tested for eligibility, accidental loops, missing responses and inconsistent denominators. Reports should state the base for each percentage: all respondents, eligible respondents, users of a service or completed cases.

Ordering and context effects

Earlier questions can change later answers by making an issue salient. Asking about a controversial programme before measuring general trust may inflate the apparent importance of that programme. Response options can also influence answers, especially when respondents choose from long lists.

Useful controls include placing general questions before specific prompts, randomizing suitable item lists, rotating response options when order effects are plausible, separating factual recall from evaluation and pretesting alternative versions. Randomization is inappropriate when it destroys a natural time sequence or makes the instrument confusing.

5. Modes of Administration

Face-to-face surveys

Interviewers can establish rapport, clarify procedures and reach people with limited digital access. Visual materials and longer instruments are possible. However, face-to-face surveys are costly and vulnerable to interviewer effects, social desirability, noncontact and location bias. Interviewers require standardized training, neutral conduct and clear rules for probing.

Telephone surveys

Telephone research can be faster and geographically broad. It is suitable for shorter questionnaires and verbal scales. Challenges include declining answer rates, call screening, unavailable numbers, hearing or language barriers and the absence of visual response aids. Interviewers should read scales consistently and avoid rushing.

Online surveys

Online instruments support complex routing, multimedia stimuli and automated data capture. They are efficient for digitally connected populations and can offer privacy for sensitive questions. Risks include coverage gaps, duplicate or fraudulent participation, professional respondents, satisficing and unclear recruitment probabilities. A survey link posted publicly is not equivalent to a representative sample.

Mobile surveys

Mobile questionnaires must be designed for small screens, intermittent connectivity and touch interaction. They should use short pages, readable type, limited grids, minimal typing and progress indicators that do not mislead. Mobile push or SMS invitations raise additional consent, cost and privacy issues. Long matrix questions encourage straight-lining and abandonment.

Mode effects

Mode effects occur when answers differ because of the administration mode rather than the underlying attitude or behaviour. Interviewer presence may increase socially desirable answers; self-administration may increase disclosure. A visual scale may be easier online than by telephone. Differences can also result from sample composition, not mode itself.

To assess mode effects, researchers can conduct a mode experiment with comparable samples, keep wording and routing constant, record mode-specific paradata and compare distributions after appropriate adjustment. Mode should not be treated as a minor technical detail; it is part of measurement design.

6. Interviewer Effects and Fieldwork Control

Interviewers influence participation and answers through introductions, tone, pace, appearance, probing and recording decisions. Standardization does not mean robotic interaction. It means that every respondent receives the same essential information and that discretion is limited to approved procedures.

A fieldwork protocol should specify:

• eligibility rules;
• contact attempts and scheduling;
• introduction and consent wording;
• permitted neutral probes;
• treatment of interruptions and proxy respondents;
• language and accessibility procedures;
• refusal conversion, if ethically permitted;
• quality checks and escalation procedures.

A neutral probe might be “Which option best describes your view?” An improper probe suggests an answer or signals approval. Interviewer monitoring, verification callbacks and encrypted systems can identify fabricated interviews or recording errors, but monitoring must respect staff privacy and applicable law.

7. Panels: Design, Retention and Analysis

Panel recruitment should define who can join, how eligibility is verified, how often contact occurs and what incentives are offered. A panel may be recruited through probability methods, address-based recruitment, customer records or an online provider. The recruitment source determines the claims that can reasonably be made about representativeness.

Attrition occurs when panel members fail to complete later waves. It may be related to age, workload, satisfaction, technology or the topic itself. Retention strategies include predictable schedules, reasonable length, accessible modes, transparent communication, fair incentives, reminders and respect for refusal.

Repeated measurement can create conditioning: respondents become more aware of an issue, learn the questionnaire, or alter behaviour because they are being observed. Panels can also experience seam effects, in which reported events appear to cluster around the boundary between waves because respondents remember periods differently.

Researchers should distinguish:

• gross change: the percentage whose answer differs between two waves;
• net change: the difference between population percentages at two times;
• transition: movement from one category to another.

A panel’s changing composition can make unweighted trends misleading. Attrition analysis should compare continuing and departing members using available baseline data, apply appropriate adjustments and report the limitations of those adjustments.

8. Piloting and Instrument Evaluation

A pilot is a small-scale test of the complete research process, not merely a grammar check. Cognitive interviewing asks respondents to explain how they understood a question, what information they retrieved, how they formed an answer and how they selected a response. This can expose ambiguous terms, hidden assumptions and inappropriate recall periods.

A field pilot should examine:

• recruitment and eligibility rates;
• completion and break-off rates;
• time to completion;
• item nonresponse;
• routing accuracy;
• interviewer variation;
• device compatibility;
• unusual response patterns;
• respondent comments and complaints.

A high completion rate does not prove validity. Respondents may complete an instrument by selecting arbitrary answers. Examine straight-lining, implausibly rapid completion, inconsistent answers, duplicate records and excessive use of “prefer not to say.” Such checks should identify low-quality records through transparent rules rather than silently deleting inconvenient responses.

For multi-item scales, internal consistency may be assessed with Cronbach’s alpha:

Here, k is the number of items, σ²ᵢ is the variance of item i, and σ²ₜ is the variance of the total score formed from the items. Alpha assumes that items are intended to measure a common construct and is influenced by the number of items and their covariance. A high value does not establish validity or unidimensionality; it can result from redundant items. It should therefore be used alongside theory, item inspection and other evidence.

9. Reliability, Validity and Interpretation

Reliability concerns consistency. A measure may be internally consistent but consistently wrong. Validity concerns whether the evidence supports the intended interpretation. Content validity asks whether important aspects of the construct are covered. Construct validity considers relationships with related and distinct measures. Criterion-related validity examines association with a credible external criterion when one exists.

Audience researchers should distinguish reported behaviour from observed behaviour. Self-reported viewing may overestimate exposure because popular or socially valued content is easier to recall. Passive measurement can provide behavioural data, but it introduces consent, device, identity-resolution and privacy challenges and may not reveal attention or meaning.

Percentages should include denominators and, where relevant, uncertainty intervals. A difference between two groups is not automatically meaningful: sample size, design, weighting, measurement reliability and practical importance matter. Statistical significance, where calculated, does not establish managerial importance or causation.

10. Ethics, Privacy and Governance

Ethical survey research requires informed consent, voluntary participation, data minimization, confidentiality and the right to withdraw where feasible. Participants should know who is conducting the research, why data are collected, how long they will be retained, whether incentives are offered and whom to contact with concerns.

Sensitive media questions may reveal political views, health information, sexuality, religion or other protected characteristics. Collect only what is necessary, restrict access and separate identifying information from response data. Panel databases require especially careful identity, contact and retention controls. Children and other vulnerable groups require additional safeguards and, where applicable, guardian permission and age-appropriate assent.

Researchers must comply with applicable data-protection, telecommunications, consumer-protection and accessibility requirements. Consent to receive a research invitation is not automatically consent for marketing. AI tools may assist with translation review, fraud detection, open-ended coding or questionnaire diagnostics, but human researchers must validate outputs, protect confidential data, document model use and avoid automated decisions that unfairly exclude respondents.

11. Applied Case: Measuring a Streaming Service’s Declining Retention

The following is an illustrative case, not a report of an actual company.

Background

A regional streaming service observed a fall in monthly renewals among households aged 18–34. Management suspected that price, content discovery and competing services were involved, but the existing customer survey asked only whether subscribers were “satisfied.”

Problem

The survey had a leading question—“How satisfied are you with our excellent content?”—a long online matrix, no distinction between account owners and other household users, and a sample drawn only from active subscribers. It could not explain why former subscribers left or whether different users experienced different problems.

Analysis and approach

The research team defined three populations: current subscribers, recently lapsed subscribers and eligible non-subscribers aware of the service. It designed a mixed-mode study with a short mobile-friendly questionnaire and a follow-up panel of consenting respondents. Questions measured recent use, discovery success, content relevance, price perceptions, technical problems, competing services, cancellation reasons and renewal intention.

The team used neutral wording: “Which of the following, if any, were reasons you considered cancelling or did cancel?” Multiple reasons were allowed, followed by a question asking for the most important reason. Current and former subscribers received different routes. A small cognitive pilot found that respondents interpreted “content variety” inconsistently, so it was replaced by separate items about genres, new releases and local content.

The main survey used stratification by age and region and documented recruitment limitations. A panel wave was conducted at sign-up and again eight weeks later. Retention incentives were modest and disclosed in advance. The team compared self-reported use with available aggregate platform records, without exposing individual viewing histories to managers.

Outcome

The research found that price was frequently mentioned but was not the dominant reason in every segment. Younger users often reported difficulty finding relevant content and perceived insufficient new releases. The panel showed that low discovery satisfaction preceded reduced use for some users, while technical interruptions were more important for others. Management tested improved recommendations and clearer content communication rather than applying a blanket discount.

Learning points

The case demonstrates that a single satisfaction item cannot diagnose a complex business problem. Defining populations, including lapsed users, separating constructs, using neutral wording and following respondents over time produced more actionable evidence. It also shows why survey results should inform—not replace—experiments, platform analytics and qualitative research.

12. Best-Practice Checklist

Before launch, the researcher should be able to answer yes to the following questions:

• Is the target population explicitly defined?
• Is the sampling or recruitment mechanism documented?
• Does every key construct have a defensible measure?
• Are wording, recall period and response options clear?
• Are eligibility, routing and denominators tested?
• Has the instrument been cognitively tested and piloted?
• Is the mode appropriate for the population and topic?
• Are interviewer procedures standardized?
• Are panel retention and attrition plans in place?
• Are privacy, accessibility, consent and data security addressed?
• Are weighting, exclusions and limitations pre-specified?
• Will reporting distinguish association, change and causation?

Chapter Summary

Surveys and panels are structured systems for measuring audience behaviour, attitudes and change. Their credibility depends on more than sample size. Researchers must define the population, evaluate the sampling frame, operationalize constructs, write neutral and answerable questions, design logical routing and select an appropriate administration mode. Face-to-face, telephone, online and mobile modes each create distinctive coverage, burden, interviewer and measurement effects.

Panels provide evidence about individual change but face attrition, conditioning and privacy challenges. Piloting through cognitive interviews and field tests is essential for identifying ambiguity, routing errors and respondent burden. Reliability supports consistency but does not prove validity; weighting and statistical formulas cannot repair biased measurement or inadequate coverage. Ethical research requires voluntary participation, data minimization, confidentiality, accessibility and transparent governance. Properly designed audience research connects evidence to decisions while remaining cautious about what the data can and cannot establish.

Key Terms

Target population: The complete group to which the researcher intends to generalize findings.

Sampling frame: The operational list or mechanism from which a sample is selected.

Cross-sectional survey: A survey measuring a population at one point in time.

Panel: A group of respondents measured repeatedly across waves.

Repeated cross-sectional design: Surveys of the same target population using new samples at successive times.

Construct: An abstract concept, such as trust or engagement, that requires measurement.

Indicator: An observable variable used to represent a construct.

Operationalization: The process of translating a construct into measurable variables.

Coverage error: Bias caused when the sampling frame omits or overrepresents parts of the population.

Nonresponse error: Bias arising when participation is related to the variables being measured.

Measurement error: Difference between the recorded response and the intended value or concept.

Mode effect: A difference in responses attributable to the administration mode.

Interviewer effect: Influence of interviewer characteristics or conduct on participation or answers.

Attrition: Loss of panel members across waves.

Conditioning: Change caused by repeated participation in a panel.

Weight: A numerical value used to adjust a case’s contribution to estimates.

Routing: The questionnaire logic that determines which questions a respondent receives.

Cognitive interview: A pretest method examining how respondents understand and answer questions.

Reliability: The consistency of a measurement procedure.

Validity: The degree to which evidence supports the intended interpretation of a measure.

Review Questions

Short Answer Questions

1. What is the difference between a panel and a repeated cross-sectional survey?
2. Define coverage error and give one media-research example.
3. Why is a double-barrelled question problematic?
4. What is the purpose of cognitive interviewing?
5. Name two possible mode effects in audience research.

Long Answer Questions

1. Explain how a researcher should move from a broad media research problem to a set of measurable questionnaire variables.
2. Compare face-to-face, telephone, online and mobile survey modes in terms of strengths, limitations and likely sources of error.
3. Explain the principal sources of survey error and describe practical methods for reducing each one.
4. Discuss panel attrition, conditioning and seam effects, and recommend strategies for managing them.
5. Explain why reliability and validity are different, using an audience-measurement example.

Analytical / Application Questions

1. A broadcaster wants to know why young viewers are abandoning its news app. Design a survey and panel strategy, including populations, sampling or recruitment, key constructs, modes, routing and follow-up waves.
2. A mobile survey shows that 72% of respondents are satisfied, while a telephone survey shows 58%. Develop a plan to determine whether the difference reflects mode effects, sample composition, wording, response scales or nonresponse.
3. Critique this question: “How satisfied are you with our affordable prices, excellent programmes and easy-to-use app?” Rewrite it as a defensible set of questions and explain the changes.

Multiple Choice Questions with Answers

Question 1

Which design measures the same respondents repeatedly over time?

A. Cross-sectional survey

B. Panel study

C. One-time census

D. Intercept sample

Correct Answer: B. Panel study

Question 2

In the formula n₀ = z²p(1 − p)/e², what does e represent?

A. The population size

B. The selection probability

C. The desired margin of error

D. The design effect

Correct Answer: C. The desired margin of error

Question 3

Which is an example of a double-barrelled question?

A. How many days did you use the app last week?

B. Which platform did you use most recently?

C. How satisfied are you with the programme and the website?

D. Have you heard of this podcast?

Correct Answer: C. How satisfied are you with the programme and the website?

Question 4

What is the primary purpose of a cognitive interview?

A. To calculate the final response rate

B. To examine how respondents understand and answer questions

C. To replace probability sampling

D. To estimate panel attrition after several years

Correct Answer: B. To examine how respondents understand and answer questions

Question 5

Which problem is especially associated with repeated panel measurement?

A. Conditioning

B. Finite-population correction

C. Random assignment

D. Content validity

Correct Answer: A. Conditioning

Question 6

A sampling frame excludes people who use only mobile phones. This is primarily an example of:

A. Coverage error

B. Interviewer error

C. Processing error

D. Recall bias

Correct Answer: A. Coverage error

Question 7

Which practice best reduces leading-question bias?

A. Using promotional adjectives

B. Presuming the respondent has used the service

C. Using neutral wording and balanced response options

D. Removing all ‘don’t know’ options

Correct Answer: C. Using neutral wording and balanced response options

Question 8

A survey has a sample size of 400 and an estimated design effect of 2. Using n_eff ≈ n/DEFF, what is the approximate effective sample size?

A. 100

B. 200

C. 400

D. 800

Correct Answer: B. 200

Question 9

Why should a high Cronbach’s alpha not be treated as proof of validity?

A. Alpha measures only sample size

B. Alpha can be high even when items are redundant or measure the wrong construct

C. Alpha applies only to nominal variables

D. Alpha proves causal relationships

Correct Answer: B. Alpha can be high even when items are redundant or measure the wrong construct

Question 10

Which action is most appropriate when reporting a weighted survey estimate?

A. Hide the weighting procedure to simplify the report

B. Report the benchmark source and weighting approach

C. Treat weighting as a substitute for valid measurement

D. Claim that weighting eliminates nonresponse bias

Correct Answer: B. Report the benchmark source and weighting approach