Sampling, Measurement and Research Instruments

Introduction

Mass media management research depends on evidence about audiences, organisations, content, technologies and markets. Whether a project evaluates a streaming service, measures audience trust, analyses news bias, or investigates newsroom innovation, the quality of its conclusions depends heavily on three connected decisions: who or what will be studied, how concepts will be measured, and which instruments will generate the evidence.

Sampling determines the relationship between the study population and the cases actually observed. Measurement translates abstract concepts—such as credibility, engagement, satisfaction or organisational culture—into observable indicators. Research instruments provide the practical means of collecting those indicators through surveys, interviews, focus groups, observation, content analysis and case-study protocols.

Poor sampling can produce biased conclusions even when data collection is technically flawless. Ambiguous concepts can make a large dataset analytically meaningless. An unreliable questionnaire, inconsistent coding scheme or leading interview guide can undermine an otherwise valuable dissertation. This chapter therefore treats sampling, measurement and instrument design as an integrated process rather than as separate administrative tasks.

Learning Objectives

By the end of this chapter, students should be able to:

• distinguish probability from non-probability sampling;
• select and justify an appropriate sampling design;
• define populations, sampling frames, units and sample sizes;
• operationalise abstract media-management concepts;
• design valid and reliable surveys, interviews and focus groups;
• construct observation protocols and content-analysis coding schemes;
• prepare case-study instruments using multiple sources of evidence;
• identify sampling, measurement, ethical and technological limitations; and
• document methodological decisions transparently in a dissertation.

1. The Logic of Sampling and Measurement

A population is the complete set of units relevant to a research question. It may consist of all employees in a media organisation, subscribers to a platform, news stories published during a period, or social-media posts containing a specified hashtag. A sample is the subset actually studied.

The unit of analysis is the entity about which conclusions are drawn: an individual, household, organisation, article, programme, episode, advertisement, social-media post or event. The sampling unit is the entity selected at a stage of sampling. These may be identical, but they are not always so. For example, a researcher may sample media organisations and then analyse employees within them. The organisation is the first-stage sampling unit; the employee is the unit of analysis.

A sampling frame is the operational list or procedure from which a sample is selected. Examples include a subscriber database, a register of licensed broadcasters, a list of news articles, or a platform-generated archive. A frame may be incomplete, duplicated, outdated or inaccessible. Coverage error occurs when the frame excludes relevant population members or includes ineligible ones.

The research process can be represented as follows:

1. Define the research question and target population.
2. Specify the unit of analysis and eligibility criteria.
3. Identify or construct a sampling frame.
4. Select a sampling strategy.
5. Determine sample size or information requirements.
6. Recruit cases and document non-response or refusal.
7. Measure concepts with appropriate instruments.
8. Evaluate validity, reliability and ethical quality.

Sampling and measurement are related to inference. A probability sample supports statistical generalisation when selection probabilities are known and non-response is adequately addressed. A purposive qualitative sample generally supports analytic or theoretical generalisation: it helps explain processes, meanings or mechanisms rather than estimate population percentages.

2. Probability Sampling

Probability sampling gives each eligible population element a known, non-zero probability of selection. It is most appropriate when the purpose is estimation or comparison across a defined population and a credible sampling frame is available.

Simple Random Sampling

In simple random sampling, every possible sample of a specified size has an equal chance of selection. A random-number generator can select cases from a numbered frame. The design is conceptually strong but may be impractical when the population is geographically dispersed or the frame is incomplete.

Systematic Sampling

Systematic sampling selects every kth element after a random starting point. If the frame contains N units and the desired sample size is n, the interval is:

Here, N is the number of eligible frame units, n is the intended sample size, and k is the sampling interval. The interval should be based on a random start between 1 and k. Researchers must examine the frame for periodic patterns. For example, a list ordered by publication day could create bias if every seventh record has a distinctive content pattern.

Stratified Sampling

Stratified sampling divides the population into mutually exclusive strata—such as age groups, regions, job roles or platform-use categories—and samples within each stratum. Proportionate allocation preserves population composition. Disproportionate allocation may deliberately oversample a small but analytically important group; weighting may then be required during analysis.

Stratification is useful when subgroups differ substantially and each must be represented. It can improve precision compared with a simple random sample, provided strata are correctly defined and the frame supports classification.

Cluster and Multistage Sampling

Cluster sampling selects naturally occurring groups, such as media outlets, households, schools or cities, and then studies units within selected clusters. It reduces travel and administrative costs but usually produces less statistical efficiency than a simple random sample because members of a cluster may resemble one another.

Multistage sampling combines stages—for example, select regions, then media organisations, then departments, then employees. The researcher must document selection at every stage and recognise that standard errors may be larger than those from a simple random sample of the same size.

Sample Size for a Proportion

For a large population and an estimated proportion, an introductory planning formula is:

where n₀ is the initial sample size, z is the standard-normal critical value associated with the desired confidence level, p is the anticipated population proportion, and e is the maximum tolerable margin of sampling error expressed as a proportion. The formula assumes a probability sample, an approximately independent selection process and a proportion estimate as the principal objective. It does not automatically correct for non-response, clustering or design effects.

When p is unknown, p = 0.50 is commonly used because it produces the largest required sample for a given z and e. For a 95% confidence level, z is approximately 1.96. With p = 0.50 and e = 0.05:

The planned minimum is therefore about 385 completed responses before allowance for non-response. This is not a guarantee of accuracy: response bias, poor coverage and weak measurement can remain serious problems.

For a finite population of size N, a finite-population correction may be used:

The correction is appropriate when sampling without replacement from a known, relatively small population and the sampling fraction is not negligible. If N = 2,000 and n₀ ≈ 384.16, the adjusted sample is approximately 323. The calculation does not compensate for non-response or a biased frame.

3. Non-Probability Sampling

Non-probability sampling does not provide known selection probabilities. It is often appropriate for exploratory, qualitative, hard-to-reach or rapidly changing populations, but statistical estimates should not be presented as if they were based on a representative probability sample.

Convenience sampling selects accessible participants. It is inexpensive but vulnerable to severe selection bias. Purposive sampling deliberately selects information-rich cases that meet theoretically relevant criteria, such as editors who introduced an AI workflow. Quota sampling fills specified subgroup quotas without random selection within each quota. It can improve visible diversity but does not establish probability-based representativeness.

Snowball sampling uses participant referrals and can assist access to hidden or networked populations. It may overrepresent highly connected participants. Theoretical sampling, associated with grounded-theory research, proceeds iteratively: emerging concepts guide the selection of later cases until additional data provide diminishing theoretical insight.

Qualitative sample adequacy should not be justified solely by a fixed numerical rule. Researchers should consider information power, diversity of perspectives, research scope, quality of dialogue and saturation. Saturation must be specified carefully: code saturation may mean identifying recurring topics, whereas meaning saturation requires deeper understanding of dimensions and relationships.

4. Operationalising Concepts

Conceptualisation defines what an abstract idea means in a particular study. Operationalisation identifies observable indicators and procedures for measuring it. For example, “audience engagement” might mean behavioural interaction with content, emotional involvement, cognitive attention, or a combination. A defensible study states which meaning is adopted.

A useful operationalisation chain is:

Concept → dimensions → indicators → items or coding rules → measurement scale → interpretation.

Consider perceived news credibility:

Definitions must include inclusion and exclusion rules. “Digital engagement,” for instance, should specify whether passive viewing, comments, shares, completion rates and repeat visits are included. Administrative metrics may measure behaviour but not motivation or meaning.

Levels and Types of Measurement

The four traditional levels are nominal, ordinal, interval and ratio. Nominal variables classify without order, such as platform type. Ordinal variables rank categories, such as satisfaction from low to high, without guaranteeing equal distances. Interval scales assume equal intervals but lack a meaningful absolute zero; ratio scales have equal intervals and a meaningful zero, as with minutes watched or number of posts.

A Likert-type item—such as “The platform provides reliable news”—usually produces an ordinal response. Researchers sometimes construct a multi-item scale and analyse its composite score as approximately continuous, but this assumption should be justified rather than treated as automatic.

5. Validity, Reliability and Measurement Quality

Validity concerns whether an instrument measures the intended construct and supports the interpretation made from its results. Reliability concerns consistency, not truth. A measure can be reliable but invalid—for example, a consistently miscalibrated coding rule.

• Face validity: the instrument appears appropriate on inspection; useful but weak evidence.
• Content validity: items adequately cover the conceptual domain, usually assessed through literature and expert review.
• Construct validity: evidence that the measure behaves as theory predicts, including convergent and discriminant evidence.
• Criterion-related validity: association with an appropriate external criterion, where one exists.
• Internal reliability: consistency among items intended to measure the same construct.
• Inter-coder reliability: agreement among content analysts using the same coding scheme.
• Test–retest reliability: stability when the same measure is repeated under comparable conditions.

For multi-item scales, Cronbach’s alpha is often reported, but it depends on assumptions including related items and a broadly coherent scale; a high value may also result from redundant items. Reliability should therefore be interpreted alongside item content, dimensionality and the purpose of the instrument. For categorical coding, percentage agreement alone can be misleading when categories are imbalanced. Cohen’s kappa is commonly used for two coders correcting for chance agreement, but its value is sensitive to prevalence and marginal distributions. The coding manual and disagreements should also be examined substantively.

6. Designing Surveys

A survey instrument should move from research objectives to constructs, indicators, items, response options and analysis plans. Each item should address one idea, use language appropriate to respondents and specify a meaningful reference period where necessary.

Avoid double-barrelled questions (“How satisfied are you with the speed and accuracy of the service?”), vague terms (“often”), unnecessary jargon, leading wording and assumptions that all respondents have relevant experience. Use filter questions where appropriate: first establish whether a respondent used a service, then ask about satisfaction.

A well-designed questionnaire commonly contains:

1. an informed-consent statement;
2. screening and eligibility questions;
3. core construct measures;
4. behavioural or factual questions;
5. demographic or professional variables needed for analysis; and
6. a debrief or contact route.

Response categories should be exhaustive and mutually exclusive where possible. “Don’t know,” “not applicable” and “prefer not to answer” should be available when conceptually appropriate. Randomising item order can reduce order effects, but it should not disrupt logical scales or respondent comprehension.

Pilot testing should examine completion time, comprehension, missing data, technical performance and unexpected interpretations. Cognitive interviewing—asking participants to explain how they understood and answered an item—is particularly valuable before full deployment.

7. Interviews and Focus Groups

A semi-structured interview uses a flexible guide containing core questions and probes. It is suitable for exploring managerial decisions, professional identities, organisational conflict and interpretations of change. Questions should be open, neutral and sequenced from accessible background topics to more sensitive issues. Probes such as “Can you give an example?” and “What happened next?” elicit evidence without imposing an answer.

Interviewers should distinguish participants’ descriptions from the researcher’s assumptions. They should record context, pauses, emotional responses and relevant non-verbal observations where ethically appropriate. Reflexive notes should identify how the interviewer’s role, status or expectations may have shaped the exchange.

Focus groups generate data through interaction. They are appropriate when group norms, shared language, audience interpretations or professional disagreement are central. A moderator needs a clear purpose, balanced participation, respectful ground rules and a plan for dominant speakers or conflict. Confidentiality cannot be guaranteed fully because participants hear one another’s contributions. Sensitive personal topics may therefore be better addressed through individual interviews.

Audio or video recording requires explicit consent. Transcripts should identify speakers consistently, preserve meaning and remove unnecessary identifying details. Translation and transcription decisions are part of the method and should be documented.

8. Observation Protocols

Observation may be participant or non-participant, structured or unstructured, overt or—in exceptional circumstances—covert. In media management, observation can examine newsroom meetings, production routines, audience-service interactions or platform moderation workflows.

An observation protocol should specify:

• setting, date, duration and access conditions;
• behaviours, interactions or events to be observed;
• a sampling rule, such as time, event or activity sampling;
• descriptive versus interpretive notes;
• a method for recording interruptions and missing observations; and
• consent, privacy and data-security arrangements.

Researchers should separate descriptive notes (“the editor rejected the headline and requested two sources”) from analytic notes (“authority appears centralised in the editor’s role”). This distinction reduces premature interpretation while preserving analytical insight.

9. Content-Analysis Coding Schemes

Content analysis systematically classifies recorded communication. The corpus may include articles, broadcasts, advertisements, videos, headlines, comments or posts. A coding scheme must define the population of content, sampling period, unit of analysis, variables, categories and decision rules.

Categories should be exhaustive enough to handle relevant cases and mutually exclusive when only one code is permitted. If multiple frames or themes may occur in one item, the scheme should explicitly allow multiple coding. Each category needs inclusion criteria, exclusion criteria and examples. “Political bias,” for instance, requires a defensible operational definition; it cannot be left to a coder’s general impression.

A coding manual should include:

Train coders using practice material that is not part of the final dataset. Conduct a pilot reliability exercise, revise definitions, then freeze the manual before full coding where possible. Disagreement is not merely a technical inconvenience; it can reveal that a concept is vague, multidimensional or culturally dependent.

10. Case-Study Instruments and Triangulation

A case study investigates a bounded contemporary phenomenon in context. The case may be a media organisation, platform policy, campaign, crisis, merger or innovation project. A case-study instrument is not a single questionnaire; it is a structured protocol for collecting and integrating multiple forms of evidence.

A strong protocol includes the case boundary, research questions, propositions or sensitising concepts, source-selection criteria, interview guide, observation schedule, document-extraction form, chronology template and evidence matrix. Triangulation compares evidence across sources, methods, investigators or theories. It does not mean that sources must produce identical accounts. Contradictions may reveal power differences, strategic communication or changing conditions.

Evidence should be stored with source identifiers, dates, consent status, analytical memos and an audit trail. Claims in the dissertation should be traceable to evidence rather than based on a single memorable quotation.

Applied Case: An Illustrative Study of a Streaming Service

Background

A fictional regional streaming service, ViewNow, reports declining subscriber retention despite increasing its original programming. Management wants to know whether dissatisfaction reflects content quality, user experience, pricing or weak communication.

Problem

The initial proposal plans an online survey shared through the company’s social-media page. This would reach highly engaged followers but exclude lapsed subscribers and users who do not follow the page. The proposed single question—“Are you satisfied with ViewNow?”—also fails to distinguish relevant dimensions.

Analysis

The target population is current and recently lapsed subscribers in the service region. The study defines retention intention, perceived content value, platform usability and price fairness as separate constructs. The frame combines the subscriber database with a documented recruitment procedure for lapsed users, while acknowledging that users with invalid contact details may remain undercovered.

Approach and Solution

The researcher proposes a stratified survey by current versus lapsed status and subscription tier, with invitations issued through the database and a controlled reminder schedule. Survey items are piloted cognitively. A purposive qualitative sample of current, lapsed and high-use subscribers is interviewed to explore reasons behind survey patterns. Usability observation records task completion difficulties while participants search for and play programmes. Finally, a content and service-document review examines promotional promises, cancellation information and customer-support responses.

The instrument records non-response, separates “not applicable” from missing answers and avoids collecting unnecessary names. The analysis plan is written before data collection so that each research question maps to an indicator or qualitative theme. Contradictory evidence—for example, high reported content value but poor retention—triggers further examination of price, interface friction or billing practices rather than an immediate conclusion.

Outcome

The illustrative design would provide stronger evidence than a social-media convenience survey because it combines subgroup coverage, construct-level measurement and qualitative explanation. It still cannot claim perfect population representation if the frame excludes unreachable lapsed users. That limitation is reported rather than concealed.

Learning Points

The case demonstrates that sampling, operationalisation and instruments must be designed together. A representative-looking sample cannot repair a vague construct, and a sophisticated instrument cannot remove coverage bias. Mixed evidence is most useful when each method has a defined role.

11. Ethics, Governance and Contemporary Technology

Participants must receive understandable information about purpose, procedures, risks, voluntary participation, withdrawal, recording, data storage and dissemination. Consent must be appropriate to the study and not treated as a one-time administrative signature. Employees may feel pressured when research is conducted inside their organisation; recruitment should protect voluntariness and confidentiality.

Researchers should minimise personal data, separate identifiers from responses, restrict access, use secure storage and define retention and deletion procedures. Sensitive media data may include political opinions, workplace allegations, private messages or information about minors. Copyright, platform terms of service and data-protection law may constrain collection and quotation.

Automated sampling, transcription, translation, sentiment analysis and AI-assisted coding can improve efficiency, but they introduce risks of opaque decisions, biased training data, privacy leakage and fabricated or distorted transcripts. AI outputs should be treated as provisional research assistance. Researchers remain responsible for checking samples, quotations, classifications and consent boundaries. A dissertation should disclose substantial automated assistance and retain an audit trail where feasible.

12. Best-Practice Checklist

Before data collection, confirm that:

• the population and unit of analysis are explicit;
• eligibility and exclusion criteria are reproducible;
• the sampling frame and its limitations are documented;
• the sampling strategy matches the research question;
• sample-size reasoning or qualitative adequacy is justified;
• every major concept has an operational definition;
• instruments have been reviewed and piloted;
• response options and missing-data codes are clear;
• interview, observation and coding procedures are standardised;
• validity and reliability evidence is appropriate;
• consent, confidentiality and data security are addressed; and
• the analysis plan is consistent with the measurement level and design.

Chapter Summary

Sampling determines which cases enter a study and what kinds of inference are defensible. Probability designs support population estimation when frames, selection and non-response are responsibly managed; non-probability designs are valuable for exploratory, qualitative and hard-to-reach research but require appropriate claims. Operationalisation translates concepts into dimensions, indicators and instrument items. Validity concerns the credibility of interpretations, while reliability concerns consistency.

Effective surveys use clear, neutral and piloted items. Interviews and focus groups require purposeful guides, skilled moderation and reflexivity. Observation protocols distinguish description from interpretation. Content-analysis schemes require explicit units, categories, coding rules and coder training. Case-study protocols integrate multiple sources through triangulation and an audit trail. Across all designs, ethical treatment, privacy, transparent limitations and careful use of technology are essential to credible mass media management research.

Key Terms

Population: The complete set of units relevant to a research question.

Sample: A subset of the population selected for study.

Sampling frame: The list or procedure from which cases are selected.

Unit of analysis: The entity about which conclusions are drawn.

Probability sampling: Sampling with known, non-zero selection probabilities.

Non-probability sampling: Sampling without known selection probabilities.

Stratified sampling: Sampling separately within defined population subgroups.

Cluster sampling: Sampling naturally occurring groups rather than isolated elements.

Purposive sampling: Deliberate selection of information-rich cases.

Saturation: The point at which additional qualitative data add limited new insight.

Conceptualisation: Defining the meaning and boundaries of an abstract concept.

Operationalisation: Translating a concept into observable indicators and procedures.

Validity: The extent to which evidence supports an intended interpretation.

Reliability: The consistency of a measurement or coding procedure.

Inter-coder reliability: Agreement among coders applying a coding scheme.

Likert-type item: A statement rated across ordered response categories.

Coding manual: A document specifying variables, categories and coding rules.

Triangulation: Comparing evidence across sources, methods, investigators or theories.

Coverage error: Bias caused by mismatch between the population and sampling frame.

Reflexivity: Critical examination of the researcher’s influence on the study.

Review Questions

Short Answer Questions

1. Distinguish the unit of analysis from the sampling unit.
2. What is the principal difference between probability and non-probability sampling?
3. Define operationalisation and provide one media-management example.
4. Why does reliability not guarantee validity?
5. State two essential elements of a content-analysis coding manual.

Long Answer Questions

1. Explain the major probability-sampling designs and identify a suitable mass-media application for each.
2. Discuss how a researcher can develop and pilot a valid and reliable survey instrument.
3. Compare interviews, focus groups and observation as methods for studying media organisations.
4. Explain how concepts such as audience engagement or news credibility can be operationalised.
5. Discuss the role of triangulation and audit trails in case-study research.

Analytical / Application Questions

1. A researcher surveys only followers of a broadcaster’s social-media page to estimate public trust in the broadcaster. Diagnose the sampling and measurement problems and propose a stronger design.
2. Design a coding scheme for analysing whether online news articles disclose the sources of factual claims. Specify the unit, variables, categories and reliability procedure.
3. A streaming platform reports high satisfaction but declining subscriptions. Design a mixed-method study showing how sampling, measurement and qualitative instruments could investigate the contradiction.

Multiple Choice Questions with Answers

Question 1

Which feature most clearly distinguishes probability sampling from non-probability sampling?

A. Probability sampling always uses interviews.

B. Probability sampling gives eligible units known, non-zero selection probabilities.

C. Probability sampling requires a qualitative research question.

D. Probability sampling eliminates all non-response.

Correct Answer: B. Probability sampling gives eligible units known, non-zero selection probabilities.

Question 2

A population contains 2,000 units. If an initial sample-size estimate is 384.16, which approximate finite-population-adjusted sample is appropriate?

A. 96

B. 192

C. 323

D. 768

Correct Answer: C. 323

Question 3

Which example is an operational definition of audience engagement?

A. Audience engagement is important to media firms.

B. Audience engagement means whatever respondents consider engaging.

C. Audience engagement is measured through viewing completion, comments and repeat visits during a specified period.

D. Audience engagement is a positive attitude toward media.

Correct Answer: C. Audience engagement is measured through viewing completion, comments and repeat visits during a specified period.

Question 4

What is a major weakness of a convenience sample?

A. It cannot contain more than one subgroup.

B. It necessarily requires a very large budget.

C. It may systematically exclude less accessible members of the population.

D. It always produces qualitative data.

Correct Answer: C. It may systematically exclude less accessible members of the population.

Question 5

Which practice best improves a questionnaire before full deployment?

A. Removing all open-ended questions without testing them.

B. Conducting cognitive interviews and a pilot test.

C. Using technical language to increase seriousness.

D. Collecting responses before defining the constructs.

Correct Answer: B. Conducting cognitive interviews and a pilot test.

Question 6

What does inter-coder reliability assess in content analysis?

A. Whether a sample represents the entire population.

B. Whether participants enjoyed the interview.

C. The consistency of coding decisions across coders.

D. Whether a theory is morally acceptable.

Correct Answer: C. The consistency of coding decisions across coders.

Question 7

Which question is double-barrelled?

A. How often did you use the platform last week?

B. Which subscription tier do you use?

C. How satisfied are you with the service’s speed and accuracy?

D. Did you cancel your subscription?

Correct Answer: C. How satisfied are you with the service’s speed and accuracy?

Question 8

Why might a researcher use purposive sampling in a qualitative study?

A. To select cases that are theoretically or substantively information-rich.

B. To guarantee a known margin of sampling error.

C. To ensure every population member has an equal chance of selection.

D. To eliminate the need for ethical approval.

Correct Answer: A. To select cases that are theoretically or substantively information-rich.

Question 9

Which statement about reliability is correct?

A. A reliable measure is necessarily valid.

B. Reliability concerns consistency, whereas validity concerns the credibility of the intended interpretation.

C. Reliability applies only to probability samples.

D. Reliability removes coverage error.

Correct Answer: B. Reliability concerns consistency, whereas validity concerns the credibility of the intended interpretation.

Question 10

What is triangulation in case-study research?

A. Using only three interview questions.

B. Comparing evidence across sources, methods, investigators or theories.

C. Selecting exactly three organisations.

D. Replacing participant consent with automated analysis.

Correct Answer: B. Comparing evidence across sources, methods, investigators or theories.