Participants, Sampling and Research Operations

Introduction

Usability testing is often described as watching people use a product, but the quality of the evidence depends heavily on who participates, how they are recruited, and how the study is operated. A beautifully designed test can produce misleading conclusions if participants do not resemble the intended users, if the sample is too narrow, or if practical arrangements introduce bias.

Participant and research operations therefore connect UX research design with responsible execution. They include defining the target population, choosing a sampling approach, recruiting and screening participants, obtaining informed consent, scheduling sessions, preparing equipment, providing incentives, protecting data, and piloting the study. These activities are not merely administrative. They influence validity, ethics, participant welfare, and the decisions made from the findings.

This chapter presents a practical framework for planning participant-based usability studies at certificate level. It emphasizes appropriate methods rather than automatic reliance on large samples or convenient recruits.

Learning objectives

By the end of this chapter, you should be able to:

• define a target-user population and relevant participant characteristics;
• distinguish a population, sampling frame, sample, and recruitment source;
• select a suitable sampling strategy for a usability study;
• make a reasoned decision about sample size and segmentation;
• create screening questions that identify relevant participants fairly;
• organize consent, privacy, accessibility, scheduling, incentives, and logistics;
• prepare a reliable session environment and conduct a pilot test;
• recognize common sources of recruitment and operational bias; and
• evaluate a participant plan ethically and professionally.

1. Defining the target users

A target population is the broader group whose needs or behavior the research is intended to inform. A sample is the smaller group who actually participates. These concepts must not be confused. A study of an online banking application might target adults who manage a personal bank account digitally, while the sample may contain twelve recruited account holders.

A useful target-user definition combines several dimensions:

Demographic characteristics should not be collected simply because they are easy to ask for. They are justified when they affect the research question, product experience, accessibility, or interpretation of findings. For example, age may be relevant to a study of readability or digital confidence, but it should not be used as a simplistic substitute for ability.

A target-user profile should be operational. “Young people” is too vague for recruitment. “People aged 18–25 who have used a mobile ticketing application at least twice in the last three months and normally purchase their own travel tickets” is more useful because it can be screened consistently. Operational definitions should also state exclusions, such as current employees of the product company, people who helped design the interface, or professional research participants whose frequent testing experience could affect behavior.

Stakeholders and user groups

Many products have more than one important user group. A hospital scheduling system may be used by receptionists, clinicians, patients, and administrators. A recruitment plan should identify primary users, secondary users, and other affected people. Testing only the most visible group can conceal serious problems for users who have different goals or authority.

Personas can help communicate user characteristics, but they are not evidence by themselves. They should be based on existing research, domain knowledge, or clearly stated assumptions. A persona must not be treated as a replacement for recruiting real people.

2. Sampling concepts and strategies

A sampling frame is the practical list or source from which participants can be recruited, such as a customer database, university community, professional association, or public recruitment panel. It may not perfectly represent the target population. For example, recruiting from an existing customer mailing list excludes people who abandoned registration or chose not to provide contact details.

A sampling strategy determines how people are selected. In UX studies, the choice is usually guided by the research purpose, risk, access, time, and desired diversity rather than by statistical representativeness alone.

Probability and non-probability sampling

Probability sampling gives members of a defined population a known, non-zero chance of selection. Simple random, systematic, stratified, and cluster sampling are examples. It is valuable when the aim is to estimate population parameters with defensible sampling error, but it can be expensive and difficult when no complete sampling frame exists.

Non-probability sampling does not provide known selection probabilities. Common forms include convenience, purposive, quota, snowball, and criterion sampling. These methods are frequent in formative usability research because researchers need people with particular experience rather than a statistically representative cross-section of the population.

A common practical approach is purposive quota sampling. For example, a team may recruit regular and occasional users, with a planned number in each group. This improves coverage, but the findings should still be described as evidence from the recruited sample, not as statistically representative estimates of all users.

Segmentation and critical participants

Segmentation is useful when different users may encounter different problems. Relevant segments can include novice and experienced users, customers and staff, mobile and desktop users, or users of different accessibility technologies. Segments should be based on plausible differences in tasks or interaction, not an excessive list of demographic categories.

Researchers should also recruit critical or edge-case participants when failure would have serious consequences. Examples include users with accessibility requirements, users operating under poor connectivity, people completing high-stakes tasks, and users with limited language proficiency where the product is intended to support them. Edge cases should not be treated as unusual problems to be ignored; they may expose design assumptions that affect many people.

3. Choosing sample size

Sample size is a design decision, not a badge of research quality. It should be connected to the purpose of the study, the number of important segments, the severity of possible failures, the diversity of tasks, and the type of evidence required.

Formative qualitative usability testing

In formative testing, the main goal is to discover usability problems and understand user behavior. Small, carefully selected groups can reveal important issues, particularly when sessions are observed closely and the team iterates between rounds. A practical cycle may use a small initial group, redesign the product, and then test again. The precise number is not universal; it depends on heterogeneity and study quality.

A homogeneous group performing a narrow task may require fewer participants than a study covering several roles, languages, devices, and accessibility needs. If the study contains distinct segments, the effective sample for each segment is what matters. Ten participants overall may be inadequate if they are divided across five substantially different groups.

Researchers should avoid claiming that a fixed number, such as five participants, will always reveal a particular percentage of problems. Such claims depend on assumptions about problem frequency, task design, participant independence, observation quality, and problem detectability. A small sample is appropriate for exploration, not for proving that no problems exist.

Summative and quantitative studies

If the study estimates a success rate, compares variants, or supports a high-consequence decision, larger samples and more formal designs may be appropriate. Researchers should define the primary metric, comparison, expected difference, acceptable uncertainty, and analysis plan before collecting data. Statistical power calculations may be needed for controlled comparisons, and a qualified analyst should be consulted when decisions are consequential.

For estimating a proportion, a commonly used planning expression is:

Here, n₀ is the initial required sample size for a proportion, z is the standard-normal critical value associated with the selected confidence level, p is the anticipated proportion, and e is the desired margin of error expressed as a proportion. This formula assumes a simple random sample, independent observations, a sufficiently large population, and a proportion estimate. If no prior estimate of p is available, p = 0.5 is conservative because it gives the largest product p(1−p). The formula is not a justification for a small qualitative usability sample.

When the population is finite and sampling is a substantial fraction of it, a finite-population correction can be applied:

Here, N is the size of the eligible population and n is the adjusted sample size. These expressions address precision under specified statistical assumptions; they do not correct for poor recruitment, nonresponse bias, or an unrepresentative sampling frame.

A practical decision sequence

1. State the research decision: discovery, comparison, estimation, or validation.
2. Identify meaningful user segments and tasks.
3. Determine the minimum evidence needed for each segment.
4. Consider risk, accessibility, product maturity, and iteration opportunities.
5. Estimate recruitment losses and no-shows.
6. Pilot the plan and revise it if eligibility or task difficulty is unclear.
7. Document the rationale rather than presenting a number without explanation.

4. Recruitment and screening

Recruitment materials should explain the study honestly without revealing so much about the expected “correct” behavior that participants rehearse it. A good invitation states the general topic, eligibility requirements, approximate duration, location or remote format, recording arrangements, incentive, privacy information, and contact method.

Recruitment channels should match the target population. Possible channels include customer invitations, community organizations, professional networks, universities, social media, research panels, and accessibility organizations. Each channel creates coverage limitations. A panel may overrepresent people comfortable with research; a customer list may exclude former users; a workplace list may create pressure to participate.

Designing a screener

A screener is a short questionnaire or interview used to determine eligibility. It should:

• begin with essential criteria;
• use neutral, understandable wording;
• ask behavior-based questions where possible;
• avoid unnecessary sensitive information;
• separate eligibility from demographic description;
• randomize answer options when order could bias responses; and
• include quality checks without becoming deceptive or burdensome.

Behavioral questions are often more reliable than self-labels. “How many times have you completed an online tax return in the past year?” is more useful than “Are you an expert at online tax?” However, recall can still be imperfect, so screening should not be treated as absolute proof.

Researchers must avoid discriminatory recruitment. Exclusion is justified when it relates to the study, safety, data protection, language, product access, or a necessary research condition. It is not justified merely because a group is inconvenient to recruit. Screening questions should be reviewed for accessibility and translated appropriately when multilingual participation is intended.

A screener should be stored separately from research findings when it contains contact details or sensitive information. The recruitment database should record status—invited, screened, eligible, scheduled, completed, or declined—without retaining unnecessary personal data.

5. Consent, privacy, and participant welfare

Informed consent is a process, not merely a signature. Before participation, people should understand:

• the purpose and general activities of the study;
• what data will be collected, including audio, video, screen, or biometric data;
• whether observers or stakeholders may attend;
• foreseeable risks, discomforts, or sensitive topics;
• how data will be stored, used, shared, and deleted;
• whether participation is voluntary and how to withdraw;
• the incentive and any conditions attached to it; and
• who to contact with questions or concerns.

Consent must be understandable and proportionate. Participants should not be required to agree to unrelated future uses of their data. A person may consent to screen recording but decline face-camera recording, if the study can operate without it. Researchers should explain what happens if a participant becomes uncomfortable or encounters personal information during a task.

Privacy involves more than removing names. Screen recordings may reveal messages, passwords, health information, financial details, or other people’s data. The protocol should minimize collection, use test accounts where possible, mask sensitive content, control access, encrypt files, and establish retention and deletion periods. In remote testing, participants should be told whether the session platform processes data outside their country and whether observers can see their screen.

Consent requirements vary by institution and jurisdiction. Studies conducted through an educational institution, involving vulnerable populations, or collecting sensitive data may require ethics review or a formal data-protection assessment. Researchers should follow applicable law and institutional policy rather than assuming that a low-risk usability study is exempt.

Children and adults with limited decision-making capacity require additional safeguards, including appropriate permission, assent, safeguarding procedures, and staff competence. Incentives must not be so large that they create undue pressure. Payments should compensate time and inconvenience, not reward a particular answer.

6. Scheduling and incentives

Scheduling is an operational design problem. A participant who arrives late, uses an unsupported device, or cannot find the remote meeting may produce unusable data and a poor experience. Confirmation messages should include the date, time zone, expected duration, location or link, preparation instructions, accessibility contact, cancellation method, and privacy reminder.

A scheduling system should track availability, segment balance, confirmation, reminders, attendance, and replacement needs. It should avoid exposing participants’ personal information to one another. Researchers should plan for cancellations and maintain a small reserve list, but reserve participants should not be recruited in a way that creates unfair overbooking.

Incentives should be decided before recruitment and communicated clearly. Factors include session length, preparation, travel, inconvenience, expertise, and local norms. A consistent policy is preferable to ad hoc negotiation. Payment should not depend on whether the participant succeeds at the task or gives positive feedback. If a session is stopped because of a researcher or technical failure, the policy should specify whether full compensation is provided.

7. Equipment and session operations

A session plan should specify the moderator, note-taker, observer limits, tasks, timing, recording setup, backup method, and escalation process. The moderator’s role is to create realistic conditions while avoiding unnecessary assistance. Leading prompts such as “Can you find the blue button?” contaminate evidence; neutral prompts such as “What are you thinking?” or “What would you do next?” are generally safer, used sparingly.

Equipment may include a test device, charging cable, microphone, camera, screen-recording software, task materials, consent forms, and a backup device. The team should verify:

• screen and audio quality;
• battery and network reliability;
• correct prototype version;
• test accounts and reset procedures;
• recording permissions;
• accessibility settings;
• observer visibility and confidentiality; and
• a method for recording notes if technology fails.

Remote sessions require additional preparation. Participants may have limited bandwidth, small screens, unfamiliar meeting software, or privacy concerns at home. A brief technical check should not exclude users unnecessarily. If the product itself is mobile, testing on a desktop emulator may produce invalid conclusions.

A session script supports consistency. It commonly includes greeting, consent confirmation, background questions, task instructions, observation prompts, follow-up questions, and closing. The script should distinguish instructions from probes. Researchers should not turn every session into a conversation that changes the task for each participant unless the study explicitly uses an exploratory interview format.

8. Pilot testing

A pilot is a rehearsal with a person or small number of people who resemble the intended participants. It tests the research operation, not only the interface. A pilot can reveal that eligibility criteria are ambiguous, tasks take too long, consent language is unclear, recording fails, incentives are misunderstood, or the moderator accidentally gives away answers.

The pilot should examine:

• recruitment and screening flow;
• participant comprehension of instructions;
• task order and estimated duration;
• suitability of the device and environment;
• recording and data-storage procedures;
• accessibility and language needs;
• note-taking and coding feasibility; and
• participant burden and emotional comfort.

Pilot findings may require changing the protocol. If tasks or eligibility rules change substantially, the team should determine whether pilot data can be combined with formal data. Often it should not be combined, because the conditions were different. A pilot is successful when it prevents avoidable failures, not when it confirms that every original plan was correct.

9. An operational workflow

A robust workflow moves through linked stages:

Figure: Participant operations form an iterative workflow rather than a one-way administrative sequence.

The workflow is iterative. A pilot may expose a need to recruit a different segment, simplify the screener, revise consent wording, or change the equipment. Documentation should record decisions and their reasons so that later readers can distinguish planned procedures from changes made during fieldwork.

10. Illustrative case: a mobile public-transport ticketing service

Background

A transport authority is redesigning its mobile ticketing service. The main task is purchasing a same-day ticket before boarding. The service is used by commuters, occasional travelers, visitors, older adults, and people who use screen readers or enlarged text.

Problem

The team initially proposes recruiting ten university students because they are available. This would be convenient, but students may differ from regular commuters in travel routines, payment experience, urgency, and accessibility needs. The sample could miss failures affecting occasional users and people buying tickets under time pressure.

Analysis

The research question is formative: identify obstacles in ticket selection, payment, and ticket retrieval. The team maps three relevant segments—regular commuters, occasional travelers, and users of accessibility features. It decides that the segments require separate coverage because their contexts and interaction methods differ. The team also identifies a high-risk scenario: a user must find a valid ticket quickly with poor connectivity.

Approach and solution

The team uses purposive quota recruitment through a commuter organization, a public advertisement, and an accessibility group. The screener asks about recent ticket-purchasing behavior, device use, accessibility settings, and familiarity with the current service. It excludes employees involved in the redesign and professional UX researchers.

The consent form explains screen recording, the use of a test payment account, voluntary participation, and data deletion. Sessions are conducted on participants’ own phones when possible, with a prepared test device available as a backup. The team pilots the study and discovers that the original task wording reveals the expected ticket type. It replaces that wording with a realistic travel scenario.

Outcome

The study identifies confusion between “single” and “same-day” tickets, difficulty locating a purchased ticket offline, and a screen-reader problem caused by an unlabeled control. These issues would probably not have been detected reliably through the original student-only sample. The findings are not claimed to represent every transport customer statistically; they are used to prioritize design changes and plan a later validation study.

Learning points

The case demonstrates that representativeness is purpose-dependent. A small purposive sample can be more useful than a larger convenience sample when it includes the contexts and capabilities relevant to the task. It also shows that recruitment, consent, equipment, and task wording are connected parts of research validity.

11. Challenges, bias, and limitations

Recruitment bias occurs when some eligible people are more likely to be invited, respond, or complete the study than others. Common causes include overreliance on existing customers, digital-only recruitment, language barriers, inaccessible venues, inconvenient times, and incentives that appeal more to some groups.

Selection bias cannot always be eliminated, but it can be managed. Researchers should document recruitment sources, compare achieved participants with the target definition, monitor segment balance, offer accessible formats, and state limitations in reporting. A quota can improve visible coverage but cannot remove all bias.

Professional participants may be skilled at verbalizing behavior or guessing study goals. This does not make them unusable, but their experience should be recorded and considered. Workplace recruitment may create perceived pressure, especially when managers invite subordinates. Participation should be private, voluntary, and unrelated to performance evaluation.

Operational failures also threaten validity. A prototype that crashes only on a participant’s device, an observer who interrupts, or a moderator who provides unequal assistance can make sessions incomparable. Standardized preparation, pilot testing, and a deviation log help researchers interpret such events.

12. Contemporary practice and AI

Remote and unmoderated testing can increase geographic reach, but they reduce the researcher’s ability to clarify confusion and verify who completed the session. Unmoderated studies require especially clear tasks, accessibility checks, fraud prevention, and careful interpretation of incomplete data.

AI tools may assist with recruitment-message drafting, transcript search, scheduling, or preliminary clustering. They should not independently decide participant eligibility where sensitive characteristics or high-stakes exclusions are involved. Automated systems can reproduce language, demographic, or accessibility bias. Researchers must verify AI-generated summaries against recordings or notes, avoid uploading identifiable data to unauthorized services, and disclose relevant processing. AI can support operations, but accountability for consent, privacy, sampling decisions, and interpretation remains with the research team.

Best-practice checklist

Before fieldwork, confirm that:

• the target population and important segments are explicitly defined;
• every eligibility criterion has a research-based justification;
• the sample-size rationale matches the study purpose;
• recruitment sources and limitations are documented;
• screening is neutral, accessible, and privacy-conscious;
• consent covers recording, data use, withdrawal, and risks;
• incentives and cancellation policies are clear;
• devices, accounts, prototypes, and backups have been tested;
• accessibility and language needs have been planned;
• a pilot has been completed and protocol changes recorded; and
• data access, retention, deletion, and incident procedures are defined.

Chapter Summary

Participant planning is a central part of usability research quality. Researchers begin by defining the target population in terms of roles, goals, contexts, experience, devices, accessibility, and relevant risk. They distinguish that population from the sampling frame and recruited sample, then select a sampling strategy suited to the research purpose.

Formative usability testing commonly uses small, purposive samples to discover problems, while estimation and comparative studies may require larger, more formal designs. Sample size must account for user segments, task diversity, risk, and expected recruitment loss; no universal participant number guarantees complete discovery. Screening should be neutral and behavior-focused, and recruitment limitations should be reported.

Consent, privacy, accessibility, incentives, scheduling, equipment, and pilot testing are methodological concerns as well as operational tasks. Ethical research minimizes data collection, protects participants from pressure and harm, and respects withdrawal. A pilot tests the entire research operation and enables justified revisions. Finally, remote tools and AI can improve efficiency but introduce new risks involving access, bias, privacy, and accountability.

Key Terms

Target population: The broader group to which a study’s research purpose relates.

Sample: The people who actually participate in a study.

Sampling frame: The practical list or source from which participants are recruited.

Sampling strategy: The method used to select or locate participants.

Probability sampling: Sampling in which eligible members have known, non-zero selection probabilities.

Non-probability sampling: Sampling without known selection probabilities.

Purposive sampling: Deliberately recruiting people with characteristics relevant to the research question.

Quota sampling: Recruiting planned numbers from specified categories.

Criterion sampling: Recruiting people who meet defined eligibility conditions.

Segmentation: Dividing users into groups expected to differ in relevant behavior or context.

Screener: A questionnaire or interview used to determine eligibility.

Informed consent: Voluntary agreement based on understandable information about the study.

Incentive: Compensation provided for time, effort, or inconvenience.

Pilot test: A rehearsal used to evaluate and improve the research procedure.

Sampling bias: Systematic distortion caused by unequal inclusion or participation.

Formative testing: Research intended primarily to discover problems and guide improvement.

Summative testing: Research intended to evaluate or estimate performance under defined conditions.

Sampling frame coverage: The extent to which a recruitment source includes the intended population.

Research operations: The practical procedures that enable a study to be conducted consistently and safely.

Review Questions

Short Answer Questions

1. Distinguish a target population, sampling frame, and sample.
2. Why might purposive quota sampling be appropriate for formative usability testing?
3. List four items that should be explained during informed consent.
4. What is the purpose of a pilot test?
5. Why should an incentive not depend on whether a participant succeeds at a task?

Long Answer Questions

1. Explain how a UX researcher could define and segment users for a mobile banking usability study.
2. Compare probability sampling with non-probability sampling, including one strength and one limitation of each.
3. Discuss how sample size decisions differ between formative usability testing and a quantitative comparative study.
4. Explain the relationship between recruitment, privacy, informed consent, and data protection.
5. Describe the equipment, scheduling, moderation, and backup procedures required for a reliable remote usability session.

Analytical / Application Questions

1. A team recruits only its existing customers for a study of why people abandon registration. Identify likely coverage limitations and propose two alternative recruitment sources.
2. A study has three substantially different user segments but plans to recruit twelve participants in total. Analyze what additional information is needed before judging whether the sample is adequate.
3. During a pilot, participants misunderstand the task and the screen recorder captures personal notifications. What changes should the research team make before formal data collection, and why?

Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes a sample from a target population?

A. A sample is the broader group, while a target population is the recruited group.

B. A sample is the people who participate, while the target population is the broader group the study concerns.

C. A sample is always statistically representative, while a target population is not.

D. A sample is a recruitment source, while a target population is a consent form.

Correct Answer: B. A sample is the people who participate, while the target population is the broader group the study concerns.

Question 2

Which sampling approach deliberately recruits people with characteristics relevant to a research question?

A. Purposive sampling

B. Simple random sampling

C. Cluster sampling

D. Systematic sampling

Correct Answer: A. Purposive sampling

Question 3

What is the main purpose of a screener?

A. To persuade participants to give positive feedback

B. To determine whether potential participants meet eligibility criteria

C. To replace informed consent

D. To calculate the product’s conversion rate

Correct Answer: B. To determine whether potential participants meet eligibility criteria

Question 4

Which practice best supports informed consent?

A. Hiding recording details to prevent participant anxiety

B. Explaining data collection, use, storage, risks, and withdrawal rights

C. Making payment conditional on successful task completion

D. Allowing observers to join without disclosure

Correct Answer: B. Explaining data collection, use, storage, risks, and withdrawal rights

Question 5

Why can a small formative usability sample be useful?

A. It guarantees discovery of every usability problem.

B. It can reveal important problems when participants are carefully selected and sessions are observed closely.

C. It removes all sampling bias.

D. It provides statistically precise population estimates automatically.

Correct Answer: B. It can reveal important problems when participants are carefully selected and sessions are observed closely.

Question 6

Which question is most behavior-focused for a screener?

A. Are you highly digital?

B. Would you describe yourself as an expert?

C. How many times have you purchased a ticket through an app in the past three months?

D. Do you think technology is easy?

Correct Answer: C. How many times have you purchased a ticket through an app in the past three months?

Question 7

What should a pilot primarily evaluate?

A. Whether the final report looks attractive

B. Whether the research procedure, tasks, equipment, and consent process work as intended

C. Whether participants agree with the design team

D. Whether the sample is automatically nationally representative

Correct Answer: B. Whether the research procedure, tasks, equipment, and consent process work as intended

Question 8

Which is an example of sampling-frame coverage limitation?

A. A recorder has a low battery.

B. A customer mailing list excludes people who never registered for the service.

C. A moderator asks a neutral follow-up question.

D. A participant receives a clearly stated incentive.

Correct Answer: B. A customer mailing list excludes people who never registered for the service.

Question 9

In the proportion-planning expression n₀ = (z² × p × (1 − p)) ÷ e², what does e represent?

A. The sample’s average age

B. The desired margin of error

C. The population size

D. The incentive amount

Correct Answer: B. The desired margin of error

Question 10

Which action is most appropriate when a remote session may expose personal notifications?

A. Ignore the issue because usability data are never sensitive.

B. Use privacy precautions such as test accounts, participant warnings, and controlled recording.

C. Ask the participant to share passwords with the moderator.

D. Publish the recording without removing identifying information.

Correct Answer: B. Use privacy precautions such as test accounts, participant warnings, and controlled recording.