Research Questions, Study Planning and Success Criteria

Introduction

Usability testing is most valuable when it answers a clearly defined decision problem. A study should not begin with the vague intention to “test the interface.” It should begin with an understanding of what the organisation needs to learn, what users need to accomplish, what uncertainty exists, and what evidence will justify a design decision.

This chapter explains how to translate product goals and user needs into research questions, hypotheses, tasks, scenarios, measures, and practical test plans. It also distinguishes formative evaluation, which supports improvement during design, from summative evaluation, which assesses performance against defined criteria. The emphasis is on practical planning suitable for certificate-level UX work, while maintaining the logic and discipline of applied research.

Learning Objectives

By the end of this chapter, you should be able to:

• distinguish product goals, user needs, research questions, hypotheses, and usability objectives;
• write focused and answerable usability research questions;
• select formative or summative evaluation appropriately;
• design realistic scenarios and task instructions;
• choose behavioural, attitudinal, and outcome metrics;
• define success criteria before collecting data;
• prepare a feasible usability test plan;
• identify threats to validity, ethical risks, and practical limitations; and
• interpret findings without confusing usability evidence with proof of causation.

1. From Product Goals to Research Questions

A product goal describes a desired organisational or product outcome. Examples include increasing successful account registration, reducing support calls, or enabling customers to complete a purchase. A user need describes a problem or expectation from the user’s perspective, such as understanding delivery costs before payment or recovering access without contacting support.

Neither statement is yet a research question. A research question specifies what the team needs to learn in order to make a decision.

For example:

• Product goal: increase completion of mobile checkout.
• User need: understand the total cost and delivery date before committing.
• Design uncertainty: users may not notice delivery information until late in the process.
• Research question: How do mobile shoppers interpret delivery cost and timing during checkout, and where do they experience uncertainty?

A useful research question is specific enough to guide recruitment, task design, observation, and analysis. It should also be open to evidence. “Do users like the new design?” is usually too broad because liking can mean visual appeal, trust, familiarity, or satisfaction. A stronger question is: “Can first-time users locate and understand the delivery-date information before placing an order?”

1.1 A planning chain

A practical planning chain is:

Goal → user need → uncertainty → research question → method → task → measure → success criterion → decision.

The chain prevents a common error: selecting a method first and then searching for a question that fits it. The question should determine the method.

1.2 Types of research questions

Research questions commonly address five areas:

1. Discoverability: Can users find a feature, control, or item?
2. Comprehension: Do users understand labels, instructions, status, or consequences?
3. Task performance: Can users complete an important task accurately and efficiently?
4. Experience: Do users feel confident, comfortable, or satisfied?
5. Comparison: Does one design perform better than another for a defined population and task?

A single study may include several question types, but excessive scope weakens the study. A short certificate-level usability study is usually better served by two to four priority questions than by an extensive list of loosely related questions.

2. Research Questions, Hypotheses, and Objectives

These terms are related but not interchangeable.

A research objective states what the study intends to accomplish: “Assess whether new users can set up two-factor authentication.”

A research question asks what the study must discover: “Where do new users become confused when setting up two-factor authentication?”

A hypothesis is a provisional, testable prediction: “Because the security option is placed under an unfamiliar label, first-time users will overlook it.”

A usability hypothesis should connect a design condition to an observable outcome. For example:

If the form groups optional fields under a collapsible section, users will complete the mandatory information with fewer unnecessary errors than when all fields are displayed together.

The hypothesis contains a condition, a population or context, and an expected outcome. It should not be treated as a fact. In exploratory formative research, hypotheses help focus observation but should not prevent researchers from noticing unexpected problems.

2.1 Qualitative and quantitative questions

Qualitative questions explore meaning, interpretation, strategies, and causes. They often use wording such as “how,” “why,” or “what do participants understand?” Quantitative questions concern measurable performance, such as completion rate, time, error frequency, or rating.

For example:

• Qualitative: “What does the phrase ‘continue securely’ mean to participants at this stage?”
• Quantitative: “What proportion of participants submit the form without a critical error?”

A mixed question can combine them: “How many participants complete the form successfully, and what explains the failures?” Numbers describe the pattern; observations and interviews help explain it.

3. Choosing Formative or Summative Evaluation

3.1 Formative evaluation

Formative evaluation is conducted to identify problems and improve a design. It is normally performed during discovery, prototyping, or iterative development. The study may use a small number of participants, especially when the aim is to uncover obvious usability problems rather than estimate a population parameter.

Typical formative questions include:

• Where do users hesitate?
• Which labels are misunderstood?
• What incorrect assumptions do users make?
• What prevents task completion?
• Which design alternative appears easier to understand?

Formative testing values diagnostic richness. A moderator may ask participants to explain their interpretation, although questioning should not lead them toward an answer.

3.2 Summative evaluation

Summative evaluation assesses a relatively mature product against predetermined criteria. It may support release decisions, procurement, compliance evidence, or comparison with a previous version. It requires tightly specified tasks, consistent procedures, and pre-defined measures.

Typical summative questions include:

• What percentage of the target population completes the critical task?
• How long does successful completion take?
• How many critical errors occur?
• Does the product meet the organisation’s usability benchmark?

Summative results should not be overstated. A small convenience sample can show that participants encountered a problem, but it cannot reliably establish population performance. Claims should match the design, sample, and evidence.

3.3 Selecting the evaluation type

Use formative evaluation when the design is still changeable and the primary need is learning. Use summative evaluation when the design is stable enough for consistent measurement and the primary need is judgment against a criterion.

A study can contain both components, but they should be distinguished. For example, a benchmark task may be followed by interviews, yet exploratory comments should not be allowed to alter the predefined scoring rules.

4. Writing Strong Research Questions

A strong question is focused, neutral, answerable, and connected to a decision. The following structure is useful:

Among [target users] in [context], how well can they [perform an important activity], and what prevents or supports successful completion?

For comparison:

For [target users] completing [defined task], does version A or version B produce better [specified outcome] under [specified conditions]?

Avoid questions that contain an assumed answer, such as “How can we prove the redesign is better?” Replace them with “What differences, if any, occur between the current design and the redesign for the defined task?”

Avoid questions that are too broad, such as “Is the application usable?” Usability is multidimensional. Break it into effectiveness, efficiency, satisfaction, learnability, accessibility, or error recovery as appropriate.

4.1 Prioritising questions

Prioritise questions using three factors:

• Impact: How serious would the problem be?
• Uncertainty: How little does the team currently know?
• Decision value: How likely is the answer to change the design or product decision?

A question about a safety-critical payment confirmation usually deserves priority over a question about a minor icon preference. Questions should also be feasible within the available time, participant pool, prototype fidelity, and technical environment.

5. Designing Tasks and Scenarios

A usability task asks participants to attempt an activity. A scenario provides the context, motivation, and desired outcome. Good tasks resemble realistic goals rather than interface instructions.

Weak instruction: “Click the Profile icon and select Notification Preferences.”

Stronger scenario: “You have started receiving too many promotional messages. Adjust your account so that you continue receiving order updates but stop promotional notifications.”

The second version tests whether users can understand the organisation of notification choices, not merely follow the researcher’s directions.

5.1 Characteristics of effective tasks

An effective task:

• reflects a real or plausible user goal;
• contains enough context to make the goal meaningful;
• does not reveal the intended control or path;
• has a clear endpoint;
• can be completed within the session time;
• is safe and ethically appropriate; and
• produces observable evidence relevant to a research question.

Tasks should avoid unnecessary reading burden, hidden assumptions, and artificial difficulty. If a task requires information not visible in the interface, provide it consistently to all participants.

5.2 Task sequencing

Begin with a low-risk orientation task so participants learn the test procedure. Place critical tasks when participants are attentive, but avoid giving away answers through the order of tasks. If prior tasks teach the interface, record the potential learning effect. For comparisons, counterbalance the order of versions where practical so that fatigue or practice does not systematically favour one version.

5.3 Avoiding leading tasks

A leading task tells participants what to notice or how to act. “Find the blue security button” tests recognition of the researcher’s clue rather than natural discoverability. The task should state the user’s objective, not the interface solution.

6. Selecting Metrics and Measures

A metric is a defined way of representing evidence. Measures should be selected because they answer a research question, not because they are easy to collect.

6.1 Core usability dimensions

• Effectiveness: whether users achieve the intended goal accurately.
• Efficiency: the resources required, often time or interaction effort.
• Satisfaction: users’ perceived comfort, confidence, or acceptability.
• Learnability: how performance changes as users gain experience.
• Error tolerance and recovery: whether users avoid, understand, and recover from errors.
• Accessibility: whether people with diverse abilities can perceive, operate, understand, and recover within the product.

A simple completion rate is:

The denominator must be defined. If a participant is excluded because of a technical failure unrelated to the design, the exclusion should be documented rather than silently removed. Completion criteria must also be stated in advance.

For a task attempted by 10 participants, with 8 successful completions:

This calculation describes the observed sample; it does not automatically mean that exactly 80% of all users would succeed.

Task time is commonly recorded from the agreed start event to the agreed completion event. Units must be consistent, usually seconds or minutes. Time should be interpreted alongside success: a fast but incorrect completion is not evidence of effective use.

Error rate requires an explicit error definition. One possible descriptive measure is:

This measure is meaningful only when errors are coded consistently. A backtrack, hesitation, or wrong selection may be coded differently depending on the research question.

6.2 Severity and prioritisation

A practical severity scheme can classify findings as:

• Critical: prevents completion, causes serious harm, or creates a high-risk misunderstanding;
• Major: substantially delays or impairs completion and is likely to affect many users;
• Minor: causes annoyance or inefficiency but has an easy workaround; and
• Cosmetic: does not materially affect task performance.

Severity should consider impact, frequency, and recoverability. A problem observed once is not automatically minor if it could lead to financial loss or safety harm. Conversely, a frequent cosmetic preference may not justify immediate redesign.

6.3 Attitudinal measures

Post-task questions can capture confidence or perceived difficulty. A short rating scale is useful only when its wording and anchors are consistent. Standardised questionnaires can support comparison, but researchers should follow the instrument’s instructions and avoid presenting a locally modified scale as a validated measure.

Think-aloud comments are valuable qualitative evidence, but speaking aloud may increase task time and alter behaviour. For summative timing, a retrospective interview or separate think-aloud phase may be more appropriate.

7. Defining Success Criteria Before Testing

Success criteria convert vague expectations into observable decision rules. They should be defined before data collection to reduce hindsight bias.

A criterion may specify:

• the target population;
• the task and completion standard;
• the minimum success level;
• an acceptable error or time threshold;
• the confidence or satisfaction expectation; and
• the action if the criterion is not met.

Example:

At least 8 of 10 representative participants will locate the cancellation option and complete the cancellation without a critical error. No participant should mistake cancellation for account deletion.

This criterion is appropriate for a formative gate, but it is not a universal statistical standard. The number 8 of 10 must be justified by risk, available resources, and the study’s purpose.

Success criteria should distinguish must-have outcomes from nice-to-have outcomes. A payment confirmation task may require zero critical misunderstandings, while a secondary preference task may tolerate minor inefficiency.

7.1 Avoiding arbitrary thresholds

A threshold should be based on user risk, business consequences, prior performance, accessibility requirements, contractual commitments, or an explicit team decision. Researchers should record the rationale. Changing the threshold after seeing the results undermines the credibility of the evaluation.

8. The Usability Test Plan

A test plan is a working research document that aligns stakeholders and protects consistency. It should include:

1. Background and decision context: why the study is needed.
2. Research objectives and questions: what must be learned.
3. Evaluation type: formative, summative, or clearly separated mixed evaluation.
4. Participants: eligibility, recruitment source, sample rationale, and accessibility needs.
5. Prototype or product: version, platform, data state, and known limitations.
6. Tasks and scenarios: exact wording and completion rules.
7. Procedure: greeting, consent, orientation, task order, probes, and closing.
8. Measures: operational definitions, recording method, and units.
9. Success criteria: thresholds and decision rules.
10. Analysis plan: coding, aggregation, comparison, and reporting.
11. Risks and ethics: privacy, distress, compensation, recording, and withdrawal.
12. Logistics: equipment, timing, backup plan, and responsibilities.

A pilot session is essential. It can reveal ambiguous task wording, broken links, unrealistic timing, recording failures, or moderator prompts that unintentionally provide answers. Pilot data should normally be labelled separately if the procedure changes materially afterward.

9. Moderation, Observation, and Analysis

The moderator should create a supportive environment without solving the task. A neutral prompt such as “What are you thinking?” is generally preferable to “Did you see the settings menu?” The moderator should distinguish a participant’s natural discovery from assistance provided by the study team.

Observation notes should separate:

• what the participant did;
• what the participant said;
• the researcher’s interpretation; and
• the potential design implication.

This separation reduces premature conclusions. For example, “paused for 12 seconds and selected the wrong option” is an observation. “The label is confusing” is an interpretation that should be supported by patterns across participants or by the participant’s explanation.

A basic analysis process is:

1. clean and organise notes or recordings;
2. code task outcomes using predefined rules;
3. identify recurring behaviours and breakdowns;
4. examine differences by task, participant characteristic, or version;
5. assess severity and confidence in each finding;
6. compare observations with success criteria; and
7. connect findings to specific design recommendations.

Do not count comments as independent participants if several comments arise from one person. Similarly, do not treat every observed difficulty as a confirmed universal problem. Report the evidence and its limitations.

10. Illustrative Case: Improving a Mobile Prescription-Refill Flow

This is a fictional illustrative case.

Background

A community pharmacy is redesigning its mobile prescription-refill service. The product goal is to reduce calls to the pharmacy and increase successful refill requests. Interviews indicate that customers want to know whether a prescription is eligible, when it will be ready, and whether insurance information has been accepted.

Problem and research questions

The prototype places eligibility information behind an expandable panel. The team is uncertain whether users will find it and whether the phrase “request refill” clearly communicates that the request is not yet approved.

The study asks:

1. Can returning customers submit a refill request without assistance?
2. Do users understand the difference between submitting a request and receiving approval?
3. Can users locate the expected pickup time and identify what to do if insurance information is rejected?

The evaluation is formative because the prototype can still be changed.

Participants and procedure

The team recruits 8 adults who have refilled a prescription within the previous year and who use a smartphone. Participants receive a consent explanation and are told that the product, not the participant, is being evaluated. Each session lasts approximately 30 minutes. The moderator observes screen interaction and asks brief neutral probes after each task.

Tasks and measures

Task 1: “You are running low on your regular medication. Use this service to request a refill and tell us what you expect to happen next.”

Task 2: “Before submitting, find out when the pharmacy expects the medication to be ready.”

Task 3: “The service tells you that your insurance information needs attention. Decide what you would do next.”

Measures include successful completion, critical misunderstandings, time to locate pickup information, observed errors, and confidence rated from 1 to 5 after each task. A critical misunderstanding is coded when a participant believes that the medication is approved or ready solely because the request was submitted.

Success criteria

• At least 7 of 8 participants submit a refill request without moderator assistance.
• No more than 1 participant interprets submission as approval.
• At least 6 of 8 participants locate the expected pickup information.
• The team identifies a clear next action for every major error observed.

Findings and solution

Six participants submitted the request successfully. Three interpreted “request refill” as confirmation that the prescription was ready, and five overlooked the expandable eligibility panel. Participants also differed in their interpretation of the insurance warning: some expected a telephone call, while others searched for an upload function.

The team changed the primary action label to “Send refill request,” added a confirmation message stating “Your pharmacy must approve this request,” displayed the expected response time on the confirmation screen, and replaced the expandable eligibility panel with a visible summary. The insurance message included a direct explanation and a phone contact option.

Outcome and learning points

A second formative round showed improved understanding, although the team did not claim that the design was universally usable. The case demonstrates several principles:

• the research questions arose from product goals and user uncertainty;
• tasks described realistic goals rather than interface routes;
• critical misunderstandings were defined before testing;
• qualitative observations explained why completion failed; and
• the outcome was a design decision, not merely a list of participant opinions.

11. Challenges, Ethics, and Limitations

Usability studies involve people, personal information, and sometimes sensitive tasks. Participants must understand the purpose, activities, recording practices, compensation, foreseeable risks, and their right to stop without penalty. Do not collect unnecessary health, financial, password, or identity information. Use synthetic accounts and test data whenever possible.

Recruitment should not exploit dependent relationships. Employees, students, or patients may feel pressured to participate if the researcher controls access to a benefit. Compensation should be fair and should not make withdrawal impractical.

Accessibility is both an ethical and methodological concern. A study that excludes keyboard users, screen-reader users, older adults, people with limited vision, or people with cognitive disabilities cannot support claims about those groups. Recruitment criteria and task materials should accommodate participants’ needs, and accommodations should be recorded because they may affect procedure and interpretation.

Remote testing introduces additional limitations: network delay, unfamiliar devices, privacy risks, and reduced visibility of physical interaction. In-person testing introduces travel and observer effects. Prototype fidelity can also distort findings. A static mock-up may reveal comprehension problems but cannot validly measure actual loading time or error recovery.

AI tools may assist with drafting task alternatives, organising notes, or identifying repeated terms, but researchers remain responsible for confidentiality, coding accuracy, and interpretation. Sensitive recordings should not be uploaded to an unapproved service. AI-generated summaries can omit context or reproduce bias; important findings should be checked against source observations.

12. Best-Practice Checklist

Before the study:

• define the decision the study will inform;
• prioritise a small number of research questions;
• state hypotheses as predictions rather than facts;
• write realistic, non-leading tasks;
• define completion, error, and severity rules;
• select measures that answer the questions;
• set success criteria before collecting data;
• pilot the procedure; and
• obtain informed consent and protect participant information.

During the study:

• use the same core procedure for all participants;
• avoid teaching the interface unless teaching is part of the research question;
• distinguish observation from interpretation;
• record assistance and technical failures;
• remain neutral and respectful; and
• monitor accessibility and participant wellbeing.

After the study:

• analyse evidence using the predefined rules;
• report both successes and failures;
• avoid generalising beyond the sample and method;
• prioritise findings by impact, frequency, and recoverability;
• connect each recommendation to evidence; and
• state what remains unknown and what should be tested next.

Chapter Summary

Effective usability testing begins with a decision problem, not with a favourite method. Product goals and user needs are translated into uncertainties, research questions, hypotheses, tasks, measures, and success criteria through a connected planning chain. Formative evaluation supports iterative improvement and diagnostic learning, whereas summative evaluation judges a stable design against predefined standards. Strong tasks express realistic user goals without revealing the interface solution. Metrics such as completion, time, errors, and confidence are useful only when operational definitions and limitations are clear. Success criteria must be established before testing and justified by risk and decision needs. A sound test plan specifies participants, procedure, tasks, measures, analysis, ethics, and logistics. Finally, credible UX research respects participants, protects data, acknowledges sampling and prototype limitations, and turns evidence into actionable design decisions.

Key Terms

Research question: A focused question describing what a study must learn.

Product goal: A desired organisational or product outcome.

User need: A user’s problem, expectation, or desired capability.

Hypothesis: A testable prediction about a design and an observable outcome.

Formative evaluation: Evaluation intended to identify problems and improve a design.

Summative evaluation: Evaluation intended to judge performance against predefined criteria.

Usability task: An activity that a participant attempts during a study.

Scenario: Context and motivation surrounding a task.

Effectiveness: Accuracy and completeness with which users achieve goals.

Efficiency: Resources required to achieve a goal, often time or effort.

Satisfaction: Users’ perceived comfort, confidence, or acceptability.

Completion rate: The proportion of valid task attempts completed successfully.

Critical error: An error that prevents completion or creates serious risk or misunderstanding.

Success criterion: A predefined standard used to judge whether an outcome is acceptable.

Operational definition: A precise rule for identifying or measuring a variable.

Pilot test: A small preliminary test of the study procedure.

Think-aloud: A method in which participants verbalise thoughts during interaction.

Moderator: The researcher who guides a test session without solving tasks.

Validity: The extent to which evidence supports the intended interpretation.

Severity: The seriousness of a usability problem and its consequences.

Review Questions

5 Short Answer Questions

1. What is the difference between a product goal and a research question?
2. Give one reason why a usability task should not reveal the intended interface control.
3. Define formative evaluation.
4. Why should success criteria be set before data collection?
5. Name two behavioural usability measures and explain what each indicates.

5 Long Answer Questions

1. Explain the planning chain from product goal to design decision, using an example of your own.
2. Compare formative and summative evaluation in terms of purpose, task structure, data, and appropriate design maturity.
3. Describe how you would write a realistic, non-leading scenario for an online banking task.
4. Explain how completion rate, time, errors, and satisfaction should be interpreted together rather than in isolation.
5. Outline the sections that should appear in a professional usability test plan and explain the purpose of each.

3 Analytical / Application Questions

1. A prototype of a university enrolment system has a completion rate of 6 out of 10 participants, but most successful participants finish quickly. Identify what additional evidence you would collect before recommending a redesign and explain why.
2. Design three research questions, three tasks, and two success criteria for a food-delivery application in which users report uncertainty about delivery fees and arrival times.
3. A moderator helped four participants by naming interface controls. Analyse how this affects the validity of the findings and propose a revised procedure for the next round.

Multiple Choice Questions with Answers

Question 1

Which statement best describes a usability research question?

A. A broad opinion about whether the interface looks attractive

B. A focused question about what must be learned to support a product decision

C. A list of interface controls that participants must click

D. A final design recommendation written before testing

Correct Answer: B. A focused question about what must be learned to support a product decision

Question 2

A team tests an early prototype to discover why users overlook a filter. What type of evaluation is most appropriate?

A. Formative evaluation

B. Summative evaluation

C. Financial evaluation

D. Regulatory evaluation

Correct Answer: A. Formative evaluation

Question 3

Which task instruction is least leading?

A. Click the blue settings icon to change your alerts

B. Open notification settings and disable promotions

C. You are receiving too many promotional messages; adjust your account so you still receive order updates

D. Find the notification menu at the top right

Correct Answer: C. You are receiving too many promotional messages; adjust your account so you still receive order updates

Question 4

Eight of ten participants successfully complete a task. What is the observed completion rate?

A. 8%

B. 20%

C. 80%

D. 125%

Correct Answer: C. 80%

Question 5

Why should a critical error be defined before a summative test?

A. To ensure that participants receive the correct answers

B. To support consistent scoring and reduce hindsight bias

C. To guarantee that every participant finishes quickly

D. To eliminate the need for observation

Correct Answer: B. To support consistent scoring and reduce hindsight bias

Question 6

Which measure primarily concerns users’ perceived comfort or confidence?

A. Satisfaction

B. Completion rate

C. Task duration

D. Error count

Correct Answer: A. Satisfaction

Question 7

What is the main purpose of a pilot usability session?

A. To publish final population estimates

B. To test the procedure, task wording, and logistics

C. To replace informed consent

D. To guarantee statistically significant results

Correct Answer: B. To test the procedure, task wording, and logistics

Question 8

Which practice best protects participant privacy in a usability study?

A. Collecting real passwords for realistic testing

B. Uploading all recordings to any available AI service

C. Using synthetic accounts and collecting only necessary information

D. Sharing identifiable notes with the entire organisation

Correct Answer: C. Using synthetic accounts and collecting only necessary information

Question 9

A participant completes a task quickly but misunderstands the result. How should this outcome be interpreted?

A. As strong evidence of efficiency

B. As successful because speed is the only relevant measure

C. As evidence that effectiveness and comprehension must also be considered

D. As unusable because time should never be recorded

Correct Answer: C. As evidence that effectiveness and comprehension must also be considered

Question 10

Which statement about a small convenience sample is most accurate?

A. It proves the exact performance of the entire user population

B. It can reveal problems but limits broad generalisation

C. It eliminates the need for predefined measures

D. It is suitable only for summative regulatory certification

Correct Answer: B. It can reveal problems but limits broad generalisation