Usability Testing and Design Evaluation

Introduction

A design is not successful merely because it is attractive, innovative, or technically functional. It must enable intended users to achieve meaningful goals accurately, efficiently, safely, and with reasonable satisfaction. Usability testing and related evaluation methods provide systematic ways to discover where an interface supports or obstructs those goals.

This chapter examines four complementary approaches: usability testing, cognitive walkthroughs, heuristic evaluations, and accessibility checks. It also explains how to plan and moderate studies, select participants and tasks, collect evidence, analyse errors, and translate findings into design decisions. The methods are useful throughout the design lifecycle, from early sketches to live products.

Learning Objectives

By the end of this chapter, you should be able to:

• distinguish usability testing from inspection-based evaluation;
• plan a suitable evaluation study and write realistic tasks;
• moderate sessions ethically and consistently;
• measure effectiveness, efficiency, satisfaction, task success, and error patterns;
• conduct a basic cognitive walkthrough, heuristic evaluation, and accessibility check;
• prioritise findings according to user impact and evidence;
• communicate results as actionable design recommendations.

1. Usability as a Quality of Use

The international standard ISO 9241-11 defines usability in relation to the extent to which specified users can achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use. This definition is important because usability is not an absolute property of a screen. The same interface may be usable for an experienced employee in an office but difficult for a first-time user on a mobile device in bright sunlight.

Three central dimensions are:

• Effectiveness: whether users complete intended goals accurately and completely.
• Efficiency: the resources required to achieve those goals, commonly time, effort, or number of steps.
• Satisfaction: users’ perceived comfort, confidence, and acceptability.

Other relevant qualities include learnability, memorability, safety, error tolerance, accessibility, and usefulness. A fast system that causes irreversible errors is not necessarily usable; nor is a pleasant system that cannot support the user’s actual goal.

A useful evaluation therefore begins with a clear context of use: target users, their goals, the environment, the device, the content, and the consequences of failure.

2. Evaluation Methods and Their Purposes

2.1 Usability testing

Usability testing observes representative users attempting realistic tasks with a product or prototype. It provides direct evidence of behaviour and experience. Testing may be moderated or unmoderated, remote or in person, formative or summative, qualitative or quantitative.

Formative testing is conducted during design to discover and correct problems. Summative testing is conducted later to assess performance against defined criteria. A certificate-level practitioner should normally favour small, iterative formative studies when the design is still changing.

2.2 Cognitive walkthrough

A cognitive walkthrough is an expert inspection focused on learnability for new or infrequent users. Evaluators step through each action in a task and ask questions such as:

1. Will the user try to achieve the correct outcome at this point?
2. Will the user notice the correct control or action?
3. Will the user understand that the control produces the intended result?
4. After acting, will the user recognise progress toward the goal?

The method is especially useful for onboarding, self-service systems, and interfaces where users cannot rely on training. It is not a substitute for observing real users.

2.3 Heuristic evaluation

In a heuristic evaluation, one or more evaluators inspect an interface against recognised usability principles. Nielsen’s commonly used heuristics include visibility of system status, match between system and the real world, user control and freedom, consistency and standards, error prevention, recognition rather than recall, flexibility and efficiency of use, minimalist design, support for recovery from errors, and help and documentation.

The evaluator records the violated principle, the location, evidence, likely consequence, and recommendation. Severity should reflect impact and persistence, not personal preference. Multiple evaluators often identify more problems than one evaluator because inspection is affected by individual expertise and attention.

2.4 Accessibility evaluation

Accessibility checks examine whether people with diverse abilities can perceive, operate, understand, and interact with a product. They should include people with disabilities where possible, together with standards-based and technical inspection.

A practical check may include keyboard-only operation, visible focus, logical heading structure, meaningful link names, sufficient colour contrast, captions or transcripts for relevant media, form labels, error identification, scalable text, and compatibility with assistive technologies. The Web Content Accessibility Guidelines provide an important reference for digital products, but compliance alone does not guarantee an accessible experience.

Accessibility is both a quality responsibility and an ethical matter. It should be considered from the beginning rather than treated as a final compliance exercise.

3. Planning a Usability Test

3.1 Define the decision to be supported

A weak research question asks, “Do people like the design?” A stronger question asks, “Can first-time customers find and compare two plans without confusing monthly and annual prices?” The second question identifies users, a goal, a behaviour, and a possible risk.

A test plan should state:

• research objectives and questions;
• target user characteristics and recruitment criteria;
• product version, device, and test environment;
• tasks and success criteria;
• measures and recording procedures;
• session length and moderator roles;
• consent, privacy, compensation, and data-retention arrangements;
• analysis and reporting procedures.

3.2 Recruit appropriate participants

Participants should resemble the intended users in relevant experience, knowledge, language, physical or cognitive access needs, and context. Recruiting only colleagues is convenient but can produce misleading results because colleagues know the product, the organisation, and the purpose of the study.

For formative work, a small number of carefully selected participants can reveal recurring problems, particularly when testing is repeated across design iterations. There is no universal sample size that guarantees discovery. The appropriate number depends on research questions, user diversity, risk, and whether quantitative precision is required.

Recruit ethically. Participation must be voluntary, informed, and capable of withdrawal. Avoid collecting unnecessary personal data, and protect recordings and transcripts.

3.3 Design realistic tasks

A task should describe a goal rather than prescribe the interface. “Find a pair of waterproof walking shoes under £100 and save them” is better than “Click the filter icon, select waterproof, choose £100, and press Save.” The latter tests whether participants can follow instructions rather than whether the design communicates effectively.

Good tasks contain a plausible context, a clear objective, and an observable completion condition. They should avoid unnecessary clues, jargon, and assumptions about the interface. Pilot tasks before formal sessions: confusing task wording can create artificial failures.

3.4 Select measures

Measures should correspond to the research question. Common measures include:

Operational definitions must be decided before testing. For example, “success” may mean complete success without help, while “partial success” may mean the user reaches the correct area but cannot finish. Ambiguous scoring creates unreliable conclusions.

3.5 Basic quantitative measures

For a set of tasks, the task-success rate can be calculated as:

This measure assumes that each attempt has a defined success criterion and that the denominator includes all valid attempts. If partial success is important, report it separately rather than silently combining it with full success.

For a binary success measure, an error rate may be expressed as:

The unit is errors per attempt. The analyst must define what counts as an error and avoid counting every hesitation as one. If comparing tasks with different complexity, errors per task or per opportunity may be more meaningful than a single overall rate.

Efficiency can be reported using completion time, but time must be interpreted with success. A short time with frequent failures is not superior to a longer time with accurate completion. When appropriate, report median time because a few unusually long sessions can distort the mean.

Satisfaction may be measured with a validated questionnaire such as the System Usability Scale (SUS), but the instrument must be administered and scored according to its published procedure. A single satisfaction question can be useful for quick formative work, but it should not be treated as a complete usability diagnosis.

4. Moderating a Session

A moderator creates conditions in which participants can behave naturally while ensuring that the study remains ethically and methodologically sound.

Before the session

Prepare the prototype, account credentials, devices, recording tools, consent form, task script, observation sheet, backup plan, and accessibility arrangements. Test links and reset the product between sessions. Decide in advance when assistance will be offered.

Opening the session

Explain that the product is being tested, not the participant. Describe recording and confidentiality, obtain consent, and clarify that participants may stop at any time. Encourage them to think aloud when appropriate, but explain that thinking aloud may affect task time and should not be overinterpreted.

During tasks

Use neutral prompts such as “What are you looking for?” or “What do you expect to happen?” Avoid leading prompts such as “Did you try the menu?” Do not immediately rescue participants. A useful protocol may allow a standardised hint after a defined period, recording that assistance as part of the result.

Observe behaviour and language separately. A participant may say that a process is easy while repeatedly making errors, or may complain about visual style while completing the task effectively. Both data types matter, but they answer different questions.

Closing

Ask open-ended questions about expectations, difficulties, confidence, and improvement ideas. Do not ask participants to design the solution unless that is an explicit research activity. Thank them, provide compensation where promised, and explain how their data will be used.

5. Analysing Evidence and Error Patterns

Analysis begins with organised data, not premature recommendations. Create a task-by-participant matrix showing completion, time, errors, assistance, and notable observations. Add qualitative notes describing what happened and possible causes.

Useful error categories include:

• Slip: an unintended action despite a correct goal, such as selecting the adjacent control.
• Mistake: an action based on an incorrect understanding, such as interpreting “Archive” as “Delete.”
• Navigation error: entering an irrelevant path or becoming lost.
• Recognition error: failing to notice a relevant control, message, or state.
• Recovery error: failing to understand or recover from an error message.
• Accessibility barrier: an interaction that cannot be completed because of sensory, motor, cognitive, or technological constraints.

Look for patterns across participants and tasks. A single observation can still be important if the consequence is severe, such as exposure of private information or loss of payment details. Conversely, repeated minor comments may not justify redesign if they do not affect goals or experience.

A finding should connect evidence to consequence and recommendation. For example: “Four of six first-time users selected ‘Submit’ before completing the required address field. The form does not indicate required fields until after submission. Add persistent field guidance and validate progressively.” This is more useful than “The form is confusing.”

Prioritising findings

A practical prioritisation scheme considers:

1. Impact: How seriously does the issue affect user goals, safety, privacy, inclusion, or business outcomes?
2. Frequency: How often did it occur, or how many users are plausibly exposed?
3. Persistence: Does it recur across tasks or user groups?
4. Evidence strength: Is it supported by observed behaviour, multiple participants, analytics, or standards?
5. Cost and feasibility: What resources are needed to address it?

Severity ratings should guide discussion, not replace judgement. High-severity findings deserve prompt action, but low-cost improvements may also be worthwhile.

6. Conducting Inspections in Practice

A cognitive walkthrough is best documented as a task sequence. For each step, record the intended user goal, available control, expected knowledge, evaluator answer to the walkthrough questions, observed risk, and design implication.

A heuristic evaluation should identify the exact interface location and explain the violated principle. “Inconsistent” is insufficient; specify which control differs, from what standard, and how the difference harms recognition or operation. Independent inspection followed by a comparison meeting can reduce groupthink.

Accessibility checking should combine several perspectives:

• manual keyboard navigation;
• screen-reader or text-to-speech sampling where expertise and tools permit;
• zoom and text resizing;
• colour and contrast inspection;
• captions, transcripts, and alternative text review;
• form, focus, error, and timeout testing;
• testing with people who have relevant disabilities.

Automated tools can detect some code-level problems, such as missing labels or contrast issues, but they cannot judge whether instructions are understandable, focus order is meaningful, or a task is cognitively accessible. Automation is therefore a supplement, not a replacement for human evaluation.

7. Illustrative Case: Improving a Community Clinic Booking Service

Background

A fictional community clinic introduced an online appointment service. The service allowed patients to choose a department, select a date, provide contact information, and confirm a booking. Clinic staff reported that many patients telephoned after attempting to book online.

Problem

The design team suspected that the booking form was too long. However, the more important question was whether patients could identify the correct appointment type and understand the confirmation state.

Analysis and approach

The team recruited six participants: four first-time online users of the clinic service and two returning users. Participants completed three tasks: book a routine consultation, change a selected date, and verify whether a booking had been confirmed. Sessions were moderated on a laptop and with keyboard-only navigation for one accessibility-focused session.

The team defined full success as completing the intended booking without moderator assistance and correctly identifying the confirmation reference. They recorded success, time, errors, assistance, and comments.

Testing showed that five participants selected the wrong appointment category initially because the categories used clinical terminology unfamiliar to patients. Three participants changed the date but did not realise that the previously selected time had become unavailable. Four participants reached the final screen but were unsure whether the booking had been completed because the confirmation message appeared below the fold. The keyboard-only participant could not reach the date picker reliably.

A heuristic inspection supported these findings: the system status was weak, terminology did not match users’ language, and the date component had poor keyboard operation. The accessibility check identified inadequate focus visibility and an unclear error announcement.

Solution

The team replaced clinical category names with patient-centred descriptions, added short explanatory text, displayed the selected date and time persistently, placed a prominent confirmation heading and reference number at the top of the final page, and provided a keyboard-operable date alternative. Errors were presented next to the relevant field and in a summary region that could be detected by assistive technology.

Outcome

In a second small formative test, all participants selected the intended appointment type. One participant still needed help changing a date, so the team continued refinement. The result illustrates an important principle: the dominant problem was not form length but communication of meaning, state, and recovery.

Learning points

• Test the user’s actual goal, not merely the interface components.
• Combine observation with inspection; each revealed different evidence.
• Accessibility barriers can be functional blockers, not cosmetic defects.
• Confirmation and system status deserve explicit testing.
• Small iterative studies can expose high-impact problems before launch.

8. Ethics, Governance, and Professional Practice

Usability research involves people, recordings, behavioural data, and sometimes sensitive information. Obtain informed consent in understandable language. Explain what will be recorded, who will access it, how long it will be retained, and whether quotations will be anonymised.

Do not pressure participants to reveal health, financial, or identity information unless necessary and approved. Use test accounts and synthetic data whenever possible. Protect files with appropriate access controls and delete them according to the agreed retention schedule.

Researchers must not manipulate findings to satisfy a stakeholder. Report inconvenient evidence, uncertainty, and limitations. Distinguish observed facts from interpretations and recommendations. When testing vulnerable groups, provide additional safeguards, accessible materials, and suitable support.

AI tools can assist with transcript summarisation, coding suggestions, or clustering of comments, but they may omit context, reproduce bias, or expose confidential data. Human researchers must verify outputs, preserve original evidence, and follow organisational privacy rules. AI should support analysis rather than make unreviewed decisions about participants or accessibility.

9. Best Practices and Limitations

Effective evaluation is iterative, mixed-method, and risk-based. Start early with inexpensive prototypes, test realistic tasks, include diverse users, define measures before sessions, pilot the protocol, and triangulate user behaviour with inspection and analytics. Report both successes and failures.

No method is complete. Usability testing can be affected by artificial tasks, small or unrepresentative samples, moderator influence, prototype limitations, and the novelty of the session. Heuristic evaluation depends on evaluator skill and may overproduce subjective issues. Cognitive walkthroughs can miss problems that emerge only through real use. Accessibility audits may overlook lived experience if disabled users are excluded. Quantitative metrics can create false precision when task definitions are weak.

The appropriate response is not to select one “best” method, but to match methods to decisions. Use inspection for early, rapid diagnosis; use user testing for behavioural evidence; use accessibility checks throughout; and use follow-up testing to determine whether changes actually improved the experience.

Chapter Summary

Usability testing evaluates whether representative users can achieve realistic goals effectively, efficiently, and satisfactorily. Strong studies begin with explicit research questions, appropriate participants, realistic tasks, operational definitions, ethical procedures, and suitable measures. Moderators should remain neutral, record behaviour and comments carefully, and avoid rescuing participants prematurely.

Cognitive walkthroughs focus on learnability, heuristic evaluations inspect designs against established principles, and accessibility checks examine inclusive operation across abilities and technologies. Task success, time, errors, assistance, abandonment, and satisfaction provide complementary evidence; none should be interpreted in isolation. Error patterns become useful findings when linked to user consequences and actionable design recommendations.

Professional evaluation is iterative and evidence-based. It combines methods, protects participants, recognises limitations, and treats accessibility, privacy, and inclusion as integral design responsibilities.

Key Terms

Usability: The extent to which specified users achieve specified goals effectively, efficiently, and satisfactorily in context.

Effectiveness: Accuracy and completeness with which users achieve goals.

Efficiency: Resources required to achieve a goal, often time or effort.

Satisfaction: Users’ perceived comfort, confidence, and acceptability.

Usability testing: Observation of users performing realistic tasks with a product or prototype.

Formative evaluation: Evaluation used to discover and correct problems during design.

Summative evaluation: Evaluation used to assess a completed or mature design against criteria.

Moderator: The researcher who guides a usability session.

Task success: A defined level of achievement of the intended task goal.

Error rate: The number of defined errors relative to a stated denominator.

Cognitive walkthrough: An inspection method focused on learnability for new users.

Heuristic evaluation: Expert inspection against recognised usability principles.

Heuristic: A general design principle used to identify usability problems.

Accessibility: The ability of people with diverse abilities to perceive, operate, understand, and interact with a product.

Slip: An unintended action arising while pursuing a correct goal.

Mistake: An action based on an incorrect goal or understanding.

Think-aloud: A technique in which participants verbalise thoughts during tasks.

Triangulation: Combining different evidence sources or methods to strengthen interpretation.

Severity: An assessment of the impact and importance of a usability problem.

Review Questions

5 Short Answer Questions

1. Define effectiveness, efficiency, and satisfaction in usability evaluation.
2. What is the principal purpose of a cognitive walkthrough?
3. Why should usability tasks describe goals rather than interface actions?
4. Give two examples of accessibility checks.
5. Distinguish a slip from a mistake.

5 Long Answer Questions

1. Explain how to plan and moderate a formative usability test from recruitment through debriefing.
2. Compare usability testing, heuristic evaluation, and cognitive walkthroughs in terms of evidence, strengths, and limitations.
3. Explain how task success, completion time, error rate, and satisfaction should be interpreted together.
4. Discuss ethical and governance responsibilities when recording usability sessions.
5. Explain why automated accessibility tools cannot replace testing with people with disabilities.

3 Analytical / Application Questions

1. A checkout prototype has a high completion rate, but users frequently pause before payment and cannot explain whether payment succeeded. Identify likely evaluation methods, measures, and design evidence you would collect.
2. Design a three-task usability test for a university course-registration system. Define participants, success criteria, at least three measures, and one ethical safeguard.
3. An interface receives many heuristic violations but users complete all tasks accurately. How would you prioritise the findings, and what additional evidence would you seek before recommending redesign?

Multiple Choice Questions with Answers

Question 1

Which measure most directly represents effectiveness in a usability test?

A. Number of successfully completed tasks

B. Average screen brightness

C. Number of interface colours

D. Length of the participant consent form

Correct Answer: A. Number of successfully completed tasks

Question 2

What is the main focus of a cognitive walkthrough?

A. Learnability for new or infrequent users

B. Server performance under heavy traffic

C. Brand recognition among existing customers

D. Legal ownership of interface content

Correct Answer: A. Learnability for new or infrequent users

Question 3

Which task instruction is best designed for unbiased usability testing?

A. Click the blue filter icon and select waterproof shoes

B. Find waterproof walking shoes under £100 and save one

C. Use the menu exactly as demonstrated by the moderator

D. Press the third button and then return to the previous page

Correct Answer: B. Find waterproof walking shoes under £100 and save one

Question 4

A participant chooses the wrong goal because they misunderstand the meaning of a control. This is best classified as a:

A. Slip

B. Mistake

C. Timing observation

D. Satisfaction rating

Correct Answer: B. Mistake

Question 5

Which activity is an accessibility check?

A. Testing whether the interface works using only a keyboard

B. Counting the number of brand photographs

C. Comparing the product logo with competitors’ logos

D. Measuring the office temperature

Correct Answer: A. Testing whether the interface works using only a keyboard

Question 6

If 8 of 10 valid task attempts are successfully completed, what is the task-success rate?

A. 20%

B. 50%

C. 80%

D. 125%

Correct Answer: C. 80%

Question 7

Which moderator prompt is most neutral?

A. Did you try the menu I showed you?

B. Why did you ignore the obvious button?

C. What are you looking for at this point?

D. Would you agree that this layout is clear?

Correct Answer: C. What are you looking for at this point?

Question 8

Why should completion time be interpreted together with task success?

A. A short time may conceal inaccurate or incomplete performance

B. Time is never relevant to efficiency

C. Successful tasks cannot be timed

D. Completion time measures accessibility by itself

Correct Answer: A. A short time may conceal inaccurate or incomplete performance

Question 9

Which statement best describes heuristic evaluation?

A. Users are observed completing realistic tasks without an evaluator

B. Experts inspect an interface against recognised usability principles

C. A system automatically rewrites all interface text

D. Participants vote on their preferred colour palette

Correct Answer: B. Experts inspect an interface against recognised usability principles

Question 10

Which practice strengthens the credibility of a usability finding?

A. Reporting only comments that support the preferred design

B. Connecting observed evidence to user consequence and recommendation

C. Treating every hesitation as a serious error

D. Removing inconvenient results from the final report

Correct Answer: B. Connecting observed evidence to user consequence and recommendation