Usability Evaluation and Iterative Design

Introduction

A user interface is not successful merely because it looks attractive or functions technically. It must enable intended users to achieve meaningful goals accurately, efficiently, safely, and with reasonable satisfaction. Usability evaluation is the systematic process of examining an interface to identify problems and opportunities for improvement. Iterative design uses the results of evaluation to revise the interface, test the revision, and continue learning.

Evaluation is therefore not a final inspection performed immediately before release. It is an evidence-based activity integrated throughout the design process. Different methods reveal different kinds of evidence: expert reviews can identify violations of established principles; cognitive walkthroughs can examine learnability for new users; usability tests can reveal what people actually do, say, and experience.

Learning Objectives

By the end of this chapter, you should be able to:

• distinguish heuristic evaluation, cognitive walkthroughs, and usability testing;
• define realistic usability tasks and participant profiles;
• select suitable qualitative and quantitative metrics;
• conduct moderated and unmoderated evaluations ethically;
• analyse observations, comments, and performance data;
• prioritise usability issues according to severity and evidence;
• translate findings into design changes and repeatable iterations.

1. Usability as an Evaluative Concept

Usability is commonly understood through five attributes: effectiveness, efficiency, satisfaction, learnability, and error tolerance. Effectiveness concerns whether users complete their goals. Efficiency concerns the resources required, such as time or effort. Satisfaction concerns users’ perceived comfort and confidence. Learnability concerns how easily a new user can begin. Error tolerance concerns whether errors are prevented, explained, and recoverable.

Usability is contextual. An interface may be usable for experienced employees but confusing for occasional customers. Evaluation must therefore specify users, goals, environment, devices, and accessibility needs. A finding is meaningful only in relation to a defined context of use.

A useful distinction is between a usability problem and a preference. A usability problem prevents, delays, or unnecessarily burdens task completion. A preference reflects an individual taste that may not affect outcomes. Evidence, rather than the evaluator’s personal design taste, should guide decisions.

2. Evaluation Methods

2.1 Heuristic Evaluation

A heuristic evaluation is an inspection method in which one or more evaluators examine an interface against recognised usability principles. Nielsen’s widely used heuristics include:

1. visibility of system status;
2. correspondence between the system and the real world;
3. user control and freedom;
4. consistency and standards;
5. error prevention;
6. recognition rather than recall;
7. flexibility and efficiency of use;
8. aesthetic and minimalist design;
9. helping users recognise, diagnose, and recover from errors;
10. help and documentation.

Evaluators normally inspect independently before consolidating findings. Independence reduces group conformity. Each finding should identify the interface location, violated principle, evidence, consequence, and recommendation.

Heuristic evaluation is relatively fast and inexpensive, but it is an expert prediction rather than direct evidence of user behaviour. It may miss domain-specific problems and can produce false positives. Multiple evaluators often produce broader coverage because no evaluator notices every issue.

2.2 Cognitive Walkthrough

A cognitive walkthrough examines whether a novice user can learn a task through exploration. It is especially useful for unfamiliar systems, infrequent tasks, and onboarding flows. For each action, evaluators ask:

• Will the user form the correct goal?
• Will the user notice the correct control?
• Will the user understand that the control produces the desired result?
• After acting, will the user recognise that progress has occurred?

The walkthrough should use a specific user profile, task goal, starting condition, and sequence of actions. It is not a general visual inspection. Its central concern is learnability and the visibility of action possibilities.

2.3 Usability Testing

Usability testing observes representative users attempting realistic tasks. It can be formative, helping improve a design, or summative, assessing performance against defined criteria. Testing may be moderated, in which a facilitator interacts with participants, or unmoderated, in which participants complete tasks independently through a platform.

Moderated testing allows probing questions and observation of unexpected behaviour. It requires more time and facilitator skill. Unmoderated testing can reach more participants and reflects more natural independence, but it offers less opportunity to clarify confusing responses and may provide less contextual information.

A test session should distinguish between observing and helping. Excessive assistance invalidates the evidence. If a participant is unable to proceed, the facilitator should record the difficulty, follow the protocol, and provide only the assistance specified in advance.

3. Planning an Evaluation

3.1 Research Questions and Scope

Begin with decisions the evaluation must inform. Examples include:

• Can first-time users locate and apply a discount code?
• Can users understand whether a payment was successful?
• Which step causes abandonment in account creation?
• Do users with screen readers receive equivalent information?

A clear question determines the method, participants, tasks, and metrics. Testing everything is inefficient and often produces unfocused findings.

3.2 Participants and Sampling

Participants should resemble the intended users in relevant characteristics, such as experience, language, physical ability, domain knowledge, and device use. A small formative study can reveal many severe problems, but it should not be described as statistically representative of an entire population. Recruitment criteria must be documented, and incentives should be fair rather than coercive.

Accessibility evaluation should include people with disabilities when accessibility is a central requirement. Automated tools and expert checks are useful, but they cannot replace testing with people who use assistive technologies.

3.3 Task Design

A usability task states a goal and necessary context without prescribing the interface action. “Find a train arriving in Manchester before 10:00 and save it” is preferable to “Click the search icon, select Manchester, and press Save.” The latter tests compliance with instructions rather than discoverability.

Good tasks are:

• realistic and connected to a user goal;
• specific enough to establish completion;
• neutral and free from hints;
• ordered from simple to complex when appropriate;
• safe, especially when real transactions or personal information are involved.

A task script should define the starting state, success condition, stopping rule, allowed assistance, and data needed. Pilot testing identifies ambiguous wording and technical failures before the formal study.

3.4 Metrics

Common measures include completion rate, time on task, error frequency, assistance requests, abandonment, satisfaction, and confidence. Metrics must have operational definitions. For example, completion may require the participant to reach a confirmation screen without facilitator intervention.

Completion rate is calculated as:

The numerator and denominator must use the same task definition and participant group. A task completed only after facilitator intervention should not automatically be classified as independent success.

Average time on task is:

Here, tᵢ is the valid time for participant i, and n is the number of valid observations. The researcher must define how pauses, technical failures, and incomplete tasks are treated. A mean can be distorted by unusually long times, so the median and range may also be useful.

Error rate can be expressed as:

The unit is errors per attempt. Researchers should define what counts as an error before reviewing results. For small formative samples, metrics describe the observed sample; they should not be presented as precise population estimates without appropriate statistical design.

Satisfaction questionnaires may use a standard instrument such as the System Usability Scale, provided it is administered and scored according to its published procedure. A single informal rating should not be treated as equivalent to a validated scale.

4. Conducting Moderated and Unmoderated Studies

A moderated session generally follows this sequence: welcome and consent; explanation of the session; collection of relevant background information; task briefing; observation; post-task questions; and debriefing. The facilitator should use neutral prompts such as “What are you looking for?” rather than leading prompts such as “Do you see the blue button?”

Think-aloud protocols ask participants to verbalise thoughts while working. They can reveal expectations and interpretation, but speaking may alter task time and behaviour. Retrospective questioning, in which participants discuss a recording afterward, reduces interruption but depends on memory.

Unmoderated studies require especially clear instructions, robust logging, privacy controls, and a defined procedure for technical failure. Participants should be able to stop without penalty. Researchers should distinguish genuine usability behaviour from misunderstanding caused by the remote platform or incomplete instructions.

Ethical practice requires informed consent, voluntary participation, confidentiality, data minimisation, secure storage, and the right to withdraw. Screen recordings, voices, names, and interaction logs may be personal data. Researchers must follow applicable institutional policies and data-protection law. Never ask participants to expose passwords, financial information, or sensitive health data in a test environment.

5. Analysing Findings

Analysis begins by organising evidence rather than immediately proposing solutions. Useful evidence sources include observation notes, recordings, task outcomes, participant quotations, error logs, and questionnaire responses. A finding should be traceable to evidence.

A practical analysis table is:

Group observations by underlying problem, not merely by screen. For example, repeated failures to understand labels across three screens may represent one content and terminology problem. Separate symptoms from causes: “users clicked the wrong control” is an observation; “the controls have indistinguishable labels” is a possible cause requiring design judgement.

Severity and Prioritisation

A simple severity model can consider impact, frequency, and persistence. One practical rating is:

• 0—Not a usability problem: preference or confirmed false positive.
• 1—Cosmetic: minor inconvenience with little effect on task success.
• 2—Moderate: causes delay, confusion, or recoverable error.
• 3—Major: prevents completion for some users or creates substantial risk.
• 4—Critical: prevents essential use, causes serious harm, or creates a severe accessibility or privacy failure.

Severity is a judgement, not an objective measurement. Frequency should not be the only criterion: a rare failure in a safety-critical or payment task may deserve immediate attention. Prioritisation should also consider implementation effort, strategic importance, legal obligations, and dependencies.

A recommendation is stronger when it is testable. “Improve navigation” is vague; “Group account settings under a labelled menu and test whether first-time users locate notification controls without assistance” is specific.

6. Iterative Design

Iteration is a cycle of evidence, interpretation, change, and reassessment. A useful cycle is:

Figure: A cycle linking evaluation evidence to redesign and reassessment.

Changes should be recorded in a decision log: the original problem, evidence, chosen intervention, rejected alternatives, owner, and validation plan. Designers should avoid changing several unrelated variables at once when they need to know which change produced improvement. After redesign, test the revised flow, but also regression-test important tasks because solving one problem can create another.

7. Worked Application

Consider an illustrative case: a university library is redesigning its mobile room-booking service. The key question is whether first-year students can find and reserve an available room without assistance.

The team defines the task: “Book a two-hour room for tomorrow afternoon for four people, and confirm the reservation.” Success requires an independent confirmation screen. Six representative students complete a moderated prototype test. Four complete the task independently, one completes it after a prompt, and one abandons the task.

The independent completion rate is:

The prompt-assisted case is recorded separately rather than counted as independent success. Observation shows that participants interpret “spaces” as study materials rather than rooms. They also overlook a date selector because its current value resembles plain text. The confirmation message appears below the fold, so one participant assumes the reservation failed.

The team classifies the terminology problem as major, the date-control visibility problem as major, and the confirmation placement problem as moderate. It replaces “spaces” with “Rooms,” styles the date selector as an explicit control with a calendar icon and label, and places a clear confirmation message at the top of the result. A second evaluation uses comparable participants and the same success rule. The team compares independent completion, errors, time, and confidence rather than relying only on positive comments.

The case illustrates several principles: tasks must represent real goals; partial success must be defined in advance; observations should be connected to causes cautiously; and redesign is incomplete until the change has been reassessed.

8. Limitations, Technology, and Professional Practice

No evaluation method provides complete certainty. Expert reviews may reflect evaluator bias. Walkthroughs may overemphasise novice behaviour. Usability tests may be affected by artificial settings, small samples, learning effects, or participants who behave differently when observed. Metrics can conceal frustration, while quotations can be memorable but unrepresentative.

Triangulation improves judgement: combine inspection with user observation, behavioural metrics with participant explanations, and accessibility checks with assistive-technology testing. Treat disagreement between methods as a question to investigate rather than a reason to select the preferred result.

Remote testing platforms, analytics, session replay, automated accessibility scanners, and AI-supported transcription can reduce effort. However, automated systems can misinterpret behaviour, reproduce bias, or expose sensitive data. AI-generated summaries must be checked against original evidence. Researchers remain responsible for consent, anonymisation, secure processing, and the accuracy of conclusions.

Professional teams should involve developers, content specialists, accessibility practitioners, product owners, and representative users. Findings should be communicated without blaming participants or colleagues. The purpose of evaluation is to improve the system, not to judge the intelligence of users or the competence of designers.

Best-Practice Checklist

Before evaluation, define the decision, users, tasks, success rules, metrics, consent process, and analysis plan. During evaluation, remain neutral, protect participants, record observable evidence, and distinguish assistance from independent success. After evaluation, consolidate duplicate findings, assess severity and confidence, prioritise transparently, create testable design changes, and verify the revised experience.

Chapter Summary

Usability evaluation provides evidence about how effectively, efficiently, safely, and satisfactorily people use an interface. Heuristic evaluation is an expert inspection against principles; cognitive walkthroughs focus on learnability and action discovery; and usability testing observes representative users performing realistic tasks. Reliable evaluation depends on clear research questions, appropriate participants, neutral tasks, operational definitions, and ethically collected data. Completion, time, error, assistance, and satisfaction measures must be interpreted within their methodological limits. Findings should be documented with evidence, separated from personal preference, rated for severity, and prioritised according to impact, risk, frequency, effort, and strategic importance. Iterative design then converts prioritised findings into specific changes and tests whether those changes improve the experience without introducing new problems.

Key Terms

Heuristic evaluation: Expert inspection of an interface against usability principles.

Cognitive walkthrough: Structured examination of whether a novice can learn a task through exploration.

Usability testing: Observation of users attempting realistic tasks with an interface.

Formative evaluation: Evaluation intended to guide improvement during design.

Summative evaluation: Evaluation intended to assess performance against defined criteria.

Moderated testing: A study in which a facilitator guides or observes participants in real time.

Unmoderated testing: A study completed independently by participants, usually through a platform.

Task: A realistic goal or activity given to a participant.

Completion rate: The percentage of attempts meeting a predefined success condition.

Error rate: The number of defined errors relative to task attempts.

Think-aloud: A protocol in which participants verbalise thoughts while working.

Triangulation: Combining methods or evidence sources to improve interpretation.

Severity: The seriousness of a usability problem’s effect.

Learnability: The ease with which new users understand and perform tasks.

Accessibility: The extent to which people with diverse abilities can use a system.

Representative participant: A person whose relevant characteristics resemble those of intended users.

Regression testing: Rechecking important functions after a design change.

Iteration: Repeated cycles of evaluation, redesign, and reassessment.

Review Questions

Short Answer Questions

1. What is the main difference between heuristic evaluation and usability testing?
2. Why should a usability task state a goal rather than prescribe interface actions?
3. Define completion rate and identify the two quantities used in its calculation.
4. What is the purpose of a cognitive walkthrough?
5. Why should usability findings be traceable to evidence?

Long Answer Questions

1. Explain the strengths and limitations of heuristic evaluation, cognitive walkthroughs, and usability testing.
2. Describe how a researcher should plan and conduct a moderated usability study.
3. Discuss how ethical considerations affect the collection and storage of usability data.
4. Explain how severity, frequency, risk, and implementation effort can influence prioritisation.
5. Describe an iterative design cycle from research question through redesign and reassessment.

Analytical / Application Questions

1. A participant completes a checkout task only after the facilitator points to the payment button. Should this count as independent completion? Justify your answer.
2. A usability test shows that most users finish a task quickly, but one user with a screen reader cannot access the confirmation message. How should the team interpret and prioritise this finding?
3. Design a usability task, success criterion, and two metrics for evaluating a mobile appointment-booking interface. Explain why your task wording is neutral.

Multiple Choice Questions with Answers

Question 1

Which evaluation method primarily examines an interface against established usability principles?

A. Heuristic evaluation

B. Diary study

C. Market segmentation

D. A/B advertising test

Correct Answer: A. Heuristic evaluation

Question 2

What is the main focus of a cognitive walkthrough?

A. Long-term brand loyalty

B. Novice learnability and action discovery

C. Server capacity

D. Advertising reach

Correct Answer: B. Novice learnability and action discovery

Question 3

Which task is best written for a usability test?

A. Click the blue Search button

B. Open the third menu and select Rooms

C. Find and reserve a room for tomorrow afternoon

D. Use the interface correctly

Correct Answer: C. Find and reserve a room for tomorrow afternoon

Question 4

Six participants attempt a task and four complete it independently. What is the independent completion rate?

A. 33.3%

B. 50.0%

C. 66.7% approximately

D. 75.0%

Correct Answer: C. 66.7% approximately

Question 5

Which practice best reduces facilitator bias during moderated testing?

A. Give frequent hints

B. Use neutral prompts

C. Explain the correct route beforehand

D. Correct every error immediately

Correct Answer: B. Use neutral prompts

Question 6

What should a usability finding contain to support professional decision-making?

A. Only the evaluator’s personal preference

B. A visual redesign with no explanation

C. Evidence, user impact, and relevant context

D. A sales forecast

Correct Answer: C. Evidence, user impact, and relevant context

Question 7

Why might a rare usability problem still receive critical priority?

A. It is visually unattractive

B. It occurs during a high-risk payment or safety task

C. It affects only expert users

D. It is easy to fix

Correct Answer: B. It occurs during a high-risk payment or safety task

Question 8

Which statement about unmoderated testing is accurate?

A. It always produces more reliable findings than moderated testing

B. It allows no collection of behavioural data

C. It can reach participants independently but provides less opportunity for clarification

D. It eliminates the need for consent

Correct Answer: C. It can reach participants independently but provides less opportunity for clarification

Question 9

What is regression testing after a design change?

A. Repeating important checks to ensure existing functions still work

B. Removing all previous test data

C. Testing only the visual colour palette

D. Replacing participants with experts

Correct Answer: A. Repeating important checks to ensure existing functions still work

Question 10

Which principle is most directly concerned with preventing users from having to remember information between screens?

A. Recognition rather than recall

B. User control and freedom

C. Aesthetic and minimalist design

D. Visibility of system status

Correct Answer: A. Recognition rather than recall