Usability Metrics and UX Analytics

Introduction

Usability testing produces observations, but observation alone is not enough for sound design decisions. UX metrics convert user behaviour, performance, and perceptions into evidence that can be compared across users, versions, tasks, and time. When used carefully, metrics help a team answer questions such as: Can users complete the task? How long does it take? Where do they fail? Is the interface becoming easier to learn? Do users feel confident and satisfied?

Metrics do not replace qualitative research. A completion rate can show that a problem exists, while observation and interview evidence may explain why it exists. Effective usability evaluation therefore combines behavioural data, self-reported measures, and contextual interpretation.

Learning Objectives

By the end of this chapter, you should be able to:

• distinguish common usability and UX metrics;
• calculate task success, completion time, error rate, efficiency, and confidence;
• explain learnability and the effect of practice;
• administer and interpret the System Usability Scale (SUS) and Single Ease Question (SEQ);
• interpret funnels, heatmaps, and session recordings responsibly;
• select appropriate metrics for a research question;
• identify limitations, ethical risks, and sources of misleading conclusions.

1. Measurement Foundations

A metric is a defined measure of an observed phenomenon. In UX work, the phenomenon may be behaviour, such as task completion; performance, such as time on task; or perception, such as perceived ease. A useful metric has a clear numerator, denominator, unit, population, and collection procedure.

A measurement plan should begin with a decision, not with a tool. For example, “Should the checkout form be redesigned?” is a decision question. Relevant evidence might include completion rate, abandonment location, errors, completion time, confidence, and comments. Recording every possible event often creates noise rather than insight.

1.1 Units of analysis

The analyst must identify what is being counted:

• User: a distinct person or account.
• Session: a period of activity, defined by a stated timeout or closing event.
• Task attempt: one effort by a participant to complete a specified task.
• Event: an interaction such as a click, form submission, or error message.
• Page or screen view: an observed display of a page or application state.

These units are not interchangeable. Ten events from one user do not necessarily represent ten users. A dashboard that reports “conversion” must state whether the denominator is users, sessions, or attempts.

1.2 Reliability and validity

Reliability concerns consistency. If the same task is measured under comparable conditions, a reliable procedure should produce reasonably stable results. Validity concerns whether the measure represents the intended construct. A click count may be reliable but invalid as a measure of task success if users can click without achieving their goal.

Construct validity is especially important for satisfaction and confidence. A single rating may be influenced by visual appeal, brand attitude, or the testing environment. Triangulation—combining behavioural evidence, ratings, and qualitative observations—provides stronger interpretation.

2. Core Usability Metrics

2.1 Task success

Task success indicates whether a participant achieves the predefined outcome. The outcome must be specified before testing. A binary measure records success or failure; a graduated measure can distinguish complete success, partial success, and failure.

For binary success:

The formula assumes that each attempt has a clearly coded outcome and that invalid or interrupted attempts are handled according to a documented rule. If one participant performs several tasks, task attempts—not automatically unique participants—are the denominator.

Example: 18 of 24 valid attempts are successful.

A success rate should be accompanied by its task definition and sample size. “The product has 75% usability” is not a valid conclusion; the evidence applies only to the tested task and population.

2.2 Completion time

Completion time is the elapsed time from the defined task start to the defined completion point. Start and end rules must be consistent. For example, the timer may begin when the task instruction is shown and end when the confirmation message appears.

Because times are often skewed by unusually slow attempts, the median is frequently more informative than the mean. The median is the middle value after observations are ordered. The arithmetic mean is:

where xᵢ is the completion time for attempt i, n is the number of valid attempts, and x̄ is the mean time. If times are measured in seconds, both sides are measured in seconds. The mean is appropriate when extreme values are meaningful and the distribution is reasonably suitable; otherwise report the median and range or interquartile range as well.

Completion time is not automatically better when shorter. A very short time may indicate efficient performance, but it may also indicate skipped information, accidental submission, or superficial use. Time must be interpreted with success and error data.

2.3 Errors and assists

An error is an observable action or outcome that deviates from the intended path, such as entering an invalid value, selecting the wrong control, or triggering an avoidable warning. An assist is help from a moderator, documentation, another person, or an interface hint that enables progress.

Possible measures include:

The first measure can exceed 1 because one attempt may contain several errors. The second measure is conditional on success and should not be confused with overall success. Error definitions must be established before coding; otherwise analysts may selectively count inconvenient actions as errors.

2.4 Efficiency

Efficiency relates successful output to resources used. In a basic usability test, a practical measure is successful completion per unit of time:

If 8 successful attempts require 1,600 seconds in total, time efficiency is 0.005 successful attempts per second, or 0.3 successful attempts per minute. This measure is useful for comparing equivalent designs under the same task conditions, but it is not a universal property of an interface. It combines success and time and can conceal important differences between users.

For more detailed analysis, record clicks, keystrokes, errors, assists, and navigation steps. However, fewer actions are not always better: a single action can be cognitively demanding, while several simple actions may be clear and safe.

2.5 Satisfaction, ease, and confidence

Satisfaction is a subjective judgement about the experience. Ease concerns perceived effort for a particular task. Confidence concerns the user’s belief that they performed the task correctly and can repeat it successfully.

These constructs should not be treated as synonyms. A participant may complete a task successfully but report low confidence because the confirmation state is unclear. Another may enjoy an interface but find a particular task difficult. Rating scales require consistent wording, scale direction, timing, and response options.

A useful task-level measure is the Single Ease Question (SEQ), commonly presented immediately after a task using a seven-point scale from very difficult to very easy. The analyst can report the mean or median and the distribution of responses. A high SEQ indicates perceived ease for that task; it does not measure the entire product.

Confidence can be collected with a question such as “How confident are you that you completed the task correctly?” A five- or seven-point scale is acceptable if its anchors are clearly labelled. Confidence should be compared with actual success. The combination reveals calibration:

• high success and high confidence: likely clear performance;
• high success and low confidence: users may need stronger feedback;
• low success and high confidence: the design may permit undetected errors;
• low success and low confidence: the task is both difficult and uncertain.

3. Learnability and Change Over Time

Learnability is the ease with which new users become competent, while memorability concerns the ability to regain competence after an interval. Learnability is not adequately represented by one completion-time measurement.

A simple study can measure the same or equivalent task over repeated trials. Report success and time by trial number. A declining time trend, together with stable or increasing success, suggests improvement. If time declines while errors increase, apparent learning may represent rushing.

A learning rate can be described as a change score rather than assumed to follow a universal law:

This percentage is meaningful only when the tasks are comparable and the same timing rules are used. For example, a reduction from 120 seconds to 90 seconds is:

Repeated exposure introduces practice effects. Participants may remember the task rather than learn the interface. To study learnability, use equivalent tasks, document exposure, and consider a delayed retest for memorability.

4. Standardized Measures: SUS and SEQ

4.1 System Usability Scale

The System Usability Scale is a ten-item questionnaire with alternating positive and negative statements, usually answered on a five-point agreement scale from 1 (strongly disagree) to 5 (strongly agree). SUS produces a score from 0 to 100, but it is not a percentage of tasks completed.

Scoring procedure:

1. For odd-numbered items, subtract 1 from the response.
2. For even-numbered items, subtract the response from 5.
3. Add the ten adjusted values.
4. Multiply the sum by 2.5.

Each adjusted item score ranges from 0 to 4, so the unscaled sum ranges from 0 to 40 and the final score ranges from 0 to 100. The transformation assumes the standard ten-item SUS wording, response scale, and scoring order. If wording, scale, or item count changes, the standard interpretation may no longer apply.

Worked example: suppose the adjusted item scores are 3, 2, 3, 1, 4, 2, 3, 1, 3, and 2. Their sum is 24, so:

A SUS score of 60 should be interpreted comparatively and contextually. There is no universal point at which a product becomes “usable.” Scores can be compared across versions when the instrument, sample, task exposure, and administration procedure are comparable. Report the distribution and sample size, not only the average.

SUS is a global perceived-usability measure. It cannot identify which screen caused a problem. Pair it with task-level performance, SEQ, and qualitative evidence.

4.2 SEQ in relation to SUS

SUS asks about perceived usability of a system after interaction, whereas SEQ asks about perceived difficulty of a particular task immediately after it. They answer different questions:

A high SUS with one low SEQ may indicate that a generally acceptable product contains a critical task problem. Conversely, consistently high SEQ scores with a low SUS may indicate broader issues such as visual quality, trust, or consistency.

5. Behavioural Analytics

Behavioural analytics records activity from real or simulated use. It is especially useful for identifying scale, frequency, and locations of abandonment. It is less useful for directly determining intention or emotional cause.

5.1 Funnels

A funnel represents a sequence of stages, such as product page → cart → shipping → payment → confirmation. For stage i, stage conversion is:

Overall conversion from the first to the final stage is:

The denominator must be stated. If 1,000 users enter checkout, 800 reach shipping, 600 reach payment, and 480 reach confirmation, stage conversion rates are 80%, 75%, and 80%; overall conversion is 48%. These figures identify where loss is concentrated, but they do not prove that the interface caused the loss. Traffic source, device, price, technical faults, and user intent may also influence it.

Funnels should be segmented by meaningful variables such as device type, new versus returning users, accessibility technology, region, or entry route. Segmentation can reveal that an overall average hides a severe problem for one group.

5.2 Heatmaps

Heatmaps visualize aggregate patterns, including clicks, taps, scrolling, or pointer movement. They can reveal ignored controls, concentrated activity, and possible confusion. A heatmap is not an attention map. Pointer movement does not equal visual attention, and a high click area may simply reflect a frequently used control.

Interpretation requires page context, viewport dimensions, sample size, and device type. A desktop click map cannot be directly compared with a mobile tap map. Privacy-preserving aggregation is preferable to exposing identifiable individual behaviour.

5.3 Session recordings

Session recordings replay selected interactions. They can reveal repeated backtracking, hesitation, rage clicks, form corrections, and unexpected paths. Analysts should create a coding scheme before review—for example, “repeated submission,” “dead click,” “validation recovery,” or “navigation reversal.” This reduces confirmation bias.

Recordings are samples, not complete explanations. A visible hesitation may be caused by a slow network, unfamiliar content, or an external interruption. Review recordings alongside performance data and, when possible, user interviews.

5.4 Event instrumentation

Reliable analytics depends on a tracking specification. Each event should define its name, trigger, properties, user or session identifier, timestamp, and expected destination. Events should be tested in development and monitored after release. Duplicate events, missing events, ad blockers, consent withdrawal, and offline use can distort results.

A useful event taxonomy separates:

• object: the entity, such as checkout or search;
• action: what occurred, such as submitted or selected;
• context: device, route, experiment version, or error type.

Avoid collecting sensitive free text when a categorical event is sufficient.

6. Interpreting Metrics Together

Metrics are most powerful when interpreted as a pattern. Consider the following diagnostic matrix:

Do not average away important variation. Report subgroup results where sample sizes and privacy conditions permit. A redesign that improves the mean but harms keyboard users, older users, or screen-reader users is not an unqualified success.

For comparisons, define the change explicitly:

If completion rate rises from 60% to 72%, the absolute increase is 12 percentage points, while the relative increase is 20%. These are different statements and should not be confused.

When making statistical claims, consider sample size, uncertainty, repeated observations, and experimental design. A small numerical difference may be ordinary variation. In a certificate-level evaluation, it is often safer to report descriptive results transparently than to claim causality from uncontrolled analytics.

7. Applied Case: Illustrative Online Appointment Service

Background

A fictional public clinic, CarePoint, reports that many visitors begin online appointment booking but do not finish. The service team believes the form is too long and proposes removing several fields.

Problem and analysis

The team defines the funnel as: appointment search → date selection → patient details → confirmation. In a two-week period, 10,000 users begin the process. Of these, 8,200 select a date, 6,150 submit patient details, and 5,535 reach confirmation. Overall conversion is 55.35%.

A device segment shows that mobile users convert at 43%, compared with 70% for desktop users. Session recordings show repeated attempts to open a date selector and frequent correction of a phone-number field. A moderated test with 12 participants finds 9 successful completions, a median time of 4 minutes 10 seconds, and a median SEQ of 4 out of 7. Three participants submit an incorrect phone number because the format requirement is not visible until after submission.

Approach and solution

The team does not immediately remove fields. It first:

1. makes the date selector larger and keyboard accessible;
2. displays the phone-number format before entry;
3. preserves entered data after validation errors;
4. adds a clear confirmation summary;
5. instruments field-level validation and abandonment by device.

A follow-up test uses comparable tasks. Eleven of 12 participants complete successfully, median time falls to 3 minutes 20 seconds, and median SEQ rises to 6. Analytics after release show improved mobile completion, but the team checks whether the event implementation is complete and whether the change affects users requiring assistive technology.

Outcome and learning points

The case demonstrates why a funnel identifies location but not cause. The form may have contributed to abandonment, but the deeper problems were date-selector discoverability, late validation, and weak recovery. Combining success, time, errors, SEQ, recordings, and segmented analytics supported a more targeted redesign than simply deleting fields.

8. Ethical, Privacy, and Governance Considerations

UX analytics can capture sensitive behaviour. Collect only data necessary for the research purpose, disclose tracking clearly, obtain required consent, and provide appropriate choices. Do not record passwords, payment details, health information, private messages, or other sensitive content. Mask inputs and restrict access using role-based permissions.

Retention periods should be defined, and raw recordings should be deleted or anonymized when no longer needed. Analysts should consider whether re-identification is possible from combinations of time, location, device, and behaviour. Vulnerable populations require additional care.

Metrics can also create harmful incentives. Optimizing conversion may pressure users into unwanted actions, hide cancellation paths, or reward dark patterns. A good UX metric system includes safety, accessibility, error severity, user control, and long-term trust—not only speed or revenue.

AI-assisted analytics can classify sessions, summarize comments, or detect unusual interaction patterns. Such systems require validation because automated labels can misinterpret disability-related interaction, cultural behaviour, or technical problems. Human review remains necessary for consequential design decisions, and organizations should document model use, uncertainty, and data provenance.

9. Best Practices for a Measurement Plan

A practical measurement plan should:

1. state the decision and research question;
2. define the user population, task, start point, and success condition;
3. specify the unit of analysis and denominator;
4. select a small set of complementary metrics;
5. define error and assist coding before testing;
6. use consistent scales and administration timing;
7. segment results by relevant device and user characteristics;
8. report counts, not only percentages;
9. triangulate analytics with observation and qualitative evidence;
10. document limitations, missing data, privacy controls, and release changes.

A concise report might state: “Among 24 valid mobile checkout attempts, 18 succeeded (75%); median completion time was 90 seconds; the median SEQ was 5/7; six attempts contained at least one validation error. Analytics showed the largest funnel loss at payment, particularly for new mobile users. Further testing is required before attributing abandonment to the payment form.” This statement is more useful than a single claim that usability “improved.”

Chapter Summary

Usability metrics translate interaction into evidence about performance and experience. Task success measures achievement, completion time measures duration, errors reveal breakdowns, and efficiency combines successful output with resources used. Satisfaction, ease, and confidence are distinct subjective constructs that require consistent questions and scales. Learnability is assessed through change across comparable exposure, while practice effects must be considered.

The SUS provides a standardized global perceived-usability score from 0 to 100 when its standard procedure is followed. The SEQ measures perceived difficulty for an individual task. Funnels show where users leave a multi-stage process; heatmaps show aggregate interaction patterns; and session recordings provide behavioural examples. None of these tools, alone, establishes cause.

Reliable UX analysis depends on clear definitions, suitable denominators, triangulation, segmentation, transparent reporting, and ethical data governance. The strongest decisions connect quantitative patterns with direct observation and user explanations.

Key Terms

Metric: A defined measure of an observed phenomenon.

Task success: Achievement of a predefined task outcome.

Completion time: Elapsed time between defined task start and completion points.

Error: An action or outcome that deviates from the intended task path.

Assist: External help that enables or supports task progress.

Efficiency: Successful output considered in relation to resources used.

Satisfaction: A user’s subjective evaluation of the experience.

Confidence: A user’s belief that the task was completed correctly.

Learnability: The ease with which users become competent through use.

Memorability: The ability to regain competence after an interval.

SUS: A ten-item standardized measure of perceived system usability.

SEQ: A post-task rating of perceived task ease.

Funnel: A staged representation of progression and abandonment.

Heatmap: An aggregate visual representation of interaction activity.

Session recording: A replay or trace of an individual interaction session.

Event taxonomy: A structured naming and property system for analytics events.

Denominator: The population or set of observations against which a count is compared.

Triangulation: Combining different evidence sources to strengthen interpretation.

Validity: The extent to which a measure represents its intended construct.

Reliability: The consistency of a measurement procedure or result.

Review Questions

Short Answer Questions

1. Why must the denominator be reported with a task-success percentage?
2. Distinguish completion time from efficiency.
3. What does the SEQ measure, and when is it administered?
4. Why can a heatmap not be treated as a direct map of visual attention?
5. State two privacy precautions for session recordings.

Long Answer Questions

1. Explain how task success, time, errors, satisfaction, and confidence should be interpreted together.
2. Describe the SUS scoring procedure and explain why a SUS score is not a percentage of tasks completed.
3. Compare funnels, heatmaps, and session recordings in terms of purpose and limitations.
4. Explain how learnability can be evaluated while accounting for practice effects.
5. Discuss why analytics should be segmented by device or user group rather than reported only as an overall average.

Analytical / Application Questions

1. A task is completed successfully by 21 of 30 valid attempts. Calculate the success rate and explain one limitation of this result.
2. A checkout funnel has 2,000 entrants, 1,500 users reaching shipping, 1,050 reaching payment, and 840 reaching confirmation. Calculate each stage conversion rate and the overall conversion rate. Identify the largest proportional loss.
3. A redesign reduces median time but increases errors and lowers confidence. Propose a research plan to determine whether the redesign is actually better.

Multiple Choice Questions with Answers

Question 1

Which metric directly measures whether a participant achieved a predefined task outcome?

A. Task success

B. SUS

C. Heatmap density

D. Session duration

Correct Answer: A. Task success

Question 2

Twenty-one of 30 valid task attempts succeed. What is the success rate?

A. 60%

B. 65%

C. 70%

D. 75%

Correct Answer: C. 70%

Question 3

What is the correct first step when scoring an odd-numbered SUS item?

A. Subtract the response from 5

B. Subtract 1 from the response

C. Multiply the response by 2.5

D. Add the response to 5

Correct Answer: B. Subtract 1 from the response

Question 4

What does the SEQ primarily measure?

A. Overall brand loyalty

B. Frequency of returning visits

C. Perceived ease of a particular task

D. Number of interface events

Correct Answer: C. Perceived ease of a particular task

Question 5

A funnel is most useful for identifying:

A. Where users abandon a staged process

B. Users’ exact emotional reasons for leaving

C. The visual hierarchy of a screen

D. The reliability of a questionnaire

Correct Answer: A. Where users abandon a staged process

Question 6

Why should completion time be interpreted with success and error data?

A. Shorter time always indicates failure

B. Time has no measurable unit

C. A short time may result from skipping information or rushing

D. Errors can only occur in long tasks

Correct Answer: C. A short time may result from skipping information or rushing

Question 7

A SUS respondent has adjusted item scores totalling 24. What is the SUS score?

A. 24

B. 40

C. 60

D. 96

Correct Answer: C. 60

Question 8

Which statement about a heatmap is most accurate?

A. It directly records visual attention

B. It shows aggregate interaction patterns that require context

C. It proves why a user clicked

D. It replaces moderated usability testing

Correct Answer: B. It shows aggregate interaction patterns that require context

Question 9

If 1,000 users enter a funnel and 480 reach confirmation, what is the overall conversion rate?

A. 48%

B. 52%

C. 80%

D. 2.08%

Correct Answer: A. 48%

Question 10

Which combination most strongly supports a responsible UX conclusion?

A. One average metric only

B. A heatmap without a defined task

C. Behavioural data, user ratings, and qualitative evidence

D. Session recordings selected only after the preferred conclusion

Correct Answer: C. Behavioural data, user ratings, and qualitative evidence