Data Analysis and Insight Synthesis

Introduction

Usability testing produces evidence in several forms: spoken comments, observed actions, task outcomes, timings, error records, survey responses, and researcher notes. The value of a study does not come from collecting these materials alone. It comes from converting them into defensible findings that can guide design decisions.

Data analysis is the systematic process of organizing, examining, comparing, and interpreting study data. Insight synthesis is the subsequent process of combining findings into a coherent explanation of user behaviour and deciding what those findings mean for product improvement. These activities require more than counting failures or selecting memorable quotations. They require disciplined coding, pattern recognition, comparison of evidence sources, awareness of bias, and clear separation between what was observed and what is inferred.

For example, a participant may say, “This form is confusing,” pause for ten seconds, select the wrong option, and eventually complete the task. These are related but distinct pieces of evidence. The statement is a subjective report; the pause is an observed hesitation; the wrong selection is a behavioural error; and completion is an outcome measure. A sound analysis keeps these distinctions visible while examining how they may be connected.

Learning Objectives

By the end of this chapter, you should be able to:

1. distinguish raw data, observations, findings, insights, and recommendations;
2. create and apply a practical coding scheme for usability evidence;
3. identify recurring behavioural patterns without overstating the evidence;
4. calculate and interpret basic usability measures;
5. triangulate qualitative and quantitative evidence;
6. distinguish findings from opinions, assumptions, and design preferences;
7. recognize common sources of researcher and participant bias;
8. report study limitations and confidence appropriately; and
9. translate synthesized evidence into prioritized UX actions.

From Raw Data to Design Action

A usability analysis normally moves through several levels of interpretation:

This progression is not strictly linear. Analysts often move back and forth between levels. A new observation may require a code to be revised, and a suspected pattern may disappear when all sessions are reviewed. The central discipline is to label the status of each statement. “Three participants opened the menu twice” is evidence. “The menu is badly designed” is an evaluative opinion unless supported by a defined criterion and relevant comparison.

A useful finding usually contains four elements:

• Context: where or when the issue occurred;
• Behaviour: what users did or failed to do;
• Impact: how the behaviour affected task performance or experience; and
• Scope: how often, for whom, and under what conditions it occurred.

For example: “During account recovery, four of six first-time participants overlooked the secondary verification link. They searched the page, made at least one incorrect selection, and took longer to proceed. This suggests that the link’s visual hierarchy and wording do not sufficiently communicate its role for this user group.”

Preparing the Dataset

Before interpretation begins, the analyst should create an organized evidence base. At certificate level, this may be a spreadsheet; larger studies may use a research repository or qualitative analysis platform. The tool is less important than the structure and auditability of the record.

A practical session-level dataset may contain:

• participant identifier and relevant characteristics;
• task identifier and task wording;
• task success status and success definition;
• time on task, where timing is reliable;
• errors, assists, and abandoned actions;
• notable quotes;
• observed behaviours;
• moderator interventions;
• environmental or technical issues; and
• analyst comments and confidence notes.

Participant identifiers should be pseudonyms or codes rather than unnecessary personal information. Audio, video, transcripts, and screen recordings should be stored securely, with access limited to authorized people.

Defining the unit of analysis

A unit of analysis is the entity being compared. It might be a participant, task attempt, error event, screen, session, or interaction sequence. Confusion occurs when analysts mix these units. For example, “eight errors” does not necessarily mean “eight participants had problems.” One participant may generate several errors.

Before counting, specify the denominator. A task completion rate is commonly calculated as:

The calculation assumes that success has been defined in advance and that invalid attempts—such as those interrupted by a system outage—are excluded consistently. If ten valid attempts produce seven successes, the completion rate is 70%. This does not mean that 70% of the general population will succeed; it describes this sample under these test conditions.

For a sample-level mean time on task:

Here, t_i is the valid time for attempt i, and n is the number of valid attempts. The mean can be distorted by an unusually long attempt, so the median and the individual values should also be inspected. Timing should not be compared across tasks unless task scope, starting conditions, interruptions, and success criteria are reasonably comparable.

Coding Qualitative Evidence

Coding is the process of assigning meaningful labels to segments of data. A segment may be a sentence, an action, a sequence of actions, or an event in a recording. Codes make large volumes of evidence searchable and comparable.

Deductive and inductive coding

Deductive coding begins with categories established before review. A test plan may specify codes such as navigation, content comprehension, error recovery, accessibility, trust, and technical failure. Deductive codes support research questions and make analysis efficient.

Inductive coding allows categories to emerge from the data. An analyst may discover that participants repeatedly “scan the page for reassurance” or “interpret delivery date as dispatch date.” These patterns may not have been anticipated and can reveal unexpected needs.

A combined approach is often effective: begin with a small framework based on the study objectives, remain open to new codes, and record changes to the codebook.

A practical codebook

A codebook should define each code and explain how it is applied.

Good codes are specific enough to be applied consistently but broad enough to recur. “Bad design” is too vague. “Cannot identify the primary action after reading the page” is more observable and useful.

Coding procedure

1. Review the research question and define what counts as relevant evidence.
2. Read or watch a small portion of the material without coding to understand context.
3. Create initial codes and record definitions.
4. Apply codes to all relevant segments, allowing multiple codes where justified.
5. Review similar segments and merge, split, or rename codes when necessary.
6. Search for negative cases—instances that contradict an emerging pattern.
7. Record uncertainty rather than forcing ambiguous evidence into a category.
8. Build a matrix linking codes to participants, tasks, and outcomes.

A code is not automatically a finding. “Hesitation” may result from difficult interface design, unfamiliar subject matter, a distracting environment, or a participant thinking aloud. Coding identifies evidence; interpretation requires context.

Consistency and disagreement

When multiple analysts code the same material, they should discuss disagreements and refine definitions. Agreement is useful, but mechanical agreement is not the sole goal. Analysts may agree on a label while sharing the same mistaken interpretation. Discussion should therefore examine the underlying evidence and alternative explanations.

Formal agreement statistics may be appropriate for a substantial research project, but they are not essential for every small usability study. In a certificate-level project, a practical method is to independently code a small sample, compare differences, document decisions, and then apply the revised codebook consistently.

Recognizing Behavioural Patterns

A pattern is more than repetition. It is a meaningful relationship among events, people, tasks, or conditions. Analysts should examine four dimensions:

• Frequency: How often did the behaviour occur?
• Distribution: Which participants, tasks, or segments showed it?
• Severity: What was its effect on success, time, confidence, safety, or trust?
• Consistency: Did the same mechanism appear across cases?

Frequency should not be treated as the only measure of importance. A rare failure involving payment security or medical information may deserve greater attention than a frequent minor annoyance. Conversely, a single dramatic incident may not represent a general usability problem.

A pattern matrix can support disciplined comparison:

Qualitative sample sizes are usually too small to justify population estimates. The phrase “four of six participants experienced this issue” is accurate. The phrase “67% of users experience this issue” is usually unjustified unless supported by an appropriately designed quantitative study.

Pattern versus coincidence

To assess whether a pattern is credible, ask:

1. Did the behaviour occur independently in more than one case?
2. Did it occur in a comparable task or context?
3. Is there a plausible mechanism explaining it?
4. Are there contradictory cases?
5. Could the moderator, prototype, device, or task wording have produced it?

Negative cases are particularly valuable. If five participants miss a control but one notices it immediately, examine what differed. The successful participant may have had prior domain knowledge, or may have used a different device. Alternatively, the case may show that the issue is conditional rather than universal.

Triangulating Evidence

Triangulation means examining a research question through multiple evidence sources, methods, analysts, or perspectives. It does not mean that every source must produce identical results. Differences can reveal important distinctions between what users say, what they do, and what they remember.

Common forms include:

• Data triangulation: comparing participants, contexts, devices, or time periods;
• Method triangulation: combining observation, interviews, task measures, and surveys;
• Analyst triangulation: involving more than one analyst;
• Theory or perspective triangulation: interpreting evidence through alternative conceptual lenses.

Consider the following evidence about a checkout form:

• Six of eight participants completed the form.
• Four participants paused at the address field.
• Three asked whether the billing address was required.
• Survey responses rated the checkout “easy” on average.
• Two participants later reported that they were unsure whether the order had been submitted.

The evidence suggests a specific problem with address requirements and submission feedback, even though the overall ease rating appears positive. The quantitative outcome shows that most participants completed the task; observations reveal friction; comments identify uncertainty; and the survey provides broader but less specific perception data.

Triangulation should be explanatory rather than merely numerical. Analysts should ask:

• Do the sources converge on the same issue?
• Do they address different stages of the experience?
• Does one source qualify or contradict another?
• Could the measurement method itself explain the difference?

A survey may show high satisfaction because participants judge the overall experience, while observation exposes a serious problem affecting a smaller but important subgroup. Neither source should automatically be discarded.

Findings, Insights, Opinions, and Recommendations

These categories must remain distinct.

• A finding is a claim supported by collected evidence.
• An insight is an interpretation that explains significance, cause, or user need.
• An opinion is a personal judgment or preference.
• A recommendation is a proposed action based on findings and insights.

Compare the statements:

1. “The blue button looks ugly.” This is an opinion.
2. “Five participants did not notice the primary action until the moderator pointed to it.” This is a finding.
3. “The action’s low visual prominence makes it compete poorly with secondary links.” This is an interpretation supported by the observation, but should be presented as a hypothesis unless other evidence confirms it.
4. “Increase the primary action’s visual prominence and test whether discovery improves.” This is a recommendation.

Recommendations should not exceed the evidence. If the study shows that users miss a control, it may support improving discoverability. It may not establish that a particular colour, icon, or layout is the correct solution. Design alternatives should be prototyped and tested.

A useful finding statement follows this structure:

When [context], [user group] performed [behaviour], resulting in [impact]. Evidence came from [sources], and confidence is [level] because [reason].

This structure makes assumptions visible and helps stakeholders distinguish evidence from proposed action.

Basic Quantitative Analysis for Usability Studies

Quantitative measures are useful when their definitions are clear. Common measures include completion, abandonment, error frequency, time on task, assists, and ratings.

For an error proportion:

The numerator must be defined carefully. “Attempts containing the error” differs from “total number of error events.” If two participants each make three errors, the first measure is 2 affected attempts; the second is 6 events.

For a task success rate, partial success may require categories such as success without assistance, success with assistance, partial completion, and failure. Combining all categories into “success” can conceal important usability differences. Report the scoring rule.

Small usability samples support diagnosis more readily than precise estimation. A result from five participants can reveal a severe design problem, but it cannot reliably estimate the exact percentage of all users who would encounter that problem. Avoid unnecessary decimal places and avoid implying statistical certainty that the design supports.

When comparing versions, keep the task, participant characteristics, instructions, and environmental conditions as similar as possible. A faster completion time may reflect learning, a simpler task, or a more experienced sample rather than a better interface.

Bias and Threats to Validity

Bias is a systematic influence that can distort collection, analysis, or interpretation. Common sources include:

Sampling bias

Participants may not represent the intended audience. Recruiting colleagues, students, or highly motivated volunteers can produce unusually knowledgeable or patient participants.

Moderator bias

Tone, facial expression, hints, question wording, and unequal assistance can influence behaviour. A moderator who expects a new design to succeed may unintentionally guide participants toward it.

Confirmation bias

Analysts may notice evidence supporting an early theory while overlooking contradictory cases. Reviewing all sessions and actively searching for disconfirming evidence helps reduce this risk.

Recall and self-report bias

Participants may not accurately remember what they did or may provide socially desirable answers. Observed behaviour and self-reported opinion should be treated as complementary, not interchangeable.

Novelty and observer effects

Participants may behave differently because they know they are being observed or because the prototype is unfamiliar. A single session may not reflect routine use.

Instrumentation and technical bias

Recording failures, inconsistent timing, prototype bugs, device differences, or accessibility-tool incompatibility can create misleading results.

Bias cannot be eliminated completely. It should be identified, reduced where possible, and disclosed in the report.

Study Limitations and Confidence

A credible report states what the study cannot establish. Limitations may concern sample composition, task realism, prototype fidelity, number of sessions, moderator involvement, technical instability, or missing data.

Confidence is not a substitute for statistical significance. It is a reasoned judgment about how strongly the evidence supports a claim. Confidence increases when evidence is observed directly, repeated across independent cases, consistent across methods, and supported by a plausible explanation. Confidence decreases when evidence is isolated, ambiguous, dependent on moderator assistance, or contradicted by credible cases.

A simple reporting scale may be useful:

• High confidence within the tested context: repeated, clear evidence with little contradiction;
• Moderate confidence: a recurring pattern with some uncertainty or limited coverage;
• Low confidence: an isolated or ambiguous observation requiring further investigation.

The phrase “within the tested context” is important. A finding may be highly credible for keyboard users completing the tested form but not necessarily for mobile users or people with different assistive technologies.

From Synthesis to Prioritization

After findings are developed, teams need to decide what to address first. Prioritization should consider user impact, frequency, severity, strategic importance, implementation effort, and confidence in the evidence.

A simple qualitative matrix can classify issues:

Do not reduce every decision to a single score. A numerical priority score can create false precision, especially when severity and confidence are subjective. If a team uses scoring, define each scale and retain the underlying evidence.

A recommendation should identify the user problem, proposed direction, success criterion, and next step. For example: “Expose the date filter beside the results heading; retest with first-time users; success means at least five of six participants locate it without assistance.” The criterion is not a guarantee, but it makes the next evaluation concrete.

Applied Case: Illustrative Online Appointment Booking Study

Background

A clinic tests a prototype for booking appointments. Six participants who have not used the prototype before are asked to find a suitable appointment, select a clinician, and confirm the booking. The study uses observation, think-aloud comments, task outcomes, and a short post-task interview.

Problem

The product team believes the booking process is straightforward because participants generally complete it and post-task ratings are positive. However, support staff report that patients sometimes book the wrong appointment type.

Analysis

The analyst defines valid completion as selecting the requested appointment type, choosing a suitable time, and reaching the confirmation screen without moderator instruction. The evidence matrix is:

The completion rate without assistance is:

This percentage describes the six valid attempts; it is not a population estimate. Coding shows related behaviours: scanning labels, opening descriptions, choosing an incorrect type, and asking for clarification. The pattern is not simply “users dislike the page.” It is a comprehension and categorization problem concentrated at the appointment-type step.

Triangulation strengthens the finding. Observation shows hesitation, task outcomes show incorrect initial choices, and comments reveal uncertainty about terminology. Positive overall ratings do not cancel the specific problem because participants may have rated the complete experience after receiving assistance or after eventually succeeding.

Approach and solution

The team proposes three design directions: use patient-centred labels, add concise descriptions, and display examples of when each appointment type is appropriate. Because the study does not establish which wording is best, the team creates two alternative prototypes rather than adopting one design based on preference.

The next test uses comparable participants and the same task. The success criterion is correct appointment-type selection without moderator assistance, accompanied by reduced hesitation and fewer clarification requests.

Outcome

In the subsequent test, five of six participants select the correct type immediately, while one compares two options before making a correct choice. The result is encouraging, but the team records that the sample remains small and that repeat use, accessibility, and multilingual wording have not yet been evaluated.

Learning points

1. Overall completion concealed a serious intermediate problem.
2. Coding connected actions, comments, and outcomes without treating them as identical.
3. Triangulation explained the problem more convincingly than any single measure.
4. The analysis supported a design direction, not certainty about one final solution.
5. Limitations remained visible after the redesign appeared to improve performance.

Ethical, Governance, and AI Considerations

Usability data may contain names, health information, financial details, voice recordings, or screen content. Collect only what is necessary, obtain informed consent, explain recording and retention, and allow participants to withdraw where feasible. Reports should use anonymized examples and avoid publishing identifiable quotations without permission.

AI tools can assist with transcription, search, clustering, and preliminary coding. They should not be treated as independent researchers. Automated systems may miss sarcasm, accessibility context, hesitation, cultural meaning, or the difference between an observation and an interpretation. Analysts should check transcripts, protect confidential data, document tool use, and retain human responsibility for coding decisions and conclusions. Sensitive recordings should not be uploaded to an unapproved service.

Best Practices

• Define success, failure, error, and assistance before analysis.
• Preserve links between claims and original evidence.
• Use a codebook and document revisions.
• Separate participant statements from observed actions.
• Search deliberately for disconfirming evidence.
• Report denominators, not only percentages.
• Avoid generalizing beyond the sample and context.
• Explain contradictions among data sources.
• State limitations beside important findings.
• Recommend testable design changes rather than unsupported preferences.
• Protect participant privacy throughout storage, analysis, and reporting.

Chapter Summary

Data analysis converts usability materials into organized evidence, while insight synthesis explains what that evidence means for users and design decisions. A disciplined workflow distinguishes raw data, observations, codes, patterns, findings, insights, and recommendations. Coding requires clear definitions, consistent application, and attention to context. Patterns should be judged by frequency, distribution, impact, consistency, and contradictory cases rather than frequency alone.

Triangulation compares different participants, methods, analysts, and perspectives. Qualitative observations, quantitative measures, and self-reports may converge or reveal different aspects of an experience. Findings must be distinguished from opinions, and recommendations must not claim more certainty than the evidence allows. Bias may arise from sampling, moderation, self-reporting, analysis, instrumentation, and observation. Limitations and confidence should therefore be reported explicitly.

Basic measures such as completion rate, error proportion, and mean time can support diagnosis when their denominators and assumptions are clear. Small usability studies are especially valuable for discovering problems, not for producing precise population estimates. Ethical analysis protects participant data, and AI-assisted analysis requires human review, transparency, and secure handling. The final purpose of synthesis is actionable learning: a clear account of user behaviour, its consequences, the strength of the evidence, and the next design or research step.

Key Terms

Raw data: Original recordings, transcripts, measures, and other unprocessed study material.

Observation: A concise description of what occurred during a session.

Code: A label applied to a relevant segment of evidence.

Codebook: A documented set of codes, definitions, and application rules.

Deductive coding: Coding based on categories established before data review.

Inductive coding: Coding in which categories emerge from the data.

Pattern: A meaningful recurring relationship among coded observations.

Finding: An evidence-supported statement about behaviour, performance, or experience.

Insight: An interpretation explaining why a finding matters.

Recommendation: A proposed action based on findings and insights.

Triangulation: Comparison of evidence from multiple sources, methods, analysts, or perspectives.

Unit of analysis: The entity being examined, such as a participant, task, or error event.

Completion rate: The proportion of valid task attempts meeting the defined success criterion.

Error proportion: The proportion of valid attempts containing a specified error.

Negative case: Evidence that contradicts or qualifies an emerging pattern.

Bias: A systematic influence that distorts data collection or interpretation.

Confidence: A reasoned judgment about the strength and reliability of a finding.

Validity: The extent to which a study supports the claims made from its evidence.

Denominator: The total valid cases used as the basis for a rate or proportion.

Insight synthesis: The process of combining evidence into explanations and actionable conclusions.

Review Questions

Short Answer Questions

1. What is the difference between an observation, a finding, and an insight?
2. Why must analysts define the unit of analysis before counting errors?
3. What is the purpose of a codebook?
4. Name two forms of triangulation used in usability research.
5. Why should a small usability sample not normally be used to estimate the exact percentage of all users affected by an issue?

Long Answer Questions

1. Explain a complete workflow for coding and synthesizing qualitative usability evidence.
2. Discuss how frequency, distribution, severity, and consistency help analysts assess behavioural patterns.
3. Explain why positive survey ratings may coexist with serious observed usability problems.
4. Compare findings, insights, opinions, and recommendations, using an example of each.
5. Discuss at least four sources of bias in usability testing and describe practical ways to reduce or disclose them.

Analytical / Application Questions

1. In a study of eight valid attempts, six users complete a task, three make at least one navigation error, and one attempt is abandoned. Calculate the completion rate and error proportion. Explain why the two measures describe different aspects of usability.
2. Five of six participants hesitate at a form field, but only two fail to complete the task. Construct a finding statement, identify one plausible insight, and propose a follow-up test.
3. A team’s observation data show repeated confusion, while its satisfaction survey is positive. Design a triangulation plan that could explain the apparent contradiction and identify what further evidence should be collected.

Multiple Choice Questions with Answers

Question 1

Which statement is a finding rather than an opinion?

A. The interface looks unattractive.

B. The button should be green.

C. Four of six participants overlooked the primary action.

D. The layout feels old-fashioned.

Correct Answer: C. Four of six participants overlooked the primary action.

Question 2

What is the main purpose of a codebook?

A. To store participant passwords

B. To define codes and rules for applying them consistently

C. To calculate population estimates

D. To replace usability testing

Correct Answer: B. To define codes and rules for applying them consistently

Question 3

Six of eight valid task attempts are successful. What is the completion rate?

A. 25%

B. 60%

C. 75%

D. 80%

Correct Answer: C. 75%

Question 4

Which practice best supports triangulation?

A. Using only the most memorable participant quotation

B. Comparing observations, task outcomes, and participant reports

C. Ignoring contradictory evidence

D. Replacing observations with survey averages

Correct Answer: B. Comparing observations, task outcomes, and participant reports

Question 5

What is a negative case?

A. A participant who refuses consent

B. Evidence that contradicts or qualifies an emerging pattern

C. A task with a negative time value

D. A code that cannot be used twice

Correct Answer: B. Evidence that contradicts or qualifies an emerging pattern

Question 6

Which statement about a small usability sample is most appropriate?

A. It can always estimate the exact population failure rate.

B. It is useful for discovering problems but has limited precision for population estimates.

C. It cannot provide any meaningful evidence.

D. It eliminates the need for qualitative analysis.

Correct Answer: B. It is useful for discovering problems but has limited precision for population estimates.

Question 7

Two participants each make three errors. How many attempts contain an error, assuming each participant made errors during one attempt?

A. 2

B. 3

C. 5

D. 6

Correct Answer: A. 2

Question 8

Which is a likely example of moderator bias?

A. A participant uses a keyboard instead of a mouse.

B. A moderator gives more hints to one version of a prototype.

C. A task has a predefined success criterion.

D. A recording is stored securely.

Correct Answer: B. A moderator gives more hints to one version of a prototype.

Question 9

What should a recommendation normally include?

A. Only the analyst’s personal design preference

B. A proposed action and a way to evaluate whether it helps

C. A claim that the solution is permanently proven

D. The removal of contradictory cases

Correct Answer: B. A proposed action and a way to evaluate whether it helps

Question 10

Why should AI-generated coding of usability data receive human review?

A. AI cannot process any text

B. Human review is unnecessary for sensitive data

C. Automated systems may miss context and confuse observations with interpretations

D. AI tools always produce numerical results only

Correct Answer: C. Automated systems may miss context and confuse observations with interpretations