AI-Enabled Research and Emerging Practice

Introduction

Artificial intelligence (AI) is changing how user researchers recruit participants, collect evidence, analyse qualitative data, create prototypes and communicate findings. In a UI/UX context, AI may transcribe an interview in minutes, suggest codes across hundreds of comments, generate alternative interface concepts or simulate a conversation with a proposed service. These capabilities can increase speed and broaden access to research. They do not, however, turn weak evidence into strong evidence or remove the need for professional judgement.

AI systems generate outputs from patterns in data and instructions. They do not possess human understanding, accountability or direct access to a participant’s lived experience. A responsible researcher therefore treats AI as an assistive instrument rather than an autonomous researcher. The central question is not “Can AI do this task?” but “What role can AI safely play, what evidence supports its output, and where must a human remain responsible?”

This chapter examines AI-assisted transcription, coding, synthesis, prototype creation and research support. It also addresses hallucination, bias, privacy, security, synthetic-user risks, human oversight, remote research, personalization and digital transformation.

Learning objectives

By the end of this chapter, students should be able to:

• describe appropriate uses of AI throughout the user-research and design-thinking process;
• distinguish automation, augmentation and synthetic generation;
• verify AI-generated transcripts, codes, summaries and design outputs;
• identify risks involving hallucination, bias, privacy, confidentiality and fabricated users;
• design a human-in-the-loop research workflow;
• evaluate emerging practices in remote research, personalization and digital transformation; and
• communicate AI use transparently in a research report.

1. AI in the research workflow

AI can support nearly every stage of a research project, but the level of risk differs by task. A useful classification is:

A responsible workflow preserves the distinction between primary evidence—what participants actually said or did—and interpretive or generated material. A transcript is closer to primary evidence than an AI summary, while a generated persona is not evidence of any real user group.

A practical AI research cycle is:

1. Define the research question and evidence requirements.
2. Obtain informed consent and establish data-handling rules.
3. Collect data using appropriate human methods.
4. Use AI for a bounded, documented task.
5. Inspect outputs against recordings, notes or source files.
6. Compare interpretations among researchers where possible.
7. Report the tool, purpose, limitations and human checks.
8. Retain, delete or anonymise data according to policy.

The sequence matters. If AI is used before the problem is clearly framed, it can encourage premature solutions or produce a large volume of irrelevant analysis.

2. Transcription and translation

Automatic speech recognition can produce searchable transcripts, timestamps, speaker labels and translations. This is valuable for interviews, usability tests and diary studies. Accuracy varies with accent, background noise, overlapping speech, specialist vocabulary, code-switching and recording quality.

A transcript should therefore be treated as an analysis aid, not an unquestionable record. Researchers should review important passages against the audio, especially quotations used in reports, moments involving negation (“I did not…”) and statements that could change a design decision. Speaker attribution should also be checked; a wrongly assigned statement can distort the interpretation of an entire interview.

Before uploading recordings, researchers should ask:

• Is the provider permitted to process this type of data?
• Is data used for model training, and can that use be disabled?
• Where are recordings and transcripts stored?
• Are retention, deletion and access controls documented?
• Has each participant consented to recording and relevant processing?

Translation introduces a further interpretive layer. Idioms, culturally specific expressions and politeness may be lost. For consequential research, bilingual review is preferable to relying entirely on machine translation.

3. AI-assisted coding and synthesis

Qualitative coding assigns labels to segments of data. Codes may be descriptive, such as “difficulty finding settings,” or interpretive, such as “low perceived control.” AI can propose codes, cluster similar comments, identify apparent contradictions and retrieve examples. It can reduce mechanical effort, but it cannot determine the significance of an experience without contextual judgement.

A sound coding process begins with a codebook. A codebook states each code’s name, definition, inclusion criteria, exclusion criteria and example. For instance:

The researcher may ask AI to apply provisional codes, but a sample of outputs must be compared with the codebook and original data. New codes proposed by AI should not be accepted merely because they sound plausible. A useful audit records the prompt, model or tool, date, dataset version, accepted changes and rejected suggestions.

From codes to themes

A theme is more than a frequently repeated word. It is a meaningful pattern related to the research question. AI may count mentions of “privacy,” but it may not distinguish concern about surveillance from confusion about a privacy setting. Synthesis should therefore connect:

• evidence: what participants said or did;
• interpretation: what pattern the researcher identifies;
• implication: what the pattern means for design or service decisions; and
• uncertainty: what remains unknown or contradictory.

One useful discipline is the evidence ladder:

1. Quote or observation.
2. Code attached to the evidence.
3. Pattern supported by multiple items or cases.
4. Interpretation linked to the research question.
5. Design implication, stated with appropriate confidence.

AI-generated summaries often compress disagreement and remove qualifiers. Researchers should preserve minority views, negative cases and uncertainty rather than presenting an artificially coherent narrative.

4. Hallucination, bias and reliability

A hallucination is an output that presents unsupported or false content as if it were reliable. In research, an AI system might invent a quotation, attribute a comment to the wrong participant, cite a nonexistent source or infer a user need not present in the data. Fluent language is not evidence of correctness.

Common safeguards include:

• require outputs to reference source segments or participant IDs;
• compare every important claim with the underlying data;
• prohibit invented quotations and label generated examples clearly;
• use retrieval from an approved dataset rather than unrestricted generation where possible;
• ask a second researcher to review consequential interpretations; and
• retain an audit trail of prompts, outputs and corrections.

Bias can enter through the training data, the model, the prompt, the sample, the researcher’s interpretation or the design decision. AI may overrepresent dominant language varieties, associate occupations or behaviours with demographic stereotypes, or interpret disability-related behaviour as error. Human researchers also bring assumptions; AI does not eliminate those assumptions and may make them harder to see because its output appears objective.

Bias review should consider:

Accuracy should be assessed by task and subgroup, not only by an overall average. In a simple audit, transcription error rate can be expressed as:

Here, WER is word error rate; S is the number of substitutions, D the number of deletions, I the number of insertions, and N the number of words in the reference transcript. The measure assumes a defined reference transcript and is useful for comparing systems, but it does not capture whether an error changes meaning or misidentifies a speaker. It should not be interpreted as a complete measure of research quality.

5. Privacy, consent and governance

Research data can include names, voices, faces, health information, financial information, locations, behavioural traces and sensitive opinions. A transcript may remain identifiable even after names are removed because combinations of details can reveal a person. Data minimisation means collecting and exposing only what is necessary for the stated purpose.

Responsible governance includes:

• informed consent that explains recording, AI processing, storage and possible human review;
• purpose limitation: using data only for approved research purposes;
• access control based on role and necessity;
• encryption in transit and at rest where appropriate;
• pseudonymisation or anonymisation before analysis when feasible;
• defined retention and deletion schedules;
• vendor due diligence and contractual restrictions on secondary use; and
• an incident process for accidental disclosure or misuse.

Legal requirements differ by jurisdiction. Researchers should follow applicable data-protection, copyright, accessibility and sector-specific rules, and consult institutional policy or a data-protection officer when uncertainty exists. Consent is not a universal solution: a participant may agree to research participation without reasonably expecting their data to train a general-purpose model or be transferred to another country.

A research report should disclose the AI tool or class of tool, the task performed, whether personal data were processed, the validation procedure and important limitations. Transparency supports reproducibility and allows stakeholders to judge the strength of the evidence.

6. Synthetic users and generated personas

A synthetic user is an AI-generated representation that imitates a user, demographic group or behavioural profile. Synthetic users can support brainstorming, edge-case exploration and early scenario writing. They can help a team ask, “What might we be overlooking?” They cannot replace recruitment, observation or usability testing with real people.

The danger is false empirical authority: a generated persona may sound specific and realistic while being based on stereotypes or statistical associations. A synthetic user has no lived experience, cannot provide informed consent and cannot reveal how an actual interface performs for a real participant.

Use synthetic users only with explicit labels and bounded claims. They are suitable for:

• generating alternative scenarios;
• stress-testing language for potentially confusing interpretations;
• exploring accessibility questions before recruitment; and
• producing hypotheses to investigate.

They are unsuitable for estimating prevalence, validating usability, proving desirability or replacing underrepresented participants. In design thinking, synthetic outputs belong in ideation and hypothesis formation, not in the evidence column of a research findings table.

7. AI-assisted prototype creation

Generative tools can produce interface sketches, content variants, code snippets, image assets and interactive prototypes. Their value is greatest when the design question is explicit. For example, a team may ask for three low-fidelity alternatives for a medication-reminder flow, each with different levels of explanation. The outputs can accelerate comparison, but they remain proposals.

Every generated prototype requires human review for:

• alignment with user needs and research evidence;
• accessibility, including keyboard use, contrast, readable language and assistive technology;
• information architecture and interaction coherence;
• security, privacy and dark-pattern risks;
• technical feasibility and maintainability;
• cultural appropriateness and inclusive representation; and
• intellectual-property or licensing restrictions.

AI-generated code is not automatically secure or production-ready. It may contain obsolete dependencies, insecure data handling or inaccessible components. A designer should document what was generated, what was modified and what was tested.

8. Human oversight and decision rights

Human oversight is not simply reading an AI response quickly. It requires a person with suitable competence, authority and time to challenge the output. A useful responsibility matrix distinguishes:

• AI task owner: defines the prompt and operating limits;
• researcher: checks evidence, context and interpretation;
• subject-matter or accessibility reviewer: examines domain-specific risks;
• decision owner: accepts or rejects the design implication; and
• data custodian: controls access, retention and compliance.

For high-impact decisions, no single generated output should determine the result. Escalation is appropriate when data are highly sensitive, outputs conflict with primary evidence, performance differs across groups, or a design decision could materially affect rights, safety or access.

A compact quality test is the TRACE model:

• T — Traceability: Can each important claim be traced to source evidence?
• R — Reliability: Has output quality been tested for the relevant task and users?
• A — Accountability: Is a named person responsible for the decision?
• C — Confidentiality: Are privacy, consent and security requirements satisfied?
• E — Equity: Have differential harms and excluded groups been considered?

TRACE is a practical checklist rather than a substitute for legal or ethical review.

9. Remote research, personalization and digital transformation

Remote research has expanded through video interviews, unmoderated usability tests, mobile diaries, screen sharing, online communities and product analytics. AI can assist with scheduling, session summaries, multilingual moderation and large-scale feedback analysis. Remote methods can increase geographic reach and support participants who prefer familiar environments, but they may exclude people with limited connectivity, private space, suitable devices or digital confidence.

Good remote practice includes accessibility checks, alternative participation channels, clear recording indicators, careful management of screen-sharing data and plans for technical failure. Researchers should not assume that a silent participant is disengaged; connection quality, language and cultural communication norms may affect behaviour.

Personalization uses behavioural or contextual data to adapt content, recommendations or interfaces. It can improve relevance, but excessive personalization may reduce user control, expose sensitive inferences or create unequal experiences. Research should test not only whether personalization increases engagement, but whether users understand it, can change it and consider it fair. Meaningful control includes understandable explanations, opt-out mechanisms and the ability to correct inaccurate assumptions.

Digital transformation refers to organisational change enabled by digital technologies, not merely the introduction of an AI tool. It changes workflows, roles, data practices and measures of success. A successful transformation therefore combines technology with research capability, governance, training and organisational learning. Automating a broken research process can make poor decisions faster.

Applied case: an AI-supported public-service appointment redesign

This is an illustrative case. A municipal health service wants to redesign its appointment-booking website. The service receives 4,000 monthly support messages and has limited research capacity.

Problem. Staff believe that users cannot find cancellation instructions, but the claim is based on anecdotal complaints. Some messages contain health information and cannot be sent casually to a public AI service.

Analysis. The research team defines three questions: where users fail, which language causes uncertainty, and whether barriers differ for mobile users or people using screen readers. They obtain governance approval, remove direct identifiers, and create a restricted analysis environment. A sample of support messages is manually reviewed to establish a reference set.

Approach. AI clusters anonymised messages and suggests candidate codes. Researchers compare those codes with the reference set, revise the codebook and inspect contradictory cases. Interview recordings are transcribed internally, with human review of quotations and speaker labels. A generative tool creates three low-fidelity booking flows, but these are treated as design hypotheses. Five real participants, including two screen-reader users, test the prototypes remotely. Participants are told when sessions are recorded and can choose a non-recorded option.

Outcome. The team finds that “cancellation problem” combines two issues: users cannot locate the option, and some users fear losing their appointment when they select it. The redesign separates cancellation from rescheduling, explains consequences before confirmation and adds a visible support route. The AI clustering accelerated discovery, but participant testing revealed the trust concern that the initial automated summary had missed.

Learning points. Anonymisation reduced but did not eliminate risk; a restricted tool and deletion schedule were still needed. AI was useful for organising evidence, not for deciding what the evidence meant. Real users were necessary to validate comprehension and accessibility. The final report documented the AI tasks, quality checks and unresolved limitations.

Challenges and limitations

AI-assisted research has several structural limitations. Models may be unavailable or change behaviour over time, reducing reproducibility. Interfaces may conceal how data are processed. Subscription costs and technical requirements can widen inequality between organisations. Automation can deskill junior researchers if they no longer learn close observation, interviewing and analysis. Generated content can also increase volume without increasing insight, creating review burden and encouraging superficial decision-making.

The remedy is not to reject AI categorically. It is to match tool capability to research risk, preserve primary evidence, test performance, include affected communities and maintain accountable human judgement.

Best-practice workflow

1. State the research question and what counts as sufficient evidence.
2. Classify the proposed AI use as automation, augmentation, generation or simulation.
3. Conduct a privacy, bias and accessibility risk review.
4. Obtain appropriate consent and minimise the data shared.
5. Use a bounded prompt and an approved tool.
6. Keep source data separate from generated interpretations.
7. Validate outputs against primary evidence and, where possible, independent reviewers.
8. Test designs with real users, especially affected or underrepresented groups.
9. Record tool use, corrections, uncertainty and decisions.
10. Delete or retain data according to a documented policy.

Chapter Summary

AI can accelerate transcription, coding, synthesis, prototyping and research administration, but it does not replace user evidence or professional accountability. Automation handles bounded tasks; augmentation proposes material for human review; generation creates design possibilities; and simulation produces hypotheses rather than empirical findings. Hallucinations, biased outputs, transcription errors and fabricated users require systematic checking against source data. Privacy protection depends on consent, minimisation, access control, secure processing, retention rules and transparency. Human oversight must include real decision authority, not superficial approval. Remote research and personalization expand reach and relevance while creating risks of exclusion, surveillance and unequal experiences. Digital transformation succeeds when AI is combined with sound research practice, governance, accessibility and organisational learning.

Key Terms

• Artificial intelligence: Systems that perform tasks commonly associated with human intelligence, such as language processing or pattern recognition.
• Automation: Use of technology to perform a defined task with limited human intervention.
• Augmentation: Use of AI to support, rather than replace, human judgement.
• Synthetic user: An AI-generated imitation of a user or user group.
• Hallucination: Unsupported or false content presented as if it were reliable.
• Bias: A systematic tendency that produces unfair or distorted results.
• Codebook: A documented set of codes, definitions and application rules.
• Qualitative coding: Labelling segments of data to support interpretation.
• Theme: A meaningful pattern related to a research question.
• Primary evidence: Data directly collected from participants or observed behaviour.
• Data minimisation: Collecting and processing only data necessary for a stated purpose.
• Pseudonymisation: Replacing direct identifiers while retaining a controlled re-identification possibility.
• Informed consent: Voluntary agreement based on understandable information about participation and data use.
• Human-in-the-loop: A process in which a person reviews, directs or can override an AI system.
• Personalization: Adapting content or interaction to a user or context.
• Remote research: Research conducted when participants and researchers are not co-located.
• Digital transformation: Organisational change enabled by digital technology, data and new practices.
• Word error rate: A comparison of substitutions, deletions and insertions against a reference transcript.

Review Questions

Short Answer Questions

1. Distinguish automation, augmentation, generation and simulation in AI-enabled research.
2. Why must an AI-generated summary be checked against primary evidence?
3. Identify three privacy controls relevant to AI-assisted research.
4. Why are synthetic users unsuitable as substitutes for usability testing?
5. What does human oversight require beyond reading an AI output?

Long Answer Questions

1. Explain how AI can support qualitative coding while preserving researcher accountability.
2. Discuss the main sources of hallucination and bias in AI-enabled user research.
3. Describe a responsible workflow for using AI to transcribe and analyse interviews.
4. Evaluate the opportunities and risks of AI-assisted prototype generation.
5. Explain how remote research and personalization can improve and also undermine inclusive design.

Analytical / Application Questions

1. A team uploads identifiable interview recordings to a free AI transcription service. Analyse the risks and propose a compliant alternative workflow.
2. An AI summary claims that all participants want more personalization, although two participants expressed discomfort. Explain how you would investigate and report this contradiction.
3. Design a TRACE-based review plan for an AI-generated prototype intended for a public-service website, including evidence checks, accessibility testing and decision authority.

Multiple Choice Questions with Answers

Question 1

Which AI use is best classified as augmentation?

A. Automatically converting an audio recording into text

B. Suggesting provisional themes for a researcher to verify

C. Replacing participants with generated personas

D. Deleting all source data after analysis

Correct Answer: B. Suggesting provisional themes for a researcher to verify

Question 2

Why should important transcript quotations be checked against the recording?

A. AI transcripts are always longer than recordings

B. Speaker labels and wording may be incorrect

C. Checking quotations removes the need for consent

D. Audio recordings cannot contain primary evidence

Correct Answer: B. Speaker labels and wording may be incorrect

Question 3

What is an AI hallucination in user research?

A. A participant changing their opinion

B. A false or unsupported claim presented as reliable

C. A researcher using multiple methods

D. A prototype with a visual defect

Correct Answer: B. A false or unsupported claim presented as reliable

Question 4

Which item is primary evidence?

A. An AI-generated persona

B. A generated interface concept

C. A participant’s observed behaviour during a usability test

D. An AI-written executive summary

Correct Answer: C. A participant’s observed behaviour during a usability test

Question 5

Which practice best protects confidential research data?

A. Uploading identifiable recordings to any free tool

B. Sharing all transcripts publicly for transparency

C. Minimising data and using an approved, access-controlled environment

D. Removing only first names and ignoring other identifiers

Correct Answer: C. Minimising data and using an approved, access-controlled environment

Question 6

What is a major limitation of synthetic users?

A. They cannot produce any text

B. They provide no lived experience or empirical validation

C. They always require a physical laboratory

D. They eliminate the need for design hypotheses

Correct Answer: B. They provide no lived experience or empirical validation

Question 7

What does a codebook usually contain?

A. Only the names of research participants

B. Definitions and rules for applying codes

C. A list of software licences

D. A schedule for deleting prototypes

Correct Answer: B. Definitions and rules for applying codes

Question 8

Which statement about personalization is most responsible?

A. Higher engagement always proves personalization is beneficial

B. Personalization should be assessed for control, fairness and user understanding

C. Users should never be allowed to opt out

D. Personalization removes the need for user research

Correct Answer: B. Personalization should be assessed for control, fairness and user understanding

Question 9

In the word error rate formula, what does N represent?

A. The number of reference words

B. The number of researchers

C. The number of themes

D. The number of interview sessions

Correct Answer: A. The number of reference words

Question 10

Which case most clearly requires escalation and additional human review?

A. AI suggests alternative button labels for brainstorming

B. A generated output conflicts with evidence and may disadvantage a group

C. A researcher uses timestamps to find a quotation

D. A prototype includes three colour variations

Correct Answer: B. A generated output conflicts with evidence and may disadvantage a group