11. Artificial Intelligence and Machine Learning in Research

Introduction and Context

Artificial intelligence (AI) and machine learning (ML) have become important instruments of contemporary research. They support literature discovery, translation, coding, image interpretation, simulation, prediction, document classification, qualitative analysis, and the generation of text, code, images, and synthetic data. Their value, however, does not arise merely from automation. In research, the central question is whether the use of an AI system produces knowledge that is valid, reproducible, transparent, ethically defensible, and appropriately attributed.

AI-assisted research creates a distinctive governance problem. A conventional research tool generally performs a relatively stable and inspectable operation. An AI system may instead produce probabilistic outputs influenced by training data, model architecture, prompts, system instructions, retrieval sources, and user interaction. The same prompt may produce different results, and a fluent answer may be factually wrong. Consequently, researchers must treat AI outputs as claims or proposals requiring evaluation, not as authoritative evidence.

The issue is particularly significant for doctoral researchers. At doctoral level, responsible AI use involves more than following institutional rules. It requires critical examination of epistemic authority, bias, accountability, intellectual property, privacy, reproducibility, and the relationship between human judgment and automated inference. This chapter develops a governance-oriented approach to AI and ML in research, with particular attention to generative AI, algorithmic bias, explainability, validation, hallucination, disclosure, and human oversight.

Learning Objectives

By the end of this chapter, the reader should be able to:

1. Distinguish AI, ML, deep learning, generative AI, and automated research analysis.

2. Evaluate appropriate and inappropriate uses of AI across the research lifecycle.

3. Design validation procedures for AI-assisted discovery and analysis.

4. Identify algorithmic bias, hallucination, opacity, privacy, and intellectual-property risks.

5. Apply principles of explainability, documentation, disclosure, and human oversight.

6. Develop a responsible-use protocol for an AI-enabled research project.

7. Critically assess emerging regulatory and professional expectations.

From Computational Assistance to Epistemic Dependence

AI refers broadly to computational systems designed to perform tasks associated with human intelligence, such as classification, prediction, language processing, planning, or perception. ML is a subset of AI in which systems learn patterns from data rather than being programmed solely through explicit rules. Deep learning uses multilayer neural networks to learn complex representations. Generative AI produces new content, including text, code, audio, images, or synthetic records, in response to learned patterns and user instructions.

These categories should not be treated as interchangeable. A supervised ML model trained to classify medical images presents different risks from a large language model used to draft a literature review. A retrieval system that ranks articles is different from a generative model that summarizes them. Governance must therefore address the specific system, task, data, consequence, and degree of human dependence.

AI enters research in at least five ways. First, it can assist discovery by identifying topics, relationships, citations, and candidate hypotheses. Second, it can automate analysis through classification, clustering, regression, image recognition, speech processing, or natural-language coding. Third, generative systems can produce drafts, code, synthetic data, or research instruments. Fourth, AI can support research operations, including transcription, translation, screening, and project administration. Fifth, AI can become the object of research itself.

The epistemic danger is not simply error. It is the possibility that researchers accept a system's output without understanding how it was generated, what data shaped it, or where it fails. Automation can thus create epistemic dependence: the researcher's conclusions become dependent on a computational process that is insufficiently examined. Responsible research retains the capacity to challenge, reproduce, and explain important outputs.

Core Concepts and Terminology

A model is a computational representation that maps inputs to outputs. Training is the process through which model parameters are adjusted using data. Inference is the production of an output from a trained model. A feature is an input variable used by a model, while a label is a target category or value. A foundation model is trained on broad data and adapted to multiple tasks. A large language model predicts sequences of tokens and may generate coherent but unsupported text.

A prompt is an instruction or input supplied to a generative model. Prompt engineering concerns the design of instructions, context, constraints, and examples. Retrieval-augmented generation combines model generation with external document retrieval; it can improve grounding but does not guarantee that retrieved material is relevant or correctly interpreted. Hallucination is the production of an apparently credible but unsupported or false output. It may involve invented references, incorrect quotations, fabricated data, or unjustified causal claims.

Validation concerns whether a system performs adequately for its intended purpose. Verification asks whether the system was implemented correctly, whereas validation asks whether it is appropriate and reliable for the research task. Generalizability concerns performance beyond the data or conditions used in development. Reproducibility concerns whether the same analysis can be repeated with the same materials and procedures. Replicability is often used for obtaining consistent findings in a new study or implementation, although terminology varies across disciplines.

Explainability refers to the degree to which the operation or output of a model can be meaningfully understood. Interpretability often denotes a model's intrinsic comprehensibility, whereas post hoc explanation provides an account after a complex model has produced an output. Fairness concerns whether a system produces unjustified disparities across groups. Accountability identifies who is responsible for decisions and consequences. Data provenance records the origin, transformations, permissions, and lineage of data and outputs.

A Governance Framework for AI-Assisted Research

A useful governance framework follows the research lifecycle rather than treating AI as a single tool. Each use should be assessed through six questions: What is the research purpose? What is the system doing? What data does it use? What could go wrong? How will the output be validated? Who remains responsible?

Defining the purpose and risk

Researchers should first specify whether AI is being used for low-risk assistance, substantive analysis, or consequential decision-making. Correcting grammar in a private draft is different from classifying participants by mental-health risk. Risk increases when outputs affect human participants, public policy, clinical decisions, employment, educational opportunity, or the interpretation of sensitive data. A risk assessment should consider severity, likelihood, reversibility, affected populations, and the availability of human review.

Selecting and documenting the system

Tool selection should consider training-data provenance, privacy terms, retention practices, model version, access controls, known limitations, and institutional approval. A free public system may transmit prompts to an external provider, making it inappropriate for confidential interviews or identifiable participant data. Researchers should record the tool name, version or access date, relevant settings, prompts, retrieved sources, code, datasets, and human interventions. If the system changes during the project, the change should be documented.

Preparing and governing data

AI does not remove the obligations attached to research data. Data must be collected lawfully and ethically, with attention to informed consent, purpose limitation, minimization, confidentiality, and secure storage. De-identification reduces risk but does not guarantee anonymity, especially where datasets can be linked. Training data may contain copyrighted works, personal information, discriminatory patterns, or unverified material. Researchers must establish whether they have the right to use data for the intended purpose and whether participants were informed about automated processing.

Validation and triangulation

Validation should be designed before relying on an output. For predictive models, relevant procedures may include train-test separation, cross-validation, calibration assessment, error analysis, subgroup performance analysis, and external validation. For generative systems, researchers should verify factual claims against primary sources, inspect quotations and references, test prompts across cases, and maintain an audit sample of outputs. Agreement with a model is not evidence of validity; independent evidence is required.

Triangulation is especially important. AI-assisted coding of interview data can be compared with human coding, a second analyst, participant interpretation, or an established coding framework. Agreement statistics may be useful, but disagreement is also informative because it can reveal ambiguity, conceptual weakness, or culturally specific meaning that the model fails to recognize.

Human oversight and escalation

Human oversight should be substantive rather than ceremonial. The reviewer must have the expertise, authority, time, and information necessary to challenge the system. A researcher who merely approves every output is not exercising meaningful oversight. Projects should define escalation conditions, such as low confidence, unusual cases, disagreement with expert judgment, potential participant harm, or a decision affecting rights or opportunities.

Disclosure and attribution

Research reports should disclose material AI use. Disclosure should state the system used, its role, the stage of research affected, the nature of human review, and any known limitations. AI tools should not be listed as authors because they cannot assume responsibility, consent to publication, or respond to allegations of misconduct. Human authors remain accountable for accuracy, originality, citations, confidentiality, and compliance with research standards. Disclosure is not a substitute for validation; it enables readers and reviewers to evaluate the work.

Generative AI: Productivity and Epistemic Risk

Generative AI can help researchers brainstorm search terms, create preliminary outlines, translate text, generate code templates, produce interview-protocol alternatives, and summarize documents. It may be particularly valuable where researchers face large volumes of material or work across languages. Yet its fluency can conceal weak reasoning. A generated literature review may combine real and nonexistent sources, flatten disagreements, or present a minority interpretation as consensus. A generated statistical script may run successfully while implementing an inappropriate test.

The proper role of generative AI is therefore assistive and contestable. Researchers may use it to propose possibilities, but they should independently establish the evidential basis of each important claim. Primary sources should be located and read. Code should be inspected, tested, and adapted. Synthetic examples should never be confused with empirical observations. Where generated text is substantially incorporated into a publication, the researcher must follow the relevant publisher and institutional disclosure requirements.

Prompt records can be treated as methodological material when they materially affect results. A prompt should specify the task, source boundaries, output format, uncertainty requirements, and prohibited inferences. For example, asking a model to “summarize these articles” is less controlled than instructing it to extract research questions, methods, sample characteristics, limitations, and direct quotations only when verified against the supplied documents.

Algorithmic Bias and Fairness

Bias can enter through sampling, measurement, labeling, historical practices, missing data, model design, deployment context, and human interpretation. A model trained on unequal historical decisions may reproduce those decisions. A language model may associate occupational competence with particular demographic groups because of patterns in its training data. A classifier may perform differently across groups because one group is underrepresented or because the target variable is measured differently.

Fairness is not a single technical property. Possible criteria include equal error rates, equal opportunity, demographic parity, calibration, and individual or group consistency. These criteria can conflict, particularly when groups have different base rates. Consequently, researchers must justify the fairness concept relevant to the research purpose rather than selecting a metric merely because it produces a favorable result.

Bias auditing should combine quantitative and qualitative methods. Quantitative analysis examines performance by subgroup, missingness, false positives, false negatives, and calibration. Qualitative inquiry asks whose categories are represented, whose experiences are absent, and whether the model's errors have unequal consequences. A system can have acceptable average accuracy while causing serious harm to a small or marginalized population.

Explainability, Interpretability, and Causal Restraint

An explanation is useful only if it answers a relevant question. Feature importance may indicate which variables influenced a prediction, but it does not necessarily show causation. A post hoc explanation can be unstable or misleading if it approximates rather than reveals the model's actual reasoning. Researchers should avoid claims such as “the model discovered the cause” when it identified only a predictive association.

Explainability requirements should be proportionate to consequence. A transparent baseline model may be preferable to a more accurate opaque model when decisions are high stakes and explanations are essential. In other contexts, a complex model may be acceptable if its performance is rigorously validated and its limitations are clearly communicated. Explanations should be tested with domain experts and, where relevant, with affected communities.

Intellectual Property, Authorship, and Research Integrity

AI-assisted research raises unresolved and context-dependent questions concerning copyright, database rights, licensing, trade secrets, and ownership of generated material. Researchers should not assume that publicly accessible content is freely reusable. They should review the terms governing datasets and software, retain evidence of permissions, and avoid uploading confidential manuscripts, proprietary code, or restricted data to systems lacking suitable protections.

Research integrity also concerns plagiarism and misrepresentation. Generated prose may reproduce distinctive language from training material or obscure the origin of an idea. Automatic paraphrasing does not eliminate the obligation to cite sources. Researchers must verify references rather than citing them because a model supplied them. Supervisors and institutions should distinguish legitimate assistance from outsourcing intellectual work that the researcher is expected to perform and understand.

Applied Case: AI-Assisted Qualitative Analysis of Public-Health Interviews

Background

The following is an illustrative case. A doctoral team studies how recently relocated families experience access to public-health services. The team has 180 transcribed interviews in three languages. To manage the volume, it proposes using a commercial language model to identify themes and generate summaries.

Situation and problem

The interviews contain indirect identifiers, descriptions of illness, immigration experiences, and references to small communities. The model provider's terms allow retention of submitted prompts for service improvement. Early testing shows that the model produces fluent summaries but sometimes interprets hesitation as agreement and translates culturally specific expressions literally. The team also discovers that the model assigns fewer “barrier” codes to interviews in one language.

Analysis

The project presents privacy, fairness, validity, and governance risks. Consent forms did not mention external AI processing. Removing names is insufficient because combinations of location, occupation, and family history may re-identify participants. The language disparity indicates that average agreement with human coding would conceal subgroup-specific error. Moreover, a summary is an interpretation, not a neutral compression of the interview.

Approach and solution

The team pauses external processing and consults its ethics committee and data-protection officer. It establishes a secure, locally controlled environment, uses a minimum necessary dataset, and conducts a renewed consent assessment. Researchers develop a codebook from close reading and pilot coding before asking the model to suggest candidate passages. Two trained human coders independently review all model suggestions, with complete human coding of a stratified sample from each language. Disagreements are recorded rather than silently resolved.

The team compares model performance by language, topic, and participant characteristics where ethically and statistically appropriate. It excludes the model from making final claims and uses participant and community advisers to assess whether summaries preserve meaning. The publication discloses the model's limited role, validation procedures, language-specific limitations, and the fact that no raw transcripts were sent to a public service.

Outcome

The model reduces the time required for initial document organization, but it performs poorly on culturally embedded expressions and indirect accounts of discrimination. Human researchers revise the codebook and retain several themes that the model failed to identify. The final analysis is slower than full automation would have been, but it is more defensible, transparent, and responsive to participants.

Learning points

The case demonstrates that efficiency is not the only outcome that matters. Data governance must precede tool use; subgroup validation is essential; qualitative meaning cannot be inferred from surface linguistic patterns alone; and human oversight must include genuine interpretive authority. The case also illustrates that a responsible decision may be to restrict AI to a narrow supporting function rather than reject it entirely or adopt it without conditions.

Common Failure Modes and Practical Controls

Several mistakes recur in AI-assisted research. Researchers may treat model output as a source rather than a hypothesis, report accuracy without examining subgroup error, omit tool versions and prompts, upload confidential data to public systems, or assume that a human glance constitutes validation. Another failure is automation bias: users accept machine recommendations because they appear objective or technically sophisticated. There is also deskilling, in which researchers lose the substantive competence needed to recognize a plausible but invalid result.

Practical controls include a written AI-use plan, data classification before upload, approved-tool lists, access restrictions, reproducible scripts, prompt and output logs, independent source verification, predefined stopping and escalation rules, subgroup error analysis, and periodic review by subject experts. Researchers should preserve the original data and human decisions so that AI-assisted findings can be audited. Where an AI system materially shapes an analysis, sensitivity analysis should examine whether conclusions change when the system is removed or replaced.

Professional and Regulatory Governance

Institutions should establish policies that differentiate permissible assistance, restricted uses, and prohibited uses. Policies should address research data, publication disclosure, examination and assessment, procurement, authorship, security, and incident reporting. Ethics review should ask not only whether participants face direct risk, but also whether automated processing changes the meaning of consent, confidentiality, or researcher responsibility.

Regulatory expectations increasingly emphasize risk management, transparency, human oversight, data governance, security, and accountability. Requirements differ across jurisdictions and sectors, so researchers must consult applicable law, institutional policy, funder conditions, and publisher guidance. Compliance, however, is a minimum standard. A legally permissible use may still be methodologically weak or ethically questionable.

Emerging Directions

Research is moving toward multimodal models, autonomous research agents, synthetic data, federated learning, foundation-model adaptation, and AI systems that search, code, analyze, and draft with limited intervention. These developments may increase productivity but also expand the chain of dependence and make responsibility harder to assign. Agentic systems can select tools, call external services, and generate intermediate outputs that researchers do not fully inspect.

Future research governance will therefore require stronger provenance systems, machine-readable audit trails, model and dataset documentation, evaluation benchmarks that include minority contexts, and methods for testing robustness under distribution shift. Researchers will also need new forms of methodological literacy: the ability to interrogate model behavior, assess uncertainty, understand data lineage, and communicate limitations to non-specialists. The enduring principle is that sophisticated automation does not reduce the need for judgment; it increases the need for informed judgment.

Best-Practice Protocol

Before using AI, define the research purpose, expected benefit, affected persons, and foreseeable harms. Confirm that the data may be processed by the selected system and that the use is consistent with consent and institutional approval. Choose the least powerful tool that adequately serves the task. Establish validation criteria in advance, including source verification, subgroup analysis, expert review, and reproducibility requirements. Record the model, version, prompts, inputs, outputs, settings, and human interventions. Keep humans responsible for interpretation and consequential decisions. Disclose material use in reports and preserve sufficient documentation for audit. Finally, review whether the AI contribution improved the research without displacing the intellectual and ethical responsibilities of the researcher.

Chapter Summary

AI and ML can support discovery, analysis, generation, and research operations, but their outputs are probabilistic and require independent evaluation. Responsible use depends on purpose-sensitive risk assessment, lawful and secure data governance, validation, subgroup fairness analysis, explainability appropriate to consequence, and meaningful human oversight. Generative AI is useful for proposing and organizing material but can hallucinate sources, reproduce bias, and generate plausible code or interpretations that are incorrect. Researchers must document tools and prompts, verify claims against reliable evidence, respect intellectual-property and confidentiality obligations, and disclose material AI use. The central governance principle is that AI may assist research, but responsibility for the validity, integrity, and consequences of research remains human.

Key Terms

Artificial intelligence: Computational systems designed to perform tasks associated with human intelligence.

Machine learning: Methods that learn patterns from data to produce predictions, classifications, or decisions.

Generative AI: AI that produces new text, code, images, audio, or other content.

Foundation model: A broadly trained model adaptable to many downstream tasks.

Training: Adjusting model parameters using data.

Inference: Producing an output from a trained model.

Hallucination: A fluent but unsupported, inaccurate, or fabricated AI output.

Validation: Determining whether a system is suitable and reliable for its intended purpose.

Verification: Checking whether a system has been implemented correctly.

Explainability: The extent to which a model's operation or output can be meaningfully understood.

Interpretability: The degree to which a model is intrinsically understandable.

Algorithmic bias: Systematic, unjustified differences in outputs or errors across people or groups.

Fairness: The ethical and technical treatment of relevant groups without unjustified disadvantage.

Data provenance: A record of the origin, transformation, permissions, and lineage of data.

Human oversight: Competent and empowered human review of AI processes and outputs.

Automation bias: Unwarranted reliance on an automated recommendation.

Prompt: An instruction or input supplied to a generative model.

Retrieval-augmented generation: Generation supported by retrieving external documents or data.

Reproducibility: The ability to repeat an analysis using the same materials and procedures.

Epistemic dependence: Reliance on a process whose production and limitations the researcher cannot adequately evaluate.

Review Questions

Short Answer Questions

1. Distinguish verification from validation in AI-assisted research.

2. What is algorithmic bias, and how can it enter a research model?

3. Why should AI-generated references be independently checked?

4. Define meaningful human oversight.

5. What information should normally be included in an AI-use disclosure?

Long Answer Questions

1. Explain how AI can support different stages of the research lifecycle and evaluate the risks associated with each stage.

2. Discuss the relationship among explainability, interpretability, predictive accuracy, and research accountability.

3. Develop a validation framework for using a generative AI system to assist qualitative data analysis.

4. Analyze the intellectual-property, privacy, authorship, and research-integrity issues created by generative AI.

5. Explain why fairness cannot be reduced to a single numerical metric in algorithmic research.

Analytical / Application Questions

1. A researcher uploads identifiable interview transcripts to a public language model and uses its summaries as findings. Identify the major risks and propose a revised workflow.

2. An image-classification model has high overall accuracy but substantially higher false-negative rates for one demographic group. How should the research team analyze and govern its use?

3. A generative model produces a polished literature review containing several nonexistent citations. Design a procedure for detecting, correcting, documenting, and disclosing the problem.

22. Multiple Choice Questions with Answers

Question 1

Which action best demonstrates meaningful human oversight of an AI-assisted research analysis?

A. Accepting every output after a quick visual inspection

B. Reviewing outputs against predefined criteria and having authority to reject or escalate them

C. Using the model with a more detailed prompt

D. Reporting only the model's average accuracy

Correct Answer: B. Reviewing outputs against predefined criteria and having authority to reject or escalate them

Question 2

A language model invents a journal article that appears plausible. This is best classified as:

A. Data provenance

B. Calibration

C. Hallucination

D. External validation

Correct Answer: C. Hallucination

Question 3

Which practice most directly improves the reproducibility of a material generative-AI contribution?

A. Deleting intermediate outputs after publication

B. Recording the system, version, prompts, settings, and human interventions

C. Using the most powerful available model

D. Replacing citations with generated summaries

Correct Answer: B. Recording the system, version, prompts, settings, and human interventions

Question 4

Why can high overall model accuracy conceal an ethical problem?

A. Accuracy always measures causation

B. Average performance can conceal unequal errors across groups

C. High accuracy prevents external validation

D. Accuracy is unrelated to model outputs

Correct Answer: B. Average performance can conceal unequal errors across groups

Question 5

Which statement about explainability is most accurate?

A. Feature importance automatically establishes causation

B. Every complex model is inherently interpretable

C. An explanation should answer a relevant question and should not be treated as proof of causation

D. Explainability eliminates the need for validation

Correct Answer: C. An explanation should answer a relevant question and should not be treated as proof of causation

Question 6

Before uploading confidential participant data to an AI service, the researcher should first:

A. Remove only participants' first names

B. Confirm data permissions, consent, security, retention, and institutional requirements

C. Ask the model to anonymize the data after upload

D. Assume public availability makes the data unrestricted

Correct Answer: B. Confirm data permissions, consent, security, retention, and institutional requirements

Question 7

In the illustrative qualitative case, why was language-specific testing necessary?

A. Different languages require different statistical significance levels

B. The model showed unequal coding performance across languages

C. Human coding is never appropriate for multilingual research

D. Translation automatically removes cultural meaning

Correct Answer: B. The model showed unequal coding performance across languages

Question 8

Which use of generative AI creates the greatest epistemic risk?

A. Suggesting alternative search terms for later evaluation

B. Correcting spelling in a private draft

C. Making unreviewed causal claims from participant data

D. Producing a preliminary outline

Correct Answer: C. Making unreviewed causal claims from participant data

Question 9

Why should AI systems not normally be listed as authors?

A. They cannot assume responsibility, consent to publication, or answer misconduct allegations

B. They cannot generate text

C. They are always less accurate than humans

D. They are prohibited from being used in research

Correct Answer: A. They cannot assume responsibility, consent to publication, or answer misconduct allegations

Question 10

Which governance response is most appropriate when an AI model performs poorly for a culturally specific subgroup?

A. Report only the aggregate result

B. Remove the subgroup from the dataset

C. Investigate the error, consult relevant expertise, revise or restrict use, and disclose the limitation

D. Increase automation so the model can decide consistently

Correct Answer: C. Investigate the error, consult relevant expertise, revise or restrict use, and disclose the limitation