Generative AI and Emerging Technology Governance

Introduction

Generative artificial intelligence (generative AI) refers to computational systems that produce new text, images, audio, video, software code, or other outputs in response to instructions or data. Its rapid adoption has changed the scope of data governance. Traditional governance focused mainly on how organizations collect, store, classify, access, retain, and delete data. Generative AI adds questions about how data is used to train models, how prompts and retrieved documents are processed, how outputs are evaluated, and who is accountable when an apparently authoritative output is wrong or harmful.

For business analytics professionals, generative AI is neither simply a productivity tool nor merely an information-security issue. It is a socio-technical system involving data, models, interfaces, people, vendors, organizational processes, and legal obligations. Effective governance therefore requires controls across the entire lifecycle: acquisition, development, deployment, use, monitoring, incident response, and retirement.

This chapter examines foundation models, prompt and retrieval systems, hallucination, intellectual property, confidential-data leakage, content provenance, deepfakes, agentic AI, shadow AI, red teaming, and responsible-use policies. It emphasizes risk-based governance rather than either unconditional adoption or blanket prohibition.

Learning Objectives

By the end of this chapter, readers should be able to:

1. Explain how foundation models, prompts, embeddings, retrieval-augmented generation, and agents interact.
2. Distinguish model error, hallucination, data leakage, copyright risk, and provenance failure.
3. Design governance controls for generative-AI systems across their lifecycle.
4. Evaluate the organizational implications of deepfakes, agentic systems, and shadow AI.
5. Develop a proportionate responsible-use policy and assurance program.
6. Apply red-teaming and monitoring methods to a realistic business case.

1. The Generative-AI Governance Problem

A conventional analytics application generally follows a recognizable path: data is collected, transformed, analyzed, and presented. A generative-AI application may introduce several additional transformations:

• a user supplies a prompt or uploads material;
• the system retrieves internal or external information;
• a model generates a probabilistic response;
• tools may be invoked to search, calculate, send messages, or modify records;
• the output may be copied into a business decision or customer communication.

Each transformation creates a governance boundary. A prompt can contain personal or confidential information. A retrieval system can expose documents to an unauthorized user. A model can produce a confident but unsupported statement. An agent can take an action that was never explicitly approved. A generated image or voice recording can be mistaken for authentic evidence.

The central governance question is therefore not “Is the model accurate?” It is: Under what conditions may this system produce or act on information, for whom, with what evidence, and subject to whose review?

A useful risk formulation is qualitative:

Risk = likelihood of harm × severity of harm × exposure

This is not a universal statutory formula, but a practical prioritization model. Likelihood concerns how often a failure may occur; severity concerns the consequences; exposure concerns the number of people, systems, decisions, or jurisdictions affected. Organizations should document their definitions rather than treating the expression as a precise probability calculation.

2. Foundation Models and Their Governance Implications

A foundation model is a broadly trained model that can be adapted to many downstream tasks. Large language models are one important class, but foundation models also support image, audio, video, multimodal, and code generation. They are typically trained on very large datasets and then made available through an application interface, an enterprise platform, or a model hosted internally.

The term “foundation” highlights reuse. A single base model may support customer service, document summarization, coding assistance, recruitment, and market analysis. Reuse creates efficiency, but it also creates concentration risk: one weakness, undocumented training source, or configuration error may affect many applications.

Governance should distinguish at least four layers:

Model cards, system cards, technical documentation, evaluation results, and supplier contracts are important evidence. However, documentation supplied by a vendor does not replace an organization’s own assessment. Performance varies by language, domain, prompt style, data quality, and user behavior.

A model’s fluency should not be confused with knowledge, truth, intent, or understanding. Generative models produce outputs according to learned patterns and system instructions. They may provide useful synthesis while lacking a reliable mechanism for determining whether every statement is factually justified.

3. Prompts, Embeddings, and Retrieval-Augmented Generation

A prompt is the instruction and contextual material supplied to a model. Prompts may include user questions, system instructions, examples, retrieved documents, conversation history, or data from an application. Prompt governance concerns both content and authority: who may submit prompts, what data may be included, and which instructions take precedence?

Prompt injection is an attempt to manipulate a system through crafted instructions. Direct injection occurs when a user instructs the model to disregard its rules. Indirect injection occurs when untrusted content—such as a web page, email, or document—contains instructions that the model interprets as commands. Prompt injection is especially important when the model can call tools or access private information.

An embedding represents content as a numerical vector so that semantically similar items can be located. A retrieval-augmented generation (RAG) system commonly performs the following sequence:

1. A source document is collected and divided into passages.
2. Passages are converted into embeddings and stored in an index.
3. A user query is converted into an embedding.
4. The system retrieves relevant passages according to a similarity method and access rules.
5. Retrieved passages are inserted into the model’s context.
6. The model generates an answer, ideally grounded in those passages.

RAG can reduce unsupported answers by giving the model current, organization-specific evidence. It does not guarantee truth. Retrieval may select the wrong passage, omit a critical qualification, use an outdated version, or expose a document to an unauthorized user. Governance must therefore control document ingestion, metadata, versioning, access permissions, retention, citation, and retrieval evaluation.

A strong RAG design separates retrieval authorization from language generation. The model should not decide independently whether a user is entitled to a document. Authorization should be enforced by the underlying information system before content enters the model context. Sensitive documents should be excluded or appropriately filtered, and logs should record which sources supported an answer.

4. Hallucination, Reliability, and Human Oversight

A hallucination is a generated output that is unsupported, fabricated, or materially misleading. Examples include invented citations, nonexistent legal provisions, incorrect financial figures, and false claims that a task was completed. Hallucination is not limited to obviously absurd responses; a polished paragraph containing one false statement can be operationally dangerous.

Several conditions increase risk:

• ambiguous or incomplete prompts;
• questions outside the model’s knowledge or retrieval coverage;
• conflicting or low-quality source documents;
• demands for a single answer when uncertainty is substantial;
• long context windows containing distracting information;
• automated workflows that reward completion rather than abstention.

Mitigations include retrieval with citations, structured output, domain-specific evaluation sets, confidence or uncertainty indicators where meaningful, refusal and escalation rules, calculation through deterministic tools, and human review. A model should not be asked to perform a task that a conventional database, rules engine, or validated statistical program can perform more reliably.

Human oversight must be substantive. A person who merely approves every output without time, expertise, or authority to challenge the system provides nominal rather than effective oversight. Organizations should specify review triggers, reviewer competence, documentation requirements, and the authority to stop or reverse an automated process.

Evaluation should use task-relevant measures. For a summarization system, reviewers may assess factual faithfulness, omission, readability, and privacy. For a customer-support system, they may assess resolution, escalation correctness, harmful advice, and consistency with policy. Aggregate accuracy can hide severe failures in minority cases, rare events, or high-impact decisions.

5. Intellectual Property and Confidential Data

Generative AI creates intellectual-property questions at several stages:

1. Training: whether copyrighted or licensed material was used in model development and under what legal basis.
2. Input: whether a user has permission to submit third-party text, images, code, or recordings.
3. Output: whether generated material resembles protected works or incorporates protected content.
4. Use: whether the organization’s intended publication, sale, or integration creates infringement or contractual risk.
5. Ownership: whether applicable law recognizes the organization or user as holding rights in a substantially machine-generated output.

Legal rules differ by jurisdiction and continue to develop. Governance should therefore avoid categorical claims such as “AI output is always owned by the user” or “all training use is lawful.” Organizations should maintain an approved-use register, review vendor terms, document material sources, and require human contribution and review where ownership or originality matters.

Confidential-data leakage may occur when a user places trade secrets, personal data, source code, privileged communications, or regulated information into a public or inadequately controlled service. Leakage may result from provider retention, human error, excessive logging, insecure plugins, model fine-tuning, retrieval misconfiguration, or output disclosure to another user.

Controls include data classification, loss-prevention rules, enterprise tenancy, encryption, contractual restrictions on provider training, retention limits, access control, redaction, tokenization, private deployment where justified, and monitoring for sensitive prompts and outputs. Privacy impact assessments should address purpose limitation, data minimization, transparency, access rights, retention, cross-border transfers, and automated decision-making obligations where applicable.

6. Content Provenance and Deepfakes

Content provenance concerns evidence about where content came from, how it was created, and what transformations it underwent. Provenance is different from a simple label stating “AI-generated.” A useful provenance record may include the creator or system, time, source materials, editing history, model or tool used, and whether a person reviewed the result.

Provenance systems can use cryptographic signatures, tamper-evident metadata, trusted capture devices, or content-credential standards. Such mechanisms improve traceability but do not prove that the underlying content is true. Authentic provenance can show that a camera captured an image; it cannot establish that the depicted event was interpreted correctly. Conversely, removal of metadata does not prove that content is fake.

Deepfakes are synthetic or manipulated media that imitate a person’s appearance, voice, or actions. Risks include fraud, reputational harm, harassment, political manipulation, extortion, impersonation of executives, and false evidence. Organizations should establish procedures for verifying urgent payment instructions, executive requests, identity claims, and sensitive recordings through independent channels.

Detection tools should be treated as one signal rather than a final authority. Detection performance changes as generation methods evolve, and false positives can harm legitimate speakers. A defensible response combines provenance, source verification, behavioral context, independent confirmation, and incident escalation.

7. Agentic AI and Tool-Using Systems

An agentic AI system uses a model to plan or sequence actions, maintain state, call tools, observe results, and continue toward a goal. Unlike a text-only assistant, an agent may query a database, create a ticket, change a record, send an email, or initiate a transaction.

Agentic systems introduce authority and control risks:

• unclear boundaries around the agent’s objective;
• excessive permissions;
• unintended loops or repeated actions;
• prompt injection through tool outputs;
• inadequate separation between planning and execution;
• irreversible or high-impact actions;
• weak audit trails;
• difficulty assigning responsibility among the user, developer, and vendor.

A mature design applies least privilege, tool allowlists, transaction limits, sandboxing, approval gates, rate limits, timeouts, state validation, and complete action logs. High-impact actions should require explicit confirmation or dual control. Read-only access should be preferred during early deployment. The system should also distinguish between proposing an action and executing it.

Agent governance can be expressed as an authority ladder:

The appropriate level depends on harm potential, reversibility, transaction value, regulatory context, and quality of monitoring.

8. Shadow AI and Organizational Accountability

Shadow AI is the use of AI tools outside approved organizational processes. It may arise because employees seek faster work, approved tools are unavailable, or policies are unclear. Treating users as the primary problem is ineffective. Shadow use often signals a gap between operational needs and governance design.

An organization should create a safe path to adoption: publish approved tools, define prohibited data, provide secure alternatives, train employees, and establish a rapid review process for new use cases. Discovery methods may include procurement records, network monitoring, browser telemetry, surveys, and interviews, subject to privacy and labor requirements.

A register of AI use cases should record the business owner, purpose, data categories, model and supplier, affected individuals, risk classification, human review, evaluation results, retention, incident contacts, and retirement conditions. Inventory is a prerequisite for accountability; an organization cannot govern systems it does not know exist.

9. Red Teaming and Assurance

Red teaming is a structured attempt to find weaknesses by simulating adversarial, abusive, or unexpected use. It should be performed before deployment and periodically afterward because models, prompts, retrieval sources, users, and threats change.

A generative-AI red-team plan should include:

• prompt injection and jailbreak attempts;
• extraction of system instructions;
• unauthorized retrieval and cross-user data access;
• sensitive-data exfiltration;
• fabricated citations and unsupported claims;
• harmful, discriminatory, or abusive outputs;
• unsafe code generation;
• deepfake and impersonation scenarios;
• agent tool misuse and excessive actions;
• denial-of-service, cost escalation, and resource exhaustion.

Testing should use both expert-designed scenarios and realistic user journeys. Results should be severity-rated, assigned to owners, remediated, retested, and reported to an accountable governance body. Red teaming is not a single certification event. It is one component of assurance alongside vulnerability management, access reviews, privacy assessments, model evaluation, internal audit, and incident exercises.

10. Designing a Responsible-Use Policy

A responsible-use policy should be specific enough to guide behavior and flexible enough to accommodate new tools. A useful policy contains the following elements:

Purpose and scope. Identify covered employees, contractors, systems, and uses.

Risk categories. Define prohibited, restricted, approved, and experimental uses. Prohibited uses may include unlawful discrimination, deceptive impersonation, unauthorized surveillance, or submission of highly confidential data to an unapproved service. Restricted uses may include employment, credit, health, legal, safety, or customer-impacting decisions.

Data rules. State what information may be entered, what must be redacted, retention expectations, and approved environments.

Human accountability. Identify the decision owner, reviewer qualifications, escalation process, and requirement to disclose material AI assistance where appropriate.

Quality and provenance. Require verification of important claims, citations or source records, version control, and labeling of synthetic media when disclosure is relevant.

Security controls. Address authentication, authorization, logging, plugin approval, secrets management, incident reporting, and supplier assurance.

Procurement and change management. Require risk assessment before acquiring or materially modifying an AI system.

Training and enforcement. Provide role-specific education, accessible guidance, reporting channels, and proportionate sanctions for violations.

Policies should be accompanied by procedures and technical controls. A statement that employees “must verify outputs” is weak if the organization supplies no verification method, time, source access, or reviewer authority.

11. Illustrative Case: Meridian Retail’s AI Service Assistant

Background. Meridian Retail, a fictional multinational retailer, deploys an AI assistant for service agents. The assistant summarizes customer conversations, retrieves warranty rules, drafts responses, and recommends refunds. It is connected to a document repository and a refund application.

Problem. After deployment, an agent receives a malicious customer message containing instructions to ignore system rules and reveal internal refund procedures. In another incident, the assistant retrieves an outdated warranty document. A third incident occurs when an employee pastes a customer’s medical information into an external public chatbot to improve a response.

Analysis. The incidents involve three distinct risks. The malicious message is indirect prompt injection. The outdated warranty answer is a retrieval governance and version-control failure. The public chatbot submission is a confidentiality and privacy breach. The common underlying weakness is that Meridian treated the assistant as a writing tool rather than as a system operating across data and action boundaries.

Approach. Meridian takes the following measures:

1. It separates customer content from system instructions and treats retrieved text as untrusted data.
2. It enforces repository permissions before retrieval and removes obsolete documents through version controls.
3. It changes the assistant’s refund capability from automatic execution to recommendation only; refunds above a threshold require supervisor approval.
4. It blocks sensitive-data categories in prompts and provides an approved enterprise environment.
5. It requires source citations and displays document dates beside recommendations.
6. It establishes red-team tests for injection, data leakage, unsupported refund advice, and excessive tool use.
7. It records the use case in an AI register and assigns accountability to the service-operations director.

Outcome. The assistant remains useful for summarization and retrieval, but its authority is narrowed. Service quality is evaluated using factual faithfulness, correct escalation, refund error rates, privacy incidents, and customer outcomes rather than response speed alone.

Learning points. First, controls must match the system’s actual capabilities. Second, retrieval quality depends on information governance outside the model. Third, human review is meaningful only when authority and time are provided. Finally, an incident can reveal a design problem rather than merely an individual employee mistake.

12. Implementation Roadmap and Contemporary Developments

A practical implementation roadmap has five stages:

1. Discover: inventory tools, use cases, data flows, vendors, and affected stakeholders.
2. Classify: assess impact, reversibility, data sensitivity, autonomy, and regulatory exposure.
3. Control: apply access restrictions, approved environments, retrieval permissions, review gates, logging, and contractual safeguards.
4. Assure: evaluate outputs, red-team threats, test representative cases, and monitor incidents and drift.
5. Improve or retire: remediate weaknesses, update documentation, retrain users, and deactivate systems that no longer meet requirements.

Contemporary governance is moving toward lifecycle accountability, documented risk management, transparency about synthetic content, stronger supplier obligations, and differentiated requirements for high-impact uses. International and national frameworks vary, but common themes include risk classification, human oversight, security, data governance, transparency, accountability, and technical documentation. Organizations operating across jurisdictions should map obligations by use case rather than assuming that one global policy is sufficient.

Important limitations remain. Technical detection of generated content is imperfect. Vendor transparency may be limited by trade secrets. Bias evaluation can be difficult when protected attributes are unavailable or legally sensitive. Human review can become routine rubber-stamping. Smaller organizations may lack specialist staff. Finally, governance can reduce but never eliminate uncertainty. Responsible use is therefore an ongoing management discipline, not a one-time approval.

Chapter Summary

Generative-AI governance extends data governance into model development, prompting, retrieval, generation, tool use, and organizational behavior. Foundation models are reusable but introduce shared and concentration risks. Prompt and RAG systems require strong authorization, source management, version control, and protection against injection. Hallucinations require task-specific evaluation, grounded evidence, deterministic tools where appropriate, and meaningful human oversight.

Intellectual-property and confidentiality risks arise through training, input, output, and use. Provenance improves traceability but does not establish truth, while deepfakes require independent verification rather than reliance on a single detector. Agentic systems require least privilege, approval gates, action limits, and auditability. Shadow AI should be addressed through usable approved alternatives and an inventory of actual practices. Red teaming, monitoring, incident response, and policy enforcement provide continuing assurance.

The central managerial principle is proportionality: the greater the potential impact, autonomy, sensitivity, and irreversibility of a system, the stronger its governance must be.

Key Terms

Foundation model: A broadly trained model adaptable to many downstream tasks.

Generative AI: AI that produces new content such as text, images, audio, video, or code.

Prompt: Instructions and contextual material supplied to a model.

Prompt injection: An attempt to manipulate a model through crafted instructions.

Embedding: A numerical representation of content used to support similarity-based retrieval.

Retrieval-augmented generation: A design that retrieves source material and supplies it to a generative model.

Hallucination: An unsupported, fabricated, or materially misleading generated output.

Model card: Documentation describing a model’s intended uses, limitations, and evaluation.

Data leakage: Unauthorized disclosure or exposure of information through an AI system.

Intellectual property: Legally protected creations, works, inventions, brands, or confidential know-how.

Content provenance: Evidence about content origin, creation, transformation, and review.

Deepfake: Synthetic or manipulated media that imitates a person or event.

Agentic AI: AI that plans or sequences actions and may use tools to pursue a goal.

Least privilege: Granting only the permissions necessary for an authorized task.

Shadow AI: Unapproved organizational use of AI tools.

Red teaming: Structured adversarial testing intended to discover system weaknesses.

Human oversight: Meaningful human ability and authority to review, challenge, modify, or stop an AI-supported process.

Responsible-use policy: Organizational rules governing acceptable, restricted, prohibited, and accountable AI use.

Review Questions

5 Short Answer Questions

1. What is a foundation model, and why does reuse create governance risk?
2. How does retrieval-augmented generation differ from generation using only a model’s internal parameters?
3. Define hallucination and identify two conditions that increase its likelihood.
4. Why is provenance not equivalent to proof that content is true?
5. What is shadow AI, and why should organizations provide approved alternatives?

5 Long Answer Questions

1. Explain the governance requirements for a business RAG system, including authorization, document quality, versioning, citations, and monitoring.
2. Discuss intellectual-property and confidential-data risks at the training, input, output, and use stages.
3. Compare text-only assistants with agentic AI systems in terms of authority, reversibility, auditability, and human approval.
4. Design a responsible-use policy for a graduate business school that permits low-risk productivity uses while restricting high-impact decisions.
5. Explain why red teaming should be integrated with lifecycle assurance rather than treated as a one-time predeployment test.

3 Analytical / Application Questions

1. A bank’s customer-service agent retrieves policy documents and can issue fee refunds. Identify at least five risks and propose controls for each, distinguishing model, data, security, and operational controls.
2. An executive receives a voice message apparently authorizing an urgent payment. Develop an evidence-based verification procedure that addresses deepfake risk without assuming that detection software is conclusive.
3. An organization discovers that employees regularly paste confidential material into public AI tools. Analyze the organizational causes and propose a 90-day remediation plan covering technology, policy, training, monitoring, and accountability.

Multiple Choice Questions with Answers

Question 1

Which statement best describes a foundation model?

A. A model designed for one narrowly defined business report

B. A broadly trained model adaptable to many downstream tasks

C. A database containing only verified organizational facts

D. A rules engine that cannot generate new content

Correct Answer: B. A broadly trained model adaptable to many downstream tasks

Question 2

What is the principal purpose of retrieval-augmented generation?

A. To eliminate the need for access controls

B. To replace all organizational databases

C. To provide the model with relevant source material at response time

D. To guarantee that every generated statement is legally correct

Correct Answer: C. To provide the model with relevant source material at response time

Question 3

Which is an example of indirect prompt injection?

A. A user asks the model to summarize a report

B. A developer changes a model’s temperature setting

C. A document supplied for summarization contains instructions telling the model to reveal system secrets

D. A reviewer corrects an inaccurate answer

Correct Answer: C. A document supplied for summarization contains instructions telling the model to reveal system secrets

Question 4

Which control most directly reduces confidential-data leakage through an unapproved public AI service?

A. Increasing the model’s creativity setting

B. Allowing unrestricted browser access

C. Using data classification, blocking rules, and an approved enterprise environment

D. Removing all human review

Correct Answer: C. Using data classification, blocking rules, and an approved enterprise environment

Question 5

Why does provenance not by itself prove that content is truthful?

A. Provenance can show origin and transformations but not whether the depicted claim is accurate

B. Provenance is only applicable to printed documents

C. Provenance always removes the creator’s identity

D. Provenance guarantees that content was generated by a human

Correct Answer: A. Provenance can show origin and transformations but not whether the depicted claim is accurate

Question 6

Which design is most appropriate for an agent that can issue customer refunds?

A. Unlimited permissions with no action logs

B. Least privilege, transaction limits, logging, and approval for high-value refunds

C. Permission to modify all financial records

D. No testing because the agent uses a commercial model

Correct Answer: B. Least privilege, transaction limits, logging, and approval for high-value refunds

Question 7

What is the defining characteristic of shadow AI?

A. Use of AI only in a research laboratory

B. Use of AI outside approved organizational governance processes

C. Use of an AI system with human review

D. Use of a model that produces synthetic images

Correct Answer: B. Use of AI outside approved organizational governance processes

Question 8

Which activity is most characteristic of red teaming?

A. Designing a marketing slogan for an AI product

B. Attempting adversarial prompts and unauthorized-access scenarios to expose weaknesses

C. Increasing employee adoption without testing

D. Replacing all system documentation with vendor advertising

Correct Answer: B. Attempting adversarial prompts and unauthorized-access scenarios to expose weaknesses

Question 9

Which statement best describes meaningful human oversight?

A. A person automatically approves every output

B. A person has the competence, time, and authority to review, challenge, or stop the process

C. A person is informed only after an irreversible action

D. A person accepts the model’s confidence score as final proof

Correct Answer: B. A person has the competence, time, and authority to review, challenge, or stop the process

Question 10

Which factor generally warrants stronger governance controls?

A. Low-impact, reversible text drafting with no sensitive data

B. A system that makes high-impact, irreversible decisions using sensitive personal data

C. A system used only for formatting public information

D. A system disconnected from users and business processes

Correct Answer: B. A system that makes high-impact, irreversible decisions using sensitive personal data