14. Governance of Generative AI and Foundation Models

Introduction and Context

Generative artificial intelligence has altered the governance problem from controlling a single-purpose prediction system to governing adaptable, general-purpose computational capabilities. Large language models (LLMs), image and video generators, multimodal systems, speech models, coding assistants, and AI agents can produce content, interpret instructions, call tools, retrieve information, and take actions. Their broad usefulness is also the source of their governance difficulty: the same model may support medical research, draft a contract, generate harmful instructions, impersonate a person, or make an apparently authoritative but false claim.

A foundation model is a model trained on broad and substantial data, usually through self-supervised learning, and subsequently adapted for many tasks. Governance must therefore address the entire lifecycle rather than only the final application. Relevant stages include data acquisition, pre-training, fine-tuning, evaluation, deployment, monitoring, incident response, retirement, and the management of downstream providers. Responsibility is distributed among model developers, deployers, users, vendors, auditors, regulators, and affected communities.

The central governance question is not whether a model is “safe” in the abstract. It is whether a particular system is acceptably reliable, lawful, secure, explainable, and controllable for a defined context of use. This requires evidence, documented assumptions, proportionate controls, meaningful human oversight, and mechanisms for correction. Governance is consequently both a technical discipline and an organizational practice.

Learning Objectives

After studying this chapter, students should be able to:

1. Explain the governance challenges created by foundation models, LLMs, multimodal systems, and agents.

2. Distinguish hallucination, harmful content, synthetic-media risk, alignment failure, and ordinary software error.

3. Apply lifecycle, risk-based, and defense-in-depth governance frameworks.

4. Design controls for data rights, evaluation, guardrails, provenance, human oversight, and third-party risk.

5. Interpret the organizational and regulatory responsibilities associated with generative AI.

6. Develop a governance strategy for a realistic enterprise deployment.

Why Foundation Models Require a Distinct Governance Approach

Traditional machine-learning governance often assumes a relatively stable relationship between input, model, and output. A credit-scoring model, for example, is designed for a defined prediction task and can be evaluated against a specific outcome. A foundation model has a much wider and less predictable capability surface. Prompt wording, conversation history, retrieved documents, tools, system instructions, and model updates can all affect behavior.

Three characteristics are especially important. First, capability is general rather than narrowly task-specific. Second, behavior is emergent and difficult to infer fully from training objectives. Third, deployment frequently involves adaptation: organizations add retrieval systems, fine-tuning, plugins, external tools, or autonomous workflows. A model that appears acceptable in a laboratory may become risky when connected to customer records or financial systems.

Governance must therefore consider both model risk and system risk. Model risk concerns the behavior and limitations of the model itself. System risk includes interfaces, data pipelines, access controls, prompts, tools, users, organizational incentives, and operational procedures. Many serious incidents result not from a model alone but from an unsafe combination of model and surrounding system.

A useful governance principle is proportionality. A low-risk internal tool that summarizes publicly available documents should not be controlled in exactly the same way as an agent that recommends medical treatment, evaluates employees, or executes payments. Controls should reflect potential harm, scale, reversibility, affected rights, and the degree of autonomy.

Core Concepts and Terminology

A language model estimates patterns in sequences of tokens. It does not automatically possess a verified database of facts or an intention to communicate truth. Generative output is produced by probabilistic inference, which explains both its fluency and its occasional fabrication. Multimodal models extend this process to combinations of text, images, audio, video, and other data types.

A prompt is an instruction or input supplied to a model. System prompts establish higher-level behavior, while user prompts express a task. Prompt injection occurs when untrusted content attempts to override instructions or manipulate a model into disclosing information or taking an unauthorized action. In retrieval-augmented generation (RAG), a model receives documents retrieved from an external knowledge source. RAG can improve factual grounding, but it does not guarantee that retrieval is correct, current, or secure.

An AI agent is a system that uses a model to plan or select actions, often through tools such as search, databases, code execution, email, or enterprise applications. Agency increases risk because an erroneous output can become an external action. Human-in-the-loop systems require human approval at specified points; human-on-the-loop systems allow automated operation but provide monitoring and intervention; human-out-of-the-loop systems permit autonomous execution and demand particularly strong justification and controls.

Hallucination is a confident, plausible, but unsupported or false output. It may result from incomplete knowledge, ambiguous prompts, retrieval failure, reasoning error, or optimization for linguistic coherence rather than truth. Hallucination is not eliminated merely by asking a model to be accurate. It must be managed through grounding, verification, uncertainty handling, restricted use cases, and human review.

Alignment refers to the extent to which a system’s behavior reflects intended objectives, constraints, values, and legitimate instructions. Alignment is broader than politeness or refusal behavior. A model may comply with a user while violating privacy, organizational policy, or safety requirements. Misalignment can arise from flawed objectives, incomplete specifications, reward-proxy optimization, or conflict between instructions.

Guardrails are technical, procedural, or organizational controls that constrain unsafe behavior. They may include input and output filters, policy classifiers, access permissions, tool restrictions, rate limits, content moderation, human approval, and incident escalation. Guardrails reduce risk but are not a substitute for sound use-case selection and accountability.

Provenance is information about the origin, history, transformation, and ownership of data or content. Content credentials, cryptographic signatures, metadata, watermarking, and audit logs can support provenance. No single provenance mechanism is universally reliable: metadata may be removed, watermarks may be attacked, and signed content can still be misleading.

A Lifecycle Governance Framework

A practical governance program begins before model selection. The organization should identify the intended purpose, affected stakeholders, prohibited uses, legal basis, data flows, and possible failure consequences. A use-case register can classify systems according to sensitivity, autonomy, scale, and reversibility. High-impact uses should require formal approval, documented risk assessment, and senior accountability.

Data and Training-Data Rights

Training and adaptation data raise questions of copyright, licensing, privacy, confidentiality, consent, provenance, and representativeness. An organization should know what data it is permitted to use, for what purpose, and under which contractual conditions. Public availability does not automatically establish unrestricted rights of reuse. Personal data may remain protected even when accessible online.

Data governance should include source documentation, licensing records, retention rules, exclusion and deletion processes, privacy-preserving techniques, and procedures for handling data-subject requests where applicable. Dataset quality assessment should examine duplication, toxic material, stereotypes, language imbalance, and historical bias. Synthetic data can reduce exposure to some sensitive information, but it may reproduce or amplify errors from the data used to create it.

Model Development and Documentation

Developers should document intended use, known limitations, evaluation conditions, training-data characteristics at an appropriate level of detail, model version, adaptation methods, safety testing, and change history. Model cards, system cards, data statements, and risk registers are useful documentation instruments. Documentation should communicate uncertainty rather than presenting benchmark scores as universal evidence of safety.

Third-party models require additional scrutiny. Procurement teams should evaluate the provider’s security practices, service commitments, data-use terms, incident notification, model-update policy, audit rights, subcontractors, geographic processing, intellectual-property commitments, and exit arrangements. An organization remains accountable for decisions made through a vendor’s model; outsourcing computation does not outsource responsibility.

Evaluation Before Release

Evaluation should combine quantitative benchmarks, adversarial testing, red teaming, expert review, and realistic scenario testing. General benchmarks may measure knowledge or reasoning but often fail to reveal risks in a specific deployment. A customer-service system should be tested on privacy leakage, discriminatory responses, escalation failures, and misleading advice, not merely on language quality.

Important evaluation dimensions include:

• Factuality and groundedness: whether claims are supported by reliable evidence.
• Robustness: whether small changes in wording or context cause unacceptable behavior.
• Safety: whether the system generates prohibited, dangerous, or abusive content.
• Fairness: whether performance and harms differ systematically across relevant groups.
• Privacy: whether the system reveals personal, confidential, or memorized information.
• Security: whether prompts, tools, or data pipelines can be manipulated.
• Reliability: whether performance remains acceptable over time and under operational load.
• Explainability and traceability: whether users and investigators can understand the basis and history of an output.

Evaluation should define thresholds and actions. A failing result may require changing the prompt, adding retrieval constraints, limiting tools, retraining, narrowing the use case, introducing human review, or refusing deployment. Testing without a decision rule becomes documentation rather than governance.

Deployment Controls and Guardrails

Effective deployment uses defense in depth. At the input layer, systems can authenticate users, classify requests, detect sensitive information, and block disallowed instructions. At the model layer, developers can apply safety tuning, constrained decoding, specialized classifiers, and refusal policies. At the grounding layer, retrieval can be restricted to approved sources, with citations and freshness checks. At the action layer, tools should follow least-privilege principles, schema validation, sandboxing, transaction limits, and approval gates.

Output controls should not rely only on keyword filters. Harmful content can be indirect, multilingual, coded, or embedded in apparently benign requests. Conversely, rigid filters may block legitimate educational or professional activity. Context-sensitive policy engines, risk scores, human review, and appeal channels are often more suitable than a single binary filter.

For agents, the distinction between generating a recommendation and executing an action is critical. An agent may draft an email automatically but require approval before sending it. It may query a database but not alter records. It may prepare a payment but require dual authorization. These controls create bounded autonomy and preserve reversibility.

Monitoring, Incident Response, and Retirement

Post-deployment monitoring should examine error rates, refusal patterns, harmful outputs, user complaints, drift, prompt attacks, privacy incidents, and unexpected tool use. Logs must be sufficiently detailed for investigation while respecting data-minimization and confidentiality requirements. Monitoring should include both automated detection and human sampling.

An incident-management process should define severity levels, ownership, containment actions, notification obligations, root-cause analysis, remediation, and lessons learned. Examples of serious incidents include disclosure of confidential data, discriminatory automated decisions, unsafe autonomous actions, coordinated deepfake campaigns, and repeated hallucinations in a high-impact workflow.

Models should be versioned and change-controlled. A provider’s silent model update can alter accuracy, tone, refusal behavior, or data handling. Revalidation is necessary after significant changes. Retirement should include disabling access, preserving appropriate records, migrating users, revoking credentials, and evaluating residual copies or downstream integrations.

Governing Hallucination and Reliability

Hallucination governance begins with recognizing that fluent language is not evidence. Systems should distinguish generated text from verified information and should communicate uncertainty in a meaningful way. A citation that merely points to a document is insufficient if the cited document does not support the claim.

Grounding strategies include retrieval from curated sources, database lookup, tool-based calculation, structured output, source quotation, and post-generation verification. Retrieval quality itself must be measured for recall, relevance, freshness, and resistance to document poisoning. In high-stakes settings, answers should be limited to supported sources, with abstention when evidence is inadequate.

Organizations should establish an error budget appropriate to context. A creative-writing assistant may tolerate occasional factual invention; a legal research system or clinical support tool may require substantially stricter thresholds and mandatory professional review. Users need training on automation bias, the tendency to accept machine recommendations because they appear confident or technically sophisticated.

Harmful Content, Misuse, and Synthetic Media

Generative systems can produce harassment, extremist propaganda, fraud scripts, malware assistance, sexual exploitation material, self-harm encouragement, and targeted disinformation. Harm depends on content, context, audience, distribution, and capability. Governance must therefore address misuse, not merely whether the model can produce an isolated harmful sentence.

Synthetic media creates particular challenges because audio, images, and video can imitate real people and events. Deepfakes may facilitate fraud, reputational injury, political manipulation, non-consensual sexual imagery, or evidentiary confusion. Controls include consent requirements, identity and likeness protections, provenance mechanisms, disclosure labels, platform policies, rapid reporting, and human investigation. Detection tools can assist but should not be treated as infallible; detection performance varies with compression, editing, language, and adversarial adaptation.

Organizations should maintain clear policies for impersonation, public release, biometric use, political communication, and sensitive-content generation. A responsible synthetic-media workflow records who authorized the content, what source material was used, which model generated it, what edits occurred, and where it was distributed.

Alignment and the Limits of Refusal

Alignment methods commonly include supervised fine-tuning, preference optimization, reinforcement learning from human or AI feedback, constitutional or rule-based guidance, and post-deployment policy controls. These approaches can improve helpfulness and reduce obvious misuse, but they involve trade-offs. Excessive refusal can impede legitimate research; weak refusal can enable harm. A model can also follow a harmful instruction that is technically consistent with a poorly defined objective.

Alignment should be treated as a sociotechnical specification problem. Organizations must identify whose interests count, which instructions are legitimate, how conflicts are resolved, and what behavior is unacceptable. Policies should be explicit enough for implementation and evaluation, while allowing escalation when context is ambiguous.

Prompt injection and jailbreaks demonstrate that alignment at the model level is fragile when the model receives untrusted instructions. A secure architecture separates data from commands, treats retrieved content as untrusted, limits authority, validates tool calls independently, and requires confirmation for consequential actions. The model should not be the sole judge of whether its own action is safe.

Provenance, Transparency, and Explainability

Generative AI requires several kinds of explanation. A user may need to know that content was AI-generated. An affected person may need an understandable account of how a decision was reached. An engineer may need a trace of prompts, retrieval documents, model versions, tools, and outputs. A regulator or auditor may require evidence of governance and control effectiveness.

For a RAG application, a useful explanation can include the question, retrieved sources, passages relied upon, generation timestamp, model version, and an indication of uncertainty. For an agent, the audit trail should include planned steps, tool requests, authorization checks, results, and human approvals. Explanations must not expose confidential prompts, security secrets, or personal data unnecessarily.

Provenance improves accountability but does not prove truth. A fully traceable false statement remains false. Similarly, a technically accurate explanation may be incomprehensible to the affected user. Explainability should therefore be purpose-driven, audience-appropriate, and connected to contestability: people should have a route to question, correct, or appeal an output.

Organizational Governance and Third-Party Risk

A mature organization assigns accountability across a governance structure. A board or executive committee sets risk appetite; a responsible-AI or model-risk function establishes standards; legal and compliance teams interpret obligations; security and privacy teams manage threats; technical teams implement controls; business owners remain accountable for the use case; and internal audit assesses independence and effectiveness.

A model inventory should record owners, providers, versions, purposes, data categories, users, integrations, risk ratings, evaluation evidence, and review dates. Procurement should avoid contracts that permit unrestricted reuse of confidential inputs or provide no notice of material model changes. Service-level agreements should address availability, incident reporting, support, deletion, portability, and access to relevant assurance evidence.

Concentration risk is also significant. Dependence on one model provider may create operational, financial, geopolitical, and continuity risks. Mitigation may involve abstraction layers, fallback models, local processing for sensitive tasks, tested export procedures, and contractual transition assistance. However, multi-model architectures introduce their own complexity and may make consistent evaluation more difficult.

Illustrative Case: An AI Claims-Processing Agent

Background

A fictional insurance company, Meridian Mutual, proposes an AI agent to summarize claim documents, identify missing evidence, draft correspondence, and recommend whether a claim should proceed to human assessment. The system uses a third-party language model, an internal document repository, optical character recognition, and a workflow tool.

Situation and Problem

During a pilot, the agent confidently states that a claimant submitted a repair invoice when the document actually belonged to another file. In a second incident, a prompt embedded in an uploaded document instructs the agent to email confidential claim data to an external address. The business team argues that the incidents are rare and that automation is necessary to reduce processing time.

Analysis

The first incident is a grounding and identity-resolution failure. Retrieval found a semantically similar document but failed to verify claim identifiers. The second is a prompt-injection and authorization failure. The agent treated document content as an instruction and had excessive email permissions. The proposed recommendation function also creates fairness and due-process concerns because claimants may be affected by an opaque system without adequate review.

Governance Strategy

Meridian Mutual narrows the initial use case to document classification and missing-evidence suggestions. The agent cannot approve, deny, or communicate a final decision. Retrieval is restricted by claim identifier, and every extracted fact must link to a page-level source. Uploaded documents are treated as untrusted data, not instructions. Email and database-writing tools are removed from the agent; drafts are placed in a queue for trained staff approval.

The company introduces testing with adversarial documents, cross-file contamination tests, privacy leakage tests, demographic fairness analysis, and human-factors research on whether staff over-trust fluent summaries. The vendor contract requires notification of material model changes, prohibits training on confidential claim data without authorization, and establishes incident-reporting and deletion requirements. A monitoring dashboard tracks unsupported claims, correction rates, escalation rates, and complaints.

Outcome

The revised system delivers narrower efficiency gains but substantially improves traceability and reversibility. Human assessors report that source-linked summaries are more useful than unrestricted narrative recommendations. The company postpones autonomous correspondence until it can demonstrate reliable identity checks and adequate authorization controls.

Learning Points

The case illustrates that governance is not equivalent to adding a content filter. The principal controls were use-case restriction, retrieval integrity, least privilege, source-linked output, human approval, vendor accountability, and continuous evaluation. It also shows that a smaller and more controllable system may create greater organizational value than a highly autonomous system with poorly bounded authority.

Challenges, Limitations, and Common Mistakes

A common mistake is treating benchmark performance as a comprehensive safety assessment. Benchmarks may be narrow, static, contaminated, or unrelated to the deployment context. Another is assuming that a model card transfers responsibility to users; documentation is valuable only when connected to decisions and controls.

Organizations also frequently overestimate content moderation and underestimate authorization. A system may refuse offensive text while still leaking confidential information or taking an unauthorized action. Excessive reliance on automated detection creates false positives and false negatives, while inadequate logging prevents investigation.

Other recurring problems include unclear ownership, undocumented model updates, inadequate user training, weak vendor contracts, failure to test multilingual or accessibility conditions, and the absence of a shutdown procedure. Governance can itself become burdensome when every low-risk experiment requires the same approval process. Proportionality and risk tiering are therefore essential.

There are technical limits to current methods. Hallucination cannot be reduced to zero in open-ended generation. Bias evaluation is constrained by incomplete demographic information and contested definitions of fairness. Provenance can be stripped or forged. Explainability may clarify a process without establishing causal validity. These limitations should be disclosed rather than hidden behind confidence scores or marketing language.

Regulatory and Professional Considerations

Generative-AI governance operates across privacy, intellectual property, consumer protection, employment, equality, product safety, cybersecurity, records management, and sector-specific rules. Regulatory approaches increasingly emphasize risk classification, transparency, accountability, documentation, human oversight, and obligations for providers of general-purpose systems and high-impact applications. The precise legal duties depend on jurisdiction and use case; organizations should obtain current legal advice rather than treating a general framework as a substitute for law.

Professional responsibility remains important where legislation is incomplete. Engineers and managers should avoid misleading claims, protect confidential information, assess foreseeable misuse, document limitations, and provide meaningful human recourse. High-impact decisions should not be delegated merely because automation is cheaper. Individuals affected by an AI-supported process should receive appropriate notice, review, and correction opportunities.

Current Developments and Emerging Directions

Governance is moving toward continuous evaluation rather than one-time certification. Automated red teaming, model behavior monitoring, secure evaluation environments, and incident databases are becoming important operational capabilities. Smaller specialized models are increasingly used for privacy, cost, latency, and controllability, although they may introduce their own accuracy limitations.

Agentic systems are driving renewed attention to identity, authorization, memory, tool security, and action verification. Standards for provenance and content credentials are developing alongside detection and labeling approaches. Organizations are also experimenting with model routing, confidential computing, privacy-enhancing techniques, and policy-as-code systems that enforce rules consistently across applications.

An important trend is the separation of capability from authority. A model may be capable of generating a payment instruction without being authorized to execute payment. This distinction supports bounded autonomy and is likely to become central to enterprise AI architecture.

Best-Practice Governance Checklist

A responsible generative-AI deployment should, at minimum:

• Define purpose, users, affected parties, prohibited uses, and risk tolerance.
• Maintain an inventory of models, versions, data flows, tools, and owners.
• Verify data rights, privacy obligations, provenance, and retention requirements.
• Evaluate factuality, safety, fairness, privacy, security, robustness, and usability in context.
• Ground consequential outputs in approved sources and support abstention.
• Apply least privilege, sandboxing, rate limits, validation, and approval gates to tools.
• Provide appropriate notices, explanations, citations, appeal routes, and human review.
• Monitor performance, misuse, drift, complaints, and incidents after deployment.
• Control model changes and revalidate after provider or system updates.
• Contractually manage third-party risk, data use, security, service continuity, and exit.
• Train users to recognize hallucination, automation bias, prompt injection, and synthetic-media manipulation.
• Maintain a tested incident-response and retirement process.

Chapter Summary

Foundation models create governance challenges because they are general-purpose, adaptive, probabilistic, and often connected to tools and sensitive data. Effective governance treats the deployed application as a sociotechnical system and manages its complete lifecycle. Key concerns include training-data rights, hallucination, harmful content, deepfakes, alignment, prompt injection, provenance, explainability, human oversight, monitoring, and third-party model risk.

A risk-based and defense-in-depth approach combines use-case classification, documentation, contextual evaluation, grounding, guardrails, least privilege, human approval, auditability, incident response, and controlled change management. No single filter, benchmark, watermark, or alignment technique is sufficient. Sound governance limits authority, makes uncertainty visible, preserves contestability, and assigns responsibility clearly. For organizations, the strategic objective is not unrestricted automation but reliable and accountable augmentation within defined boundaries.

Key Terms

Foundation model: A broadly trained model adaptable to many downstream tasks.

Large language model: A neural model trained to process and generate language at scale.

Multimodal model: A model that handles two or more modalities such as text, image, audio, or video.

AI agent: A system that uses a model to plan or perform actions through tools.

Hallucination: A plausible but unsupported or false model output.

Alignment: The correspondence between system behavior and legitimate objectives, values, and constraints.

Guardrail: A technical, procedural, or organizational control limiting unsafe behavior.

Retrieval-augmented generation: Generation supported by documents retrieved from an external knowledge source.

Prompt injection: An attempt to manipulate a system through instructions embedded in user or retrieved content.

Grounding: Linking an output to reliable external evidence or verified data.

Provenance: Information about the origin, ownership, transformation, and history of content or data.

Synthetic media: Artificially generated or manipulated text, image, audio, or video content.

Deepfake: Synthetic or altered media that convincingly imitates a real person or event.

Red teaming: Adversarial testing designed to expose weaknesses or harmful behaviors.

Human-in-the-loop: A design requiring human intervention or approval at defined decision points.

Model drift: Change in performance or behavior over time or after environmental or model changes.

Least privilege: Granting only the minimum access required for a task.

Third-party model risk: Risk arising from reliance on an external model provider or service.

Model inventory: A controlled record of models, uses, owners, versions, and governance evidence.

Explainability: The provision of understandable information about a system’s output or operation.

Review Questions

Short Answer Questions

1. Why does a foundation model require lifecycle governance rather than only pre-release testing?

2. Define hallucination and identify two methods for reducing its impact.

3. What is prompt injection, and why is it especially significant for AI agents?

4. Distinguish provenance from explainability.

5. What does the principle of least privilege mean in an agentic system?

Long Answer Questions

1. Explain a risk-based, defense-in-depth governance framework for generative AI from data acquisition through retirement.

2. Discuss the governance challenges associated with training-data rights, privacy, copyright, and provenance.

3. Analyze the relationship between alignment, guardrails, human oversight, and organizational accountability.

4. Explain how organizations should evaluate and manage third-party foundation-model providers.

5. Discuss why synthetic media and deepfakes require both technical and institutional responses.

Analytical / Application Questions

1. Design controls for an AI agent that retrieves internal documents and drafts procurement contracts. Identify risks, evaluation methods, permissions, and approval gates.

2. A customer-support chatbot provides fluent but occasionally false information and sometimes reveals details from other customers’ conversations. Diagnose the likely governance failures and propose a remediation plan.

3. An enterprise wants to deploy an autonomous financial agent capable of initiating payments. Using the principles in this chapter, determine whether and how bounded autonomy could be introduced safely.

22. Multiple Choice Questions with Answers

Question 1

Which statement best explains why foundation models create distinctive governance challenges?

A. They perform only one narrowly defined prediction task.

B. Their broad capabilities and adaptation make behavior dependent on context, integration, and use case.

C. They do not require training data once deployed.

D. Their outputs are deterministic and therefore easy to audit.

Correct Answer: B. Their broad capabilities and adaptation make behavior dependent on context, integration, and use case.

Question 2

A model produces a convincing legal citation that does not exist. What type of failure is this?

A. Model drift

B. Data poisoning

C. Hallucination

D. Provenance labeling

Correct Answer: C. Hallucination

Question 3

Which control most directly addresses an agent sending confidential data through an unauthorized email tool?

A. Increasing the model’s temperature

B. Adding more general-language benchmark tests

C. Applying least privilege and requiring approval for external communication

D. Removing all audit logs

Correct Answer: C. Applying least privilege and requiring approval for external communication

Question 4

Why is retrieval-augmented generation not a complete solution to hallucination?

A. Retrieved documents may be irrelevant, outdated, incorrect, or manipulated.

B. Retrieval prevents a model from generating any text.

C. Retrieved documents are always legally unrestricted.

D. RAG eliminates the need for source verification.

Correct Answer: A. Retrieved documents may be irrelevant, outdated, incorrect, or manipulated.

Question 5

Which evaluation approach is most appropriate for a high-impact claims-processing system?

A. Testing only creative writing quality

B. Using a general benchmark without deployment-specific scenarios

C. Combining factuality, privacy, fairness, security, adversarial, and human-review testing

D. Measuring only response speed

Correct Answer: C. Combining factuality, privacy, fairness, security, adversarial, and human-review testing

Question 6

What is the principal governance value of content provenance mechanisms?

A. They guarantee that all content is truthful.

B. They provide information about origin and transformation that can support accountability.

C. They eliminate the need for content moderation.

D. They prevent every form of deepfake creation.

Correct Answer: B. They provide information about origin and transformation that can support accountability.

Question 7

A document uploaded to an agent contains instructions to ignore system rules and disclose data. The safest architectural response is to:

A. Treat the document as an authoritative system instruction.

B. Treat retrieved content as untrusted data and independently validate tool actions.

C. Give the agent broader permissions to interpret the document.

D. Disable all monitoring to preserve autonomy.

Correct Answer: B. Treat retrieved content as untrusted data and independently validate tool actions.

Question 8

Which statement best describes third-party model risk?

A. It disappears when the provider hosts the model.

B. It concerns only the provider’s electricity consumption.

C. It includes risks from provider data practices, updates, security, continuity, and contractual limitations.

D. It can be managed solely through user training.

Correct Answer: C. It includes risks from provider data practices, updates, security, continuity, and contractual limitations.

Question 9

Why should an organization revalidate a deployment after a provider changes the model version?

A. Model updates can alter accuracy, refusal behavior, privacy characteristics, or tool interactions.

B. Version changes never affect system behavior.

C. Revalidation is needed only for image models.

D. The update automatically proves compliance.

Correct Answer: A. Model updates can alter accuracy, refusal behavior, privacy characteristics, or tool interactions.

Question 10

Which design best demonstrates bounded autonomy for an AI financial agent?

A. Allowing unrestricted payment execution based on generated text.

B. Giving the model administrator privileges for convenience.

C. Allowing the agent to prepare a transaction while requiring independent authorization before execution.

D. Removing transaction logs to protect privacy.

Correct Answer: C. Allowing the agent to prepare a transaction while requiring independent authorization before execution.