9. AI Governance Frameworks and Accountability

Introduction and Context

Artificial intelligence governance is the system through which an organisation directs, controls, monitors, and accounts for the design and use of AI. It is broader than technical model validation and different from compliance alone. Governance determines who may approve an AI system, what evidence is required before deployment, how risks are accepted, how affected persons can seek redress, and who is answerable when outcomes are harmful.

The strategic importance of AI governance arises from the distinctive characteristics of AI systems. Models may operate at scale, adapt to changing data, influence consequential decisions, and produce outputs that are difficult to interpret. Responsibility can become blurred among data suppliers, model developers, vendors, business users, senior executives, and boards. A technically accurate model can still be inappropriate if its purpose is unjustified, its data are unlawfully obtained, its use is opaque, or its consequences are not monitored.

Effective governance therefore combines organisational authority, risk management, technical assurance, ethical deliberation, legal interpretation, and continuous oversight. This chapter examines governance structures and compares four influential approaches: the NIST AI Risk Management Framework (AI RMF), the OECD AI Principles, ISO/IEC 42001, and related standards and regulatory approaches. The central argument is that frameworks are useful only when translated into clear accountability, documented decisions, operational controls, and credible escalation.

Learning Objectives

By the end of this chapter, students should be able to:

1. Explain the purpose, scope, and strategic value of AI governance.

2. Distinguish governance, management, risk ownership, assurance, and operational responsibility.

3. Design an accountability structure involving the board, executives, ethics committees, model owners, and independent assurance functions.

4. Compare the NIST AI RMF, OECD AI Principles, ISO/IEC 42001, and related frameworks.

5. Apply governance processes to model approval, monitoring, incident response, and escalation.

6. Evaluate common weaknesses in AI governance and propose practical improvements.

From Responsible Intent to Accountable Governance

Governance begins with a decision about authority. A policy stating that an organisation values fairness or transparency is not, by itself, governance. Governance exists when those values are connected to decision rights, resources, controls, evidence, and consequences. A responsible organisation can answer five questions for every material AI system: What is the system intended to do? Who owns the risk? What evidence supports deployment? Who may stop or change the system? How will affected stakeholders obtain explanation or remedy?

Several concepts require careful distinction. AI governance is the organisational framework for directing and controlling AI-related activities. AI risk management identifies, assesses, treats, and monitors risks. Model governance focuses more narrowly on model development, validation, documentation, change control, and performance. Assurance is an independent evaluation of whether governance arrangements and controls are designed and operating effectively. Accountability is the obligation to justify decisions and accept responsibility for their consequences. Responsibility refers to assigned duties, while accountability normally remains with a person or governing body that can be held answerable.

Governance is also different from explainability. Explainability concerns whether the operation or output of a system can be meaningfully understood. Governance determines when explanation is required, to whom it must be provided, and what action follows from an explanation. Similarly, human oversight is not satisfied merely because a person is placed somewhere in a workflow. The person must have competence, authority, time, relevant information, and the ability to override or suspend the system.

A Governance Architecture for Organisations

A proportionate governance architecture normally operates at several levels. The exact design depends on organisational size, sector, risk appetite, and regulatory obligations, but the allocation of roles should be explicit.

Board and Board Committees

The board retains ultimate oversight of material enterprise risks, including AI risks that could affect strategy, legal exposure, reputation, safety, or stakeholder rights. It should approve the organisation’s AI risk appetite, receive reports on high-impact systems, challenge management assumptions, and ensure that adequate expertise and resources exist. A risk, audit, technology, or ethics committee may conduct detailed review, but delegation does not eliminate the board’s oversight responsibility.

Board reporting should focus on decisions and exposures rather than technical detail alone. Useful information includes the inventory of high-impact systems, unresolved incidents, material changes to models, control failures, concentration in third-party providers, complaints, and trends in fairness, reliability, security, and privacy indicators.

Accountable Executive

An accountable executive translates board expectations into an enterprise programme. This role may be held by a chief risk officer, chief data and AI officer, chief information officer, or another senior leader. The executive establishes policies, assigns budgets, coordinates legal and technical functions, and ensures that unacceptable uses are stopped. The role must possess sufficient authority to challenge business units whose commercial objectives conflict with governance requirements.

AI Governance or Risk Office

A central office maintains the AI inventory, classification method, policy library, approval workflow, training programme, and reporting system. It should not become a bureaucratic gate that approves every low-risk experiment. Instead, it should apply proportionality: lightweight controls for low-impact uses and stronger evidence, review, and monitoring for systems affecting rights, safety, access to essential services, employment, credit, health, or public safety.

Ethics Committee

An ethics committee offers multidisciplinary deliberation on questions that cannot be resolved through technical metrics alone. Its membership may include representatives from law, compliance, security, data science, affected business units, social science, accessibility, and, where feasible, external or community perspectives. The committee should have a defined mandate and escalation route. It should not be used as symbolic endorsement or as a substitute for executive accountability.

Ethical review is particularly valuable when a proposed purpose is contestable, vulnerable groups may be affected, benefits and burdens are distributed unequally, or the system changes human behaviour. The committee should record alternatives considered, affected stakeholders, reasons for approval or rejection, residual risks, and conditions of use.

Model Owner and System Owner

The model owner is responsible for a model’s lifecycle, including requirements, data provenance, testing, documentation, deployment conditions, monitoring, and retirement. A system owner has broader responsibility for the complete sociotechnical system: interfaces, human decisions, vendor components, operational processes, and consequences. These roles must not be confused. A model may be statistically sound while the surrounding workflow produces harmful outcomes.

Owners should maintain a model or AI system card containing purpose, scope, users, data sources, limitations, performance results, fairness analysis, security assumptions, human oversight arrangements, approval status, and known residual risks. Ownership must continue after deployment; handover to an operations team should not end responsibility.

Independent Assurance

Independent assurance may be performed by internal audit, model validation, compliance testing, information security assurance, or qualified external assessors. Independence means more than technical competence. The assessor must be sufficiently separate from the team that designed or operates the system and must be able to report adverse findings without retaliation.

Assurance can examine design adequacy, evidence quality, control operation, legal compliance, robustness, privacy, security, fairness, and monitoring. It should be risk-based and recurring. A one-time pre-deployment review cannot address data drift, changing populations, adversarial attacks, new uses, or automation bias after deployment.

Escalation and Speak-Up Channels

Governance requires channels through which employees, customers, affected persons, and suppliers can raise concerns. Escalation thresholds should include severe harm, unexplained performance deterioration, material bias, unauthorised use, privacy or security incidents, loss of human control, and failure to meet deployment conditions. Possible responses range from additional monitoring and restricted use to rollback, suspension, notification, remediation, and system retirement.

A mature process protects good-faith reporting, preserves evidence, assigns an incident commander, communicates with affected stakeholders, and conducts a post-incident review. The purpose is not merely to identify an individual at fault but to correct organisational conditions that allowed the failure.

Comparing Major Frameworks

Framework comparison is most useful when it distinguishes purpose, level of abstraction, and implementation mechanism.

NIST AI Risk Management Framework

The NIST AI RMF is a voluntary, risk-based framework designed to help organisations manage risks to individuals, organisations, and society. Its core functions are Govern, Map, Measure, and Manage.

Govern establishes policies, accountability, culture, legal alignment, inventory, and risk tolerance. Map identifies context, intended purpose, stakeholders, impacts, and assumptions. Measure applies qualitative and quantitative methods to assess validity, reliability, safety, security, transparency, explainability, privacy, and fairness. Manage prioritises and treats identified risks, including through deployment restrictions, monitoring, and response.

The framework is valuable because it links context to measurement and action. It recognises that trustworthy characteristics can involve trade-offs and that risk management is iterative. Its voluntary character, however, means that it does not itself create legal duties or guarantee implementation. Organisations must translate its functions into accountable roles, evidence requirements, and escalation rules.

OECD AI Principles

The OECD AI Principles provide a high-level, internationally oriented normative foundation. They emphasise inclusive growth and sustainable development; human-centred values and fairness; transparency and explainability; robustness, security, and safety; and accountability. They also support democratic values, human rights, and the ability to challenge outcomes.

The principles are useful for policy formation and cross-border dialogue because they state what responsible AI should support without prescribing one organisational control system. Their abstraction is also a limitation for operational teams. An organisation must convert principles into procedures such as impact assessment, approval criteria, audit trails, user notices, and complaint mechanisms.

ISO/IEC 42001

ISO/IEC 42001 specifies requirements for an AI management system (AIMS). It follows a management-system logic: establish organisational context, leadership, planning, support, operation, performance evaluation, and improvement. It addresses AI policy, objectives, risk and impact assessment, documented information, competence, operational controls, monitoring, internal audit, management review, and corrective action.

Its principal contribution is institutionalisation. AI governance becomes part of a repeatable management system rather than a collection of isolated projects. Organisations may seek certification through an independent conformity-assessment process, although certification should not be treated as proof that every AI outcome is ethical or harmless. A management system can be formally compliant while its objectives, risk thresholds, or stakeholder engagement remain weak.

Related Standards and Regulatory Approaches

ISO/IEC 23894 provides guidance on AI risk management, while ISO/IEC 22989 supports AI concepts and terminology. ISO/IEC 24027 addresses bias in AI systems, and ISO/IEC 25059 concerns quality models for AI systems. The ISO/IEC 27001 family is relevant to information security, and privacy-management standards may complement AI governance where personal data are involved. The IEEE 7000 series contributes standards-oriented approaches to ethical considerations in system design.

Regulatory approaches add binding obligations that voluntary frameworks do not. Depending on jurisdiction and use case, organisations may face requirements concerning privacy, discrimination, consumer protection, safety, cybersecurity, employment, financial services, medical devices, public procurement, or high-risk AI systems. A governance programme must therefore map voluntary frameworks to applicable law rather than treating a framework as a substitute for legal analysis.

An Integrated View

The frameworks are complementary. OECD principles answer, in broad terms, what values responsible AI should embody. NIST AI RMF provides a practical risk-management cycle. ISO/IEC 42001 provides an auditable management-system structure. Related standards supply specialised controls, terminology, and evaluation methods. A useful organisational model is to use OECD principles as a normative reference, ISO/IEC 42001 as the governance operating system, NIST as the risk workflow, and specialised standards and law as control sources.

The Governance Lifecycle in Practice

A governance lifecycle begins before model development. During intake and classification, the organisation records the purpose, users, affected groups, data, deployment setting, provider involvement, and potential impact. The system is then classified according to factors such as consequence severity, scale, autonomy, reversibility, vulnerability of affected persons, and legal sensitivity.

During impact assessment, teams examine foreseeable benefits, harms, rights implications, accessibility, privacy, security, discrimination, environmental effects, and alternatives. The assessment should ask not only whether the model works, but whether the problem should be solved through automation and whether less intrusive means exist.

During development and procurement, requirements are converted into measurable acceptance criteria. Data lineage, consent or lawful basis, representativeness, labelling quality, security controls, test sets, documentation, and vendor obligations are reviewed. Contracts should specify audit rights, incident notification, data use restrictions, model-change notification, support, and exit arrangements.

During validation and approval, independent reviewers test technical and sociotechnical performance. Results should be disaggregated across relevant groups where lawful and meaningful. Approval should state conditions, prohibited uses, monitoring thresholds, responsible roles, and expiry or review dates. Approval is not a permanent licence: significant changes in model, data, purpose, user population, or context should trigger reassessment.

During deployment and monitoring, organisations track performance, drift, data quality, access, complaints, overrides, security events, and real-world outcomes. Monitoring should include leading indicators, such as rising override rates, and lagging indicators, such as confirmed harm. A dashboard that reports only accuracy can conceal deterioration in fairness, reliability, or human decision quality.

Finally, retirement and learning require secure data and model disposal, preservation of records, communication to users, transition arrangements, and a review of lessons learned. Retirement may be appropriate where the model no longer meets its purpose, risks exceed benefits, a safer alternative exists, or controls cannot be sustained.

Illustrative Case: Automated Recruitment Screening

The following is a fictional illustrative case.

Background

Northstar Services, a large outsourcing company, proposes an AI tool to rank applicants for customer-service positions. The tool analyses application forms, work history, assessment responses, and language patterns. Recruiters will review the highest-ranked candidates, while low-ranked candidates may not receive further consideration.

Situation and Problem

The vendor reports strong predictive performance based on historical hiring outcomes. Northstar’s legal team identifies possible employment-discrimination exposure, while the recruitment team argues that automation will reduce processing time. An internal review discovers that historical hiring data reflect a workforce concentrated in certain regions and that the model’s language features may disadvantage applicants using accessibility tools or non-standard varieties of English.

Analysis

The system is high impact because it influences employment opportunities and operates at scale. The model owner cannot decide alone whether deployment is appropriate. A system owner must examine the entire process, including job descriptions, assessment design, recruiter behaviour, candidate notice, accommodation, appeals, and vendor controls.

Using the NIST sequence, Northstar maps affected candidates and the decision context, measures group-level performance and error patterns, and manages risks through design and process changes. Under an ISO/IEC 42001-oriented system, the organisation documents objectives, responsibilities, operational controls, monitoring, internal audit, and management review. OECD principles direct attention to fairness, transparency, robustness, and accountability.

The ethics committee concludes that ranking candidates solely by historical similarity would reproduce past preferences. It recommends removing unnecessary language proxies, testing accessibility, conducting structured human review, informing candidates that AI is used, providing a channel for correction and accommodation, and prohibiting automatic rejection based only on the model score.

Approach and Governance Decision

The accountable executive requires a restricted pilot rather than immediate full deployment. Independent validation is commissioned. The model owner must provide documentation and subgroup performance results; the recruitment system owner must define human-review duties; legal and compliance teams assess applicable employment and privacy obligations; and internal audit reviews the control design.

Approval is conditional on minimum performance thresholds, documented reviewer training, monthly monitoring, quarterly committee review, vendor notification of model changes, and immediate escalation of material disparities or complaints. A red-team exercise tests whether applicants can be unfairly excluded through formatting or language manipulation.

Outcome

The pilot reveals that overall predictive performance is acceptable but error rates differ materially for several applicant groups. Northstar does not deploy the original ranking model. It replaces some automated features with structured application criteria, improves accommodation procedures, and uses the model only to organise recruiter workloads rather than to determine exclusion. After a further review, a limited deployment is approved with explicit controls and a sunset date.

Learning Points

The case demonstrates that governance examines purpose and consequences, not merely model accuracy. It also shows why accountability must be distributed but not diluted: the model owner manages technical evidence, the system owner manages the workflow, independent assurance challenges the evidence, and the executive decides whether residual risk is acceptable. Finally, a documented decision to restrict or reject deployment is a successful governance outcome, not a failure of innovation.

Challenges, Limitations, and Common Governance Failures

A frequent failure is unclear ownership. Many organisations assign responsibility to a committee while no individual has authority to stop deployment. Another is the “ethics washing” problem, in which principles are prominently stated but lack budgets, metrics, sanctions, and escalation. Committees may also suffer from insufficient diversity, technical dependence on the project team, or conflicts of interest.

Framework overload creates a different problem. Multiple standards can produce duplicated assessments, inconsistent terminology, and documentation that staff complete mechanically. A common control library and crosswalk can reduce this burden. Organisations should also avoid treating certification, an impact assessment, or a model card as conclusive evidence of trustworthiness. These are inputs to judgment, not substitutes for it.

Measurement itself has limitations. Fairness indicators may conflict, small subgroup samples may make estimates uncertain, and technical metrics may not capture dignity, autonomy, chilling effects, or unequal access to remedy. Monitoring can also create privacy risks. Governance must therefore combine quantitative evidence with qualitative inquiry and stakeholder engagement.

Third-party and foundation-model dependencies complicate accountability. A buyer may lack access to training data, model internals, or meaningful test conditions. Contracts, independent testing, usage restrictions, provenance records, fallback procedures, and concentration-risk analysis become essential. Generative systems add risks involving fabricated outputs, confidential-data leakage, prompt injection, copyright, and uncontrolled downstream use.

Strategic Implementation and Best Practice

A practical implementation programme should begin with an enterprise inventory rather than a policy document. Each system should have a named owner, purpose, risk classification, lifecycle status, provider information, approval record, and monitoring plan. The organisation should then define a small number of mandatory gates: use-case intake, impact assessment, validation, approval, change review, incident response, and retirement.

Best practice includes the following:

• align board reporting with material risks and decisions;
• give accountable executives authority over commercial escalation;
• separate development from independent validation;
• apply proportional controls rather than identical controls to every model;
• document assumptions, limitations, residual risks, and approval conditions;
• involve affected stakeholders early, especially where rights or access are concerned;
• test real-world outcomes, not only laboratory performance;
• link monitoring thresholds to predetermined actions;
• maintain audit trails for data, versions, approvals, overrides, incidents, and changes; and
• rehearse suspension, rollback, communication, and remediation procedures.

Automation can support governance through model registries, evidence repositories, policy-as-code checks, access controls, drift detection, privacy scanning, and automated alerts. Nevertheless, automated governance tools are themselves fallible. Their thresholds require validation, and a dashboard cannot replace accountable judgment.

Current Developments and Strategic Implications

AI governance is moving from voluntary principle statements toward management systems, sector-specific regulation, procurement requirements, and demonstrable assurance. Organisations increasingly need to show not only that they have a policy but that controls operate, decisions are traceable, and affected people can obtain remedy. Supply-chain governance is becoming more significant as organisations rely on external models, cloud platforms, data brokers, and embedded AI features.

Another development is the shift from model-centric to system-centric accountability. Risks arise from interactions among models, interfaces, incentives, workers, data, and institutions. Governance must therefore examine how humans use outputs, whether automation changes professional judgment, and whether users can meaningfully contest decisions. The strategic capability is not simply building accurate models; it is building institutions capable of knowing when to use, constrain, question, or discontinue them.

Chapter Summary

AI governance establishes authority, accountability, assurance, and escalation across the AI lifecycle. A robust structure connects board oversight and executive authority with ethics review, model and system ownership, independent assurance, stakeholder remedy, and incident response. The NIST AI RMF offers the functions Govern, Map, Measure, and Manage; the OECD principles provide a high-level normative foundation; ISO/IEC 42001 organises governance as an auditable AI management system; and related standards and law provide specialised or binding requirements. Effective implementation requires proportional classification, impact assessment, evidence-based approval, continuous monitoring, change control, and the willingness to restrict or retire systems. Governance succeeds when responsibility is visible, decisions are defensible, and organisations can act before harm becomes irreversible.

Key Terms

AI governance: Organisational direction and control of AI activities, risks, and impacts.

Accountability: The obligation to justify decisions and accept responsibility for consequences.

AI risk management: The identification, assessment, treatment, and monitoring of AI-related risks.

Model owner: The person or unit responsible for a model’s lifecycle and technical controls.

System owner: The person or unit responsible for the complete AI-enabled sociotechnical process.

Independent assurance: Objective evaluation of governance arrangements and control effectiveness.

AI inventory: A maintained record of AI systems, owners, purposes, risks, and lifecycle status.

Risk appetite: The level and type of risk an organisation is willing to accept.

Impact assessment: A structured examination of likely effects on people, rights, groups, and society.

Human oversight: Competent and empowered human involvement capable of understanding, intervening in, or stopping an AI process.

NIST AI RMF: A risk-management framework organised around Govern, Map, Measure, and Manage.

OECD AI Principles: International principles addressing human-centred, transparent, robust, safe, and accountable AI.

ISO/IEC 42001: A standard specifying requirements for an AI management system.

Residual risk: Risk remaining after controls and treatments are applied.

Model drift: Change in model performance or data relationships over time.

Escalation: The formal movement of a concern or incident to an authority able to decide and act.

Audit trail: A traceable record of activities, evidence, decisions, changes, and approvals.

Conformity assessment: Evaluation of whether specified requirements are met.

Remedy: Corrective action or redress available to persons affected by an AI decision.

Review Questions

Short Answer Questions

1. Why is AI governance broader than model validation?

2. Distinguish a model owner from a system owner.

3. What are the four core functions of the NIST AI RMF?

4. Why should independent assurance be separate from development?

5. What is the purpose of an AI system inventory?

Long Answer Questions

1. Explain how a board, accountable executive, ethics committee, model owner, and internal audit function should work together.

2. Compare the purposes and limitations of the NIST AI RMF, OECD AI Principles, and ISO/IEC 42001.

3. Describe a complete AI governance lifecycle from intake to retirement.

4. Discuss why certification or a model card cannot, by itself, establish that an AI system is trustworthy.

5. Explain how governance should address third-party and foundation-model risks.

Analytical / Application Questions

1. An organisation wants to use an AI system to prioritise applicants for public housing. Design a governance structure, approval conditions, monitoring indicators, and escalation triggers.

2. A deployed fraud model shows stable accuracy but a sharp increase in complaints from one community. Analyse the governance response using the NIST AI RMF functions.

3. A vendor refuses to disclose training-data information for a high-impact model. Decide whether and under what conditions deployment should proceed, using accountability, assurance, procurement, and residual-risk principles.

22. Multiple Choice Questions with Answers

Question 1

Which role has primary responsibility for ensuring that an organisation’s AI risk appetite and material AI exposures receive board-level oversight?

A. The model owner

B. The board or relevant board committee

C. The software developer

D. The end user

Correct Answer: B. The board or relevant board committee

Question 2

A model performs well in testing, but its deployment workflow causes recruiters to reject applicants automatically. Which concept best explains why model ownership alone is insufficient?

A. System ownership

B. Data compression

C. Conformity assessment

D. Model drift

Correct Answer: A. System ownership

Question 3

Which NIST AI RMF function focuses on identifying context, intended purpose, stakeholders, and potential impacts?

A. Govern

B. Map

C. Measure

D. Manage

Correct Answer: B. Map

Question 4

What is the distinctive organisational contribution of ISO/IEC 42001?

A. It provides only a list of ethical principles.

B. It guarantees that every model outcome is fair.

C. It structures AI governance as a management system with documented processes and continual improvement.

D. It replaces all sector-specific law.

Correct Answer: C. It structures AI governance as a management system with documented processes and continual improvement.

Question 5

An ethics committee rejects immediate deployment of a recruitment model and requires a restricted pilot with monitoring. What does this decision illustrate?

A. Ethical review is a substitute for executive accountability.

B. Governance may appropriately constrain or stage innovation when residual risks are uncertain.

C. Technical accuracy is irrelevant to governance.

D. Independent assurance is unnecessary after committee review.

Correct Answer: B. Governance may appropriately constrain or stage innovation when residual risks are uncertain.

Question 6

Which monitoring result should most clearly trigger escalation for a high-impact AI system?

A. A stable user-interface colour scheme

B. A material increase in unexplained group-level error disparities

C. A reduction in document length

D. A minor change in employee job titles

Correct Answer: B. A material increase in unexplained group-level error disparities

Question 7

Why are OECD AI Principles generally insufficient as an organisation’s complete operational control system?

A. They reject transparency and accountability.

B. They are high-level normative principles that must be translated into procedures and controls.

C. They apply only to software engineers.

D. They prohibit risk assessment.

Correct Answer: B. They are high-level normative principles that must be translated into procedures and controls.

Question 8

Which practice best supports independent assurance?

A. Allowing the development team to approve its own model without review

B. Separating assurance from the team being evaluated and protecting adverse reporting

C. Reviewing only the model’s accuracy score

D. Conducting assurance only after the system is retired

Correct Answer: B. Separating assurance from the team being evaluated and protecting adverse reporting

Question 9

A vendor changes a foundation model underlying a customer-facing AI service. Which governance control is most appropriate?

A. Ignore the change if the vendor retains the same product name.

B. Require change notification, reassessment, and approval according to materiality.

C. Delete all previous approval records.

D. Transfer accountability entirely to the vendor.

Correct Answer: B. Require change notification, reassessment, and approval according to materiality.

Question 10

In the illustrative recruitment case, why was restricting the model to organising recruiter workloads a governance improvement?

A. It eliminated the need for any human involvement.

B. It reduced the model’s direct authority to exclude applicants while preserving a limited efficiency benefit.

C. It proved historical hiring data were unbiased.

D. It made legal and ethical review unnecessary.

Correct Answer: B. It reduced the model’s direct authority to exclude applicants while preserving a limited efficiency benefit.