Explainable, Ethical, Secure and Governed AI

Introduction

Artificial intelligence systems increasingly influence decisions about credit, employment, education, healthcare, insurance, public services and security. Their value depends not only on predictive accuracy but also on whether their decisions can be understood, challenged, protected and governed. A model that performs well in a laboratory may be unacceptable in practice if it discriminates against a protected group, exposes personal data, fails under attack, consumes excessive energy or cannot be audited.

Responsible AI therefore treats technical performance as one dimension of system quality. Other dimensions include interpretability, fairness, privacy, robustness, security, environmental sustainability, accountability and compliance. These concerns are interconnected. For example, collecting more personal data may improve accuracy but increase privacy risk; a highly complex model may improve prediction but reduce explainability; defensive security controls may add computational cost and environmental impact.

This chapter presents a lifecycle approach to trustworthy AI. It examines explanation methods, fairness assessment, privacy protection, adversarial threats, model documentation, human oversight, auditability, risk controls and regulatory governance. The emphasis is practical: students should be able to identify risks, select appropriate controls and justify deployment decisions.

Learning objectives

By the end of this chapter, students should be able to:

1. Distinguish interpretability, explainability, transparency and accountability.
2. Select suitable global and local explanation methods and identify their limitations.
3. Calculate and interpret important fairness metrics.
4. Explain privacy risks and compare common privacy-preserving techniques.
5. Identify adversarial threats and design robustness controls.
6. Evaluate environmental and social impacts across an AI lifecycle.
7. Construct documentation, oversight and audit mechanisms.
8. Apply a risk-based governance framework to an AI deployment.

1. Trustworthy AI as a sociotechnical system

An AI system includes more than a trained model. It includes data collection, labels, preprocessing, model code, infrastructure, interfaces, human operators, organisational incentives and affected communities. Harm may therefore originate outside the algorithm itself. A technically unbiased classifier can still produce unfair outcomes if historical labels reflect discriminatory practices or if access to the system differs between groups.

A useful risk model considers five questions:

• Purpose: What decision is the system intended to support, and is that purpose legitimate?
• People: Who benefits, who may be harmed, and who can contest an outcome?
• Performance: How accurate, calibrated and robust is the system across relevant populations and conditions?
• Protection: How are data, models, interfaces and infrastructure secured?
• Proof: What evidence demonstrates that controls operate effectively over time?

These questions should be addressed before deployment and revisited after changes in data, policy, model version or operating environment.

Important terminology

Interpretability is the degree to which a human can understand how a model represents information or produces an output. Explainability concerns the methods used to communicate a model’s behaviour or a particular prediction. Transparency concerns visibility into data, processes, design choices and limitations. Accountability means that identifiable people or organisations are responsible for decisions and remedies. These terms overlap but are not interchangeable. A model may be transparent because its documentation is public without being locally interpretable; an explanation may be intelligible without proving that the model is fair.

2. Explainability and interpretability

2.1 Intrinsic and post-hoc approaches

An intrinsically interpretable model is understandable by design. Examples include small decision trees, linear models with carefully selected features, monotonic additive models and rule lists. Their advantages include auditability and direct inspection. Their limitations include restricted functional form, difficulty representing complex interactions and the danger that a supposedly simple model may still use problematic proxies.

Post-hoc explanation methods analyse a trained model after it has been built. They can be applied to neural networks, ensembles and other black-box models, but an explanation is an approximation rather than a guarantee of causal reasoning.

Explanations may be:

• Global: describing overall model behaviour, important variables or typical interactions.
• Local: explaining one prediction or a small neighbourhood around it.
• Example-based: identifying similar, contrasting or influential training examples.
• Counterfactual: stating what minimum change might produce a different outcome.
• Mechanistic: examining internal representations or computational components.

2.2 Common methods

For a linear model, a prediction can be represented as an intercept plus weighted features. If a feature is standardised and the coefficient is positive, increasing that feature generally increases the prediction, holding other variables constant. This interpretation assumes the model is correctly specified and that associations are not automatically treated as causes.

Decision trees can be explained through the sequence of tests leading to a leaf. Feature-importance methods such as permutation importance estimate how performance changes when a feature’s values are disrupted. This can be misleading when features are correlated: importance may be shared, duplicated or assigned arbitrarily.

Local surrogate methods fit a simple model near a specific observation. SHAP-style methods allocate a prediction among features using contribution values derived from a cooperative-game framework. Such values depend on the background distribution and on assumptions about feature dependence. They should not be interpreted as causal effects.

Counterfactual explanations ask for a plausible change that would alter an outcome. For example, “an application would be accepted if verified income increased” may be actionable, whereas “change postcode” may be legally or ethically unacceptable. A valid counterfactual should satisfy feasibility, actionability, proximity and non-discrimination constraints.

2.3 Explanation quality

A useful explanation should be accurate with respect to the model, understandable to its audience, relevant to the decision, stable under small perturbations and actionable where appropriate. Explanation fidelity measures how closely an explanation reproduces the original model in the stated scope. User studies are also important: a mathematically faithful explanation may cause overconfidence or be misunderstood by non-specialists.

Explanations have risks. A model may generate plausible but false rationales, reveal sensitive information, expose security weaknesses or encourage users to treat correlation as causation. In high-impact settings, explanations should be combined with model validation, documentation, human review and an appeal process.

3. Fairness and discrimination

Fairness is not a single property. Different definitions can conflict, especially when groups have different base rates. Therefore, a project must identify the relevant harm, protected characteristics, legal requirements and stakeholder values before choosing a metric.

Consider a binary classifier and a protected group G. Let TP, TN, FP and FN denote true positives, true negatives, false positives and false negatives. The principal rates are:

TPR is sensitivity or recall; FPR is the false-positive rate; PPV is precision. The denominators must be non-zero, and the definitions apply to a specified decision threshold and reference outcome.

Demographic parity requires similar positive prediction rates between groups:

Equal opportunity requires equal true-positive rates:

Equalised odds requires both equal TPR and equal FPR. Predictive parity requires equal PPV. Calibration requires that predicted probabilities have comparable empirical meaning across groups; for example, among cases assigned probability 0.7, approximately 70% should have the outcome in each group.

These criteria cannot generally all be satisfied simultaneously when groups have unequal base rates and the classifier is imperfect. This is not a mathematical defect but a decision-making trade-off. A governance process must decide which errors are most harmful and document the rationale.

Worked fairness example

Suppose a screening system produces the following results:

For Group A, TPR_A = 80 ÷ (80 + 20) = 0.80 and FPR_A = 10 ÷ (10 + 90) = 0.10. For Group B, TPR_B = 60 ÷ (60 + 40) = 0.60 and FPR_B = 20 ÷ (20 + 80) = 0.20. The system therefore has a 0.20 TPR gap and a 0.10 FPR gap. If missing eligible cases is the principal harm, equal opportunity may be prioritised; if wrongful flags are more harmful, FPR may receive greater attention. The result alone does not determine the ethically correct intervention.

Fairness interventions may occur before modelling, during training or after prediction. Examples include improving representation, auditing labels, reweighting observations, imposing fairness constraints, changing thresholds and providing human review. Removing a protected attribute does not guarantee fairness because proxy variables may remain.

4. Privacy and data protection

AI privacy risks include unauthorised collection, excessive retention, linkage of supposedly anonymous data, membership inference, model inversion and leakage through logs or explanations. Privacy is not achieved merely by deleting names. Quasi-identifiers, embeddings and behavioural patterns can identify individuals when combined with other datasets.

Core principles include purpose limitation, data minimisation, accuracy, storage limitation, security, accountability and meaningful user rights. Organisations should define a lawful and legitimate purpose, collect only necessary information, establish retention periods and provide mechanisms for access, correction, objection or appeal where applicable law requires them.

Privacy-preserving techniques

• De-identification: removes or transforms direct identifiers but may not prevent re-identification.
• Aggregation: reports group-level results, reducing individual disclosure at the cost of detail.
• Access control and encryption: protect data in storage, transit and, where feasible, processing.
• Federated learning: trains across distributed data without centrally collecting raw records; gradients and updates may still leak information.
• Secure multiparty computation and homomorphic encryption: enable computation with stronger confidentiality but may impose substantial complexity and computational cost.
• Differential privacy: adds carefully calibrated randomness so that the inclusion or exclusion of one person has limited effect on published results.

A common differential-privacy guarantee is expressed as:

Here M is a randomised mechanism, D and D′ differ in one individual’s record, S is any set of outputs, ε is the privacy-loss parameter and δ is a small failure probability. Smaller ε generally provides stronger privacy but may reduce accuracy. The guarantee depends on correct accounting across repeated releases; privacy budgets must not be treated as unlimited.

Privacy engineering should begin at design time. A data protection impact assessment can map data flows, threats, purposes, retention, access rights and residual risk. Sensitive data should not be collected merely because future use is imaginable.

5. Robustness, security and adversarial threats

Robustness is the ability to maintain acceptable performance under natural variation, distribution shift, noise and faults. Security additionally considers intentional attempts to compromise confidentiality, integrity or availability.

Threats include:

• Evasion attacks: manipulating an input at inference time to cause an incorrect result.
• Poisoning attacks: contaminating training data or labels.
• Backdoors: inserting a trigger that causes targeted behaviour.
• Model extraction: querying a service to reconstruct its function.
• Membership inference: determining whether a record was in training data.
• Prompt injection and tool misuse: steering an AI application to ignore controls or perform unauthorised actions.
• Supply-chain attacks: compromising datasets, packages, pretrained models or deployment infrastructure.

Security analysis should use threat modelling. Identify assets, actors, attack surfaces, trust boundaries, likely attack paths and consequences. Controls include input validation, least privilege, secure coding, dependency management, secrets protection, rate limiting, monitoring, access logging, segmentation, backup and incident response.

Robustness testing should include adversarial evaluation, out-of-distribution tests, subgroup tests, stress tests and red-team exercises. No single robustness score proves security. A model can resist a known perturbation while remaining vulnerable to a new attack. Updates should be versioned and regression-tested.

For generative systems, controls should include tool permissions, output filtering, retrieval-source validation, prompt isolation, human confirmation for high-impact actions and protection against sensitive-data disclosure. Safety controls should be tested against adaptive attackers rather than only fixed test prompts.

6. Environmental and social sustainability

AI impacts the environment through data collection, hardware manufacture, training, inference, cooling, networking and disposal. The relevant quantity is not only training energy. A model used millions of times may have greater lifetime impact during inference.

A basic energy estimate is:

where E_total is energy, P is average power and t is operating time. If P is measured in kilowatts and t in hours, E_total is in kilowatt-hours. Estimated emissions can be expressed as:

where C is carbon dioxide equivalent and I is the electricity carbon intensity, typically measured in kilograms CO₂e per kilowatt-hour. These are estimates and depend on workload, location, hardware utilisation and accounting boundaries.

Sustainable design may involve smaller models, transfer learning, pruning, quantisation, efficient architectures, caching, batching, renewable electricity and selecting an appropriate service level rather than maximising model size. However, efficiency must not be pursued by sacrificing accessibility, safety or fairness. Social sustainability also includes labour conditions in data annotation, accessibility for disabled users and the effects of automation on employment and public services.

7. Documentation and auditability

Documentation converts intentions into evidence. A responsible AI project should maintain an inventory of systems, owners, purposes, data sources, model versions, risk classifications, performance results, incidents and retirement decisions.

Useful artefacts include:

• Data sheets: provenance, collection method, composition, labels, missingness, restrictions and known biases.
• Model cards: intended use, excluded use, evaluation data, subgroup performance, limitations, ethical considerations and version history.
• System cards: architecture, integrations, human roles, security controls, monitoring and escalation.
• Risk register: risk description, likelihood, impact, owner, controls, residual risk and review date.
• Decision logs: significant design choices, approvals, exceptions and evidence.
• Incident records: event, detection, containment, user impact, root cause, corrective action and communication.

Auditability requires more than storing documents. Logs should be tamper-evident, time-stamped and access-controlled. A production record should allow investigators to identify the model version, relevant input, output, explanation, human intervention and policy in force at the time. Logging must itself respect privacy and retention requirements.

A model risk register can use an expected-loss concept:

where R is a simplified risk score, P(H) is the estimated probability of a harmful event and I(H) is its impact. This is not a universal law or a substitute for expert judgement. Probability and impact scales must be defined, and low-probability catastrophic harms may require special treatment rather than simple multiplication.

8. Human oversight and governance

Human oversight is meaningful only when the human has competence, authority, time, information and the ability to intervene. A person who merely confirms an automated recommendation is not an effective safeguard. Oversight design should specify:

• which decisions are automated and which require approval;
• when a case must be escalated;
• what evidence the reviewer receives;
• how conflicts of interest are managed;
• how users challenge decisions;
• who is accountable for final outcomes; and
• how reviewer performance and automation bias are monitored.

Governance should be risk-based and proportionate. Low-risk recommendation tools may require basic documentation and monitoring. High-impact systems require stronger validation, independent review, impact assessment, access controls, incident response and periodic reauthorisation.

Relevant legal frameworks vary by jurisdiction. Examples include data-protection laws such as the EU General Data Protection Regulation, the EU Artificial Intelligence Act’s risk-based obligations, the NIST AI Risk Management Framework, ISO/IEC 42001 for AI management systems and ISO/IEC 23894 for AI risk management. These instruments differ in legal status, scope and implementation. Organisations must obtain jurisdiction-specific legal advice rather than treating a framework or standard as universal compliance.

A governance committee should include technical experts, legal and compliance specialists, domain professionals, security personnel and representatives of affected users. Governance should continue after launch because data distributions, social conditions, threats and regulations change.

9. A practical lifecycle for responsible deployment

A disciplined lifecycle can be organised as follows:

1. Define and justify the use case. State the intended benefit, affected people, prohibited uses and success criteria.
2. Assess impact and legality. Identify rights, harms, sensitive data, accessibility concerns and applicable obligations.
3. Prepare and audit data. Record provenance, consent or legal basis, representativeness, label quality, missingness and retention.
4. Select a proportionate model. Prefer interpretable methods where performance is adequate; justify complexity where it is necessary.
5. Evaluate broadly. Test overall and subgroup accuracy, calibration, fairness, robustness, privacy and resource use.
6. Document decisions. Produce model, data, system and risk documentation before approval.
7. Pilot with safeguards. Use restricted deployment, human review, rollback procedures and user feedback.
8. Monitor in production. Track drift, performance, subgroup outcomes, incidents, access, explanations and environmental indicators.
9. Respond and improve. Investigate complaints, correct errors, retrain or withdraw the system when controls fail.
10. Retire responsibly. Remove access, preserve necessary evidence, delete data according to policy and communicate the retirement.

Applied case: illustrative loan-screening system

Background

A financial institution develops a machine-learning model to prioritise small-business loan applications for human underwriters. The model uses transaction history, business age, requested amount and repayment history. A protected demographic attribute is excluded from production features, but it is collected in a restricted evaluation dataset to audit outcomes.

Problem

Initial validation shows strong average AUC, but approval rates and error rates differ between demographic groups. Applicants receive only a generic “not prioritised” message. The institution cannot reproduce some historical decisions because preprocessing code changed without version control.

Analysis

The team constructs confusion matrices by group and measures TPR, FPR, PPV, calibration and approval rates. It discovers that repayment labels are partly based on earlier human decisions, which may have reflected unequal access to credit. It also finds that a postcode-derived variable acts as a proxy for socioeconomic and demographic characteristics. A privacy review identifies excessive retention of raw transaction data, while a security review finds that the scoring API lacks rate limiting.

Approach and solution

The institution first narrows the purpose: the model may prioritise applications but may not automatically reject them. It removes the postcode proxy, reviews labels with domain experts and creates a data sheet. A simpler monotonic model is compared with the original ensemble. The selected model provides reason codes linked to actionable information, such as missing documentation, while avoiding sensitive or immutable characteristics.

The organisation establishes subgroup thresholds for investigation rather than promising that every fairness metric will be equal. Applications near the decision boundary receive mandatory human review. Reviewers receive training on automation bias and must record reasons for overriding the model. Model and data versions are stored with each decision, and applicants receive a clear appeal route.

Security controls include authentication, rate limiting, encrypted logs, dependency scanning and periodic red-team testing. Retention is reduced, access is role-based and privacy impact analysis is repeated after each major change. A dashboard monitors drift, subgroup metrics, complaint rates, override patterns and service energy use.

Outcome

The revised system has slightly lower average predictive performance but substantially better documentation, reproducibility and error visibility. Some disparities remain, so deployment is authorised only with ongoing review and a reauthorisation date. The institution concludes that model quality cannot be represented by one accuracy statistic.

Learning points

The case demonstrates that fairness depends on labels, features, thresholds and institutional processes. Explainability is useful only when explanations are accurate and actionable. Human oversight requires authority and an appeal mechanism. Governance is an operational control, not a document produced at the end of development.

Challenges and limitations

Responsible AI methods have significant limitations. Fairness metrics can conflict and may conceal intersectional harms when groups are aggregated. Explanation techniques can be unstable, manipulated or misinterpreted. Privacy protections introduce utility costs and are difficult to quantify across complex data pipelines. Robustness testing cannot enumerate every future attack. Carbon estimates may omit hardware manufacturing or outsourced infrastructure. Risk scoring can create false precision when evidence is weak.

There are also institutional limitations. Small organisations may lack specialist staff; vendors may refuse to disclose model details; affected communities may have limited influence; and compliance activity may become a checkbox exercise. Effective governance therefore requires proportional controls, independent challenge, resources for monitoring and willingness to stop deployment when evidence is inadequate.

Best-practice checklist

Before deployment, confirm that:

• the purpose is necessary, lawful and clearly bounded;
• affected people and foreseeable harms have been identified;
• data provenance, quality, retention and access are documented;
• performance and fairness are reported by relevant subgroups;
• explanations are tested with intended users;
• privacy, security and adversarial risks have been assessed;
• a human can intervene and users can appeal;
• model, data and system versions are reproducible;
• monitoring thresholds, owners and escalation routes exist;
• environmental and social impacts are measured proportionately; and
• an independent approval and reauthorisation process is defined.

Chapter Summary

Trustworthy AI is a sociotechnical discipline combining technical evaluation with ethical reasoning, legal compliance and organisational accountability. Interpretability and explainability help users understand model behaviour, but explanations are not automatically causal, fair or faithful. Fairness must be defined in relation to harms and context; demographic parity, equal opportunity, equalised odds, predictive parity and calibration may conflict. Privacy requires data minimisation, secure processing, retention control and techniques such as differential privacy, whose guarantees depend on parameters and composition. Robust AI requires threat modelling, adversarial testing, secure infrastructure and monitoring for distribution shift. Environmental assessment should consider the full lifecycle, not only training. Documentation, versioning, audit logs, human oversight, appeals and incident response turn principles into operational governance. A risk-based lifecycle enables organisations to justify, monitor, improve or withdraw AI systems responsibly.

Key Terms

Interpretability: The degree to which a human can understand a model’s operation or output.

Explainability: Methods for communicating why a model produced an output.

Transparency: Visibility into data, design, processes and limitations.

Accountability: Responsibility for decisions, impacts and remedies.

Local explanation: An explanation of one prediction or a small neighbourhood.

Global explanation: A description of general model behaviour.

Counterfactual explanation: A plausible change that would produce a different outcome.

Demographic parity: Similar positive prediction rates across groups.

Equal opportunity: Equality of true-positive rates across groups.

Equalised odds: Equality of true-positive and false-positive rates across groups.

Calibration: Agreement between predicted probabilities and observed frequencies.

Differential privacy: A formal guarantee limiting the effect of one record on outputs.

Federated learning: Distributed training in which raw data remains at local sites.

Adversarial attack: Deliberate manipulation intended to compromise AI behaviour.

Data poisoning: Corruption of training data to influence a model.

Model card: Documentation of a model’s uses, performance, limitations and risks.

Auditability: The ability to reconstruct and evaluate decisions using reliable evidence.

Human oversight: Competent, empowered review and intervention in an AI process.

Model drift: Change in data or relationships that reduces performance over time.

Environmental impact: Resource use and ecological effects across an AI lifecycle.

Review Questions

Short Answer Questions

1. Distinguish interpretability, explainability, transparency and accountability.
2. Why does removing a protected attribute not necessarily eliminate discrimination?
3. What is the difference between an evasion attack and a poisoning attack?
4. State two purposes of a model card.
5. Why is human confirmation alone insufficient as human oversight?

Long Answer Questions

1. Compare intrinsic interpretability with post-hoc explanation methods, including their advantages and limitations.
2. Explain demographic parity, equal opportunity, equalised odds, predictive parity and calibration. Why can these criteria conflict?
3. Describe a privacy-preserving AI lifecycle from data collection to model retirement.
4. Develop a governance framework for a high-impact AI system, including documentation, monitoring, auditability and appeals.
5. Discuss how environmental, social, security and fairness concerns should influence model selection.

Analytical / Application Questions

1. A classifier has TPR values of 0.82 and 0.67 for two groups. Calculate the TPR gap and explain what additional information is required before recommending an intervention.
2. Design a threat model for a public AI scoring API. Identify assets, threat actors, attack surfaces, likely attacks and at least five controls.
3. A model uses a sensitive demographic variable only for fairness auditing. Evaluate the ethical and privacy benefits and risks, and propose governance controls for its use.

Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes a local explanation from a global explanation?

A. A local explanation describes one prediction or a nearby region, while a global explanation describes general model behaviour.

B. A local explanation always proves causation, while a global explanation never uses data.

C. A local explanation is used only for linear models, while a global explanation is used only for neural networks.

D. A local explanation hides the model, while a global explanation removes the need for validation.

Correct Answer: A. A local explanation describes one prediction or a nearby region, while a global explanation describes general model behaviour.

Question 2

A group has TP = 80 and FN = 20. What is its true-positive rate?

A. 0.20

B. 0.50

C. 0.80

D. 0.90

Correct Answer: C. 0.80

Question 3

Which fairness criterion requires equality of both true-positive rates and false-positive rates?

A. Demographic parity

B. Equalised odds

C. Predictive parity

D. Calibration

Correct Answer: B. Equalised odds

Question 4

In differential privacy, what does a smaller ε generally indicate?

A. A stronger privacy guarantee, often with a possible utility cost

B. A larger training dataset

C. A guaranteed increase in model accuracy

D. Removal of all security threats

Correct Answer: A. A stronger privacy guarantee, often with a possible utility cost

Question 5

Which is an example of a poisoning attack?

A. Altering an input after deployment to evade classification

B. Querying an API to approximate its function

C. Injecting manipulated records into training data

D. Encrypting a model before storage

Correct Answer: C. Injecting manipulated records into training data

Question 6

Why can a feature-importance ranking be misleading when features are highly correlated?

A. Correlated features may share or arbitrarily receive importance.

B. Correlation guarantees that all features are causal.

C. Importance methods cannot process numerical variables.

D. Correlated features always have zero predictive value.

Correct Answer: A. Correlated features may share or arbitrarily receive importance.

Question 7

If average power is 2 kW and operation lasts 5 hours, what is the estimated energy consumption?

A. 0.4 kWh

B. 2.5 kWh

C. 7 kWh

D. 10 kWh

Correct Answer: D. 10 kWh

Question 8

Which condition makes human oversight meaningful?

A. The reviewer confirms every model output without seeing evidence.

B. The reviewer has competence, authority, information and ability to intervene.

C. The organisation removes all records of human decisions.

D. The model is too complex for the reviewer to question.

Correct Answer: B. The reviewer has competence, authority, information and ability to intervene.

Question 9

Which document primarily records a model’s intended uses, limitations and subgroup performance?

A. Model card

B. Network firewall rule

C. Payroll register

D. Source-code licence only

Correct Answer: A. Model card

Question 10

Which practice most directly improves auditability?

A. Deleting model versions after deployment

B. Allowing unrestricted changes to production code

C. Recording time-stamped model versions, inputs, outputs and interventions

D. Reporting only the overall accuracy of the latest model

Correct Answer: C. Recording time-stamped model versions, inputs, outputs and interventions