Responsible AI, Ethics and Governance

Introduction

Deep learning and natural language processing (NLP) systems increasingly influence employment, education, healthcare, financial services, policing, public administration and online communication. These systems classify applicants, recommend content, summarize records, detect fraud, generate text and support decisions. Their strategic value is substantial, but so are their risks. A model can be accurate and still be unfair, unsafe, unlawful, environmentally costly or impossible to challenge.

Responsible AI is the disciplined design, deployment and management of AI so that it respects human rights, legal duties, professional standards and societal values. It is not a final testing stage added after model development. It is a lifecycle approach involving problem definition, data collection, model development, evaluation, deployment, monitoring and retirement.

Governance converts ethical intentions into accountable practice. It identifies who may make decisions, what evidence is required, which risks are acceptable, how incidents are reported and how affected people obtain explanation or remedy.

Learning Objectives

By the end of this chapter, students should be able to:

1. Explain bias, fairness, privacy, security, intellectual property, transparency and environmental impact in deep learning and NLP.
2. Distinguish important fairness metrics and explain why they may conflict.
3. Apply human oversight, documentation, risk assessment and monitoring controls.
4. Recognise major principles in contemporary AI governance, including risk-based regulation.
5. Evaluate an AI application and recommend proportionate safeguards.

1. Responsibility Across the AI Lifecycle

An AI system is more than a trained model. It includes a purpose, data, software, infrastructure, users, organisational processes and affected communities. Responsibility therefore exists at several levels:

• Individual responsibility: developers, analysts and managers must act competently and report concerns.
• Organisational responsibility: the deploying organisation must define acceptable uses, allocate accountability and provide resources for testing and oversight.
• Technical responsibility: the system should be reliable, secure, interpretable where necessary and monitored in operation.
• Public responsibility: regulators, professional bodies and civil society establish expectations concerning rights, safety and social impact.

A useful lifecycle is:

1. Problem formulation: clarify the decision, intended users, affected people and whether AI is appropriate.
2. Data governance: document sources, consent or lawful basis, representation, quality and retention.
3. Development: select models and train them using controlled, reproducible processes.
4. Validation: test performance, fairness, robustness, privacy and security before release.
5. Deployment: define human roles, user training, access controls and escalation procedures.
6. Monitoring: detect drift, changing populations, incidents, misuse and unexpected harms.
7. Retirement: withdraw systems that no longer meet safety, legal or performance requirements.

A central principle is proportionality: high-impact uses require stronger evidence, controls and human involvement than low-risk uses such as a personal writing assistant.

2. Bias and Fairness

2.1 Understanding bias

Bias is a systematic pattern that produces unjustified or harmful differences in outcomes. It can enter at every stage:

• Historical bias: past social inequalities are reproduced in training data.
• Representation bias: some groups are absent or underrepresented.
• Measurement bias: variables measure groups differently; for example, complaint records may reflect surveillance rather than actual offending.
• Label bias: human annotations contain inconsistent or stereotyped judgments.
• Aggregation bias: one model is applied to populations with different characteristics.
• Evaluation bias: a benchmark does not represent real users or real operating conditions.
• Deployment bias: a system is used for a purpose different from the one for which it was validated.

Removing protected attributes from a dataset does not automatically remove discrimination. Other variables, such as postcode, language style or browsing history, may act as proxies.

2.2 Fairness is a choice of criteria

Fairness has no single universal mathematical definition. Relevant protected attributes may include sex, race, ethnicity, disability, age, religion or socioeconomic status, subject to applicable law. A team must specify the decision context and the harm being controlled.

For a binary classifier, let TP, TN, FP and FN denote true positives, true negatives, false positives and false negatives. Common performance measures are:

Precision measures the proportion of positive predictions that are correct; recall measures the proportion of actual positive cases detected. These formulas assume a binary decision and non-zero denominators.

Important group fairness criteria include:

• Demographic parity: the positive prediction rate is similar across groups.
• Equal opportunity: recall, or the true-positive rate, is similar across groups.
• Equalized odds: both true-positive rates and false-positive rates are similar across groups.
• Calibration: among people receiving the same predicted risk, observed outcome frequencies are similar across groups.

These criteria can conflict, especially when groups have different base rates or when predictions are imperfect. Consequently, “fairness” cannot be established by choosing whichever metric produces the most favourable result. The organisation must justify the criterion according to the decision’s consequences, legal context and affected community.

2.3 Fairness assessment in practice

A responsible assessment should:

1. Define the protected groups and the decision’s potential harms.
2. Check representation and label quality before training.
3. Report performance separately by group, not only overall averages.
4. Examine intersectional groups, such as older women with disabilities.
5. Test thresholds, missing-data strategies and language varieties.
6. Consult domain experts and affected stakeholders.
7. Document trade-offs and obtain approval from an accountable authority.

Fairness interventions may occur before training (data rebalancing or improved labels), during training (fairness-aware objectives or constraints) or after training (threshold adjustment). Each intervention can affect accuracy, calibration, privacy or operational complexity. No technical adjustment can compensate for an unjust objective, poor institutional process or inappropriate use case.

3. Privacy and Data Protection

Deep learning systems often require large datasets containing personal, sensitive or confidential information. Privacy concerns include unauthorised collection, excessive retention, re-identification, inference of sensitive attributes and leakage through model outputs.

Important principles include:

• Purpose limitation: collect and use data for specified legitimate purposes.
• Data minimisation: use only what is necessary.
• Accuracy: correct errors that could affect decisions.
• Storage limitation: delete or anonymise data when no longer needed.
• Security: protect data against unauthorised access or alteration.
• Accountability: demonstrate compliance through records and controls.

Anonymisation seeks to make identification reasonably unlikely, while pseudonymisation replaces direct identifiers with codes but does not eliminate privacy obligations. Removing names is insufficient if combinations of attributes can identify individuals.

Privacy-enhancing techniques include aggregation, access restriction, encryption, secure deletion, federated learning and differential privacy. Differential privacy introduces controlled randomness so that the presence or absence of one person has limited influence on published results. A common formal statement is:

Here, M is a randomised mechanism, D and D′ are datasets differing in one person’s record, S is an output event, ε (epsilon) controls the privacy-loss level, and δ is a small failure probability. The guarantee applies under the stated threat model and accounting assumptions; smaller ε generally offers stronger privacy but may reduce statistical utility. Differential privacy does not make an unlawful purpose lawful, nor does it protect against every organisational or security failure.

Privacy also concerns generated text. A language model may reproduce memorised personal information from training data or reveal confidential prompts. Testing should therefore include extraction attempts, prompt-injection scenarios, access logging and procedures for data-subject requests where applicable.

4. Security, Safety and Robustness

AI security protects models, data, infrastructure and users. Relevant threats include:

• Data poisoning: malicious training examples alter behaviour.
• Evasion or adversarial input: carefully designed inputs cause incorrect predictions.
• Prompt injection: instructions embedded in user content manipulate a language model or connected tool.
• Model extraction: repeated queries approximate a proprietary model.
• Membership inference: an attacker infers whether a record was in training data.
• Sensitive information disclosure: outputs reveal secrets or personal data.
• Supply-chain compromise: libraries, models or datasets contain malicious components.

Safety concerns include hallucinated content, toxic generation, unsafe recommendations, over-reliance by users and failures outside the training distribution. Robustness testing should include rare cases, noisy input, distribution shift, multilingual input, accessibility needs and deliberate misuse.

Controls should follow defence in depth:

• secure development and dependency management;
• encryption in transit and at rest;
• least-privilege access and strong authentication;
• separation of development, testing and production;
• input validation and output filtering;
• rate limits and abuse detection;
• red-team testing and independent review;
• logging, incident response and rollback capability;
• business continuity and a safe fallback process.

A model card or system card should state known limitations, unsuitable uses, evaluation conditions and residual risks. Security is not achieved merely by hiding the model; secrecy cannot replace sound controls.

5. Transparency, Explainability and Contestability

Transparency means that relevant information about an AI system is available to appropriate stakeholders. It may include the system’s purpose, data provenance, model type, performance, limitations, operator, retention practices and avenues for appeal.

Interpretability concerns how understandable a model is in itself. Explainability concerns methods used to describe a model’s behaviour. A feature-importance score may help a technical team, but it is not necessarily a meaningful explanation to an affected person. Explanations should be accurate, understandable, relevant to the decision and actionable.

For high-impact decisions, a person should generally be told:

• that AI was used;
• what type of decision it supported;
• the principal factors relevant to the outcome;
• the system’s limitations and confidence where meaningful;
• how to request human review or correction.

A post-hoc explanation can be misleading if it presents plausible reasons rather than the actual causal basis of a prediction. Explanations should therefore be validated, not treated as a public-relations device.

Contestability is equally important. People need notice, a meaningful opportunity to provide information, access to review and a remedy when the system is wrong. A nominal human who merely approves automated recommendations does not provide effective oversight.

6. Intellectual Property and Data Rights

Training and deploying generative models can involve copyrighted text, images, audio, code and proprietary documents. Legal rules differ by jurisdiction and continue to develop. Responsible practice requires more than assuming that material available online is free to use.

Organisations should maintain dataset and model provenance records, check licences and contractual restrictions, respect terms of use, protect confidential information and establish procedures for takedown or correction requests where appropriate. Generated output should be reviewed for copyrighted material, confidential content, misleading attribution and licence obligations.

Developers should distinguish:

• ownership of training data;
• permission to access or process data;
• ownership or licensing of model weights;
• rights in generated output; and
• responsibility for infringement caused by use or distribution.

Open-source availability does not eliminate compliance duties. A licence may impose attribution, disclosure, usage or redistribution conditions. Legal review is especially important for commercial systems, code generation and models trained on personal or restricted data.

7. Environmental and Social Impact

Deep learning can consume significant electricity and water through hardware manufacture, data-centre operation and cooling. Impact occurs during training, repeated fine-tuning, inference and device disposal. A large model is not automatically the most responsible model.

An approximate energy calculation is:

Power is measured in watts and time in hours, producing watt-hours. If a 500 W server operates continuously for 20 hours, energy use is 500 × 20 = 10,000 Wh = 10 kWh. This estimate excludes additional facility overhead unless that overhead is included in the power figure. Carbon emissions depend on the electricity mix:

CO₂e is estimated carbon-dioxide equivalent, energy is commonly measured in kWh, and the emissions factor is measured in kg CO₂e/kWh. Results are estimates and should state system boundaries, hardware utilisation and whether embodied emissions are included.

Reduction strategies include smaller models, transfer learning, pruning, quantisation, efficient hardware, caching, renewable electricity, batching and selecting the simplest adequate method. Environmental assessment should also consider social benefits, accessibility and the effects of automation on workers. Efficiency must not be pursued by transferring hidden costs to vulnerable communities.

8. Human Oversight and Accountability

Human oversight must be designed, not assumed. Three patterns are common:

• Human-in-the-loop: a person reviews or approves each consequential recommendation.
• Human-on-the-loop: a person supervises system operation and intervenes when alerts or thresholds are triggered.
• Human-in-command: a responsible authority can define, suspend or terminate the system.

The appropriate pattern depends on impact, reversibility, speed and the possibility of harm. Reviewers need authority, time, training, relevant evidence and protection from automation bias. They must be able to disagree with the model without unreasonable penalty.

Accountability should be assigned using a responsibility matrix. It should identify the system owner, data steward, technical lead, security lead, domain decision-maker, legal or compliance adviser and incident manager. “The algorithm decided” is not an acceptable allocation of responsibility.

9. Documentation and Risk Management

Documentation makes decisions auditable and supports continuity when staff change. Useful artefacts include:

• Data sheets: source, collection method, consent or legal basis, population, labels, missingness, known limitations and licence.
• Model cards: intended use, out-of-scope use, architecture, training data, metrics, subgroup results and limitations.
• System cards: interfaces, human processes, dependencies, monitoring, security controls and incident procedures.
• Impact assessments: affected groups, benefits, harms, mitigations, residual risk and approval decision.
• Change logs: versions, modifications, tests, approvals and rollback points.
• Incident records: event, severity, users affected, containment, root cause and corrective action.

A practical risk register records likelihood, impact, affected stakeholders, controls, owner, deadline and residual risk. Risk scoring is a prioritisation aid, not a substitute for judgement. A low numerical score must not override a legal prohibition or a fundamental-rights concern.

Monitoring should include technical, social and operational indicators: accuracy, subgroup performance, drift, refusal or escalation rates, complaints, security events, privacy incidents, energy use and human override patterns. Thresholds should trigger investigation rather than automatic concealment of the problem.

10. Governance Principles and Regulatory Direction

Although legal requirements differ, major governance frameworks converge on several principles:

• human autonomy and human oversight;
• fairness and non-discrimination;
• privacy and data governance;
• transparency and explainability;
• safety, robustness and cybersecurity;
• accountability and traceability;
• sustainability;
• accessibility and inclusion.

Contemporary regulation increasingly uses a risk-based approach. Prohibited or unacceptable practices receive the strongest restrictions; high-risk systems require documentation, assessment, monitoring and human oversight; transparency obligations apply to certain user-facing or generative systems; low-risk applications may require fewer controls. Organisations must identify the laws relevant to their jurisdiction and sector rather than treating a voluntary framework as legal advice.

Governance should be integrated with existing structures such as information security management, privacy management, quality assurance, procurement, internal audit and enterprise risk management. Procurement contracts should require supplier documentation, security notification, audit rights, performance commitments and cooperation during incidents.

11. Applied Case: Illustrative University Admissions Assistant

Background

A university considers an NLP system that summarises applicant statements and assigns a recommendation score for admissions staff. The system is intended to reduce administrative workload.

Problem

Initial testing shows high overall agreement with previous admissions decisions, but lower performance for applicants whose first language is not English and for applicants using assistive technology. The training labels reflect historical decisions that may themselves contain bias. Statements also contain personal and sensitive information.

Analysis

The project team identifies representation and label bias, privacy risks, possible proxy variables and automation bias. It also recognises that a summary may omit context or reproduce stereotypes. A high overall accuracy figure is inadequate because admissions is a consequential decision and subgroup outcomes matter.

Approach and solution

The university decides that the model will assist, not decide. Before deployment it:

1. documents purpose, excluded uses and affected groups;
2. obtains legal and privacy review of data collection and retention;
3. audits performance by language background, disability-related accessibility needs and intersectional groups where lawful and statistically meaningful;
4. improves annotation guidance and uses reviewers from diverse backgrounds;
5. tests summaries for omissions, toxicity, hallucination and confidential-data leakage;
6. provides admissions staff with source text and a warning that the summary is fallible;
7. requires written human justification for adverse decisions;
8. creates an applicant correction and appeal route;
9. logs model versions, reviewer overrides and complaints;
10. pilots the system and suspends it if predefined harm thresholds are exceeded.

Outcome

The pilot reduces administrative time, but the university retains human responsibility and does not use the score as an automatic ranking. Monitoring reveals that one language group receives more manual review; the university investigates and revises the model rather than treating the difference as acceptable merely because efficiency improved.

Learning points

The case demonstrates that responsible AI combines technical testing with institutional reform. Fairness is assessed in context, privacy is designed into the process, human oversight is meaningful only when reviewers have authority, and documentation supports both accountability and improvement.

Challenges and Limitations

Responsible AI methods have important limitations. Fairness metrics can conflict and may require sensitive demographic data that creates privacy concerns. Explainability methods may be unstable or misunderstood. Impact assessments can become paperwork if leaders do not act on findings. Small organisations may lack specialist staff and resources. Supply chains can obscure how a foundation model was trained. Some harms, especially cultural or democratic harms, are difficult to quantify. Finally, technical controls cannot resolve every structural inequality; some uses should be rejected rather than optimised.

Best-Practice Checklist

Before deployment, an organisation should be able to answer “yes” to the following:

• Is the purpose necessary, lawful and clearly defined?
• Are affected people and potential harms identified?
• Are data provenance, quality, consent or lawful basis documented?
• Has performance been evaluated for relevant groups and conditions?
• Are privacy, security, intellectual-property and environmental impacts assessed?
• Are human authority, escalation and appeal procedures real and resourced?
• Are limitations, versions and decisions documented?
• Are monitoring, incident response and retirement criteria defined?
• Has an accountable decision-maker approved residual risk?

Chapter Summary

Responsible AI is a lifecycle discipline that links technical quality with human rights, organisational accountability and public trust. Bias may arise from history, representation, measurement, labels, aggregation, evaluation or deployment, and fairness requires an explicit choice of criteria. Privacy requires purpose limitation, minimisation, security, retention controls and careful handling of model outputs. Security and safety demand defence in depth, robustness testing and incident response. Transparency must be matched by meaningful explanation, contestability and human remedy. Intellectual-property responsibility requires provenance and licence management. Environmental impact can be estimated through energy and emissions calculations and reduced through efficient models and infrastructure. Effective governance assigns responsibility, documents decisions, assesses risk, monitors outcomes and provides authority to suspend or retire unsafe systems. The central professional principle is that AI should support accountable human decision-making, not conceal or replace it.

Key Terms

Accountability: The duty to explain, justify and take responsibility for AI decisions and impacts.

Adversarial input: An intentionally crafted input designed to cause an AI system to behave incorrectly.

Bias: A systematic pattern that produces unjustified or harmful differences in outcomes.

Calibration: Agreement between predicted probabilities and observed outcome frequencies.

Contestability: The ability of affected people to challenge a decision and seek review or remedy.

Data minimisation: Collecting and using only data necessary for a specified purpose.

Demographic parity: Similar positive prediction rates across defined groups.

Differential privacy: A formal privacy approach limiting the effect of one person’s record on an output.

Explainability: Methods used to communicate why or how an AI system produced an output.

Fairness: The ethically and legally justified treatment of people and groups in an AI context.

Human-in-the-loop: A design in which a person reviews or approves relevant system outputs.

Model card: Documentation describing a model’s purpose, performance, limitations and suitable uses.

Prompt injection: Instructions in input content that manipulate a language model or connected tool.

Provenance: The recorded origin, ownership, processing history and permissions associated with data or models.

Recall: The proportion of actual positive cases correctly identified.

Risk-based governance: Applying controls in proportion to the potential severity and likelihood of harm.

Robustness: The ability of a system to perform acceptably under variation, noise or attack.

Transparency: Availability of relevant information about an AI system to appropriate stakeholders.

True-positive rate: The same quantity as recall: TP ÷ (TP + FN) for a binary classifier.

Utility: The practical value produced by a system, considered alongside its costs and harms.

Review Questions

Short Answer Questions

1. What is the difference between historical bias and representation bias?
2. Define demographic parity and equal opportunity.
3. Why is removing names from a dataset not always anonymisation?
4. What information should a model card contain?
5. Explain why human oversight must include authority to disagree with a model.

Long Answer Questions

1. Explain how bias can enter an NLP lifecycle from problem formulation to deployment, and recommend controls at each stage.
2. Compare precision, recall, demographic parity, equal opportunity and calibration. Why might an organisation be unable to satisfy all fairness criteria simultaneously?
3. Discuss privacy risks in training and deploying a large language model and evaluate suitable technical and organisational safeguards.
4. Explain how transparency, explainability and contestability differ, and show why all three matter in high-impact decisions.
5. Discuss the relationship between AI governance, cybersecurity, environmental sustainability and organisational accountability.

Analytical / Application Questions

1. A loan-screening model has equal accuracy overall but a substantially higher false-positive rate for one protected group. Analyse possible causes and propose a validation and governance plan.
2. A company wants to deploy a generative assistant using confidential customer records. Design a risk register covering privacy, security, intellectual property, human oversight and monitoring.
3. A model uses 2 kW for 30 hours during training. Calculate its direct energy use in kWh, state two limitations of the estimate, and recommend three ways to reduce environmental impact in a future training run.

Multiple Choice Questions with Answers

Question 1

Which type of bias occurs when historical social inequalities are reproduced in training data?

A. Historical bias

B. Aggregation bias

C. Evaluation bias

D. Deployment bias

Correct Answer: A. Historical bias

Question 2

For a binary classifier, recall is calculated as:

A. TP ÷ (TP + FP)

B. TP ÷ (TP + FN)

C. TN ÷ (TN + FP)

D. (TP + TN) ÷ (TP + TN + FP + FN)

Correct Answer: B. TP ÷ (TP + FN)

Question 3

Which fairness criterion requires similar true-positive rates across groups?

A. Demographic parity

B. Equal opportunity

C. Data minimisation

D. Calibration only

Correct Answer: B. Equal opportunity

Question 4

What does differential privacy primarily limit?

A. The electricity used by a data centre

B. The effect of one person’s record on an output

C. The number of model parameters

D. The need for human oversight

Correct Answer: B. The effect of one person’s record on an output

Question 5

Which is an example of prompt injection?

A. Compressing a model to reduce memory use

B. Adding instructions in user content that manipulate a connected language-model tool

C. Encrypting data at rest

D. Measuring subgroup recall

Correct Answer: B. Adding instructions in user content that manipulate a connected language-model tool

Question 6

What is the main purpose of a model card?

A. To replace all security testing

B. To document a model’s purpose, performance, limitations and suitable uses

C. To guarantee that a model is unbiased

D. To identify the cheapest hardware

Correct Answer: B. To document a model’s purpose, performance, limitations and suitable uses

Question 7

A server rated at 500 W runs for 20 hours. Ignoring overhead, how much energy does it use?

A. 0.01 kWh

B. 1 kWh

C. 10 kWh

D. 100 kWh

Correct Answer: C. 10 kWh

Question 8

Which practice best supports meaningful human oversight?

A. Allowing reviewers to approve every output without seeing evidence

B. Giving reviewers authority, training, relevant information and an escalation route

C. Removing all human review from high-impact decisions

D. Hiding model uncertainty from users

Correct Answer: B. Giving reviewers authority, training, relevant information and an escalation route

Question 9

Why can fairness criteria conflict?

A. Because all models use identical base rates

B. Because groups may have different base rates and predictions may be imperfect

C. Because fairness never involves statistical measures

D. Because demographic data cannot be collected in any context

Correct Answer: B. Because groups may have different base rates and predictions may be imperfect

Question 10

Which governance approach applies stronger controls to systems with greater potential harm?

A. Risk-based governance

B. Random governance

C. Output-only governance

D. Hardware-neutral governance

Correct Answer: A. Risk-based governance