3. AI Harms, Stakeholders and Impact Analysis

Introduction and Context

Artificial intelligence systems do not produce value in a social vacuum. They classify people, recommend actions, allocate resources, shape opportunities, and influence institutional decisions. A model may be technically accurate and still cause serious harm if it is used in an inappropriate context, trained on distorted data, deployed without meaningful oversight, or evaluated only against the interests of its purchaser. Understanding AI ethics therefore requires more than identifying bias in algorithms. It requires a systematic analysis of who may be affected, how harms may arise, how severe they may become, and what responsibilities are distributed across the AI lifecycle.

AI harms may be direct or indirect, immediate or cumulative, individual or collective, intentional or unintended. A facial-recognition system may misidentify a person directly. A hiring model may indirectly reproduce historical exclusion. A recommender system may contribute cumulatively to manipulation, polarization, or unhealthy consumption. Environmental harms may arise from the energy and water consumed during model development and operation. These effects are often connected: a system designed to improve organizational efficiency may transfer costs to workers, applicants, communities, or the environment.

This chapter develops practical methods for identifying and analysing such effects. It introduces categories of harm, stakeholder mapping, risk classification, ethical impact assessment, and trade-off analysis. The emphasis is not on treating these tools as mechanical checklists, but on using them as disciplined forms of institutional reasoning.

Learning Objectives

By the end of this chapter, students should be able to:

1. Distinguish individual, organizational, societal, and environmental AI harms.

2. Identify direct, indirect, cumulative, reversible, and irreversible impacts.

3. Map stakeholders and analyse their interests, power, vulnerability, and voice.

4. Classify AI risks according to severity, likelihood, exposure, and uncertainty.

5. Conduct an ethical impact assessment across the AI system lifecycle.

6. Analyse trade-offs without reducing ethical decisions to a single numerical score.

7. Apply these methods to professional and organizational cases.

8. Recommend governance and mitigation measures appropriate to different risks.

Understanding AI Harm

An AI harm is a negative effect attributable wholly or partly to the design, development, deployment, use, or governance of an AI-enabled system. Attribution is often complex. Harm may result not from the model alone but from the interaction between data, technical architecture, institutional incentives, human decisions, and the surrounding social environment.

A useful distinction is between performance failure and ethical harm. Performance failure occurs when a system does not meet a technical objective, such as low accuracy or high latency. Ethical harm occurs when the system negatively affects rights, welfare, dignity, autonomy, fairness, safety, or the environment. The two can overlap, but they are not identical. A highly accurate system can be ethically unacceptable if it enables unjust surveillance. Conversely, a technically imperfect system may be acceptable in a low-stakes setting where users can easily correct its outputs.

Individual harms

Individual harms affect a person’s opportunities, rights, safety, dignity, privacy, or psychological well-being. Common forms include:

• Discrimination: unequal treatment or unequal outcomes based on protected or socially sensitive characteristics.
• Privacy loss: unauthorized collection, inference, disclosure, or retention of personal information.
• Autonomy reduction: manipulation, coercion, or excessive dependence on automated recommendations.
• Economic loss: denial of employment, credit, insurance, education, or public benefits.
• Physical or psychological harm: unsafe decisions, harassment, distress, or reputational damage.
• Due-process harm: inability to understand, challenge, or correct a consequential decision.

Individual harm is not limited to incorrect predictions. A correct prediction may still be harmful if the information was obtained illegitimately or used for an unjust purpose.

Organizational harms

Organizations may experience financial, operational, legal, cultural, and reputational harm. An unreliable model can create service failures, regulatory exposure, cybersecurity vulnerabilities, and discriminatory outcomes. Automation may also weaken employee expertise when staff are discouraged from exercising professional judgment. Poorly governed AI can create accountability gaps in which developers, managers, vendors, and operators each assume that another party is responsible.

Organizational harms may affect different groups unevenly. A cost-saving automation project may benefit shareholders and senior management while increasing workload, surveillance, or job insecurity for employees. Therefore, organizational benefit should not be treated as equivalent to social benefit.

Societal harms

Societal harms affect communities, institutions, democratic processes, and patterns of social opportunity. Examples include unequal access to public services, amplification of disinformation, concentration of economic power, erosion of trust in institutions, and normalization of pervasive surveillance. Some harms are distributive: benefits and burdens are allocated unequally. Others are relational: they alter the status, voice, or social standing of groups. Still others are systemic, emerging from many individually acceptable decisions that collectively produce an undesirable society.

Environmental harms

AI systems require physical infrastructure. Training and operating models may consume electricity, water, hardware, and data-centre capacity. Environmental analysis should consider the complete lifecycle: manufacturing equipment, data collection, model training, repeated experimentation, deployment, maintenance, and disposal. The relevant question is not merely how much energy a single query consumes, but whether the scale and purpose of deployment justify the environmental burden and who bears it.

Environmental harms also include e-waste, mining impacts, local water stress, and carbon emissions associated with energy sources. Efficiency improvements are valuable, but efficiency can increase total consumption if lower costs lead to much greater use—a phenomenon sometimes described as rebound.

Important Distinctions in Impact Analysis

Several distinctions improve the quality of ethical reasoning.

Direct and indirect harm: A wrong medical recommendation is direct; reduced access to human clinicians because of automation may be indirect. Indirect effects are often harder to detect because they pass through institutional practices.

Intentional and unintended harm: A harmful outcome may result from malicious use, negligence, or reasonable design choices interacting with an unexpected context. Lack of harmful intent does not remove responsibility.

Reversible and irreversible harm: A corrected recommendation may be reversible, whereas loss of a job opportunity, disclosure of sensitive data, or physical injury may be difficult or impossible to repair.

Acute and cumulative harm: A single harmful decision may be serious, but repeated low-level errors can create cumulative disadvantage. Monitoring must therefore examine patterns over time.

Known and uncertain harm: Some risks can be estimated from historical data; others involve novel capabilities, uncertain social responses, or inadequate evidence. Uncertainty should increase the need for precaution rather than be treated as evidence of safety.

Stakeholder Mapping: From Technical System to Social System

A stakeholder is any person, group, institution, community, or ecological interest that can affect or be affected by an AI system. Stakeholder analysis begins by widening the frame beyond the system owner and end user.

A practical stakeholder map may include:

1. Direct users: people who operate or interact with the system.

2. Affected persons: individuals evaluated, classified, monitored, or acted upon.

3. Rights-holders and vulnerable groups: people whose rights or opportunities may be especially exposed.

4. Design and supply-chain actors: data providers, developers, annotators, infrastructure vendors, and model providers.

5. Decision-makers: managers, public officials, clinicians, teachers, or judges who rely on outputs.

6. Oversight actors: auditors, regulators, ethics committees, courts, and civil-society organizations.

7. Indirect and future stakeholders: families, communities, future workers, and environmental interests.

For each stakeholder, analysts should ask: What benefits and harms are plausible? What information do they possess? What power do they have to influence the system? How vulnerable are they? Can they consent, opt out, appeal, or obtain remedy? Are their interests represented in design and governance?

A power-interest matrix can help prioritize engagement, but it should not be used to ignore stakeholders with low power. Those most affected may have the least institutional influence. Participatory methods such as interviews, community workshops, user testing, and independent consultation can correct this imbalance.

Stakeholder mapping should be revisited at major lifecycle transitions. The affected population during a pilot may differ from the population after national deployment. A vendor may also become a critical stakeholder when an organization loses the ability to inspect a proprietary model.

Risk Classification and Prioritization

Risk is commonly analysed through the relationship between the likelihood of an adverse event and its consequences. A simple formulation is:

Risk priority = likelihood × impact

This expression is useful for organizing attention but should not be interpreted as a complete ethical calculation. Low-probability events may deserve urgent action when consequences are catastrophic, rights are fundamental, or uncertainty is extreme. In addition to likelihood and impact, analysts should consider exposure, duration, reversibility, affected population, detectability, and the availability of remedy.

A risk register can record:

• the hazard and causal mechanism;
• affected stakeholders;
• likelihood and severity estimates;
• evidence and uncertainty;
• existing controls;
• responsible owners;
• residual risk after mitigation;
• monitoring indicators and review dates.

Risk classes may be organized as low, moderate, high, and unacceptable, or through more specialized categories such as safety-critical, rights-impacting, privacy-sensitive, discriminatory, cybersecurity, environmental, and systemic risk. Classification must be context-dependent. A small error rate in entertainment recommendations is not equivalent to the same rate in emergency triage or employment screening.

Risk assessment should distinguish inherent risk, present before controls, from residual risk, remaining after controls. It should also identify risk escalation. A model initially used to support a human decision may become a de facto automated decision system if workload, organizational culture, or interface design causes users to follow it automatically.

Ethical Impact Assessment Across the AI Lifecycle

An ethical impact assessment is a structured process for identifying, evaluating, mitigating, documenting, and monitoring the consequences of an AI system. It complements technical validation and legal compliance; it does not replace either.

1. Define purpose and necessity

The organization should state what problem the system addresses, why AI is necessary, what alternatives exist, and what uses are prohibited. Vague objectives encourage function creep. Necessity requires asking whether the same legitimate aim can be achieved with less intrusive or less risky means.

2. Describe the system and context

Documentation should cover data sources, model type, users, affected populations, decision thresholds, integrations, human roles, deployment environment, and expected scale. The same model may have different ethical implications in different contexts.

3. Identify stakeholders and harms

Stakeholder evidence should be combined with technical threat modelling, historical outcome analysis, domain expertise, and consultation. Analysts should look for both intended outputs and second-order effects.

4. Evaluate rights, values, and distribution

Relevant values may include fairness, privacy, autonomy, safety, transparency, accountability, dignity, sustainability, and solidarity. The analysis should ask who benefits, who bears risk, and whether burdens fall disproportionately on groups already disadvantaged.

5. Design mitigations

Mitigations can be technical, procedural, organizational, or legal. Examples include data minimization, representative evaluation, access controls, human review, model restrictions, user training, independent audits, appeal mechanisms, incident reporting, and system non-deployment.

6. Decide and document

The decision should identify the responsible authority, evidence considered, unresolved uncertainties, accepted residual risks, and conditions for deployment. “Human in the loop” is not sufficient unless the human has competence, time, authority, and genuine ability to disagree with the system.

7. Monitor and revise

Post-deployment monitoring should examine performance drift, subgroup outcomes, complaints, override patterns, security incidents, environmental costs, and unexpected uses. An impact assessment is a living process rather than a one-time approval document.

Trade-Off Analysis Without Ethical Reductionism

AI governance often involves competing values. A fraud-detection model may reduce financial crime while increasing false accusations. A content moderation system may limit harmful material while suppressing legitimate expression. A predictive maintenance system may improve safety while increasing worker surveillance. Trade-offs should be made explicit rather than hidden inside technical objectives.

A defensible trade-off analysis asks:

1. What values conflict?

2. Who receives the benefit and who bears the burden?

3. Is the burden necessary and proportionate to the objective?

4. Are less harmful alternatives available?

5. Are affected people informed and able to challenge the decision?

6. Can benefits and burdens be redistributed or compensated?

7. What uncertainty remains, and who decides whether it is acceptable?

Several ethical perspectives may contribute. A consequentialist analysis compares expected benefits and harms. A rights-based analysis examines whether fundamental rights are violated even when aggregate benefits are large. A justice-based analysis focuses on distribution, equality, and historical disadvantage. A care perspective emphasizes dependency, relationships, vulnerability, and the quality of human attention. In practice, responsible governance often combines these perspectives rather than selecting only one.

Quantitative scoring can support comparison, but it must not conceal value judgments. Assigning a numerical weight to dignity or privacy does not make the decision objective. Deliberation, transparency, affected-party participation, and accountable authority remain essential.

Practical and Professional Applications

In recruitment, impact analysis should examine job-ad targeting, résumé parsing, interview scoring, accessibility, subgroup selection rates, feedback, and routes of appeal. In healthcare, it should include clinical validity, patient consent, demographic performance, professional accountability, and the consequences of automation bias. In financial services, analysts should examine explainability, proxy discrimination, data provenance, adverse-action notices, and access to correction. In public administration, the threshold for deployment should be especially demanding where benefits, immigration status, policing, or liberty are involved.

Professionals should maintain an evidence trail linking system requirements to identified risks and controls. Product managers should include affected communities in requirements gathering. Data scientists should test subgroup performance and document limitations. Engineers should design logging, access controls, rollback procedures, and safe failure modes. Executives must allocate resources and accept accountability for deployment decisions. Legal and compliance teams should interpret regulation alongside broader ethical duties rather than treating compliance as the maximum standard.

Illustrative Case Study: Automated Welfare-Fraud Detection

Background

Consider a fictional public agency that introduces an AI system to identify welfare applications and existing claims that may contain fraud. The system combines application information, household records, employment data, address histories, and prior investigation outcomes. Cases receiving high risk scores are referred for investigation.

Situation and problem

The agency expects to reduce fraud and direct limited investigative resources more efficiently. However, applicants report delayed payments, confusing requests for evidence, and difficulty correcting inaccurate data. Community organizations claim that people with unstable housing, informal employment, language barriers, or complex family arrangements are being referred disproportionately.

Stakeholder analysis

Direct stakeholders include agency investigators and applicants. Other stakeholders include children dependent on benefits, social workers, data suppliers, taxpayers, advocacy groups, elected officials, and communities experiencing poverty. Applicants have high exposure and low power: they may not know why they were selected and may lack the resources to challenge a decision. Investigators have operational power but may have limited ability to inspect the model. Children are indirect stakeholders because interruption of household benefits affects them without their participation.

Risk analysis

Individual risks include loss of income, stigma, stress, and inability to meet basic needs. Organizational risks include legal claims, backlogs, loss of public trust, and dependence on unreliable data suppliers. Societal risks include deepening poverty and unequal treatment of marginalized communities. Key uncertainties concern proxy variables, historical investigation bias, and whether investigators will treat the score as a presumption of guilt.

The risk is high because the consequences are serious, the affected population is vulnerable, and errors may be difficult to reverse. A technically accurate model would not by itself resolve these concerns.

Strategy and safeguards

An ethical impact assessment would first test whether every data source is necessary and lawfully appropriate. The agency should prohibit automatic suspension of benefits based solely on a model score. A trained investigator should review evidence, document reasons, and provide timely notice. Applicants should receive an understandable explanation, access relevant records, correct data, and appeal through an independent process. The agency should conduct subgroup testing, monitor false-positive rates, audit supplier data, and publish aggregate impact information without exposing personal data.

A safer design may use the system to prioritize requests for clarification rather than label people as fraudulent. Where uncertainty is high, the agency should provide temporary support while investigation proceeds. Independent civil-society participation can improve oversight, and deployment should be paused if monitoring reveals unacceptable disparate impact.

Outcome

In this illustrative case, the agency retains some efficiency benefits but changes the system’s role from automated accusation to decision support. The approach does not eliminate all error. It reduces the severity and irreversibility of errors, strengthens procedural rights, and makes responsibility visible.

Learning points

The case demonstrates that ethical design includes purpose limitation, stakeholder voice, human authority, remedy, and monitoring. It also shows why technical accuracy, legal compliance, and ethical acceptability must be analysed separately. The most responsible solution may be to redesign the use case rather than improve the model alone.

Challenges and Common Mistakes

A frequent mistake is treating fairness as a single metric. Different fairness measures can conflict, and statistical parity cannot by itself establish justice. Another mistake is relying on consent where people have no realistic alternative. Notice without comprehension or remedy is weak accountability.

Organizations also underestimate data and institutional bias. Removing protected attributes does not remove proxies such as location, language, education, or purchasing patterns. Small datasets may make subgroup evaluation statistically unstable, but uncertainty should lead to cautious deployment, not silent exclusion of minority groups.

Overreliance on checklists is another danger. A completed assessment may create an appearance of responsibility without changing incentives or decisions. Vendor opacity, automation bias, inadequate documentation, poor incident response, and failure to reassess after system updates are recurring governance weaknesses.

Governance, Regulation, and Emerging Practice

AI impact analysis is increasingly connected to risk-based regulation, privacy and data-protection obligations, sector-specific safety rules, procurement standards, algorithmic accountability requirements, and corporate responsibility frameworks. Exact legal duties vary by jurisdiction and application, but several governance principles are broadly durable: legitimate purpose, necessity, proportionality, data governance, transparency, human oversight, security, accountability, accessibility, and effective remedy.

Emerging practice includes algorithm registers, model cards, data statements, red-teaming, continuous monitoring, independent audits, synthetic-data assessment, energy reporting, and structured incident databases. Generative AI introduces additional concerns involving fabricated content, intellectual property, confidentiality, impersonation, labour displacement, and scalable persuasion. Foundation models also complicate responsibility because a general-purpose model may be adapted by many downstream users in contexts unknown to its original developer.

The future of impact analysis is therefore likely to combine technical evaluation with institutional and participatory methods. Automated monitoring can detect drift and anomalous outcomes, but it cannot decide what a society ought to permit. Ethical governance remains a human and political responsibility.

Best Practices for Practitioners

• Define the legitimate purpose, boundaries, and prohibited uses before model development.
• Map affected stakeholders, especially groups with low power or high vulnerability.
• Assess harms across the complete lifecycle and supply chain.
• Separate technical performance from ethical acceptability.
• Test subgroup outcomes and investigate proxy discrimination.
• Design meaningful explanation, appeal, correction, and remedy mechanisms.
• Ensure human oversight includes competence, authority, time, and independence.
• Record assumptions, uncertainties, decisions, owners, and residual risks.
• Monitor real-world outcomes after deployment and establish stop conditions.
• Prefer less intrusive alternatives and redesign or abandon systems when risks cannot be justified.

Chapter Summary

AI harms arise through the interaction of models, data, institutions, users, and social conditions. They may affect individuals, organizations, society, and the environment, and may be direct, indirect, cumulative, uncertain, reversible, or irreversible. Effective analysis begins with a broad stakeholder map that includes affected people and vulnerable communities, not only system owners and users. Risk classification should consider severity, likelihood, exposure, duration, reversibility, uncertainty, and remedy. Ethical impact assessment should span purpose definition, system description, stakeholder consultation, value analysis, mitigation, accountable decision-making, and post-deployment monitoring. Trade-off analysis must make competing values and distributions of benefit and burden explicit. Responsible AI governance is therefore an ongoing practice of evidence, participation, accountability, and institutional judgment.

Key Terms

AI harm: A negative effect caused wholly or partly by an AI system or its governance.

Affected person: An individual subject to an AI-supported classification, prediction, recommendation, or decision.

Algorithmic discrimination: Unequal treatment or outcomes produced or reinforced by an algorithmic system.

Autonomy: A person’s capacity to make informed and voluntary choices.

Cumulative harm: Disadvantage created through repeated or aggregated effects over time.

Environmental impact: Effects of an AI system on energy, water, materials, emissions, and ecosystems.

Ethical impact assessment: A structured process for identifying, evaluating, mitigating, and monitoring ethical consequences.

Function creep: Expansion of a system beyond its original purpose without adequate reassessment.

Human oversight: Meaningful human capacity to understand, supervise, intervene in, or stop an AI system.

Inherent risk: Risk that exists before mitigation measures are applied.

Irreversible harm: Harm that cannot easily be undone or repaired.

Residual risk: Risk remaining after controls and mitigations.

Risk register: A documented record of hazards, likelihood, impacts, controls, owners, and monitoring.

Stakeholder: Any person, group, institution, or ecological interest affected by or able to affect a system.

Systemic harm: Harm arising from interactions across institutions or society rather than one isolated decision.

Trade-off analysis: Structured examination of competing values, benefits, burdens, and alternatives.

Vulnerability: Increased exposure to harm combined with reduced capacity to prevent or remedy it.

Proxy variable: A feature that indirectly represents a sensitive attribute or social condition.

Review Questions

Short Answer Questions

1. How does ethical harm differ from technical performance failure?

2. Distinguish inherent risk from residual risk.

3. Why should low-power stakeholders be included in stakeholder mapping?

4. What is function creep, and why does it create governance risk?

5. Identify two environmental impacts associated with AI systems.

Long Answer Questions

1. Explain the four major categories of AI harm and illustrate each with an example.

2. Describe the stages of an ethical impact assessment across the AI lifecycle.

3. Discuss why human oversight may fail even when a human formally remains in the decision process.

4. Explain how rights-based, consequentialist, justice-based, and care perspectives contribute to trade-off analysis.

5. Analyse the governance requirements for an AI system used in a high-stakes public-service decision.

Analytical / Application Questions

1. A bank uses an AI model to approve loans. Approval rates are high overall, but applicants from one neighbourhood receive substantially more manual reviews. Design a stakeholder map and identify possible harms, evidence requirements, and mitigation measures.

2. A hospital deploys an AI triage tool that improves average waiting times but performs less reliably for speakers of a minority language. Should deployment continue? Construct a reasoned decision using risk, fairness, remedy, and trade-off considerations.

3. An organization plans to use a generative AI system to monitor employee communications for misconduct. Conduct a concise ethical impact assessment, including necessity, privacy, proportionality, stakeholder interests, alternatives, and conditions for deployment.

22. Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes technical performance failure from ethical harm?

A. Technical failure concerns cost, while ethical harm concerns speed.

B. Technical failure concerns whether a system meets its technical objective; ethical harm concerns negative effects on rights, welfare, dignity, fairness, safety, or the environment.

C. Technical failure is always more serious than ethical harm.

D. Ethical harm occurs only when developers intend to cause it.

Correct Answer: B. Technical failure concerns whether a system meets its technical objective; ethical harm concerns negative effects on rights, welfare, dignity, fairness, safety, or the environment.

Question 2

Why should stakeholder mapping include people with low institutional power?

A. They always control deployment decisions.

B. They are usually the least affected by AI systems.

C. They may face substantial exposure or vulnerability despite having little ability to influence design or governance.

D. They can replace the need for technical testing.

Correct Answer: C. They may face substantial exposure or vulnerability despite having little ability to influence design or governance.

Question 3

Which situation illustrates function creep?

A. A model is tested before deployment.

B. A system designed to detect equipment faults is later used to evaluate employee loyalty without a new impact assessment.

C. A dataset is documented before training.

D. A model is retired after monitoring identifies unacceptable risk.

Correct Answer: B. A system designed to detect equipment faults is later used to evaluate employee loyalty without a new impact assessment.

Question 4

Which factor is especially important when prioritizing a low-probability AI risk?

A. Whether the model has a modern interface.

B. Whether the event could cause catastrophic or irreversible harm.

C. Whether the vendor offers a discount.

D. Whether the system uses a neural network.

Correct Answer: B. Whether the event could cause catastrophic or irreversible harm.

Question 5

What makes human oversight meaningful?

A. A human is named in the system documentation.

B. A human receives the model output but must accept it.

C. The human has competence, sufficient time, authority, and a genuine ability to challenge or stop the system.

D. The system produces a confidence score.

Correct Answer: C. The human has competence, sufficient time, authority, and a genuine ability to challenge or stop the system.

Question 6

In the welfare-fraud case, why is using AI to request clarification safer than automatically suspending benefits?

A. It eliminates all model errors.

B. It reduces the irreversibility and severity of false-positive decisions while preserving an opportunity for review.

C. It removes the need for investigators.

D. It guarantees equal outcomes for every group.

Correct Answer: B. It reduces the irreversibility and severity of false-positive decisions while preserving an opportunity for review.

Question 7

Which question is central to ethical trade-off analysis?

A. Which algorithm has the most parameters?

B. Who receives the benefits and who bears the burdens of the system?

C. Can the system be marketed internationally?

D. Does the system require a graphical interface?

Correct Answer: B. Who receives the benefits and who bears the burdens of the system?

Question 8

Why is removing a protected attribute from training data insufficient to prevent discrimination?

A. Models cannot operate without protected attributes.

B. Other variables may act as proxies for protected or socially sensitive characteristics.

C. Protected attributes are never relevant to fairness analysis.

D. Removal automatically creates disparate impact.

Correct Answer: B. Other variables may act as proxies for protected or socially sensitive characteristics.

Question 9

What is residual risk?

A. Risk that exists before any controls are applied.

B. Risk transferred entirely to a vendor.

C. Risk remaining after mitigation and control measures.

D. Risk that has been proven impossible.

Correct Answer: C. Risk remaining after mitigation and control measures.

Question 10

Which action best reflects a lifecycle approach to ethical impact assessment?

A. Assessing the model only after a public incident.

B. Completing one checklist before training and never revisiting it.

C. Defining purpose, assessing stakeholders and harms, implementing safeguards, and monitoring real-world outcomes after deployment.

D. Evaluating only average accuracy before launch.

Correct Answer: C. Defining purpose, assessing stakeholders and harms, implementing safeguards, and monitoring real-world outcomes after deployment.