Sociotechnical Impacts and Stakeholder Analysis

Introduction

Artificial intelligence is often described as a technical capability: a model classifies images, predicts demand, recommends content, or generates text. In practice, however, AI systems operate within institutions, markets, laws, workplaces, and communities. Their effects arise not only from model accuracy but also from who defines the problem, whose data are collected, how outputs are used, who can challenge decisions, and how benefits and risks are distributed.

A sociotechnical perspective therefore treats an AI system as the combination of computational components and the human, organizational, legal, economic, and cultural arrangements surrounding them. A highly accurate system can still be harmful if it automates an unjust policy, excludes people from participation, intensifies surveillance, or shifts unpaid labor and error correction onto vulnerable groups. Conversely, a technically imperfect system may be valuable when it supports human judgment, is transparently governed, and produces benefits fairly.

This chapter presents practical methods for analyzing these effects. It explains how AI redistributes benefits, risks, power, labor, and opportunity; introduces stakeholder mapping and impact-pathway analysis; and develops methods for contextual inquiry and inclusive consultation. The emphasis is on responsible innovation: designing and deploying AI in ways that are socially responsive, contestable, and attentive to affected communities.

Learning Objectives

By the end of this chapter, students should be able to:

1. Explain why AI systems must be analyzed as sociotechnical systems.

2. Identify how AI redistributes benefits, risks, power, labor, and opportunity.

3. Distinguish users, affected parties, rights-holders, implementers, and other stakeholders.

4. Construct a stakeholder map using influence, impact, dependence, and representation.

5. Trace intended and unintended consequences through an impact pathway.

6. Plan contextual inquiry and inclusive consultation with affected communities.

7. Evaluate an AI proposal using distributional, procedural, and recognition-based criteria.

8. Recommend practical governance and design measures for responsible innovation.

1. AI as a Sociotechnical System

A sociotechnical system includes at least five interacting elements:

• Technical artefacts: models, data pipelines, interfaces, sensors, databases, and infrastructure.
• Human actors: developers, operators, decision-makers, users, workers, and affected people.
• Organizational arrangements: policies, incentives, workflows, budgets, procurement practices, and accountability structures.
• Institutions and regulation: law, professional standards, public authorities, courts, and market rules.
• Social context: culture, history, inequality, language, norms, and community practices.

These elements shape one another. A predictive policing tool, for example, does not merely predict crime. It is embedded in decisions about patrol allocation, reporting practices, police discretion, public trust, and historical patterns of enforcement. The data may reflect past institutional behavior rather than an objective measure of future events.

A useful distinction is between a system’s technical performance and its social performance. Technical performance concerns measures such as accuracy, latency, robustness, or reliability. Social performance concerns whether the system advances legitimate goals, distributes effects fairly, preserves agency, and remains accountable. Technical performance is necessary in many applications, but it is not sufficient for responsible deployment.

Important terminology

• Direct user: a person who operates or interacts with the system.
• Subject: a person represented in the system’s data or whose characteristics are assessed.
• Affected party: anyone whose opportunities, treatment, workload, privacy, or well-being may be changed by the system, whether or not they use it.
• Rights-holder: a person or group whose legally or morally protected interests may be engaged.
• Stakeholder: an individual, group, organization, or institution that can affect, is affected by, or has a legitimate interest in the system.
• Sociotechnical system: a technical system together with the people, practices, institutions, and context in which it operates.
• Responsible innovation: an approach that anticipates consequences, includes relevant publics, reflects on purposes and values, and responds through revision or restraint.
• Contestability: the practical ability to question, appeal, correct, or obtain review of an AI-assisted decision.
• Distributional impact: how benefits, burdens, risks, and opportunities are allocated among groups.
• Procedural impact: how participation, explanation, due process, and accountability are arranged.
• Recognition: whether people’s identities, knowledge, dignity, and social standing are respected.

2. How AI Redistributes Social Goods and Burdens

AI does not affect everyone equally. Its distributional effects can be examined across five connected dimensions.

2.1 Benefits and risks

Benefits may include faster service, improved diagnosis, safer transport, personalized learning, reduced administrative cost, or new creative tools. Risks may include inaccurate decisions, privacy loss, discrimination, security vulnerabilities, manipulation, unsafe automation, and dependency on opaque suppliers.

Benefits and risks can be separated geographically and socially. A company may gain efficiency while workers experience intensified monitoring. A public agency may reduce processing time while applicants face opaque refusals. Consumers may receive convenience while data subjects lose control over personal information. Responsible analysis therefore asks: Who receives the benefit? Who bears the risk? Under what conditions?

2.2 Power and discretion

AI can centralize power in organizations that control data, computing infrastructure, models, and standards. Large firms may gain advantages through economies of scale, while smaller organizations and communities become dependent on proprietary systems. Automated recommendations can also shift discretion: a frontline worker may appear to make a decision even though the system’s ranking strongly constrains the available choices.

Power is not only the ability to make a decision. It also includes the ability to define categories, set thresholds, determine what counts as evidence, and decide whose complaints are credible. A data label such as “high risk” can influence a person’s treatment even when the label is probabilistic and contestable.

2.3 Labor and skill

AI may substitute for some tasks, complement others, create new occupations, and eliminate or devalue forms of expertise. Effects are often uneven within the same occupation. A language model may automate drafting but increase the need for reviewing, fact-checking, prompt design, and accountability. Content moderation systems may filter material at scale while shifting difficult edge cases and psychological harms to human reviewers.

The analysis should include hidden labor. Data annotation, content cleaning, safety testing, customer support, appeals, and error correction are frequently treated as peripheral even though they are essential to system operation. Workers may also experience algorithmic management, in which software assigns tasks, measures performance, schedules shifts, or disciplines workers.

2.4 Opportunity and exclusion

AI can widen access when it provides translation, assistive interfaces, remote services, or affordable expertise. It can also reproduce exclusion through inaccessible design, language limitations, missing data, biased labels, or requirements for digital connectivity. Historical disadvantage may be amplified when past outcomes are used as training targets.

A system’s average performance can conceal severe group-specific failures. Evaluation should therefore consider relevant subgroups, intersectional identities, accessibility needs, and cases where the cost of error is especially high. Disaggregation is not automatically sufficient: small groups may be identifiable, data may be unreliable, and categories may not reflect lived identities.

2.5 Environmental and infrastructural burdens

AI systems require energy, water, hardware, data centers, and electronic equipment. Environmental burdens may be concentrated near extraction sites, data centers, or disposal facilities, while benefits accrue elsewhere. A complete impact analysis considers the system’s life cycle: data collection, training, deployment, maintenance, updating, and retirement.

3. Frameworks for Sociotechnical Analysis

3.1 The distributional–procedural–recognition framework

A practical framework evaluates three dimensions:

1. Distribution: Who gains, loses, bears risk, or receives opportunity?

2. Procedure: Who participates, decides, reviews, explains, and appeals?

3. Recognition: Are people’s identities, knowledge, dignity, language, and historical experiences respected?

These dimensions are related but not interchangeable. A system may distribute benefits equally while excluding affected communities from decision-making. A consultation may be extensive but fail to recognize Indigenous governance or disability expertise. Conversely, respectful representation cannot compensate for severe and avoidable material harm.

3.2 The value-sensitive design perspective

Value-sensitive design integrates human values into conceptual, empirical, and technical investigations. Conceptual investigation identifies values and conflicts, such as privacy versus personalization. Empirical investigation examines how stakeholders understand and experience the system. Technical investigation studies how design features can support or undermine values, for example through access controls, explanation interfaces, or audit logs.

The approach distinguishes direct stakeholders, who interact with the technology, from indirect stakeholders, who may be affected without using it. This distinction prevents designers from treating only paying customers as legitimate participants.

3.3 Responsible innovation

Responsible innovation is commonly organized around anticipation, inclusion, reflexivity, and responsiveness:

• Anticipation: explore plausible consequences, including misuse and second-order effects.
• Inclusion: involve relevant people early enough to influence goals and design.
• Reflexivity: examine assumptions, interests, institutional power, and professional limits.
• Responsiveness: change, pause, restrict, or abandon an innovation when evidence warrants it.

These principles are not a checklist that guarantees ethical outcomes. They are habits of inquiry and governance. Inclusion, for instance, is weak if participants lack information, compensation, accessibility, or actual influence.

4. Stakeholder Mapping

Stakeholder mapping is a structured method for identifying actors, relationships, interests, influence, vulnerabilities, and responsibilities. It should begin before model development, not only before launch.

4.1 Stakeholder categories

A useful initial inventory includes:

• commissioning organizations and funders;
• developers, data scientists, product managers, and vendors;
• operators and frontline professionals;
• direct users and customers;
• people represented in training or operational data;
• people subject to predictions, rankings, or decisions;
• workers whose tasks or conditions may change;
• caregivers, families, and dependants;
• advocacy groups and community organizations;
• regulators, courts, auditors, and professional bodies;
• infrastructure providers and downstream integrators;
• future generations and the natural environment, represented through appropriate institutions.

The list should be generated through document review, process observation, interviews, domain expertise, and community outreach. A single workshop is unlikely to reveal less visible stakeholders.

4.2 Mapping dimensions

For each stakeholder, document:

• expected benefits and harms;
• degree of exposure to system decisions;
• ability to influence design or deployment;
• dependence on the organization or service;
• resources and technical knowledge;
• rights and legitimate interests;
• ability to opt out or appeal;
• barriers to participation;
• trusted representatives and preferred communication channels.

A simple two-axis influence–impact matrix can support prioritization, but it must not be treated as a measure of moral importance. People with low institutional influence may face the greatest harms and therefore require additional protections.

A more complete prioritization asks four questions:

1. Power: Can this stakeholder affect the system or its decisions?

2. Exposure: How strongly can the system affect this stakeholder?

3. Vulnerability: Would harm be difficult to avoid, repair, or contest?

4. Legitimacy: Does the stakeholder have a recognized right or valid interest in the matter?

4.3 Stakeholder mapping process

1. Define the system boundary, including procurement, data collection, operation, and retirement.

2. Map the decision process before and after AI is introduced.

3. List visible and invisible stakeholders at every stage.

4. Record interests, dependencies, risks, and forms of knowledge.

5. Identify conflicts and unequal bargaining power.

6. Validate the map with independent participants and affected communities.

7. Convert findings into engagement, mitigation, and accountability actions.

8. Revisit the map as the system, context, or evidence changes.

5. Impact-Pathway Analysis

Impact-pathway analysis traces how an intervention may produce outcomes. It helps distinguish immediate outputs from longer-term social effects and exposes assumptions that are otherwise hidden.

A generic pathway is:

Need or problem → intervention → technical output → organizational decision → immediate outcome → longer-term impact

For example:

Long emergency-department waiting times → demand-prediction tool → forecast of patient volume → staffing adjustment → shorter average waits → possible improvement in access, but possible deterioration for patients whose needs are poorly represented in historical data.

A strong pathway includes both intended and unintended routes. It also identifies feedback loops. If a risk score leads to increased monitoring, the resulting additional records may make the group appear more “high risk,” reinforcing the original allocation of attention. This is a form of institutional feedback, not proof that the group intrinsically possesses the predicted characteristic.

5.1 Building an impact pathway

For each link, ask:

• What assumption connects these stages?
• What evidence supports the assumption?
• Who controls the decision at this stage?
• Who benefits and who bears the burden?
• What alternatives exist?
• What could fail technically or institutionally?
• What indicators would reveal harm?
• What remedy is available, and who can trigger it?

The analysis should include time horizons. Some impacts are immediate, such as a denied application; others are cumulative, such as reduced employment prospects, loss of trust, or deskilling. It should also consider reversibility. A wrong recommendation may be corrected quickly, while disclosure of sensitive information may be effectively permanent.

6. Contextual Inquiry and Inclusive Consultation

6.1 Contextual inquiry

Contextual inquiry is an empirical method for understanding work and lived experience in the setting where activities occur. Rather than asking only what people claim they do, researchers observe tasks, interruptions, workarounds, informal judgments, communication patterns, and material constraints. Interviews are conducted in context and grounded in actual practice.

For AI projects, contextual inquiry can reveal:

• where data are created and altered;
• how staff handle exceptional cases;
• which judgments are invisible in formal procedures;
• how people currently resolve errors;
• what happens when systems are unavailable;
• who performs unpaid or unrecognized repair work;
• how users interpret uncertainty and recommendations.

Researchers must obtain appropriate consent, minimize collection of personal information, protect confidentiality, and avoid turning observation into employee surveillance. Participation should not expose workers or community members to retaliation.

6.2 Inclusive consultation

Consultation is meaningful when affected people can understand the proposal, express concerns, influence decisions, and receive feedback about what changed. It is not merely collecting opinions after the key design choices have been fixed.

Good practice includes:

• engaging communities during problem definition;
• compensating participants for time and expertise;
• providing accessible formats, interpretation, and disability accommodations;
• using plain language and concrete scenarios;
• offering multiple channels, including non-digital methods;
• working with trusted community organizations without assuming they speak for everyone;
• protecting anonymity where disclosure creates risk;
• reporting back decisions, disagreements, and unresolved issues;
• creating continuing participation during piloting and monitoring.

Participation should be proportionate to impact. A low-risk internal productivity tool may require limited consultation; a system affecting housing, education, employment, health, policing, immigration, or welfare requires deeper engagement and stronger procedural safeguards.

6.3 Avoiding tokenism and participation burden

A diverse meeting does not necessarily produce inclusive governance. Tokenism occurs when a person or group is included symbolically but lacks influence. Participation burden occurs when marginalized people are repeatedly asked to explain harms without compensation, protection, or evidence of change.

Organizations should disclose the decision space: what is fixed, what can change, and who has authority. They should also accept that the responsible outcome may be redesign, restricted use, or non-deployment. Consultation cannot legitimize a system whose fundamental purpose is unacceptable or whose harms cannot be mitigated.

7. Worked Application: Illustrative University Scholarship Screening System

Background

A university proposes an AI system to prioritize scholarship applications. It uses academic records, household information, prior awards, written statements, and predicted financial need. The stated goals are to reduce administrative workload and identify applicants who may need support.

Problem definition

The proposal assumes that prioritization is primarily a prediction problem. Stakeholder analysis shows that it is also a question of institutional values: What counts as merit? How should hardship be understood? Is the objective equal opportunity, efficient allocation, or maximizing graduation probability?

Stakeholder map

Direct users include scholarship officers and applicants using the application portal. Affected parties include applicants, families, recommenders, financial-aid staff, and students whose applications are deprioritized. Indirect stakeholders include disability advocates, student unions, faculty, donors, regulators, and communities with limited access to digital services. The university has high influence; applicants have high exposure but comparatively low influence. Applicants with unstable housing, caregiving responsibilities, interrupted education, disabilities, or limited English proficiency may face elevated vulnerability.

Impact pathway

Application data → model score → officer priority queue → timing and depth of review → funding decision → educational opportunity.

Potential benefits include faster processing and more consistent review. Potential harms include proxy discrimination, inaccurate interpretation of nontraditional educational histories, privacy loss, and automation bias among officers. A feedback effect may occur if prior recipients are used as the target: groups historically under-awarded may appear less likely to be “successful,” reproducing the historical allocation rather than identifying need.

Contextual inquiry

Observation of scholarship staff reveals that they routinely interpret missing data, contact applicants for clarification, and recognize hardship not captured in structured fields. Applicants report that the portal is difficult to use on mobile devices and that they are uncertain how information about disability or family circumstances will be used.

Responsible approach

The university should first clarify whether AI is necessary. It might use a system to organize applications while prohibiting automated rejection or final ranking. Applicants should receive notice that AI assists review, access to meaningful human review, and a channel to correct data. Sensitive attributes should not be used casually as predictive features, but their relevance to fairness assessment and contextual review should be considered under appropriate governance.

Before deployment, the university should conduct subgroup and accessibility testing, assess historical labels for bias, pilot with human oversight, document error patterns, and establish an appeals process. Independent student and disability representatives should participate in evaluation. The system should be suspended if it produces unexplained disparities or if staff cannot adequately review recommendations.

Outcome and learning points

The responsible outcome is not necessarily full automation. A constrained decision-support system may deliver administrative benefits while preserving human judgment and applicant rights. The case demonstrates that stakeholder analysis can change the problem definition, reveal hidden labor, identify feedback loops, and produce a less risky design.

8. Governance, Regulation, and Professional Practice

Sociotechnical analysis must connect to enforceable governance. Relevant obligations vary by jurisdiction, sector, and application, but commonly include privacy and data-protection law, anti-discrimination and equality law, employment protections, consumer protection, accessibility requirements, administrative law, intellectual-property rules, and sector-specific safety obligations. International and national AI frameworks increasingly emphasize risk management, transparency, accountability, human oversight, and impact assessment.

Legal compliance is a baseline, not a complete ethical justification. A system may be lawful yet exploitative, inaccessible, or socially harmful. Conversely, ethical analysis should not make unsupported legal claims; teams should obtain qualified legal advice for high-impact deployments.

Professional practice should establish:

• a named accountable owner with authority to stop deployment;
• documented purpose, scope, data sources, limitations, and intended users;
• impact assessment before procurement or development;
• data governance and retention controls;
• security testing and incident response;
• monitoring for performance and disparate impacts after launch;
• human oversight that is competent and empowered, not merely nominal;
• notification, explanation, correction, and appeal mechanisms where appropriate;
• supplier transparency, audit rights, and exit plans;
• records of consultation, decisions, dissent, and changes.

Human oversight is meaningful only when people understand the system, have enough time and authority to disagree, and are not penalized for overriding it. “Human in the loop” is insufficient if the human merely approves recommendations under production pressure.

9. Challenges and Limitations

Incomplete knowledge

Future impacts are uncertain, and communities may disagree about values. Scenario analysis and monitoring reduce uncertainty but cannot eliminate it.

Representation problems

No consultation can include everyone. Organizations must explain how participants were selected, whose interests remain underrepresented, and how representatives were authorized.

Power asymmetry

Participants may fear losing services, employment, or access if they criticize a system. Confidential channels, independent facilitators, and non-retaliation protections are therefore important.

Measuring social effects

Some effects, such as dignity, trust, or chilling of speech, are difficult to quantify. Quantitative indicators should be combined with qualitative evidence, complaints, case reviews, and community testimony.

Trade-offs and value conflict

Improving one fairness measure can worsen another, and privacy protections may reduce some forms of accuracy. Teams should make trade-offs explicit rather than claiming that a single metric resolves them.

Scope creep and function creep

A system introduced for one purpose may later be used for another. Purpose limitation, change-control procedures, and periodic reauthorization help prevent unauthorized expansion.

10. Best-Practice Workflow

A practical workflow for bachelor-level projects and professional teams is:

1. Define the social problem: verify that the problem is real and that AI is an appropriate intervention.

2. Set boundaries: specify users, affected groups, decisions, data, geography, and time period.

3. Map stakeholders: include low-power and indirectly affected groups.

4. Conduct contextual inquiry: observe existing practices and hidden labor.

5. Build impact pathways: trace benefits, harms, assumptions, feedback loops, and remedies.

6. Consult inclusively: provide accessible information, compensation, and genuine influence.

7. Assess alternatives: compare automation with non-AI, human-only, and lower-risk designs.

8. Specify safeguards: define oversight, access controls, appeals, monitoring, and stop conditions.

9. Pilot cautiously: use limited deployment, independent evaluation, and incident reporting.

10. Monitor and revise: review outcomes, complaints, distributional effects, and changed context.

11. Retire responsibly: remove systems that no longer meet their purpose or cannot be governed safely.

Contemporary Developments

Current AI systems, especially foundation models and generative systems, intensify stakeholder challenges because their training data and downstream uses are broad and difficult to trace. New concerns include synthetic media, automated persuasion, model extraction, copyright disputes, energy demand, and the use of AI outputs in high-stakes workflows without verification. At the same time, participatory data governance, algorithmic impact assessments, model documentation, red-teaming, accessibility engineering, and community-led technology evaluation are expanding the practical toolkit.

The central contemporary shift is from evaluating an isolated model to governing an evolving system. Updates, fine-tuning, integrations, user adaptations, and organizational incentives can change impacts after launch. Responsible innovation must therefore be continuous rather than a one-time approval event.

Chapter Summary

AI systems are sociotechnical arrangements whose effects depend on technical design, institutional context, human practice, and social history. They redistribute benefits, risks, power, labor, opportunity, and environmental burdens. Analysis should therefore examine not only accuracy but also distributional outcomes, procedures of participation and appeal, and recognition of diverse identities and knowledge.

Stakeholder mapping identifies direct and indirect users, affected parties, rights-holders, workers, communities, organizations, regulators, and less visible contributors. Influence–impact analysis is useful for planning engagement, but low-power groups with high exposure must not be overlooked. Impact-pathway analysis traces how a technical output becomes an organizational decision and then a social outcome, including feedback loops and unintended effects.

Contextual inquiry reveals real practices, informal work, workarounds, and lived constraints. Inclusive consultation requires accessible information, compensation, protection, genuine influence, and feedback. Responsible innovation emphasizes anticipation, inclusion, reflexivity, and responsiveness. Good governance connects these methods to accountability, monitoring, contestability, privacy, accessibility, security, and the authority to pause or discontinue a system.

Key Terms

Sociotechnical system: A technical system together with the people, institutions, practices, and context surrounding it.

Stakeholder: An individual, group, organization, or institution that can affect, is affected by, or has a legitimate interest in a system.

Affected party: Anyone whose treatment, opportunity, workload, privacy, or well-being may change because of a system.

Rights-holder: A person or group with legally or morally protected interests relevant to a decision.

Distributional impact: The allocation of benefits, burdens, risks, and opportunities among groups.

Procedural impact: Effects on participation, explanation, due process, review, and accountability.

Recognition: Respect for people’s identities, dignity, knowledge, language, and social standing.

Responsible innovation: Innovation that anticipates consequences, includes publics, reflects on values, and responds to evidence.

Contestability: The ability to question, appeal, correct, or obtain review of an AI-assisted decision.

Stakeholder map: A structured representation of stakeholders, interests, influence, exposure, and relationships.

Impact pathway: A chain connecting a problem and intervention to outputs, decisions, outcomes, and longer-term impacts.

Contextual inquiry: Observation and interviewing in the setting where activities and decisions occur.

Inclusive consultation: Participation designed to give affected people accessible information and meaningful influence.

Algorithmic management: Use of software to assign, monitor, evaluate, schedule, or discipline workers.

Automation bias: The tendency to over-trust or insufficiently question automated recommendations.

Function creep: Expansion of a system to purposes beyond those originally authorized.

Feedback loop: A process in which system outputs alter future data or decisions, reinforcing or changing later outputs.

Human oversight: Competent, empowered human review capable of understanding and overriding system recommendations.

Review Questions

Short Answer Questions

1. Why is technical accuracy insufficient for evaluating an AI system?

2. Distinguish a direct user from an affected party.

3. What are the three dimensions in the distributional–procedural–recognition framework?

4. What is the purpose of an impact pathway?

5. Give two characteristics of meaningful inclusive consultation.

Long Answer Questions

1. Explain how AI can redistribute power and labor within an organization. Use examples.

2. Describe a complete stakeholder-mapping process for an AI system used in public services.

3. Explain how contextual inquiry can reveal risks that are not visible in technical documentation.

4. Discuss the relationship between responsible innovation and legal compliance.

5. Analyze the university scholarship-screening case using distributional, procedural, and recognition-based criteria.

Analytical / Application Questions

1. Choose an AI system used in employment, education, health, or policing. Construct an impact pathway showing at least two intended and two unintended effects.

2. Create a stakeholder map for a generative-AI tool introduced into a university. Identify stakeholders with high exposure but low influence and propose safeguards.

3. A company claims that a human reviewer makes its automated decisions ethical. Design an inquiry to test whether human oversight is meaningful in practice.

Multiple Choice Questions with Answers

Question 1

What best explains why AI should be analyzed as a sociotechnical system?

A. AI models never require human operators

B. Social outcomes depend on technical components together with institutions, practices, and context

C. Technical accuracy is determined entirely by culture

D. Only legal rules affect AI outcomes

Correct Answer: B. Social outcomes depend on technical components together with institutions, practices, and context

Question 2

Which stakeholder may have high exposure to an AI system but relatively low influence over its design?

A. The system vendor

B. The commissioning executive

C. An applicant subject to an automated ranking

D. The product manager

Correct Answer: C. An applicant subject to an automated ranking

Question 3

Which question primarily examines distributional impact?

A. Who receives benefits and who bears the risks?

B. Who designed the user interface color scheme?

C. How quickly does the model produce an output?

D. Which programming language was used?

Correct Answer: A. Who receives benefits and who bears the risks?

Question 4

In an impact pathway, what normally follows a technical output?

A. An organizational decision or action

B. The retirement of every stakeholder

C. A guarantee of fairness

D. The elimination of uncertainty

Correct Answer: A. An organizational decision or action

Question 5

What is a defining feature of meaningful consultation?

A. It occurs only after deployment is complete

B. Participants are included symbolically but cannot affect decisions

C. Participants receive accessible information and can influence relevant choices

D. Only technical experts are allowed to participate

Correct Answer: C. Participants receive accessible information and can influence relevant choices

Question 6

Which activity is characteristic of contextual inquiry?

A. Observing work and interviewing people in the setting where activities occur

B. Testing only model accuracy on a benchmark dataset

C. Replacing all interviews with anonymous online ratings

D. Reviewing only the vendor’s marketing material

Correct Answer: A. Observing work and interviewing people in the setting where activities occur

Question 7

What is automation bias?

A. The deliberate removal of all human judgment

B. The tendency to over-trust automated recommendations

C. A guarantee that automation treats groups equally

D. The use of automation to improve accessibility

Correct Answer: B. The tendency to over-trust automated recommendations

Question 8

Why can using past scholarship recipients as a prediction target reproduce disadvantage?

A. Past recipients may reflect historical inequalities in access and allocation

B. Historical data are always more accurate than current data

C. Prediction targets cannot contain any social information

D. Scholarship systems do not involve stakeholders

Correct Answer: A. Past recipients may reflect historical inequalities in access and allocation

Question 9

Which condition is necessary for meaningful human oversight?

A. Reviewers must be able to understand and override recommendations

B. Reviewers must approve every recommendation automatically

C. Reviewers should be denied information about model limitations

D. Reviewers must never document disagreements

Correct Answer: A. Reviewers must be able to understand and override recommendations

Question 10

What is function creep?

A. A reduction in model processing speed

B. Expansion of a system to purposes beyond those originally authorized

C. A method for compensating consultation participants

D. The replacement of data with human testimony

Correct Answer: B. Expansion of a system to purposes beyond those originally authorized