Transparency, Explainability, and Contestability

Introduction

AI systems increasingly influence decisions about employment, education, credit, insurance, healthcare, public benefits, policing, and online information. In these settings, people need more than an accurate prediction. They need to know when an AI system is being used, what information it relies on, how its output should be understood, who is responsible for it, and what can be done when the result is wrong or harmful.

These needs are often described using terms such as transparency, interpretability, explainability, traceability, disclosure, and contestability. The terms are related but not interchangeable. A model may be transparent without being understandable to an ordinary user. It may be explainable without being genuinely contestable. A system may provide a detailed explanation while concealing the fact that its training data were biased or that a human decision-maker merely rubber-stamped an automated recommendation.

Responsible innovation therefore treats explanation as part of a broader sociotechnical governance system. Good practice combines technical documentation, meaningful notices, human oversight, record keeping, accessible communication, and effective appeal mechanisms. This chapter develops those ideas and applies them to the design and evaluation of responsible AI systems.

Learning Objectives

By the end of this chapter, students should be able to:

1. Distinguish transparency, disclosure, interpretability, explainability, traceability, and contestability.

2. Explain why different stakeholders require different forms of information.

3. Compare intrinsic and post hoc explanation methods and identify their limitations.

4. Design practical documentation and user notices for an AI-enabled service.

5. Evaluate whether an appeal process provides meaningful contestability.

6. Analyse ethical, legal, technical, and organisational responsibilities surrounding AI explanations.

1. A Conceptual Map of Openness and Accountability

1.1 Transparency

Transparency is the condition in which relevant information about an AI system is made available to appropriate stakeholders. Relevant information can include the system’s purpose, scope, inputs, outputs, performance, limitations, ownership, data practices, and governance arrangements.

Transparency is not equivalent to publishing source code. Source code may be difficult to understand, may omit training data and operational procedures, and may reveal little about how a deployed system behaves in practice. Conversely, a system can be transparent through carefully prepared documentation even when its proprietary source code is not publicly released.

Transparency is also relative to an audience. A regulator may need access to audit logs and validation reports; a product team may need model cards and failure analyses; an affected individual may need a concise explanation of the factors relevant to a particular decision. The appropriate question is not simply, “Is the system transparent?” but “Transparent to whom, about what, and for what purpose?”

1.2 Disclosure and notice

Disclosure is the act of communicating specified information. It is often narrower than transparency. A chatbot disclosure might state that the user is interacting with an AI system. A lending notice might state that an automated scoring tool contributed to an application decision. Disclosure is important because people cannot meaningfully respond to an AI process if they do not know that it exists.

A notice should be timely, visible, comprehensible, and relevant. A vague statement buried in a privacy policy is unlikely to provide meaningful notice. Disclosure should normally identify the system’s role, the consequences of its output, the responsible organisation, and available routes for questions or review.

1.3 Interpretability

Interpretability generally refers to the extent to which a model’s operation can be understood directly from its structure. A small decision tree, a linear model with carefully selected variables, or a rule-based system may be intrinsically interpretable. A human can inspect the model and relate its structure to its outputs.

Interpretability is not absolute. A linear model with thousands of correlated features may be difficult to understand even though its mathematical form is simple. Likewise, a decision tree can become too deep for practical comprehension. Interpretability depends on model complexity, feature meaning, audience expertise, and task context.

1.4 Explainability

Explainability concerns the production of information that helps a person understand why a system generated a particular output or how it behaves generally. An explanation may be generated from an interpretable model or produced after the fact for a complex model.

Two useful categories are:

• Global explanations: describe overall behaviour, such as important features, common rules, error patterns, or performance across groups.
• Local explanations: describe one prediction, such as why a particular application was classified as high risk.

An explanation is not automatically a causal account. If a system identifies a correlation between a postcode and repayment risk, that does not prove that changing the postcode would change the person’s financial behaviour. Explanations must therefore avoid implying causal certainty when only association has been established.

1.5 Traceability

Traceability is the ability to reconstruct the origins, transformations, decisions, and actions associated with an AI system. It can include data provenance, model versions, software dependencies, configuration changes, prediction timestamps, human interventions, and downstream outcomes.

Traceability is especially important for incident investigation. If a person challenges a decision, an organisation should be able to determine which model version was used, what input data were supplied, whether the system was operating within its validated scope, and who authorised the final action. Traceability converts an abstract claim of accountability into inspectable evidence.

1.6 Contestability

Contestability is the practical ability to question, challenge, correct, or seek review of an AI-assisted decision. It is stronger than merely receiving an explanation. A person may understand that an application was rejected because income data were incomplete, yet still lack any means to correct the data or obtain reconsideration.

Meaningful contestability normally requires:

1. Awareness that an AI-supported decision occurred.

2. Sufficient information to identify a potentially relevant error or unfairness.

3. A channel for submitting a challenge.

4. Access to a competent and sufficiently independent reviewer.

5. A genuine possibility that the result will be changed.

6. Protection from retaliation or unreasonable barriers.

7. A response within a reasonable time.

2. Why Explanation Is Difficult

AI explanation is difficult because different goals are often conflated. A developer may want debugging information, a regulator may want evidence of compliance, a manager may want operational assurance, and an affected person may want to know what they can do differently. One explanation cannot satisfy all these purposes equally well.

Explanations also involve trade-offs. A highly detailed explanation may overwhelm a non-specialist or expose sensitive information. A simplified explanation may be accessible but omit important uncertainty. Revealing too much about a fraud-detection system may enable evasion. Protecting trade secrets may conflict with the need to demonstrate fairness. Responsible design therefore requires proportionality rather than maximal disclosure in every situation.

A useful distinction is between process information and decision information. Process information explains how the system was developed and governed: its data sources, testing, monitoring, and responsible personnel. Decision information concerns a particular outcome: the relevant inputs, principal factors, confidence or uncertainty, applicable thresholds, and routes for correction. Both may be necessary for accountability.

3. Explanation Methods

3.1 Intrinsically interpretable models

Linear or logistic regression, small decision trees, scoring rules, and monotonic rule systems can be interpretable when variables are meaningful and the model remains manageable. Their advantages include auditability, stability, and the possibility of explaining the model before deployment.

However, interpretability can be reduced by proxy variables, interaction effects, transformations, missing-data procedures, and poorly understood features. A coefficient is not automatically a fair or meaningful explanation. In a logistic regression, for example, a coefficient describes the change in log-odds associated with a one-unit increase in a feature, holding other included features constant. This technical statement may not correspond to a simple real-world causal effect, particularly when features are correlated.

3.2 Feature attribution

Feature-attribution methods assign portions of a prediction to input features. They can be used locally for one case or aggregated to describe global patterns. Examples include permutation importance and Shapley-value-based methods.

Permutation importance measures how much predictive performance changes when a feature’s values are disrupted. It can be misleading when features are strongly correlated, because one feature may substitute for another. Shapley-based explanations distribute a prediction difference among features under specified assumptions about the baseline and feature coalitions. Their values depend on the chosen background distribution, treatment of dependent features, and model-access assumptions. They should not be interpreted automatically as causal effects or as proof that the listed features were morally justified.

3.3 Counterfactual explanations

A counterfactual explanation states what change would have been sufficient for a different output, such as: “If verified monthly income had been above a specified threshold, the application would have been referred for approval.” Counterfactuals can be actionable and are often intuitive.

Their limitations are significant. Some suggested changes may be impossible, unlawful, costly, or outside the person’s control. A system may recommend changing age, disability status, nationality, or another immutable characteristic indirectly encoded in a proxy. Counterfactuals also depend on the model’s decision boundary and may ignore uncertainty, institutional constraints, or multiple simultaneous errors. They should identify which variables are actionable and distinguish “possible under the model” from “realistically achievable.”

3.4 Surrogate models

A surrogate model approximates a complex model in order to make its behaviour easier to describe. For example, a simple tree may be fitted to imitate a neural network’s outputs. Surrogates can be useful for broad patterns or local regions, but fidelity must be measured. A simple explanation that does not accurately represent the original model is misleading. Local fidelity may also fail outside the narrow region used to generate the explanation.

3.5 Example-based and concept-based explanations

An example-based explanation compares a case with similar cases, prototypes, or influential training examples. Concept-based explanations describe higher-level properties, such as image texture or document structure, rather than raw features. These methods can aid human understanding, but similarity does not establish fairness or causality. The choice of comparison cases can itself encode bias, and examples may disclose personal or confidential information.

3.6 Natural-language explanations

Language models can convert technical outputs into conversational explanations. This improves accessibility but introduces a serious risk: fluent text may invent reasons, overstate certainty, or conceal ambiguity. Natural-language explanations should be generated from verified system records, constrained by templates where appropriate, and evaluated for factual faithfulness. A polished explanation is not evidence that the underlying account is correct.

4. Evaluating Explanations

An explanation should be evaluated as a communication and governance artefact, not only as a technical output. Important criteria include:

• Fidelity: Does it accurately reflect the behaviour of the underlying system?
• Stability: Does it remain reasonably consistent when insignificant inputs change?
• Completeness: Does it include the factors necessary for the intended decision or review?
• Comprehensibility: Can the intended audience understand it?
• Actionability: Does it identify realistic steps for correction or improvement?
• Consistency: Are comparable cases explained in compatible ways?
• Non-discrimination: Does it avoid presenting protected attributes or unjustified proxies as legitimate reasons?
• Privacy: Does it avoid disclosing another person’s confidential information?
• Security: Does it avoid enabling manipulation or evasion beyond what is necessary for contestability?
• Uncertainty: Does it communicate limitations, confidence, and the possibility of error?

These criteria may conflict. A highly faithful explanation could be too technical. A concise explanation could omit uncertainty. An actionable explanation could expose security vulnerabilities. Design should therefore begin by identifying the audience, decision stakes, rights affected, and foreseeable misuse.

Explanations can also be tested empirically. User studies may assess whether people can predict the model’s behaviour, identify errors, make informed decisions, or successfully challenge an outcome. Such tests are more meaningful than assuming that a technically sophisticated explanation is automatically useful.

5. Documentation Across the AI Lifecycle

Responsible transparency is continuous. Documentation should be created before deployment, maintained during operation, and preserved after incidents or retirement.

5.1 System-purpose statement

The purpose statement should define the intended use, excluded uses, affected populations, decision authority, and consequences of error. “Assess applicants” is too vague. A stronger statement specifies whether the system recommends, ranks, flags, or makes a decision; which staff use the output; and what actions remain prohibited without human review.

5.2 Data documentation

Data documentation should record sources, collection methods, permissions, representativeness, labelling procedures, missingness, known errors, sensitive attributes, retention periods, and transformations. It should identify whether data describe historical decisions that may themselves reflect discrimination.

5.3 Model documentation

A model card or equivalent record can include model type, training objective, evaluation datasets, subgroup performance, calibration, limitations, intended users, out-of-distribution risks, and monitoring plans. Documentation should state what the model has not been validated to do.

5.4 Impact and risk assessment

An impact assessment examines foreseeable effects on rights, safety, equality, autonomy, privacy, and access to services. It should include affected communities where feasible, not merely the technical development team. High-impact uses require stronger evidence, oversight, and correction mechanisms.

5.5 Operational records

Operational logs should support reconstruction without collecting unnecessary personal data. Relevant records may include model and policy versions, input provenance, output, confidence or score where appropriate, overrides, alerts, human decisions, explanations shown to users, and correction requests. Retention should be proportionate and governed by access controls.

6. Designing Meaningful User Notices and Appeals

A useful notice answers five questions:

1. What is happening? An AI system is being used, and its role is described plainly.

2. Why is it being used? The purpose and expected benefit are stated.

3. What does it affect? The decision, service, or opportunity at stake is identified.

4. Who is responsible? A named organisation or contact route is provided.

5. What can I do? The person is told how to obtain information, correct data, request human review, and challenge the outcome.

Notices should be layered. A short explanation can appear at the point of interaction, with links to fuller technical and governance documentation. Translation, disability access, and alternative channels are part of meaningful communication rather than optional extras.

An appeal process should not merely send a complaint back to the same automated workflow. It should provide a human reviewer with authority, time, relevant records, and training. The reviewer should consider new evidence and be able to depart from the model. The organisation should track appeal outcomes, reversal rates, recurring data errors, and differences across groups. A process that never changes decisions may indicate either exceptional accuracy or a merely symbolic appeal mechanism; evidence is needed to distinguish the two.

7. Worked Application: Automated University Scholarship Screening

The following is an illustrative case.

Background

A university introduces an AI system to prioritise scholarship applications. The system predicts the likelihood that an applicant will meet financial-need and academic eligibility criteria. It uses application information, academic records, household data, and historical scholarship outcomes. Staff make the final decisions, but they review high-ranked applications first.

Problem

Applicants receive a generic message stating that their application was “not prioritised.” Students do not know that an AI ranking contributed to the outcome. An audit finds that applicants from certain neighbourhoods are less likely to be shortlisted. The neighbourhood variable was included because it improved historical prediction, but it may reflect unequal access to schools and services. Some applicants also report that household income data were outdated.

Analysis

The system has a transparency failure because its AI role and purpose were not clearly disclosed. It has an explainability problem because “not prioritised” does not identify relevant factors or uncertainty. It has a traceability weakness because staff cannot consistently identify which data version or model score was used. Most importantly, contestability is weak: applicants cannot correct data before the shortlist is finalised, and reviewers have no formal process for reconsideration.

A feature-attribution explanation showing that neighbourhood contributed strongly might describe the model but would not justify using neighbourhood as a fair decision factor. A counterfactual such as “a different postcode would have increased priority” would be inappropriate because changing residence may be impossible and would expose the proxy nature of the criterion.

Approach and solution

The university should first define the system as a prioritisation aid rather than an eligibility decision-maker. It should remove or carefully justify features that act as proxies for protected or socioeconomic status, and evaluate performance and error rates across relevant groups. It should validate the system on current data rather than relying only on historical outcomes.

Applicants should receive a point-of-use notice stating that an AI tool assists scholarship prioritisation, identifying the university as responsible. They should be told the principal legitimate factors considered, that the output is not a final decision, and how to correct records or request review. The notice should avoid implying that applicants can simply change immutable circumstances.

Each application should generate a retained record containing the model version, data used, score or ranking, missing-data flags, staff action, and any subsequent correction. A trained scholarship officer independent of the initial ranking should review challenges. The officer should be able to reconsider the application using corrected data and relevant contextual evidence. The university should monitor reversal rates, processing times, and patterns of complaints, and publish an accessible summary of system performance and changes.

Outcome

After implementation, several applications are reprioritised because income records were corrected. The university discovers that applicants with interrupted schooling were systematically underrepresented in the historical training outcomes and modifies the evaluation process. The system remains in use, but only within a narrower, documented role and with a formal human review route.

Learning points

This case illustrates that explanation is not merely a technical add-on. A feature list cannot repair an unfair objective; a notice cannot compensate for an absent appeal; and human involvement is not meaningful if reviewers lack authority. Transparency, traceability, explainability, and contestability must be designed together.

8. Governance, Regulation, and Professional Responsibility

Legal requirements vary by jurisdiction and application. Data protection, equality, consumer protection, administrative law, sector-specific rules, and emerging AI legislation may all apply. Some regimes require information about automated decision-making, risk management, human oversight, record keeping, or impact assessment. Compliance should not be reduced to finding a single “right to explanation,” because legal duties differ according to context and may include access, correction, reasons, review, or non-discrimination obligations.

Professional responsibility extends beyond legal minimums. Developers should avoid deploying systems whose behaviour cannot be adequately monitored in a high-stakes context. Product managers should budget for documentation, user research, appeals, and post-deployment monitoring. Auditors should inspect actual operation rather than relying solely on policy statements. Leaders should assign responsibility clearly and ensure that commercial confidentiality does not become a blanket excuse for withholding necessary information.

Human oversight must be substantive. A person who merely accepts a score without understanding its limitations does not provide effective oversight. Oversight requires competence, access to evidence, freedom from inappropriate performance pressure, and authority to override or suspend the system.

9. Contemporary Developments and Persistent Limitations

Generative AI has expanded the demand for explanations while making reliability more difficult. Systems can produce plausible but unsupported rationales, summarise records selectively, or adapt explanations to persuade rather than inform. Retrieval from verified records, citation of evidence, confidence communication, and human review are important safeguards.

Privacy-preserving transparency is another growing concern. Detailed logs and explanations may contain personal data. Organisations should apply access controls, minimisation, redaction, secure audit mechanisms, and retention limits. Transparency to an affected person does not require exposing another person’s information.

There is also increasing interest in algorithmic impact assessments, transparency registers, provenance standards, and documentation formats that support interoperability between developers, deployers, auditors, and regulators. These tools are valuable only when kept current and connected to operational practice.

No explanation method solves all accountability problems. Explanations can create false reassurance, encourage automation bias, reveal sensitive information, or shift responsibility onto affected individuals. Some harms arise from the decision objective, institutional policy, or unequal social conditions rather than from a model’s internal mechanics. In such cases, the correct response may be to redesign or discontinue the system, not to produce a better explanation.

10. Best-Practice Design Principles

A responsible organisation should:

1. Define the system’s purpose, authority, scope, and prohibited uses before deployment.

2. Match explanations to stakeholder needs and decision stakes.

3. Disclose AI involvement at the point where it matters.

4. Use plain language while preserving material qualifications and uncertainty.

5. Document data provenance, model versions, limitations, and performance across relevant groups.

6. Prefer actionable explanations that do not suggest impossible or discriminatory changes.

7. Test explanations for fidelity, comprehension, stability, accessibility, and misuse.

8. Maintain logs sufficient for investigation while protecting privacy.

9. Provide accessible correction and appeal routes with independent, empowered review.

10. Monitor outcomes after deployment and revise or withdraw systems when harms cannot be controlled.

Chapter Summary

Transparency makes relevant information about an AI system available; disclosure communicates particular information; interpretability concerns understanding a model from its structure; explainability concerns communicating its behaviour or a particular output; traceability enables reconstruction of data, model, and decision histories; and contestability enables people to challenge and potentially change outcomes.

Explanation methods include intrinsically interpretable models, feature attribution, counterfactuals, surrogate models, example-based methods, and natural-language explanations. Each has assumptions and limitations. Explanations should be evaluated for fidelity, stability, completeness, comprehensibility, actionability, fairness, privacy, security, and uncertainty.

Responsible practice requires lifecycle documentation, point-of-use notices, operational records, competent human oversight, and appeal mechanisms with real authority. Technical explanation cannot substitute for fair objectives, accurate data, accountable institutions, or the possibility of correction. In high-impact settings, meaningful contestability is a central measure of responsible innovation.

Key Terms

Transparency: Availability of relevant information about an AI system to appropriate stakeholders.

Disclosure: Communication that a specified fact, such as AI involvement, is being made known.

Interpretability: The extent to which a model can be understood directly from its structure.

Explainability: The provision of information that helps people understand model behaviour or an output.

Global explanation: An account of how a system generally behaves.

Local explanation: An account of why a system produced a particular output.

Traceability: The ability to reconstruct data, model, decision, and action histories.

Contestability: The practical ability to question, correct, appeal, or seek review of an AI-supported decision.

Model card: Documentation describing a model’s purpose, evaluation, limitations, and appropriate use.

Data provenance: Information about the origin, collection, transformation, and use of data.

Feature attribution: Assignment of portions of a prediction to input features.

Counterfactual explanation: A statement of what change would have produced a different outcome under a model.

Surrogate model: A simpler model used to approximate a complex model’s behaviour.

Fidelity: The degree to which an explanation accurately represents the underlying system.

Actionability: The extent to which an explanation identifies realistic steps a person can take.

Proxy variable: A variable that indirectly represents another attribute, often including a sensitive characteristic.

Human oversight: Human capacity and authority to understand, review, override, or stop an AI system.

Layered notice: Communication that provides concise information first and fuller details through additional materials.

Review Questions

Short Answer Questions

1. How does transparency differ from disclosure?

2. Distinguish interpretability from explainability.

3. What is the purpose of traceability in an AI system?

4. State two requirements of meaningful contestability.

5. Why can a counterfactual explanation be misleading?

Long Answer Questions

1. Compare global and local explanations, including their uses and limitations.

2. Explain why publishing source code alone does not guarantee transparency.

3. Discuss the criteria that should be used to evaluate an explanation method.

4. Describe the documentation that should be maintained across the AI lifecycle.

5. Explain why human involvement does not necessarily constitute meaningful human oversight.

Analytical / Application Questions

1. A bank provides applicants with a feature-attribution chart after rejecting a loan application. Design a fuller notice and appeal process that would make the decision more contestable.

2. An employer uses a model that ranks applicants partly on the basis of inferred commuting distance. Analyse the potential proxy, fairness, explanation, and governance risks.

3. Evaluate whether a counterfactual explanation is appropriate for a public-benefits decision when the suggested change is to move to a different neighbourhood. Propose a more responsible alternative.

Multiple Choice Questions with Answers

Question 1

Which concept refers most directly to reconstructing the data, model version, and actions involved in an AI decision?

A. Interpretability

B. Traceability

C. Disclosure

D. Actionability

Correct Answer: B. Traceability

Question 2

What is a local explanation intended to describe?

A. The complete history of an organisation’s AI policy

B. The general social effects of automation

C. Why a particular input received a particular output

D. The source code of every deployed model

Correct Answer: C. Why a particular input received a particular output

Question 3

Which is a major limitation of feature-attribution methods?

A. They can never be applied to complex models

B. They always establish causal relationships

C. Their results can depend on correlated features and baseline choices

D. They provide no information about individual predictions

Correct Answer: C. Their results can depend on correlated features and baseline choices

Question 4

Which feature is essential to meaningful contestability?

A. A longer privacy policy

B. A genuine possibility that review can change the outcome

C. Automatic acceptance of every complaint

D. Removal of all human decision-makers

Correct Answer: B. A genuine possibility that review can change the outcome

Question 5

Why is publishing source code insufficient to guarantee transparency?

A. Source code is always illegal to disclose

B. Code does not necessarily reveal data, deployment, limitations, or operational procedures

C. Source code cannot contain mathematical operations

D. Transparency concerns only marketing information

Correct Answer: B. Code does not necessarily reveal data, deployment, limitations, or operational procedures

Question 6

Which explanation is generally most actionable when its suggested changes are feasible and within the person’s control?

A. A counterfactual explanation

B. A source-code listing

C. A training-data catalogue

D. A generic system disclaimer

Correct Answer: A. A counterfactual explanation

Question 7

What should a responsible user notice identify?

A. Only the model’s programming language

B. The AI’s role, purpose, responsible organisation, and review routes

C. Every individual record in the training dataset

D. A guarantee that the output is correct

Correct Answer: B. The AI’s role, purpose, responsible organisation, and review routes

Question 8

What does fidelity mean when evaluating an explanation?

A. How attractive the explanation looks

B. How closely the explanation reflects the underlying system’s behaviour

C. How quickly the explanation is delivered

D. How much proprietary information it reveals

Correct Answer: B. How closely the explanation reflects the underlying system’s behaviour

Question 9

In the scholarship-screening case, why was a neighbourhood feature ethically problematic?

A. Neighbourhood data can never be stored digitally

B. It might act as a proxy for unequal social conditions or protected characteristics

C. It necessarily improves accuracy too much

D. It prevents any form of human review

Correct Answer: B. It might act as a proxy for unequal social conditions or protected characteristics

Question 10

Which statement best describes meaningful human oversight?

A. A person automatically accepts the model’s recommendation

B. A person has competence, evidence, independence, and authority to override the system

C. A person signs a document without seeing the output

D. A person is involved only after the system is retired

Correct Answer: B. A person has competence, evidence, independence, and authority to override the system