6. Explainable AI Foundations

Introduction and Context

Artificial intelligence increasingly participates in decisions that affect access to credit, employment, education, healthcare, insurance, public services, and digital information. As predictive systems become more capable, a fundamental question arises: when an AI system produces an output, what can a person reasonably know about how and why that output was produced?

This question is often expressed through the broad term explainable AI (XAI). However, explainability is not a single technical property. It includes several related but distinct ideas: transparency, interpretability, explanation, justification, accountability, and, in some contexts, causal understanding. Confusing these concepts can lead to weak governance. A model may be transparent because its code is available but still be difficult to understand. A system may provide a persuasive explanation without revealing the true reasons for its prediction. A feature may be strongly associated with an outcome without causing it.

For postgraduate students and AI professionals, explainability must therefore be treated as a socio-technical design problem. It involves mathematical models, human cognition, domain expertise, organizational processes, regulatory expectations, and ethical judgment. The appropriate explanation depends on the audience, the decision context, the potential harm, and the purpose for which the explanation is requested.

This chapter develops a conceptual and practical foundation for explainable AI. It distinguishes central terms, examines major categories of explanation methods, evaluates their strengths and limitations, and considers how explanations should be designed and governed in professional settings.

Learning Objectives

After studying this chapter, the learner should be able to:

1. Distinguish transparency, interpretability, explainability, and causality.

2. Explain the difference between intrinsic and post-hoc explanations.

3. Compare local and global explanations and identify when each is appropriate.

4. Distinguish model-specific from model-agnostic explanation methods.

5. Describe major XAI techniques, including feature importance, partial dependence, surrogate models, counterfactuals, and Shapley-based methods.

6. Assess the validity, usefulness, fairness, and limitations of an explanation.

7. Apply an explanation framework to a high-impact AI scenario.

8. Identify ethical, governance, technical, and professional risks associated with explainable AI.

Why Explainability Is Difficult

A prediction and an explanation are not the same object. A prediction is an output generated under particular inputs, parameters, and decision rules. An explanation is an account intended to help someone understand that output. The explanation is therefore relational: it is produced for a particular audience and purpose.

A data scientist may need a global account of model behaviour, such as which variables generally influence predictions and where the model performs poorly. A loan applicant may need a local explanation of an individual decision, such as which factors could be changed to obtain a different result. An auditor may require evidence that the model does not systematically disadvantage a protected group. A regulator may require documentation of data provenance, validation, monitoring, and human oversight. These are different explanatory needs.

Explanations also involve a trade-off between fidelity and comprehensibility. A highly faithful description of a deep neural network may be too complex for a non-technical user. A simplified explanation may be understandable but omit important interactions. The challenge is not simply to maximize information; it is to provide information that is accurate, relevant, actionable, and appropriate to the decision context.

A Vocabulary for Understanding AI Systems

Transparency

Transparency refers to the extent to which information about a system is accessible and inspectable. It can concern the training data, feature engineering, model architecture, objective function, parameters, software implementation, deployment context, or decision process.

Transparency is broader than making source code publicly available. A transparent governance record may include documentation of data sources, known limitations, performance across demographic groups, intended use, prohibited use, human review procedures, and changes made after deployment. Transparency supports scrutiny, but it does not guarantee that people will understand the system.

Interpretability

Interpretability generally refers to the degree to which a human can understand the relationship between a model’s inputs, internal structure, and outputs. An interpretable model exposes meaningful relationships in a form that users can reason about. A small decision tree, a sparse linear model, or a generalized additive model may be considered interpretable when its variables and functional relationships are understandable in the relevant domain.

Interpretability is not absolute. A linear model with thousands of correlated variables may be difficult to interpret, while a moderately complex model may be understandable to a specialist. Interpretability is therefore influenced by model complexity, representation, user expertise, and task context.

Explainability

Explainability is the capability of an AI system, or of an associated method, to provide human-understandable reasons or evidence for its outputs. It often concerns systems that are not intrinsically interpretable, although the term is sometimes used more broadly.

An explanation should be evaluated for at least four properties:

• Fidelity: whether it accurately reflects the model’s actual behaviour.
• Comprehensibility: whether the intended audience can understand it.
• Usefulness: whether it supports an appropriate task, such as debugging, contesting, or improving a decision.
• Stability: whether similar cases receive reasonably consistent explanations.

An explanation can be understandable yet unfaithful. For example, an automated system may report that income was the main reason for a credit decision when the real prediction was driven by a complex interaction between income, location, and transaction history.

Causality

Causality concerns whether changing one factor would produce a change in an outcome under specified conditions. Causal reasoning is different from identifying association or predictive importance. A model may use a variable that is highly predictive but not causally influential. Conversely, a causally important variable may have limited predictive power if it varies little in the available data.

A statement such as “the applicant was rejected because of a low income score” may be merely descriptive. A causal statement would ask whether increasing income, while holding relevant conditions constant, would change the decision. This is a much stronger claim and may require a causal model, intervention data, or carefully justified assumptions.

Accountability and Justification

Accountability concerns who is responsible for the system, its decisions, and its consequences. An explanation can support accountability, but it cannot transfer responsibility to the model. A justification is a reason offered to defend a decision; it may be institutionally persuasive without being a complete technical explanation. Responsible practice requires that explanations not be used to conceal responsibility or create the appearance of review without meaningful oversight.

Core Dimensions of Explainable AI

Intrinsic and Post-hoc Explanations

An intrinsically interpretable model is designed so that its decision process can be understood from the model itself. Examples include sparse linear models, rule lists, small decision trees, monotonic models, and some generalized additive models. Their advantage is that explanation is integrated into the model rather than added afterward.

Intrinsic interpretability does not automatically establish fairness or correctness. A biased rule list is still biased, and a transparent model may rely on inappropriate data. In addition, simplification can reduce predictive performance or fail to represent complex interactions.

Post-hoc explanation methods are applied after a model has been trained. They attempt to describe the behaviour of a black-box or complex model using techniques such as feature attribution, partial dependence, surrogate models, counterfactual examples, or example-based explanations. Post-hoc methods can provide useful diagnostic evidence, but they do not make the original model intrinsically transparent.

A central risk is explanation-model confusion. A local surrogate may approximate a complex model near one data point while being inaccurate elsewhere. Its coefficients explain the surrogate, not necessarily the full model. Post-hoc explanations should therefore be reported with their scope, assumptions, and uncertainty.

Local and Global Explanations

A global explanation describes general model behaviour across a population or relevant subpopulation. It may show average feature effects, important interactions, calibration patterns, or regions where errors occur. Global explanations are useful for model development, policy review, risk management, and monitoring.

A local explanation describes why the model produced a particular output for one instance. Feature attribution methods, local surrogate models, counterfactuals, and influential examples are common local approaches. Local explanations are especially important when an individual wishes to understand or challenge a decision.

Global and local behaviour may differ substantially. A feature can have modest average influence while being decisive for a particular case. Likewise, a model may appear fair at the population level while producing problematic local decisions. High-stakes governance should usually require both perspectives.

Model-Specific and Model-Agnostic Explanations

Model-specific methods exploit the internal structure of a particular model family. Tree-based importance measures, neural-network gradient methods, and coefficients in linear models are examples. Such methods can be computationally efficient and closely connected to the model, but they may not transfer to other architectures.

Model-agnostic methods treat the predictive system as a function that can be queried. They can be used across different model types and are valuable in heterogeneous technology environments. However, they may require many model queries, can be sensitive to sampling choices, and may produce explanations that do not correspond to meaningful causal or semantic features.

Major Explanation Techniques

Feature Importance

Feature importance summarizes how much a feature contributes to predictions. In tree ensembles, importance may be based on impurity reduction or performance changes after permutation. Permutation importance is often more informative, but correlated features can make interpretation difficult: when two variables contain similar information, permuting one may appear unimportant even though the model relies on the underlying signal.

Feature importance is usually global. It should not be interpreted as proof that a feature causes the outcome, nor does it necessarily explain an individual prediction.

Coefficients and Additive Effects

In linear and generalized additive models, coefficients or component functions can communicate the direction and magnitude of effects. Their meaning depends on scaling, transformations, interactions, regularization, and the reference category. A positive coefficient does not necessarily imply that increasing a variable will improve an outcome in every case, particularly where interactions or nonlinearities exist.

Partial Dependence and Related Plots

A partial dependence plot estimates the average predicted response as a feature changes while other features are averaged over a dataset. It can reveal nonlinear patterns and thresholds. Individual conditional expectation plots extend this idea by displaying prediction curves for individual instances, thereby showing heterogeneity hidden by averages.

These methods become unreliable when features are strongly correlated or when the plot examines unrealistic combinations of values. For example, averaging predictions over combinations of age and medical conditions that rarely occur together may generate an apparently precise but substantively meaningless relationship.

Surrogate Models

A surrogate model approximates the predictions of a more complex model. A decision tree or sparse linear model may be trained on inputs labelled with the black-box model’s outputs. Surrogates can provide a useful high-level summary, but their quality depends on the sample used, the region of interest, and the complexity allowed.

A global surrogate with poor fidelity should not be presented as a faithful description. Local surrogates are often more accurate within a neighbourhood, but their conclusions should not be generalized beyond that neighbourhood.

Shapley-Based Attributions

Shapley-based methods allocate a prediction difference among features using ideas from cooperative game theory. They ask, conceptually, how the prediction changes as features are added to different subsets. The resulting attributions can be locally additive and can be aggregated for global analysis.

Their interpretation depends on the baseline or background distribution and on how missing or absent features are represented. With correlated features, responsibility may be divided in ways that are mathematically coherent but difficult to interpret substantively. Shapley values explain a model output relative to a reference, not necessarily the causal contribution of a feature.

Counterfactual Explanations

A counterfactual explanation identifies a minimally changed input that would produce a different output, such as “if the debt-to-income ratio were lower, the application would meet the model threshold.” Counterfactuals are often actionable and intuitive, especially for recourse.

A valid counterfactual should be feasible, sufficiently close to the original case, consistent with domain constraints, and free from prohibited or discriminatory recommendations. Changing age or ethnicity is not an acceptable form of individual recourse. A counterfactual that recommends reducing a debt level that cannot realistically be changed is technically informative but practically useless.

Example-Based Explanations

Some systems explain a decision by showing similar cases, prototypes, or influential training examples. Such explanations can help users compare a case with familiar patterns. They introduce risks concerning privacy, data leakage, spurious similarity, and the possibility that historical examples encode past discrimination.

A Practical Framework for Selecting Explanations

Explanation design should begin with the purpose rather than the technique. A useful process has six stages.

First, identify the decision and its stakes. Explanations required for a medical triage recommendation differ from those required for a low-risk content-ranking system. Second, identify the audience: developer, auditor, operator, affected person, manager, regulator, or researcher. Third, define the question being asked. Is the goal to debug the model, justify an individual outcome, discover bias, support recourse, or document compliance?

Fourth, select a suitable explanatory scope. Global analysis may be appropriate for validation, while local and counterfactual explanations may be needed for individual decisions. Fifth, test fidelity, stability, robustness, and fairness. Explanations should be compared with known model behaviour and evaluated under reasonable perturbations. Sixth, integrate the explanation into a governance process that includes documentation, human review, escalation, record keeping, and correction.

A mature XAI process does not assume that one explanation is sufficient. It may present a layered explanation: a concise user-facing reason, a technical attribution for specialists, model and data documentation for auditors, and a counterfactual or appeal pathway for affected individuals.

Applied Case: An Explainable Credit Assessment System

Background

Consider an illustrative case involving Meridian Finance, a fictional lender that uses a gradient-boosted model to estimate the probability that a loan applicant will default. The model uses income, existing obligations, repayment history, account activity, employment information, and other variables. The organization seeks to reduce manual processing while meeting internal fairness and accountability requirements.

Situation and Problem

An applicant, Samira, receives a rejection notice stating that her “financial profile does not meet the institution’s criteria.” She requests a meaningful explanation and asks what could be changed. The data science team initially generates a global feature-importance chart showing that repayment history and debt-to-income ratio are important. This does not explain Samira’s decision. A local attribution method indicates that recent account activity, debt-to-income ratio, and a short employment history contributed to the rejection.

However, the team discovers that “account activity” is a proxy for geographic access to banking services. It is predictive because customers in certain areas historically experienced different repayment patterns. The model’s explanation is therefore technically informative but raises fairness concerns.

Analysis

The global importance chart helps the organization understand overall model behaviour but cannot justify the individual decision. The local attribution is more relevant to Samira, but its values depend on the reference distribution and correlated features. The team also tests a counterfactual explanation. It finds that reducing the debt-to-income ratio would change the outcome, whereas changing account activity alone would not produce a stable result across plausible perturbations.

Further subgroup analysis shows that the model’s error rates differ across geographic and demographic groups. The explanation process has thus uncovered not only a communication problem but also a possible proxy-discrimination problem.

Approach and Solution

Meridian takes several actions. It removes or transforms the questionable geographic proxy, retrains the model, and evaluates predictive performance and error rates across relevant groups. It introduces a monotonic constraint for debt-to-income ratio so that, within defined conditions, increasing financial burden cannot improve the risk assessment. The organization also creates a layered explanation:

• a plain-language notice identifying the main legitimate factors;
• a technical record containing local attribution values and model version;
• a feasible counterfactual describing acceptable steps for improvement;
• an appeal channel allowing applicants to correct inaccurate data or request human review.

The company documents that the explanation is a model-based account rather than a causal claim. It also monitors explanation stability and subgroup performance after deployment.

Outcome

The revised process improves the usefulness of applicant communications and identifies cases requiring human review. Some predictive performance is sacrificed, but the organization gains stronger control over proxy risks, documentation, and recourse. The result illustrates an important principle: explainability is not merely a presentation layer. It can reveal deficiencies in data, features, model design, and institutional processes.

Learning Points

The case demonstrates that global and local explanations serve different purposes; an explanation can expose a proxy without resolving the underlying fairness issue; counterfactuals must be feasible and legitimate; and governance requires more than displaying feature scores. Explanations should support correction and accountability rather than simply defend an automated outcome.

Ethical, Governance, and Professional Implications

Explainability has an ethical value because affected people may deserve reasons for consequential decisions, opportunities to challenge errors, and protection against arbitrary treatment. Yet disclosure can create competing risks. Detailed explanations may expose personal data, reveal security-sensitive information, enable gaming, or disclose proprietary methods. Governance must balance openness with privacy, safety, and legitimate confidentiality.

A particularly important distinction is between explanation and justification. An explanation may faithfully describe a discriminatory model. That does not make the decision acceptable. Fairness assessment, data protection, validity, and human rights obligations remain separate requirements.

Organizations should assign responsibility for explanation quality. Model owners, data scientists, product managers, legal specialists, domain experts, and affected-user representatives may have different roles. Documentation should record the intended use, data limitations, model version, explanation method, validation results, known failure modes, and escalation procedures.

Professionals should avoid anthropomorphic claims such as “the AI decided” when the system is an instrument embedded in an organizational process. They should also avoid presenting probabilistic outputs as certain facts. An explanation should communicate uncertainty where it matters and should not imply that changing an attributed feature will necessarily change the outcome.

Challenges and Common Mistakes

One common mistake is treating a single feature-ranking chart as a complete explanation. Rankings omit interactions, local variation, uncertainty, and the difference between association and causation. Another is using post-hoc explanations as proof that a black-box model is fair. Fairness must be assessed through appropriate data, metrics, contextual judgment, and impact analysis.

Instability is another concern. Small changes in data, background samples, or model versions can produce different explanations. Explanations may also be manipulated to appear reassuring, especially when organizations choose a method because it produces a preferred narrative. Users can be misled by technically correct but irrelevant information, such as a list of factors they cannot change.

Correlated variables create attribution ambiguity. If education and occupation encode similar information, dividing responsibility between them may be arbitrary from a social or policy perspective. Distribution shift creates a further problem: explanations validated in development may become unreliable when the population, data-generation process, or operating environment changes.

Finally, explanations can increase automation bias. Users may trust a system more because it offers a reason, even when the reason is weak. Human reviewers require training to question explanations, recognize uncertainty, and investigate contradictory evidence.

Current Developments and Emerging Directions

Contemporary XAI research is moving beyond isolated attribution scores toward evaluation frameworks that measure whether explanations improve human decisions, debugging, calibration, fairness assessment, or recourse. There is increasing attention to uncertainty-aware explanations, causal and structural models, concept-based explanations, and explanations tailored to domain workflows.

Foundation models and generative AI create new challenges. Natural-language explanations can be fluent yet fabricated, incomplete, or inconsistent with the underlying computation. A language model may generate a plausible rationale after producing an answer, but fluency is not evidence of faithfulness. Systems should therefore distinguish generated summaries from verified computational traces, retrieved evidence, tool records, or causal analyses.

Regulatory and organizational practice is also shifting toward lifecycle governance. Explanation requirements are increasingly connected with risk classification, impact assessments, auditability, human oversight, data governance, incident management, and post-deployment monitoring. The emerging direction is not a universal demand for maximal transparency, but proportionate, purpose-specific, and testable explanation.

Best Practices for Professional Use

1. Define the explanation’s purpose, audience, and decision stakes before selecting a method.

2. Prefer intrinsically interpretable models when they meet the required performance and operational needs.

3. Use multiple views, combining global, local, error-focused, and subgroup analysis.

4. Report the scope, assumptions, baseline, uncertainty, and known limitations of every explanation method.

5. Do not interpret feature importance as causality without appropriate causal evidence.

6. Test explanations for fidelity, stability, robustness, and sensitivity to correlated features.

7. Design counterfactuals around feasible, lawful, and non-discriminatory actions.

8. Validate explanations with domain experts and representative users, not only technical teams.

9. Preserve model versions, explanation records, and appeal outcomes for auditability.

10. Treat explainability as one part of a broader system of fairness, privacy, safety, security, and accountability.

Chapter Summary

Explainable AI concerns the production and evaluation of human-understandable accounts of AI behaviour. Transparency refers to access to information about a system; interpretability concerns the understandability of its structure and relationships; explainability concerns reasons or evidence for outputs; and causality concerns what would happen under an intervention. These concepts overlap but are not interchangeable.

Explanations may be intrinsic or post-hoc, local or global, and model-specific or model-agnostic. Important techniques include coefficients, feature importance, partial dependence, surrogate models, Shapley-based attributions, counterfactuals, and example-based explanations. Each provides useful evidence under particular assumptions and introduces limitations involving fidelity, correlation, instability, unrealistic data combinations, privacy, and misuse.

Effective XAI begins with purpose and audience. High-stakes systems generally require layered explanations, technical validation, subgroup analysis, feasible recourse, human review, and lifecycle documentation. Explanations cannot by themselves establish fairness, legality, or causality. Their ethical value depends on whether they enable understanding, correction, contestability, and responsible governance.

Key Terms

Transparency: The accessibility and inspectability of information about an AI system and its operation.

Interpretability: The extent to which humans can understand a model’s structure and input-output relationships.

Explainability: The capacity to provide understandable reasons or evidence for an AI output.

Causality: The relationship between an intervention and a resulting change in an outcome.

Intrinsic interpretability: Understandability built into the model’s structure.

Post-hoc explanation: An explanation generated after a model has been trained.

Local explanation: An account of a model’s behaviour for one instance or a small neighbourhood.

Global explanation: An account of general model behaviour across a population or dataset.

Model-specific method: An explanation technique designed for a particular model family.

Model-agnostic method: A technique that explains models through their input-output behaviour regardless of architecture.

Feature attribution: Allocation of a prediction or prediction change among input features.

Surrogate model: A simpler model trained to approximate the outputs of a complex model.

Partial dependence: An estimate of average prediction changes as a feature varies.

Individual conditional expectation: Instance-level prediction curves showing variation hidden by averages.

Counterfactual explanation: A description of a minimal feasible change that would produce a different output.

Shapley value: A game-theoretic allocation of a prediction difference among features.

Fidelity: The degree to which an explanation accurately represents the model being explained.

Recourse: The practical ability of an affected person to improve or challenge an outcome.

Proxy variable: A feature that indirectly encodes a protected or sensitive attribute.

Automation bias: Excessive reliance on an automated recommendation or explanation.

Review Questions

Short Answer Questions

1. How does transparency differ from interpretability?

2. Why is a predictive association not necessarily a causal relationship?

3. What is the difference between a local and a global explanation?

4. State two limitations of post-hoc explanations.

5. What makes a counterfactual explanation actionable?

Long Answer Questions

1. Compare intrinsic and post-hoc interpretability, including their benefits and limitations.

2. Explain feature importance, partial dependence, surrogate models, and Shapley-based attributions.

3. Discuss why explanation quality should be assessed through fidelity, comprehensibility, usefulness, and stability.

4. Examine the ethical and governance implications of providing explanations for high-impact AI decisions.

5. Explain how a layered explanation process can serve applicants, developers, auditors, and regulators.

Analytical / Application Questions

1. A hospital uses a complex model to prioritize patients. Design an explanation strategy for clinicians, patients, and auditors, and justify the methods selected for each audience.

2. A loan model assigns high importance to postal code. Explain how you would investigate whether this feature is a legitimate predictor or a proxy for protected characteristics.

3. A counterfactual explanation tells an applicant to change a feature that cannot realistically be changed. Evaluate the explanation and propose a more responsible alternative.

22. Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes explainability from causality?

A. Explainability always requires a causal model.

B. Explainability provides understandable reasons, whereas causality concerns effects of interventions.

C. Causality is limited to global explanations.

D. Explainability applies only to intrinsically interpretable models.

Correct Answer: B. Explainability provides understandable reasons, whereas causality concerns effects of interventions.

Question 2

Which is an example of an intrinsic explanation approach?

A. A post-hoc local surrogate for a neural network

B. A small decision tree whose rules are directly inspectable

C. A permutation test applied after deployment

D. A counterfactual generated from a black-box model

Correct Answer: B. A small decision tree whose rules are directly inspectable

Question 3

What is a principal limitation of a global feature-importance chart?

A. It cannot summarize any model behaviour.

B. It necessarily proves that features are causal.

C. It may not explain why a particular individual received a prediction.

D. It can only be used with linear models.

Correct Answer: C. It may not explain why a particular individual received a prediction.

Question 4

A model-agnostic explanation method is one that:

A. Uses only neural-network gradients

B. Requires access to the model’s source code

C. Explains a model through its input-output behaviour across model types

D. Can provide causal conclusions without assumptions

Correct Answer: C. Explains a model through its input-output behaviour across model types

Question 5

Why can partial dependence plots be misleading with strongly correlated features?

A. They cannot display nonlinear relationships.

B. They may evaluate unrealistic combinations of feature values.

C. They always show individual rather than average effects.

D. They require a model with no numerical variables.

Correct Answer: B. They may evaluate unrealistic combinations of feature values.

Question 6

What does a counterfactual explanation primarily provide?

A. A list of all variables used during training

B. A minimally changed, feasible situation associated with a different output

C. A guarantee that the model is fair

D. A proof that an attributed feature caused the decision

Correct Answer: B. A minimally changed, feasible situation associated with a different output

Question 7

In the illustrative credit case, why was account activity ethically problematic?

A. It was not predictive of any outcome.

B. It could act as a proxy for unequal geographic access to banking services.

C. It was an intrinsically interpretable feature.

D. It made all counterfactuals feasible.

Correct Answer: B. It could act as a proxy for unequal geographic access to banking services.

Question 8

Which property asks whether an explanation accurately reflects the model’s actual behaviour?

A. Comprehensibility

B. Fidelity

C. Transparency

D. Recourse

Correct Answer: B. Fidelity

Question 9

Why should Shapley-based attributions not automatically be interpreted as causal effects?

A. They allocate prediction differences relative to a reference rather than necessarily representing interventions.

B. They cannot be calculated for individual cases.

C. They are always global rather than local.

D. They do not use model predictions.

Correct Answer: A. They allocate prediction differences relative to a reference rather than necessarily representing interventions.

Question 10

What is the most responsible response when a fluent language model produces a plausible rationale unsupported by its computation?

A. Treat fluency as evidence of faithfulness.

B. Use the rationale as a causal explanation.

C. Distinguish the generated narrative from verified computational evidence and evaluate its fidelity.

D. Publish the rationale without recording the model version.

Correct Answer: C. Distinguish the generated narrative from verified computational evidence and evaluate its fidelity.