7. Explainable AI Methods and Technologies

Introduction: From Predictive Accuracy to Accountable Understanding

Artificial intelligence systems increasingly influence lending, recruitment, medical diagnosis, insurance, public services, industrial maintenance, and content moderation. In many of these settings, knowing that a model is accurate is not enough. Users may need to know why a prediction was made, whether the reasoning is consistent with domain knowledge, whether a decision can be challenged, and whether the system relies on inappropriate or accidental correlations. Explainable artificial intelligence (XAI) addresses these needs by developing methods, interfaces, and governance practices that make model behaviour more understandable to relevant stakeholders.

Explainability is not a single technical property. A data scientist may need a global view of model behaviour, while a loan applicant may need an explanation of one rejected application. A regulator may require evidence of procedural fairness, whereas a clinician may need to inspect the evidence supporting a diagnosis. Consequently, an explanation must be evaluated in relation to its audience, purpose, model, data, and decision context.

This chapter examines major XAI methods for tabular, textual, and visual models. It distinguishes inherently interpretable models from post-hoc explanation techniques and considers feature importance, partial dependence, individual conditional expectation, surrogate models, LIME, SHAP, counterfactuals, rule extraction, saliency methods, and concept-based explanations. The chapter also addresses evaluation, limitations, governance, and professional use.

Learning Objectives

By the end of this chapter, students should be able to:

1. Distinguish global from local explanations and intrinsic from post-hoc interpretability.

2. Apply feature-importance, partial-dependence, ICE, surrogate, LIME, and SHAP methods appropriately.

3. Construct and assess counterfactual and rule-based explanations.

4. Explain how saliency and concept-based methods are used with image and text models.

5. Identify weaknesses such as instability, feature dependence, explanation unreliability, and misleading visualisation.

6. Select explanation methods for tabular, textual, and visual applications.

7. Integrate explainability into ethical, governance, and model-risk processes.

What an Explanation Is—and What It Is Not

An explanation is an information artefact intended to clarify the relationship between inputs, model behaviour, and an output. It may describe which variables were influential, how predictions change when inputs vary, what conditions would produce a different decision, or which regions of an image or words in a document affected a result.

Two distinctions are especially important. A global explanation describes the model’s general behaviour across a population; a local explanation describes one prediction or a small neighbourhood around it. A global explanation might show that income and debt-to-income ratio are generally influential in a credit model. A local explanation might show that a particular applicant’s high debt ratio and short employment history contributed to a rejection.

A second distinction concerns the model itself. An intrinsically interpretable model is understandable by design, such as a shallow decision tree, sparse linear model, or carefully constructed rule list. A post-hoc explanation is produced after training a more complex model, such as a neural network or gradient-boosted ensemble. Post-hoc explanations can be useful, but they describe the behaviour of an explanatory procedure and may not provide a complete account of the original model.

Explanations also have different purposes. They may support debugging, scientific discovery, user trust, fairness investigation, compliance documentation, model monitoring, or contestability. A technically correct explanation can still be unsuitable if it is too complex, inaccessible, or incapable of supporting a meaningful decision or appeal.

A Map of the Main Explanation Methods

Feature importance for tabular models

Feature-importance methods rank variables according to their contribution to predictive performance or individual outputs. In tree-based models, impurity-based importance measures how much a feature reduces a chosen split criterion. Permutation importance measures the performance decline that occurs when a feature’s values are randomly shuffled. Model-specific importance can be useful for broad screening, but it should not automatically be interpreted as causal influence.

Permutation importance is often more meaningful than split-count measures because it tests dependence on predictive information. However, correlated variables create ambiguity: if two features carry similar information, permuting one may produce little performance loss even though the underlying factor is important. Importance can also vary across subgroups, time periods, and outcome definitions. A responsible analysis therefore reports uncertainty, checks correlated features, and avoids presenting a ranking as a complete explanation.

For an individual prediction, additive attribution methods assign positive or negative contributions to features. Such attributions should be interpreted relative to a baseline, such as an average prediction or a reference population. The baseline matters: the same contribution can appear different when the reference distribution changes.

Partial dependence and individual conditional expectation

A partial dependence plot (PDP) estimates the average predicted response as one or more features vary, while averaging over the observed values of other features. For a feature x, a simplified form is:

PD(x) = average of f(x, X other) over the observed data.

PDPs provide a global view of the model’s functional relationship with a feature. They can reveal thresholds, saturation, nonlinearity, and interactions. For example, a fraud model may show that risk rises sharply after an unusual transaction amount.

PDPs rely on an important assumption: combinations of feature values used in the calculation should be plausible. With strongly correlated variables, the method may evaluate unrealistic observations, such as a very young person with an implausibly long employment history. This can produce an attractive but misleading curve.

Individual conditional expectation (ICE) plots address part of this problem by displaying a separate prediction curve for each observation rather than only the average. ICE reveals heterogeneity hidden by a PDP. Curves may have different slopes, thresholds, or directions, indicating that the model treats subgroups differently. Centered ICE plots subtract each observation’s starting prediction, making differences in shape easier to compare. PDPs are compact and global; ICE is more informative about individual variation but can become visually crowded.

Surrogate models

A surrogate model is a simpler model trained to approximate the predictions of a complex model. A decision tree may approximate a neural network’s outputs, or a sparse linear model may approximate an ensemble within a selected region. Surrogates can provide an accessible summary, but their usefulness depends on fidelity—the degree to which they reproduce the original model.

A global surrogate is trained over the complete dataset; a local surrogate is trained near one instance. A high-performing surrogate on the original outcome does not necessarily explain the complex model. Fidelity must be measured against the complex model’s predictions, not merely against ground-truth labels. If a shallow tree has low fidelity, its rules should not be presented as the black-box model’s actual logic.

LIME: local approximation through perturbation

Local Interpretable Model-agnostic Explanations (LIME) explains a prediction by sampling perturbed versions of an input, obtaining the black-box predictions for those samples, weighting samples according to proximity to the original instance, and fitting an interpretable local model. For tabular data, perturbations may modify feature values; for text, words or tokens may be removed; for images, superpixels may be hidden.

LIME is flexible and model-agnostic. Its explanation usually takes the form of influential features with positive or negative weights. Yet its result depends on the perturbation distribution, distance function, interpretable representation, and random seed. If perturbations are unrealistic, the local model may learn behaviour in regions that do not represent valid data. Explanations should therefore be tested for stability by rerunning the method and varying settings.

SHAP and additive attribution

SHAP methods are based on Shapley values from cooperative game theory. Features are treated as contributors to a prediction, and the method allocates the difference between a baseline output and the instance output among them. In an additive form:

prediction = baseline + contribution of feature 1 + contribution of feature 2 + ... + contribution of feature n.

The allocation reflects average marginal contributions over feature coalitions under a specified background or conditional distribution. SHAP can produce local explanations, global summaries, dependence plots, and interaction analyses. TreeSHAP provides an efficient implementation for many tree models, while other versions approximate values for neural or arbitrary models.

SHAP’s apparent precision should not be confused with certainty. Results depend on how missing features are represented, how feature dependence is handled, and which background data are selected. Independent-feature assumptions may be inappropriate when variables are highly related. A SHAP value describes model attribution, not whether a feature caused the outcome. It also does not prove that the model used a feature in a humanly meaningful way.

Explanations for Text and Images

Text models

Text explanations commonly identify influential words, tokens, phrases, or spans. LIME and SHAP can mask tokens and observe prediction changes. Gradient-based methods, such as integrated gradients, use changes in the model output along a path from a baseline representation to the input. Attention visualisation can show where an attention mechanism is focused, but attention weights alone should not automatically be treated as explanations of causality or decision responsibility.

Text explanations face special difficulties. Removing a word may create unnatural grammar. Tokenisation can divide one word into several subword units, making attribution difficult to interpret. Synonyms, negation, context, and long-range dependencies also complicate analysis. For example, the word “not” may reverse the meaning of a sentence even though its individual attribution is small. Explanations should therefore be inspected at the phrase or sentence level and evaluated against linguistic and domain knowledge.

Image models and saliency

Saliency methods identify pixels, regions, or superpixels associated with a model output. Occlusion methods hide part of an image and measure the change in prediction. Gradient-based maps use derivatives of the output with respect to pixels. Grad-CAM uses gradients flowing into convolutional feature maps to produce a coarse localisation map, often highlighting regions relevant to a selected class.

Saliency maps can help identify whether a classifier attends to a tumour region, a product component, or an irrelevant background. However, a visually compelling heatmap can be unstable or insensitive to meaningful changes. Saturated gradients may produce weak signals, and different preprocessing choices can alter the map. A map may indicate correlation rather than a robust basis for the prediction. Good practice includes testing with occlusion, comparing multiple methods, and checking whether highlighted regions correspond to expert-labelled evidence.

Concept-based explanations

Feature attribution identifies input elements; concept-based methods operate at a higher semantic level. Testing with Concept Activation Vectors (TCAV), for example, represents human-defined concepts such as “stripes,” “wheel,” “inflammation,” or “crowded scene” in an internal representation space. The method assesses the sensitivity of a model’s output to a concept direction.

Concept explanations are often more meaningful to experts because they connect model behaviour with domain vocabulary. They require representative concept examples and careful separation from unrelated concepts. A medical concept set must also account for variations in imaging equipment and patient populations. Concept methods can reveal whether a model relies on clinically relevant patterns or on shortcuts such as labels, borders, or acquisition artefacts.

Counterfactuals and Rule Extraction

A counterfactual explanation states what would need to change for a different outcome. For example: “The application would have been approved if annual income had been 8,000 units higher while other relevant conditions remained unchanged.” Useful counterfactuals should be valid, sufficiently close to the original case, actionable, and diverse when multiple alternatives exist.

Counterfactual generation can be formulated as an optimisation problem that balances proximity, plausibility, sparsity, and desired outcome. Constraints are essential. Age cannot usually be reduced immediately, a protected attribute should not be treated as an actionable lever, and categorical variables must take valid values. A counterfactual that changes several correlated variables independently may be mathematically valid but socially or practically impossible.

Counterfactuals support contestability and user guidance, but they do not establish that changing the proposed feature will cause the desired result. If the model encodes historical bias, a counterfactual may merely describe how to satisfy that biased model.

Rule extraction converts model behaviour into if-then statements, decision lists, or rule sets. Rules can be extracted directly from interpretable models or approximately from black-box systems. They are useful in regulated environments and operational workflows, especially when decisions must be reviewed by people. Their main risks are loss of fidelity, excessive rule complexity, and false confidence that a compact rule set captures every relevant model condition.

Choosing and Evaluating an Explanation

Method selection should begin with the decision need rather than the availability of a library. The following questions are useful:

• Is the goal global model understanding, individual justification, debugging, monitoring, or recourse?
• Is the audience technical, managerial, professional, or affected by the decision?
• Are the inputs tabular, textual, visual, sequential, or multimodal?
• Are correlations, constraints, or distribution shifts likely to affect the explanation?
• Must the explanation be stable, actionable, contrastive, or legally auditable?

Evaluation should examine at least four properties. Fidelity asks whether the explanation reflects the model’s actual behaviour. Stability asks whether small irrelevant changes produce radically different explanations. Plausibility asks whether the explanation makes sense to a knowledgeable human, although plausibility alone is not proof of truth. Usefulness asks whether it improves a task such as error detection, review, or decision-making.

Additional criteria include completeness, sparsity, comprehensibility, consistency across similar cases, and fairness across groups. Explanation quality should be assessed empirically, not inferred from attractive plots. A model card or explanation report should document the method, baseline, data distribution, assumptions, known failure modes, and intended audience.

An Illustrative Case: Explaining a Health-Risk Model

Background

Consider a fictional regional hospital that uses a gradient-boosted model to identify patients at risk of readmission within thirty days. The model uses clinical measurements, prior admissions, medication information, age, and administrative variables. It performs well overall, but clinicians report that some high-risk predictions appear inconsistent.

Problem

The hospital needs explanations for three purposes: clinicians must review individual alerts, the data-science team must investigate errors, and the governance committee must assess whether the system relies on inappropriate proxies.

Analysis

The team first computes permutation importance and finds that prior admissions, medication count, and a particular administrative code are influential. A PDP suggests that risk rises with medication count, but ICE plots show that the relationship differs substantially across patients. This warns the team against interpreting the average curve as a universal clinical rule.

A SHAP summary identifies the administrative code as a strong contributor. Local SHAP explanations show that it frequently increases risk for patients from one hospital unit. Investigation reveals that the code is associated with a discharge process that is more common among complex cases, but it also reflects documentation practices rather than physiology. The code may therefore be a useful predictor but a problematic basis for clinical interpretation.

For image data used in a separate chest-imaging component, Grad-CAM highlights image borders in several cases. Occlusion tests confirm that masking the border changes predictions more than masking some clinically relevant regions. The team concludes that the visual model has learned an acquisition shortcut.

Approach and solution

The hospital retains the predictive model temporarily but introduces a governance intervention. The administrative code is reviewed, its meaning is documented, and alternative clinically grounded variables are collected. The image model is retrained with border removal and stronger quality controls. Clinicians receive local explanations showing major contributing factors, uncertainty indicators, and a warning that attribution is not causal evidence. Counterfactuals are presented only when changes are actionable and clinically meaningful.

The team evaluates explanations by asking clinicians to identify deliberately inserted model errors and by measuring explanation stability across repeated runs. It also audits alert rates and explanation patterns across demographic and clinical subgroups.

Outcome and learning points

The case illustrates that explainability can expose data-quality and shortcut-learning problems that aggregate accuracy conceals. It also shows why several methods should be combined: global importance found a suspicious variable, local attribution clarified its role, ICE revealed heterogeneity, and saliency analysis exposed visual artefacts. No individual explanation method was treated as definitive.

Risks, Ethical Questions, and Governance

Explanations can create harm when they disclose sensitive information, reveal security vulnerabilities, or encourage users to manipulate a system. They can also produce automation bias: users may accept a confident-looking explanation without independently checking the prediction. In high-impact decisions, an explanation should support human review rather than serve as a decorative justification for an unchallengeable outcome.

A central ethical distinction is between explaining a model and justifying a decision. A model may accurately explain that a protected attribute or its proxy influenced a prediction, but that does not make the influence ethically acceptable. Fairness analysis must therefore be conducted on outcomes, error rates, data practices, and decision procedures—not replaced by feature attribution.

Governance processes should specify who receives explanations, how explanations are logged, how disputes are handled, and when a model must be suspended. Documentation should include model purpose, training data, feature definitions, explanation method, baseline population, validation results, limitations, and change-control procedures. In privacy-sensitive settings, explanation outputs should follow data-minimisation and access-control principles.

Professional practice requires communicating uncertainty. Terms such as “contributed to,” “associated with,” or “the model relied on” are generally safer than causal statements unless a causal design supports them. Explanations should be translated for the audience without concealing technical qualifications.

Emerging Directions and Best Practices

Current work increasingly combines attribution with causal modelling, uncertainty quantification, concept discovery, and interactive human-computer interfaces. Multimodal systems require explanations that connect text, image, audio, and structured inputs rather than attributing each modality in isolation. Foundation models also motivate research into mechanistic interpretability, representation-level concepts, and explanations that remain reliable under distribution shift.

A practical XAI workflow is to:

1. Define the explanation’s audience, purpose, and decision risk.

2. Establish a meaningful baseline and identify valid data constraints.

3. Begin with an interpretable model or global diagnostic where appropriate.

4. Use local methods for individual cases and test their fidelity.

5. Compare methods rather than relying on one visualisation.

6. Check correlated features, subgroup behaviour, stability, and plausibility.

7. Validate explanations with domain experts and affected users.

8. Record assumptions, limitations, and governance decisions.

9. Monitor explanations after deployment for drift and unusual patterns.

Explainability is most valuable when integrated into the model lifecycle: problem formulation, data collection, training, validation, deployment, monitoring, and appeal. It is not a final presentation layer added after technical development is complete.

Chapter Summary

Explainable AI provides methods for understanding model behaviour at global and local levels. Feature importance, PDP, and ICE support population-level analysis; surrogate models, LIME, and SHAP approximate or attribute individual predictions; counterfactuals describe possible changes leading to different outcomes; rule extraction produces human-readable decision structures; and saliency and concept-based methods support visual and semantic interpretation. Each technique has assumptions and failure modes, particularly with correlated, unrealistic, unstable, or causally ambiguous inputs. Effective XAI requires method selection based on purpose and audience, empirical evaluation of fidelity and stability, domain validation, and integration with ethical and governance controls. An explanation should improve accountability and informed action, not merely make a complex model appear understandable.

Key Terms

Explainable AI (XAI): Methods and practices that make AI behaviour understandable to relevant stakeholders.

Interpretability: The degree to which a model or its behaviour can be understood by people.

Global explanation: An account of model behaviour across a dataset or population.

Local explanation: An account of why a model produced a particular individual prediction.

Intrinsic interpretability: Understandability built into the model structure.

Post-hoc explanation: An explanation generated after a model has been trained.

Feature importance: An estimate of the influence or predictive contribution of input variables.

Permutation importance: Importance estimated from performance loss after shuffling a feature.

Partial dependence plot: A plot of average predicted response as selected features vary.

Individual conditional expectation: Individual prediction curves showing how a feature affects different observations.

Surrogate model: A simpler model trained to approximate a complex model’s predictions.

LIME: A method that fits an interpretable model locally around a prediction using perturbed samples.

SHAP: An additive attribution framework based on Shapley values from cooperative game theory.

Counterfactual explanation: A statement describing changes that would produce a different model outcome.

Rule extraction: Conversion of model behaviour into approximate or direct if-then rules.

Saliency map: A visual representation of input regions associated with a model output.

Concept-based explanation: An explanation expressed through human-understandable concepts represented in a model.

Fidelity: The extent to which an explanation accurately reflects the original model.

Stability: The consistency of an explanation under small irrelevant changes.

Recourse: Feasible action available to a person seeking a more favourable decision.

Review Questions

Short Answer Questions

1. Distinguish global and local explanations with one application example for each.

2. Why can correlated features make importance rankings difficult to interpret?

3. What is the principal difference between a PDP and an ICE plot?

4. Define fidelity in the evaluation of a surrogate model.

5. Why should a saliency map not automatically be interpreted as causal evidence?

Long Answer Questions

1. Explain the logic, uses, and limitations of LIME and SHAP for tabular data.

2. Discuss how counterfactual explanations can support contestability while creating ethical risks.

3. Compare intrinsic interpretability with post-hoc explanation methods.

4. Explain the application of saliency and concept-based methods to visual models.

5. Develop an XAI evaluation framework covering fidelity, stability, plausibility, usefulness, and fairness.

Analytical / Application Questions

1. A credit model has two highly correlated income variables. Permutation importance ranks both low, although removing them reduces performance substantially. Analyse the result and propose a better investigation.

2. An image classifier highlights a hospital image’s border rather than the suspected lesion. Design a validation and remediation process.

3. A counterfactual tells a rejected applicant to change age and nationality to obtain approval. Evaluate this explanation and specify how the system should be redesigned.

22. Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes a global explanation from a local explanation?

A. A global explanation describes one prediction, whereas a local explanation describes the whole dataset.

B. A global explanation describes general model behaviour, whereas a local explanation concerns a particular prediction.

C. A global explanation is always causal, whereas a local explanation is always statistical.

D. A global explanation applies only to images, whereas a local explanation applies only to tabular data.

Correct Answer: B. A global explanation describes general model behaviour, whereas a local explanation concerns a particular prediction.

Question 2

Why can a partial dependence plot be misleading when features are strongly correlated?

A. It cannot display nonlinear relationships.

B. It always gives identical predictions for every individual.

C. It may evaluate implausible combinations of feature values.

D. It measures only training accuracy.

Correct Answer: C. It may evaluate implausible combinations of feature values.

Question 3

A local surrogate model has high accuracy on the true labels but low agreement with the black-box model. What does this indicate?

A. The surrogate has high fidelity.

B. The surrogate may predict the task well but does not explain the black-box model well.

C. The black-box model must be causal.

D. The explanation is necessarily stable.

Correct Answer: B. The surrogate may predict the task well but does not explain the black-box model well.

Question 4

Which procedure is central to LIME?

A. Replacing the black-box model with a decision tree permanently.

B. Sampling perturbed inputs near an instance and fitting a weighted local model.

C. Computing only the model’s global training accuracy.

D. Using human concepts to label internal neural representations.

Correct Answer: B. Sampling perturbed inputs near an instance and fitting a weighted local model.

Question 5

In an additive SHAP explanation, feature contributions are generally interpreted relative to:

A. A baseline prediction.

B. The maximum possible feature value.

C. The model’s software version.

D. A randomly selected class label only.

Correct Answer: A. A baseline prediction.

Question 6

Which is the best example of a valid counterfactual constraint?

A. Allowing a person’s age to be reduced immediately.

B. Changing a protected attribute as the recommended action.

C. Requiring proposed changes to be feasible and consistent with valid data relationships.

D. Changing every feature simultaneously to guarantee a new outcome.

Correct Answer: C. Requiring proposed changes to be feasible and consistent with valid data relationships.

Question 7

What does an ICE plot reveal that an ordinary PDP may conceal?

A. The model’s source code.

B. Variation in feature effects across individual observations.

C. Whether the dataset contains labels.

D. The exact causal effect of a feature.

Correct Answer: B. Variation in feature effects across individual observations.

Question 8

A saliency map highlights image borders, and occlusion confirms that masking the borders changes predictions strongly. The most appropriate conclusion is that:

A. The model has definitely learned the clinical concept correctly.

B. The model may be relying on an acquisition or background shortcut.

C. The saliency method has proved causality.

D. The model cannot be evaluated further.

Correct Answer: B. The model may be relying on an acquisition or background shortcut.

Question 9

Which criterion asks whether an explanation remains similar after small irrelevant changes?

A. Fidelity

B. Stability

C. Sparsity

D. Recourse

Correct Answer: B. Stability

Question 10

Why should explanations not replace fairness analysis?

A. Attributions cannot be computed for any model.

B. A technically accurate attribution does not establish that the underlying influence is ethically acceptable or fair.

C. Fairness applies only to interpretable models.

D. Explanations always remove protected attributes from data.

Correct Answer: B. A technically accurate attribution does not establish that the underlying influence is ethically acceptable or fair.