Explainability, Causal Analysis and Decision Analytics

Introduction and Context

Machine-learning systems increasingly influence credit approval, medical triage, industrial maintenance, recruitment, pricing and public-sector resource allocation. High predictive accuracy is valuable, but accuracy alone does not answer several managerial and scientific questions:

• Why did the model produce this prediction?
• Does a feature merely correlate with an outcome, or does changing it cause the outcome to change?
• How certain is the prediction?
• What action should a manager take, given costs, constraints and competing objectives?

Explainability, causal analysis and decision analytics address these questions from related but distinct perspectives. Explainability studies how a model generates or is associated with predictions. Causal analysis studies whether interventions change outcomes. Decision analytics combines predictions, uncertainty, feasible actions and preferences to support decisions.

A central discipline is to avoid conflating prediction with causation. A model may accurately predict employee attrition using commute distance, for example, without implying that reducing commute distance will prevent every resignation. Similarly, a feature-importance ranking may describe model behaviour without identifying the true drivers of the phenomenon.

Learning Objectives

By the end of this chapter, the reader should be able to:

1. Distinguish global and local explanations and select appropriate methods.
2. Interpret permutation importance, partial dependence, ICE, SHAP and counterfactual explanations.
3. Explain why association in predictive models does not establish causation.
4. Formulate causal questions using treatment, outcome, confounders and counterfactuals.
5. Connect predictive distributions and uncertainty with optimization and managerial utility.
6. Evaluate explanations for fidelity, stability, fairness, actionability and governance.
7. Design an interpretable decision workflow for a realistic organizational problem.

1. A Conceptual Architecture

Let X denote observed features, Y an outcome, and f(X) a fitted prediction function. In supervised learning, the objective is commonly to minimize expected predictive loss:

Here, L is a loss function and E denotes expectation over the data-generating distribution. This objective concerns prediction, not explanation or intervention.

An explanation can concern either the model or the real-world phenomenon. A model explanation asks, “Which inputs were influential for f?” A causal explanation asks, “What would happen to Y if an intervention changed X?” These are different estimands. Decision analytics adds an action a and a utility U:

where A(x) is the set of feasible actions for case x. The optimal policy π* depends not only on predicted outcomes but also on consequences, costs, constraints and preferences. If action changes the outcome, the relevant quantity should generally be an intervention-based prediction such as E[Y | do(a), X = x], rather than merely E[Y | X = x].

Important distinctions

2. Global and Local Explainability

A global explanation summarizes behaviour across a population. It can reveal dominant variables, nonlinearities, interactions and regions where a model is unreliable. A local explanation describes one observation, such as why a loan application was declined or why a machine was assigned high failure risk.

Global explanations are useful for model development, monitoring and policy review. Local explanations support case management, contestability and human oversight. Neither is automatically truthful: a global average can conceal subgroup differences, while a local approximation can be unstable or misleading away from the observed point.

2.1 Feature importance

For a tree model, impurity-based importance aggregates reductions in a split criterion attributable to a feature. Such importance can be biased toward high-cardinality or continuous variables and can distribute credit unpredictably among correlated predictors.

Permutation importance measures the change in performance after randomly permuting one feature in a validation set. If M is a performance score, the importance of feature j can be estimated as:

D is held-out data and higher values indicate greater loss of performance after permutation when M is higher-is-better. For a loss metric where lower is better, the sign and interpretation must be adjusted. The procedure assumes that permutation produces a meaningful disruption; correlated features violate this assumption because information may remain available through correlated variables.

Feature importance is not a causal effect. A postcode may be important because it proxies for income, access or historical institutional practices. Removing it from a model does not necessarily remove the underlying proxy information.

2.2 Partial dependence

The partial-dependence function for feature j is:

X−j denotes all features other than j, and z is a selected value of feature j. In a sample of n observations, it is estimated by:

This replaces feature j by z for every observation and averages predictions. It is useful for identifying average nonlinearity, thresholds and saturation. For classification, the plotted quantity should be specified: probability, log-odds or a raw score are not interchangeable.

PDPs can create unrealistic combinations when features are correlated. For instance, setting pregnancy status and age combinations independently may produce impossible or rare records. Conditional or accumulated local effects can reduce some problems, but they require careful interpretation.

2.3 ICE and interactions

An individual conditional expectation (ICE) curve retains one curve per observation:

ICE reveals heterogeneity hidden by a population average. Curves that cross indicate that the feature has different effects across cases, often because of interactions. A two-feature PDP, PD_j,k(z,w), can visualize average interaction, but correlated features and extrapolation remain concerns.

3. SHAP and Additive Attributions

SHAP, based on Shapley values from cooperative game theory, allocates a prediction among features relative to a baseline. For an observation x:

Here, φ0 is the expected model output under a specified background distribution, p is the number of features, and φj is feature j’s contribution. The decomposition is exact for the chosen model output and background definition, subject to the SHAP implementation and feature-dependence assumptions.

For a set of features S, let v(S) be the model value when only features in S are known. The Shapley value is:

N is the set of all features. The coefficient averages the marginal contribution of feature j across all possible feature-ordering contexts. Exact calculation is exponential in p, so practical algorithms use model-specific or sampling-based approximations.

SHAP explanations are additive, which supports global summaries based on mean absolute attribution:

This measures average contribution magnitude, not direction or causality. A feature may have a high mean absolute value while increasing predictions for some cases and decreasing them for others. Explanations should therefore inspect distributions and dependence plots, not only rankings.

The background distribution is consequential. Interventional SHAP treats features as separable under intervention, whereas conditional approaches account for dependence among features. Neither interpretation is universally correct; the choice must match the scientific question. Correlated variables may share or compete for attribution, so a precise-looking SHAP number should not be overinterpreted.

4. Counterfactual Explanations

A counterfactual explanation describes a minimally changed input that would produce a different decision. If x is the original case and x′ a candidate alternative, a common formulation is:

subject to feasibility constraints, where d measures change cost, y* is the desired outcome, L penalizes failure to reach it, and λ controls the trade-off. Features may be continuous, ordinal, categorical or immutable. A valid counterfactual must respect domain constraints such as age non-decrease, valid category combinations and affordability.

Counterfactuals are not automatically causal. “Increase income by £5,000” may be a model-valid change but not a feasible or sufficient intervention. Actionability requires identifying who can change the feature, at what cost, over what time horizon, and whether the relationship is stable.

A robust counterfactual workflow should:

1. State the decision boundary and desired outcome.
2. Separate actionable from immutable variables.
3. Encode feasibility and legal constraints.
4. Penalize implausibly large or costly changes.
5. Generate multiple diverse alternatives where appropriate.
6. Test whether the explanation remains valid under model uncertainty and perturbation.

5. Prediction Is Not Causation

A predictive relationship estimates association, often represented by E[Y | X = x]. A causal question concerns a potential outcome under intervention. For a binary treatment T, define Y(1) as the outcome if treated and Y(0) if untreated. The individual treatment effect is:

Only one potential outcome is observed for each person. The average treatment effect is:

Under standard identification assumptions, observational data may estimate an adjusted effect. One common assumption is conditional exchangeability:

This states that, conditional on measured covariates X, treatment assignment is independent of potential outcomes. It is not testable from observational data alone. Identification also requires positivity, 0 < P(T = 1 | X = x) < 1 for relevant x, and consistency, meaning the observed treatment corresponds to the defined intervention.

A causal graph can clarify confounding. Suppose socioeconomic status affects both education and income. Education and income may be associated even if the desired intervention is not adequately represented by the observed variables. Conditioning on a mediator or collider can also bias an estimate. Therefore, adjustment sets should be selected using substantive knowledge and a defensible causal diagram, not by automatically including every available feature.

Causal methods

Randomized experiments support causal inference because random assignment breaks systematic treatment-outcome confounding in expectation. When experiments are infeasible, researchers may use matching, inverse-probability weighting, regression adjustment, instrumental variables, regression discontinuity, difference-in-differences or synthetic controls. Each method has assumptions that must be argued, assessed and reported.

Machine learning can assist causal analysis through flexible nuisance-function estimation, heterogeneous treatment-effect modeling and causal forests. Flexibility does not eliminate identification assumptions. A powerful model trained on biased observational data cannot recover a causal effect without a credible design.

6. Uncertainty and Reliable Interpretation

A point prediction hides uncertainty. For a predicted quantity ŷ, distinguish:

• Aleatoric uncertainty: irreducible variability in outcomes.
• Epistemic uncertainty: uncertainty due to limited data or model knowledge.
• Distribution shift: uncertainty caused by deployment conditions differing from training conditions.

For a regression model, a predictive interval concerns a future observation, whereas a confidence interval concerns an estimated parameter or mean response. They should not be conflated. Classification probabilities should be calibrated: among cases assigned probability 0.7, approximately 70% should experience the event over a suitable evaluation population and time period.

Explanations also have uncertainty. Bootstrap resampling, repeated training, subgroup analysis and sensitivity to background data can reveal whether an importance ranking or SHAP attribution is stable. A feature that appears important in one sample but not across resamples should not drive a major policy without further investigation.

Calibration can be assessed with reliability diagrams and proper scoring rules. For binary outcomes, the Brier score is:

where pi is the predicted probability and yi ∈ {0,1}. Lower values are better, although the score combines calibration and discrimination. Evaluation should also examine subgroup calibration, false-positive and false-negative costs, and performance under temporal or geographic shift.

7. From Model Output to Decision Analytics

A predictive score is not a decision rule. A decision rule requires an action space, constraints and a value model. For a binary intervention with benefit B, intervention cost C, and event probability p, a simple expected net benefit is:

The intervention is preferred to doing nothing when p·B − C > 0, or equivalently when p > C/B, assuming B and C are positive, known and expressed in comparable units. In practice, benefits may be delayed, uncertain or distributed unevenly, and actions may have capacity constraints.

For multiple periods, discounted net present value can be represented as:

Bt and Ct are benefits and costs at time t, r is the discount rate per period, and T is the planning horizon. The formula assumes commensurable monetary values and a chosen discounting convention.

Optimization may allocate limited interventions. If xi indicates whether case i receives an intervention, a basic formulation is:

subject to:

K is available capacity. This formulation is illustrative; real systems may require fairness constraints, staffing schedules, risk limits, minimum service levels and uncertainty-aware robust optimization.

The explanation should match the decision. If a manager needs to know which cases are high risk, global and local predictive explanations may suffice. If the manager must choose an intervention, treatment-effect estimates and action costs are more relevant. If the decision affects rights or access, explanations must also support due process, contestability and legal review.

8. Illustrative Case: Preventive Maintenance in a Manufacturing Plant

Background

This fictional case concerns a plant operating 400 industrial pumps. A gradient-boosted model predicts whether a pump will fail within 30 days using vibration, temperature, operating hours, pressure and maintenance history.

Problem

The maintenance team has capacity to inspect only 40 pumps weekly. Managers need to know which pumps to prioritize, why the model assigns risk, whether a proposed inspection is likely to prevent failure, and whether the allocation is fair across production lines.

Analysis

The model’s global permutation importance ranks vibration and temperature highly. A PDP shows that predicted risk rises sharply above a vibration threshold, while ICE curves reveal that the threshold is much lower for pumps operating at high pressure. This indicates an interaction that a single global curve might hide.

For one pump, the baseline predicted failure probability is 0.18. SHAP analysis in probability or an appropriately specified transformed output shows that unusually high vibration and recent pressure spikes increase the prediction, while a recent bearing replacement decreases it. The explanation describes model behaviour; it does not prove that replacing the bearing caused the lower risk.

A counterfactual search proposes reducing vibration below the model threshold. Engineers reject the proposal as incomplete because vibration cannot be directly controlled; it is a symptom. A feasible counterfactual instead recommends inspection of the coupling and alignment, with an estimated cost and downtime.

Decision approach

Suppose pump i has failure probability pi, avoided failure cost Bi, inspection cost Ci, and inspection capacity K = 40. The team ranks pumps by estimated net benefit piBi − Ci, not by risk alone. If an inspection changes future failure probability, the benefit should be based on a causal or engineering estimate of the intervention effect, not simply on the predictive risk.

The team validates calibration by production line and month, audits false negatives, and reports uncertainty intervals from repeated temporal validation. It also checks whether the model relies on maintenance codes that are entered differently across lines. A governance committee requires a human engineer to approve high-impact interventions and records the reason, evidence and outcome.

Outcome

The combined workflow reduces emergency failures in a pilot period, but the organization does not claim that the model alone caused the improvement. The evaluation compares the policy with a pre-specified baseline and examines operational changes, inspection compliance and seasonal effects. The resulting system is a decision-support tool rather than an autonomous maintenance authority.

Learning points

1. Feature importance identifies predictive reliance, not causal leverage.
2. ICE can reveal interactions hidden by PDP averages.
3. Counterfactuals must be physically and operationally feasible.
4. Capacity and costs transform prediction into prioritization.
5. Monitoring and evaluation must include calibration, drift and intervention outcomes.

9. Limitations, Ethics and Governance

Explainability methods can produce plausible but inaccurate narratives. Post-hoc explanations may approximate a complex model locally without faithfully representing its internal computation. Different background samples, model versions or correlated-feature treatments can produce different explanations. Explanations may also expose sensitive information or enable strategic gaming.

Causal analysis faces unmeasured confounding, treatment heterogeneity, interference between units and changing policies. A causal estimate is conditional on its estimand, population, intervention definition and assumptions. Claims should therefore distinguish “associated with,” “predicts,” “estimated effect,” and “causes.”

Fairness requires more than removing protected attributes. Proxies may remain, historical labels may encode unequal treatment, and fairness criteria can conflict. Calibration, equalized error rates, equal opportunity, demographic parity and individual notions of fairness answer different questions and may be mutually incompatible under unequal base rates. The selected criterion must be justified by the application and legal context.

Good governance includes:

• documented data provenance, target definitions and feature engineering;
• pre-deployment validation and subgroup analysis;
• versioned models, explanation settings and background data;
• uncertainty reporting and drift monitoring;
• human review for consequential decisions;
• appeal, correction and recourse mechanisms;
• access controls for sensitive explanations;
• post-deployment outcome and impact evaluation.

Explanations should be meaningful to their audience. A data scientist may inspect SHAP distributions, while a customer may require a concise, truthful reason and an actionable recourse path. Simplicity should not be purchased by hiding uncertainty or making causal claims the evidence cannot support.

10. Contemporary Directions and Best Practice

Current research increasingly combines predictive modeling with causal representation, heterogeneous treatment effects, distributionally robust learning and uncertainty-aware optimization. Large foundation models are also being used to generate explanations, but natural-language fluency is not evidence of faithfulness. Generated explanations should be checked against model outputs, perturbation tests and domain review.

A robust professional workflow is:

1. Define the scientific or managerial question.
2. Separate prediction, explanation, causal effect and decision objectives.
3. Establish the deployment population, time horizon and action constraints.
4. Validate predictive performance and calibration on suitable held-out data.
5. Use complementary global and local explanation methods.
6. Test sensitivity to correlated features, sampling, model versions and distribution shift.
7. Specify causal assumptions before estimating intervention effects.
8. Translate probabilities into utility-aware policies with explicit costs.
9. Evaluate subgroup impacts, recourse and operational feasibility.
10. Monitor outcomes and revise the system when evidence changes.

Chapter Summary

Explainability clarifies how a predictive model behaves, while causal analysis addresses the effects of interventions. Global tools such as permutation importance, partial dependence and aggregate SHAP summaries describe population-level behaviour; local tools such as ICE, individual SHAP values and counterfactuals explain particular predictions. Each method has assumptions and failure modes, especially with correlated features and unrealistic data combinations.

Prediction does not establish causation. Causal claims require a defined intervention, potential outcomes, identification assumptions and an appropriate research design. Uncertainty applies both to predictions and to explanations, and should be assessed through calibration, resampling, subgroup analysis and shift monitoring. Decision analytics connects predictions with actions by incorporating utility, costs, capacity and constraints. Responsible deployment requires technical validation, causal humility, fairness analysis, governance, contestability and ongoing evaluation.

Key Terms

Explainability: The study and communication of how a model produces or is associated with its outputs.

Global explanation: A summary of model behaviour across a population.

Local explanation: An explanation for one observation or decision.

Permutation importance: Performance change caused by permuting one feature in evaluation data.

Partial dependence: Average model prediction as a feature is varied while averaging over other features.

ICE: Individual conditional expectation showing a separate response curve for each observation.

SHAP value: An additive attribution based on Shapley-value principles.

Background distribution: Reference data used to define the baseline and missing-feature expectations in an explanation.

Counterfactual explanation: A feasible alternative input that would change a model output or decision.

Actionability: The extent to which a proposed change can realistically be made.

Association: A statistical relationship between variables.

Causation: A relationship in which an intervention changes an outcome.

Potential outcome: The outcome that would occur under a specified treatment or intervention.

ATE: Average treatment effect across a defined population.

Confounder: A variable influencing both treatment and outcome.

Positivity: The condition that each relevant treatment has positive probability for covariate profiles.

Calibration: Agreement between predicted probabilities and observed frequencies.

Aleatoric uncertainty: Irreducible variability in outcomes.

Epistemic uncertainty: Uncertainty arising from limited knowledge or data.

Decision analytics: The integration of predictions, uncertainty, actions, constraints and utility for decision support.

Review Questions

Short Answer Questions

1. Distinguish global from local explainability.
2. Why can permutation importance be misleading with correlated features?
3. What does the baseline term φ0 represent in a SHAP decomposition?
4. State two assumptions commonly required for causal identification from observational data.
5. Why is a predictive probability not, by itself, a decision rule?

Long Answer Questions

1. Compare feature importance, partial dependence, ICE and SHAP with respect to purpose, interpretation and limitations.
2. Explain why a counterfactual explanation may be model-valid but operationally or causally invalid.
3. Discuss the difference between predictive association and causal effect using potential outcomes.
4. Explain how calibration and uncertainty should influence deployment of a high-stakes classifier.
5. Design a governance framework for an explainable machine-learning system used in public-service allocation.

Analytical / Application Questions

1. A retention model identifies salary as an important predictor of resignation. Explain why this finding does not prove that increasing salary will prevent resignation, and propose a causal study.
2. A hospital has capacity to contact 100 high-risk patients. Formulate an expected-net-benefit policy and identify the additional information required before implementation.
3. A SHAP summary ranks postcode first, but its attribution is unstable across samples and differs by demographic group. Analyze possible causes and recommend validation and governance actions.

Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes prediction from causation?

A. Prediction always requires a randomized experiment.

B. Causation concerns the outcome under an intervention, whereas prediction concerns expected outcomes given observed information.

C. Causal effects are always larger than predictive associations.

D. Prediction and causation are identical when model accuracy is high.

Correct Answer: B. Causation concerns the outcome under an intervention, whereas prediction concerns expected outcomes given observed information.

Question 2

What does a partial-dependence function estimate?

A. The causal effect of intervening on a feature.

B. The average model prediction as a feature is varied over the data distribution of other features.

C. The probability that a feature is statistically significant.

D. The uncertainty of the model parameters only.

Correct Answer: B. The average model prediction as a feature is varied over the data distribution of other features.

Question 3

In the SHAP decomposition f(x) = φ0 + ∑φj, what is φ0?

A. The largest feature contribution.

B. The model’s classification threshold.

C. The baseline expected model output under a specified background distribution.

D. The residual error for the observation.

Correct Answer: C. The baseline expected model output under a specified background distribution.

Question 4

Why can correlated features complicate SHAP interpretation?

A. SHAP cannot produce numerical values for correlated features.

B. Credit may be shared or allocated differently among features that contain overlapping information.

C. Correlation guarantees that all features have zero importance.

D. Correlation converts a predictive model into a causal model.

Correct Answer: B. Credit may be shared or allocated differently among features that contain overlapping information.

Question 5

Which condition is required for standard observational causal identification?

A. Every treatment must have probability exactly 0.5.

B. There must be no observed covariates.

C. Relevant treatment options must have positive probability for covariate profiles of interest.

D. The outcome must be continuous.

Correct Answer: C. Relevant treatment options must have positive probability for covariate profiles of interest.

Question 6

A binary intervention has expected benefit £500 and cost £100. Ignoring other considerations, intervention is preferred when predicted event probability exceeds:

A. 0.05

B. 0.20

C. 0.50

D. 0.80

Correct Answer: B. 0.20

Question 7

What does calibration mean for a group assigned predicted probability 0.7?

A. Exactly 70% of every small subgroup must experience the event.

B. Approximately 70% should experience the event over an appropriate evaluation population and period.

C. The model must have 70% accuracy.

D. The model must produce a causal estimate of 0.7.

Correct Answer: B. Approximately 70% should experience the event over an appropriate evaluation population and period.

Question 8

Which characteristic is essential for a useful counterfactual explanation?

A. It changes every feature simultaneously.

B. It ignores immutable variables to simplify optimization.

C. It respects domain constraints and identifies changes that are realistically actionable.

D. It always provides a causal effect estimate.

Correct Answer: C. It respects domain constraints and identifies changes that are realistically actionable.

Question 9

What does ICE reveal that a standard PDP may hide?

A. The training-set size.

B. Heterogeneity of individual response curves and possible interactions.

C. The legal status of each feature.

D. The exact true causal graph.

Correct Answer: B. Heterogeneity of individual response curves and possible interactions.

Question 10

In a capacity-constrained intervention problem, why should managers not rank cases solely by predicted risk?

A. Risk scores cannot be calculated for individuals.

B. Risk does not incorporate intervention cost, benefit, feasibility or available capacity.

C. High-risk cases never benefit from intervention.

D. Optimization is unrelated to prediction.

Correct Answer: B. Risk does not incorporate intervention cost, benefit, feasibility or available capacity.