12. Explainability, Causal Boundaries and Decision Analytics

Introduction: From Predictive Accuracy to Defensible Action

Machine-learning systems are increasingly used to support decisions about credit, health, marketing, employment, maintenance, fraud, supply chains, and public services. In each setting, predictive performance matters, but accuracy alone does not establish that a model is suitable for managerial use. A model may predict loan default accurately without identifying what would prevent default. A model may rank patients by risk without proving that a treatment will improve their outcomes. A model may recommend reducing a customer’s price when the recommendation is based on historical correlations that would not survive a change in market conditions.

This distinction creates three connected responsibilities. First, analysts must explain how a model produces its predictions. Second, they must identify the boundary between prediction and causal inference. Third, they must translate model outputs into decisions while accounting for uncertainty, costs, constraints, fairness, and organizational accountability. Explainability, causal reasoning, and decision analytics therefore form a bridge between statistical modelling and responsible action.

The central question is not simply, “What does the model predict?” It is also: “Why should this prediction be trusted, what would happen under a proposed intervention, and what decision is justified given the evidence?”

Learning Objectives

After studying this chapter, students should be able to:

1. Distinguish global, subgroup, and local forms of model explanation.

2. Explain the purposes and limitations of methods such as permutation importance, partial dependence, ICE, SHAP, LIME, and counterfactual explanations.

3. Communicate predictive uncertainty, calibration, intervals, and risk thresholds to decision-makers.

4. Distinguish association, prediction, intervention, and causation.

5. Identify confounding, selection bias, data leakage, mediation, and post-treatment variables.

6. Evaluate counterfactual, scenario, and what-if analyses without overstating their evidential basis.

7. Convert model outputs into decisions using utility, constraints, thresholds, and human oversight.

8. Apply governance, ethical, and professional principles to explainable and causal analytics.

The Meaning of Explainability

Explainability concerns the extent to which a human can understand the basis, behaviour, and limitations of a model or prediction. Interpretability is sometimes used as a synonym, although a useful distinction is that an interpretable model is understandable by its construction, whereas an explainable model may require an additional explanation method. A linear regression with a small number of meaningful variables is typically interpretable. A complex neural network may be explainable through post-hoc tools, but the explanation is an approximation rather than the model itself.

An explanation has an audience and a purpose. A data scientist may need a global account of model behaviour for debugging. A compliance officer may need evidence that protected characteristics were not used improperly. A credit applicant may need a concise reason for an adverse decision. A senior manager may need to know whether an intervention is likely to produce measurable value. These are different explanatory requirements, and no single technique satisfies all of them.

Explanations should be evaluated along several dimensions:

• Fidelity: whether the explanation accurately represents the model or the underlying phenomenon.
• Stability: whether similar cases receive similar explanations.
• Comprehensibility: whether the intended audience can understand the explanation.
• Actionability: whether the explanation identifies factors that can realistically be changed.
• Completeness: whether important qualifications and uncertainty are disclosed.
• Consistency: whether explanations align with policy, domain knowledge, and observed data.

A simple explanation can be persuasive while being misleading. For example, stating that “income was the most important factor” does not show that increasing income would reduce default risk. Importance is not causality, and a model explanation is not automatically a justification for an intervention.

Levels of Model Explanation

Global explanations

Global explanations describe general model behaviour across the population. In a tree-based model, permutation importance measures how much predictive performance declines when a feature is randomly shuffled. A large decline suggests that the feature contributes predictive information, although correlated features can make the result difficult to interpret. The model may distribute importance across correlated variables or use one variable as a substitute for another.

Partial dependence plots estimate the average predicted response as one feature varies while other features are averaged over the data. They can reveal non-linearity, thresholds, and diminishing returns. However, they may evaluate combinations of feature values that are rare or impossible in the real population. When variables are strongly correlated, the averaging operation can produce unrealistic cases.

Individual conditional expectation plots extend partial dependence by showing the prediction path for each observation. They reveal heterogeneity hidden by an average curve. For example, an average effect of account age may conceal that the model responds differently for new and established customers. Accumulated local effects are often preferable when feature correlations make partial dependence unreliable, because they focus on changes within regions supported by the data.

Global surrogate models approximate a complex model using a simpler model, such as a decision tree. Their value depends on fidelity: a readable surrogate with poor agreement is not a trustworthy explanation. Global explanations are useful for model validation, feature review, and strategic diagnosis, but they do not explain an individual decision.

Local explanations

A local explanation describes why a particular observation received a prediction. SHAP methods assign contributions to features using a cooperative-game concept: the prediction is decomposed into a baseline value plus feature contributions. Positive contributions push the prediction upward and negative contributions push it downward relative to the baseline. SHAP values are useful for ranking influential features for an individual case, but their interpretation depends on the background distribution, feature dependence, and the model-specific implementation.

LIME fits a simple model around one observation by perturbing nearby inputs. It can provide an intuitive local approximation, but explanations may change when the perturbation procedure or random seed changes. Counterfactual explanations answer a different question: what is the smallest specified change that would produce a desired alternative outcome? “A missed payment was the main adverse factor” is descriptive; “with two fewer missed payments, the predicted classification would change” is counterfactual. A responsible counterfactual must also satisfy feasibility, temporal, legal, and ethical constraints. Age or protected status, for example, may be immutable or inappropriate to present as a recommended change.

Explanations for different stakeholders

A technical explanation may show calibration curves, error decomposition, and feature dependence. A manager may require a concise account of expected benefit, cost, and operational risk. An affected individual may require a reason that is specific, understandable, and contestable. These forms should be consistent with one another, but they need not contain the same level of detail. Explanations should never conceal uncertainty merely because a simpler message is more convenient.

Uncertainty Is Part of the Prediction

A prediction is an estimate, not a fact. Uncertainty arises from measurement error, sampling variation, model approximation, changing environments, and incomplete knowledge. For classification, a predicted probability should be distinguished from a hard class label. A probability of 0.70 means that the model assigns a risk estimate of 70 percent under conditions represented by its calibration data; it does not guarantee that the individual will experience the event.

Calibration assesses whether predicted probabilities correspond to observed frequencies. If among cases assigned a probability near 0.70 approximately 70 percent experience the outcome, the model is well calibrated in that region. Discrimination measures whether high-risk cases are ranked above low-risk cases; a model can discriminate well while being poorly calibrated. Both properties matter when decisions depend on expected loss.

Prediction intervals, confidence intervals, bootstrap distributions, and conformal prediction methods can communicate uncertainty in regression or classification settings. Their interpretation depends on assumptions and on whether future data resemble development data. Distribution shift, concept drift, and selective missingness can make historical uncertainty estimates too optimistic. A responsible report therefore states the population, time period, validation design, and conditions under which uncertainty estimates are credible.

Decision-makers should also see the consequences of errors. A false negative in medical triage may be more costly than a false positive, while an unnecessary maintenance visit may be less costly than a catastrophic equipment failure. The threshold should therefore be selected using costs, benefits, capacity, and policy—not merely by choosing 0.50.

The Causal Boundary: Prediction Is Not Intervention

Prediction concerns the distribution of an outcome conditional on observed information, commonly written as P(Y | X). Causal inference concerns how the outcome would change if an intervention changed X, represented conceptually by P(Y | do(X=x)). The distinction is fundamental. A variable can be highly predictive because it is a proxy for a cause, a consequence of the outcome, or a marker of an underlying process. Changing that variable may not change the outcome.

Suppose a model predicts employee turnover using participation in an internal support programme. High participation may be associated with turnover because employees already considering departure seek support. Removing the programme would not necessarily reduce turnover. The variable is predictive, but the observed association does not establish a treatment effect.

Causal analysis requires an explicit estimand, such as the average treatment effect: the expected difference in outcomes if a population received treatment compared with the same population not receiving it. Because the same unit cannot usually be observed simultaneously under both conditions, causal claims require design assumptions or experiments.

Causal diagrams and sources of bias

Directed acyclic graphs help analysts represent assumed relationships. A confounder affects both the treatment and the outcome. If prior health affects both treatment selection and recovery, comparing treated and untreated patients without adjustment may produce a biased treatment estimate. A mediator lies on the pathway from treatment to outcome; controlling for it may remove part of the total effect. A collider is influenced by two variables; conditioning on it can create a spurious association.

Common threats include:

• Confounding: important common causes are omitted or inadequately measured.
• Selection bias: the analysed sample is formed by a process related to treatment or outcome.
• Data leakage: information unavailable at decision time, or caused by the outcome, enters the model.
• Post-treatment adjustment: variables affected by the intervention are controlled for inappropriately.
• Treatment-effect heterogeneity: an average effect hides meaningful differences across groups.
• Interference: one unit’s treatment affects another unit’s outcome, violating the assumption of independent units.

Randomized experiments provide strong identification when properly designed, but practical, ethical, or financial constraints often require observational methods. Matching, inverse-probability weighting, regression adjustment, doubly robust estimation, instrumental variables, regression discontinuity, and difference-in-differences can be valuable, but each depends on assumptions that must be examined rather than merely named.

Counterfactual and Scenario Analysis

Counterfactual analysis asks what would have happened under an alternative condition. In causal inference, a counterfactual is tied to a well-defined intervention and a population of potential outcomes. In explainable AI, a counterfactual explanation often means a minimal input change that alters a model’s output. These uses are related but not identical. A model-generated counterfactual may describe how the predictor would respond, not how the real-world outcome would respond after an intervention.

Scenario analysis changes several assumptions to explore possible futures. A retailer might examine demand under different prices, advertising budgets, and delivery times. Scenario analysis is useful for planning, but it is conditional: its credibility depends on whether the specified relationships remain valid under the scenario. Stress testing goes further by examining adverse but plausible conditions, such as supply disruption or a sudden increase in fraud.

A sound analysis specifies:

1. the baseline and decision time;

2. the variables that may be changed;

3. constraints and impossible combinations;

4. the outcome and time horizon;

5. uncertainty and sensitivity to assumptions;

6. the distinction between model response and real-world causal effect.

Sensitivity analysis tests whether a recommendation changes when assumptions, costs, thresholds, or effect estimates vary. If a decision is reversed by a small plausible change, it should be presented as fragile rather than certain.

From Prediction to Decision Analytics

Decision analytics combines predictions with objectives, constraints, actions, and consequences. A useful formulation is:

Choose action a that maximizes expected utility:

EU(a | x) = Σᵧ P(y | x, a) U(a, y)

where x represents available information, y possible outcomes, and U the organization’s utility or loss function. In causal decision-making, P(y | x, a) should ideally represent the outcome under action a, not merely a prediction based on historical observations. If the model only predicts risk without estimating treatment effects, it may support prioritization but not prove that intervention will be effective.

Decision thresholds are operational rules derived from consequences. A hospital may prioritize a patient when expected harm from delay exceeds the cost of additional assessment. A bank may refer an application for human review when predicted risk is near the policy boundary or when the case is outside the training distribution. Capacity constraints can transform an individual ranking problem into a resource-allocation problem.

Decision quality should be assessed at two levels. Model-level metrics include discrimination, calibration, precision, recall, and stability. Decision-level metrics include net benefit, cost savings, service quality, equity, intervention response, and unintended consequences. A model with a slightly lower area under the curve may create greater value if it is better calibrated, easier to govern, or more effective when linked to an intervention.

An Applied Case: Predicting Customer Churn and Choosing Retention Actions

Background

Consider an illustrative telecommunications company, Northstar Mobile, serving subscription customers. The company trains a gradient-boosted model to predict whether a customer will cancel service within the next 60 days. Variables include tenure, recent service interruptions, payment history, support contacts, plan type, data usage, and recent changes in monthly charges.

Situation and problem

The model identifies customers at high risk of churn with strong validation performance. Marketing proposes sending all high-risk customers a large discount. Finance objects that discounts may be wasted on customers who would have stayed anyway. The analytics team is asked to explain the model and recommend a defensible policy.

Analysis

Global analysis shows that service interruptions, recent charge increases, and repeated support contacts are influential. Local SHAP explanations indicate that a particular customer’s high score is driven mainly by three interruptions and a recent price change. However, the explanations do not show that a discount will prevent cancellation. In fact, support contacts may be a consequence of dissatisfaction, while a price change may be associated with an unobserved change in household circumstances.

The team examines calibration by risk band and finds that the model is reliable for ordinary customers but overpredicts risk for a small segment with very short tenure. It also checks whether any features are post-outcome or unavailable at the time a retention decision would be made. A temporary “account closure inquiry” variable is excluded because it is too close to the outcome and would cause leakage in deployment.

Approach and strategy

Rather than treating the churn score as a treatment recommendation, Northstar conducts a randomized retention experiment among eligible high-risk customers. Customers are randomly assigned to no offer, a service-recovery message, a modest discount, or a technical-support intervention. Outcomes include retention, margin, complaint volume, and subsequent usage. The experiment estimates incremental retention rather than merely observed churn risk.

The decision policy combines predicted baseline risk with estimated incremental effect and intervention cost. A high-risk customer who is unlikely to respond to a discount is not automatically targeted. Customers with service interruptions receive service recovery where the experiment supports it, while uncertain cases are referred for review. The dashboard displays risk, uncertainty, recommended action, evidence source, expected net value, and a monitoring flag for data drift.

Outcome

The company discovers that discounts produce positive incremental value for one segment but little value for another. Service recovery is more effective among customers with repeated interruptions. The final policy reaches fewer customers than the original proposal but produces better net value and clearer accountability. The model remains useful for prioritization, while causal evidence determines which action to take.

Learning points

The case illustrates four principles. A predictive explanation is not a treatment effect. Leakage can make a model appear powerful while making deployment invalid. Decisions require costs, capacity, and action effectiveness. Finally, experimentation and monitoring are part of the analytics system, not activities performed only after modelling is complete.

Professional and Governance Considerations

Explainability is connected to contestability and accountability. An affected person should be able to understand the significant factors in a decision, correct inaccurate data, and request appropriate review. Organizations should document the model’s purpose, population, features, training period, validation design, known limitations, decision threshold, human roles, and escalation process.

Protected characteristics should not be treated simplistically. Excluding sensitive variables does not eliminate discrimination because proxy variables may encode social disadvantage. Fairness assessment may involve group-wise calibration, error rates, selection rates, equalized measures, or individual-consistency analyses. These criteria can conflict, particularly when base rates differ, so the selected criterion must be justified in relation to the decision context and applicable law.

Privacy, security, and data minimization remain important. Explanations should not reveal confidential information about other individuals or expose system vulnerabilities. Automated decisions require access controls, audit trails, versioning, incident response, and post-deployment monitoring. Human oversight should be meaningful rather than a ceremonial approval step; reviewers need authority, time, information, and training to challenge the system.

Common Failure Modes and Best Practices

A common mistake is to equate feature importance with causal importance. Another is to present a single local explanation as if it were a complete account of model behaviour. Analysts also frequently overlook correlated variables, extrapolate beyond the data, report probabilities without calibration, and hide uncertainty behind precise-looking scores.

Further failures occur when a model is optimized separately from the action it supports. A highly accurate risk model may have little value if no effective intervention exists. Conversely, a modestly accurate model can be valuable when paired with a low-cost, effective action. Thresholds should not be copied from another organization without examining local costs and capacity.

Best practice is to combine several forms of evidence: global and local explanation, calibration and discrimination, subgroup analysis, domain review, causal design where interventions are proposed, and decision-level evaluation after deployment. Every explanation should identify its scope, assumptions, and uncertainty. Counterfactual recommendations should be feasible and should not imply that individuals are responsible for factors they cannot control. Model cards, decision logs, monitoring dashboards, and periodic independent review support institutional memory and accountability.

Current Developments

Emerging work includes explanation methods for foundation models, causal representation learning, heterogeneous treatment-effect estimation, conformal uncertainty, and decision-focused learning. Large language models can generate fluent explanations, but fluency is not evidence of fidelity; generated rationales require validation against model behaviour and source data. Causal machine learning increasingly combines predictive algorithms with explicit assumptions about treatment, outcomes, and interventions. Digital twins and simulation environments support scenario analysis, although their usefulness depends on validation and the quality of the causal mechanisms represented.

The strategic direction is therefore not toward a single universal explainability tool. It is toward integrated systems in which prediction, causal evidence, uncertainty communication, decision rules, human governance, and monitoring are designed together.

Chapter Summary

Explainability helps users understand model behaviour, but explanations must be matched to audience and purpose. Global methods describe overall patterns, while local methods clarify individual predictions. Feature importance, SHAP, LIME, partial dependence, and counterfactual explanations are useful but depend on assumptions and may not be causal. Predictive probabilities require calibration and uncertainty communication.

Prediction estimates outcomes conditional on observed information; causal inference asks how outcomes would change under intervention. Confounding, leakage, selection bias, mediation, and post-treatment adjustment can invalidate causal claims. Scenario and counterfactual analyses must distinguish model responses from real-world effects and should include feasibility constraints and sensitivity analysis. Decision analytics converts evidence into actions through utility, costs, thresholds, resources, fairness, and governance. The most defensible systems combine predictive modelling with causal evaluation, experimentation where feasible, human oversight, and continuous monitoring.

Key Terms

Explainability: The extent to which model behaviour or a prediction can be understood by an intended audience.

Interpretability: Understandability arising from a model’s structure or representation.

Global explanation: An account of model behaviour across a population or dataset.

Local explanation: An account of why a model produced a prediction for one observation.

Permutation importance: The performance change caused by randomly shuffling a feature.

Partial dependence: The average predicted response as a feature varies across observations.

Individual conditional expectation: Observation-level prediction paths as a feature changes.

SHAP value: A contribution assigned to a feature relative to a baseline prediction.

LIME: A locally fitted simple model used to approximate a complex model near one case.

Counterfactual explanation: A description of changes that would produce an alternative model output.

Calibration: Agreement between predicted probabilities and observed event frequencies.

Confounding: Distortion caused by a variable related to both treatment and outcome.

Mediator: A variable on the causal pathway between treatment and outcome.

Collider: A variable influenced by two other variables, whose conditioning can induce bias.

Data leakage: Use of information unavailable at decision time or caused by the outcome.

Average treatment effect: The average difference between potential outcomes under treatment and non-treatment.

Scenario analysis: Examination of outcomes under specified alternative assumptions.

Expected utility: Probability-weighted value of possible outcomes from an action.

Concept drift: Change over time in the relationship between inputs and outcomes.

Contestability: The ability of affected people to question, correct, or seek review of a decision.

Review Questions

Short Answer Questions

1. Distinguish a global explanation from a local explanation.

2. Why does feature importance not establish causation?

3. What does calibration measure?

4. Define data leakage and explain why it is dangerous.

5. What is the difference between a predictive counterfactual and a causal counterfactual?

Long Answer Questions

1. Compare permutation importance, partial dependence, SHAP, LIME, and counterfactual explanations, including their uses and limitations.

2. Explain the difference between P(Y | X) and P(Y | do(X=x)) and discuss its importance in managerial decision-making.

3. Discuss how confounding, mediation, collider bias, and post-treatment adjustment affect causal analysis.

4. Explain how uncertainty, costs, capacity, and fairness should influence decision thresholds.

5. Develop a governance framework for an explainable predictive system used in a high-impact domain.

Analytical / Application Questions

1. A bank’s model identifies customers likely to default, and income change is its strongest feature. Analyse why the bank should not immediately conclude that increasing income will prevent default. Specify additional evidence required.

2. A hospital uses a calibrated risk model to prioritize patients but has limited specialist capacity. Design a decision policy that incorporates risk, error costs, uncertainty, and human review.

3. Using the Northstar Mobile case, propose an experiment to estimate which retention intervention creates the greatest incremental value and identify at least three threats to validity.

22. Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes prediction from causation?

A. A predictive feature must always be a direct cause of the outcome.

B. Prediction estimates outcomes conditional on information, whereas causation concerns outcomes under intervention.

C. Causal analysis never requires assumptions about data or design.

D. A highly accurate model automatically identifies effective interventions.

Correct Answer: B. Prediction estimates outcomes conditional on information, whereas causation concerns outcomes under intervention.

Question 2

What is the primary purpose of a calibration assessment?

A. To determine whether predicted probabilities correspond to observed frequencies.

B. To identify which features have the largest coefficients.

C. To prove that a treatment causes an outcome.

D. To measure whether a model is legally compliant.

Correct Answer: A. To determine whether predicted probabilities correspond to observed frequencies.

Question 3

Why can partial dependence plots be misleading with strongly correlated features?

A. They always remove all feature effects.

B. They may evaluate unrealistic combinations of feature values.

C. They can only be used with linear regression.

D. They measure treatment effects rather than predictions.

Correct Answer: B. They may evaluate unrealistic combinations of feature values.

Question 4

A model says that reducing a customer’s account age would lower predicted churn. What is the most appropriate interpretation?

A. It proves that reducing account age will lower actual churn.

B. It is a potentially invalid counterfactual because the proposed change may be infeasible or non-actionable.

C. It establishes the average treatment effect of account age.

D. It demonstrates that account age is a confounder.

Correct Answer: B. It is a potentially invalid counterfactual because the proposed change may be infeasible or non-actionable.

Question 5

In a causal diagram, a variable affected by both treatment and outcome-related processes is commonly called a:

A. Mediator

B. Collider

C. Baseline covariate

D. Calibration variable

Correct Answer: B. Collider

Question 6

Why might a model with strong discrimination still be unsuitable for resource allocation?

A. Discrimination guarantees perfect probabilities.

B. It may be poorly calibrated or disconnected from intervention costs and capacity.

C. Discrimination can only be measured for regression models.

D. Resource allocation never uses predictions.

Correct Answer: B. It may be poorly calibrated or disconnected from intervention costs and capacity.

Question 7

In the Northstar Mobile case, why was a randomized retention experiment needed?

A. To replace all predictive modelling.

B. To determine whether interventions changed retention incrementally rather than merely identifying high-risk customers.

C. To eliminate the need for monitoring.

D. To ensure every customer received a discount.

Correct Answer: B. To determine whether interventions changed retention incrementally rather than merely identifying high-risk customers.

Question 8

Which practice most directly addresses data leakage?

A. Including every variable available in the organization’s database.

B. Using only information available at the time the operational decision is made.

C. Removing all categorical variables.

D. Selecting the threshold of 0.50.

Correct Answer: B. Using only information available at the time the operational decision is made.

Question 9

A recommendation changes substantially when plausible costs and effect estimates are slightly altered. This indicates that the recommendation is:

A. Perfectly robust

B. Causally identified

C. Sensitive or fragile

D. Automatically fair

Correct Answer: C. Sensitive or fragile

Question 10

Which combination best represents defensible decision analytics?

A. Prediction alone and an unexplained automatic threshold

B. Feature importance alone and no human review

C. Predictions, uncertainty, causal evidence where relevant, costs, constraints, and governance

D. A complex model with no post-deployment monitoring

Correct Answer: C. Predictions, uncertainty, causal evidence where relevant, costs, constraints, and governance