11. Interpretability, Causality and Decision AnalyticsIntroduction and ContextData mining and feature engineering do not end when a predictive model produces a high accuracy score. In many organisational settings, the more difficult question is: why did the model make this prediction, and what should a decision-maker do about it? A credit institution may need to explain why an application was rejected; a hospital may need to understand why a patient was classified as high risk; a marketing team may want to know whether a campaign caused additional purchases rather than merely coincided with them. These questions move analysis beyond prediction toward interpretation, causal reasoning and decision analytics. Interpretability concerns the extent to which a human can understand the relationship between model inputs, model behaviour and model outputs. Causality asks whether changing one factor would produce a change in an outcome. Decision analytics connects evidence to action by considering objectives, constraints, risks and possible consequences. The three ideas are related but not interchangeable. A model may be highly interpretable without identifying causal effects. A causal relationship may exist even when a complex model is difficult to explain. A statistically strong pattern may be unsuitable for action if it is unstable, unfair or too costly to exploit. This chapter develops a practical and analytical framework for interpreting mined patterns and engineered features. It examines global and local explanations, partial dependence, interaction effects, SHAP-style attribution and counterfactual reasoning. It then distinguishes prediction, association and causation, introducing causal diagrams, potential outcomes and intervention-based thinking. Finally, it shows how analytical findings can be translated into defensible managerial decisions. Learning ObjectivesBy the end of this chapter, the reader should be able to: 1. distinguish global interpretability from local explanation; 2. explain the purposes and limitations of partial dependence, accumulated local effects and feature-attribution methods; 3. interpret SHAP-style values and counterfactual explanations appropriately; 4. distinguish predictive usefulness, association and causal effect; 5. use causal diagrams and potential-outcome reasoning to identify confounding and inappropriate controls; 6. evaluate explanations for stability, fairness, fidelity and actionability; 7. translate model outputs and mined patterns into decisions under organisational constraints; and 8. identify ethical, governance and professional risks in automated decision systems. From Accurate Prediction to Understandable EvidenceInterpretability is not a single property. It depends on the audience, the purpose of analysis and the type of model. A data scientist may regard a coefficient table as interpretable, whereas a customer may require a plain-language explanation of an individual decision. A regulator may need evidence that the system is consistently applied and does not use prohibited or proxy variables. A manager may be less interested in a complete mathematical description than in knowing which interventions are feasible and likely to improve outcomes. Two distinctions are especially useful. Global interpretability describes the overall behaviour of a model: which variables matter, how their values influence predictions, whether relationships are monotonic and whether interactions exist. Local interpretability explains one prediction or a small set of predictions. A local explanation might state that a particular application was assigned high risk because of recent arrears, high utilisation and short employment tenure. A model explanation can be either model-specific or model-agnostic. Linear regression and decision trees have structures that can be inspected directly, although even these models may become difficult to understand after extensive feature engineering. Model-agnostic methods treat the fitted model as a function and approximate or analyse its input-output behaviour. They can be applied to random forests, gradient-boosting models and neural networks, but their explanations may be approximations rather than literal descriptions of the model. Interpretability should also be distinguished from transparency, explainability and comprehensibility. Transparency refers to whether the internal mechanism is directly inspectable. Explainability refers to methods that communicate reasons for outputs. Comprehensibility concerns whether the explanation can be understood by its intended audience. A model may be transparent to a specialist but incomprehensible to a non-technical user. Global Views of Model BehaviourFeature importance and its cautionsA first global view often comes from feature importance. In a linear model, the sign and magnitude of a coefficient provide information about the direction and strength of association, but only after considering scaling, transformations, regularisation and correlation among predictors. In a tree ensemble, impurity-based importance measures how much a variable contributes to splits. Such measures can favour variables with many possible split points or high cardinality. Permutation importance offers an alternative: the values of one feature are disrupted and the resulting deterioration in predictive performance is measured. Permutation importance is often more meaningful operationally, but it too can mislead when features are correlated. If two variables contain overlapping information, permuting one may have little effect because the model can rely on the other. Conversely, permuting a variable in a way that creates unrealistic combinations can exaggerate its apparent importance. Importance therefore should be interpreted as contribution to predictive performance under a particular data distribution, not as proof that a variable causes the outcome. Partial dependence and related toolsThe partial dependence function estimates the average model prediction as a feature is varied across a range of values while the other features are averaged over the data. For a feature x_j, it can be represented conceptually as: PD(x_j) = E_X−j[f(x_j, X−j)] where f is the fitted model and X−j denotes the remaining features. A partial-dependence plot may reveal a linear, threshold, U-shaped or saturating relationship. For example, a churn model may show that predicted churn increases sharply after the number of unresolved complaints exceeds three. Partial dependence is a model-based description, not a causal effect. It can be unreliable when the selected value of x_j is combined with other variables in ways rarely or never observed. If income and occupation are strongly related, asking the model to predict for a very low income combined with a specialised occupation may create implausible profiles. In such cases, individual conditional expectation plots, which display predictions for each observation as the feature changes, or accumulated local effects plots, which focus on changes within observed regions, may provide a safer view. A partial-dependence curve also averages across heterogeneous cases. A positive average effect may conceal subgroups for which the relationship is negative. Therefore, analysts should examine distributions, subgroup plots and interaction effects rather than relying on one global curve. Interactions and feature engineeringAn interaction exists when the effect of one feature depends on the value of another. A promotional discount may increase purchases among occasional customers but have little effect among customers who would purchase anyway. Feature engineering can represent this relationship explicitly through interaction terms, ratios, rolling measures or domain-specific indicators. However, engineered features may also make explanations harder. A model may use a ratio whose numerator and denominator are not familiar to decision-makers, or a rolling statistic that incorporates information from a time period after the decision point. Interpretation must therefore include a review of feature construction. Analysts should document the source variables, time window, aggregation rule, missing-value treatment and intended meaning of every important feature. A feature that is highly predictive because it contains post-outcome information is not a useful basis for prospective decisions. Local Explanations: Attribution and CounterfactualsSHAP-style attributionSHAP-style methods explain an individual prediction by assigning contributions to features relative to a baseline prediction. If the baseline is the average model output, each feature receives a positive or negative contribution, and the contributions sum to the difference between the baseline and the particular prediction. In an additive form: f(x) = φ0 + φ1 + φ2 + ... + φp Here, φ0 is the baseline and φj is the contribution attributed to feature j for observation x. A positive value moves the prediction upward; a negative value moves it downward. The underlying idea is related to cooperative game theory: features are treated as contributors to a prediction, and contributions are calculated across possible feature coalitions. Different implementations use different approximations and assumptions. Tree-specific methods may calculate values efficiently for tree models, while model-agnostic approaches may rely on sampling or a background dataset. SHAP-style explanations are useful for comparing cases, identifying unusual decisions and communicating the relative direction of evidence. A loan officer might see that high debt utilisation and recent delinquency increased estimated default risk, while stable income and long tenure reduced it. Yet attribution does not mean that changing a feature will necessarily change the outcome. If debt utilisation is a consequence of financial distress, reducing it may be difficult, and the attribution does not establish that reducing it would eliminate risk. Correlated features present a further difficulty. If income and occupational status encode similar information, attribution may be divided among them or assigned unevenly depending on the method. A feature can receive a low attribution even though it is substantively important because another correlated feature carries the predictive signal. Explanations should therefore be interpreted in groups of related variables where appropriate. Counterfactual explanationsA counterfactual explanation describes a plausible change to an input that would alter the model’s output. For example: “The application would have crossed the approval threshold if monthly debt payments had been 120 units lower, all else remaining equal.” Counterfactuals are naturally action-oriented because they express the difference between the observed case and a nearby alternative. A useful counterfactual should satisfy several conditions: - Validity: it changes the model output in the intended direction.
- Proximity: it is not unnecessarily far from the observed case.
- Plausibility: the proposed values are realistic and consistent with one another.
- Actionability: the affected variables can be changed by the person or organisation.
- Sparsity: it changes a limited number of features, avoiding an unrealistic complete redesign of the case.
- Stability: similar cases should not receive radically different explanations without substantive reason.
Counterfactuals require temporal and institutional reasoning. Age cannot normally be reduced, a past missed payment cannot be erased, and changing education level may not be an immediate intervention. Some variables are immutable, some are difficult to change and some are controlled by the organisation rather than the individual. A system that tells a rejected applicant to “change nationality” or “increase age” is not merely unhelpful; it may violate ethical and legal standards. Counterfactuals can also be misleading when the model is poorly calibrated or relies on non-causal proxies. If changing a proxy variable would not alter the underlying situation, the suggested action may produce a superficial improvement without reducing the actual risk. Prediction, Association and CausationThree different questionsPredictive analysis asks: given what we know, what outcome is likely? Association asks: which variables occur together or vary together? Causal analysis asks: what would happen to the outcome if an intervention changed a factor? These questions often use overlapping data but require different reasoning. Suppose customers who receive a retention call are more likely to cancel their subscriptions. A predictive model may correctly use “received retention call” as a strong predictor of cancellation. The association may reflect the fact that calls are directed to already dissatisfied customers. It would be incorrect to conclude that the call causes cancellation. In fact, the call may reduce cancellations compared with what would have happened without it. Prediction can use variables that are not causes, including reliable proxies, future-revealing indicators and consequences of the outcome. Causal decision-making cannot generally do so. A variable may improve accuracy while being unsuitable for intervention or policy. Potential outcomes and causal effectsThe potential-outcomes framework defines a causal effect by comparing two possible outcomes for the same unit: the outcome if it receives treatment and the outcome if it does not. For individual i, these are often written as Y_i(1) and Y_i(0). The individual treatment effect is Y_i(1) − Y_i(0), but both potential outcomes cannot normally be observed for the same unit at the same time. This is the fundamental problem of causal inference. Researchers therefore estimate average effects under assumptions concerning treatment assignment, measurement, interference and missing potential outcomes. Randomised experiments address confounding through random assignment. In observational data, analysts may use matching, regression adjustment, inverse-probability weighting, instrumental variables, regression discontinuity or difference-in-differences, depending on the setting and assumptions. Causal diagramsDirected acyclic graphs (DAGs) help analysts express assumptions about causal structure. A confounder is a variable that influences both treatment and outcome. For example, prior customer dissatisfaction may influence both whether a retention call is made and whether cancellation occurs. Adjusting for dissatisfaction may be necessary to estimate the call’s effect. A mediator lies on the pathway from treatment to outcome. If a training programme improves skills and skills improve productivity, skills may mediate the effect. Controlling for the mediator can remove part of the treatment effect. A collider is a variable influenced by two other variables. Conditioning on a collider can create a spurious association. For example, conditioning on being selected for a programme when both motivation and manager recommendation influence selection may induce a relationship between motivation and recommendation that was not present initially. Causal adjustment is consequently not a mechanical instruction to include every available variable. The correct adjustment set depends on the causal question and the assumed data-generating process. Feature selection procedures optimised for prediction do not automatically identify a valid causal adjustment set. From Explanations to DecisionsDecision analytics begins when an organisation specifies an objective and a feasible action. A model score alone is not a decision rule. A decision usually depends on expected benefits, costs, capacity, uncertainty and constraints. If an intervention has benefit B when successful, cost C, and an estimated probability p of success, a simplified expected-value rule might recommend intervention when pB − C is positive. Real decisions may also include risk aversion, fairness constraints, service-level requirements and long-term effects. A practical workflow is as follows. First, define the decision, the unit of analysis, the decision time and the available actions. Second, identify the outcome and establish whether the objective is prediction, intervention, ranking or explanation. Third, audit features for leakage, proxies, instability and actionability. Fourth, evaluate predictive performance and calibration on temporally appropriate data. Fifth, use global and local explanations to understand model behaviour. Sixth, assess causal evidence before describing an action as effective. Seventh, simulate or pilot the decision policy, monitor outcomes and revise it when evidence changes. The distinction between risk prediction and uplift modelling is particularly important. A risk model identifies people likely to experience an outcome. An uplift model estimates how the outcome changes because of treatment. Targeting the highest-risk customers may be inefficient if they are unlikely to respond to the intervention. Decision analytics should seek those for whom the intervention is both needed and effective. Illustrative Case: Retention Decisions in a Subscription BusinessBackgroundConsider an illustrative case involving “StreamBox,” a fictional subscription video service. The company experiences monthly cancellations and wants to allocate a limited number of retention offers. Its data warehouse contains viewing frequency, payment history, support contacts, subscription tenure, device information, previous offers and cancellation records. Situation and problemA gradient-boosting model predicts cancellation within the next 30 days. Its area under the ROC curve is strong on a hold-out sample. The marketing team proposes sending offers to the 10 percent of customers with the highest predicted cancellation risk. However, the team also observes that customers who previously received retention calls cancel at higher rates. Some managers conclude that calls are ineffective or harmful. Interpretation and analysisGlobal importance analysis identifies recent support contacts, declining viewing hours, payment failures and previous retention calls as influential features. Partial-dependence analysis suggests that cancellation risk rises after repeated payment failures and declines with sustained viewing. Local SHAP-style explanations show why individual customers have high scores. For one customer, declining usage and two payment failures dominate the prediction; for another, a recent complaint and short tenure are more important. The analysts then inspect the time sequence. Retention calls are made primarily to customers who have already shown signs of dissatisfaction. The association between calls and cancellation is therefore confounded by prior dissatisfaction and declining engagement. The predictive model is using the call indicator as a marker of risk, not necessarily as evidence that calls cause cancellation. The business question is not simply who will cancel, but who will cancel because the company fails to intervene and who will respond to an offer. The company conducts a controlled experiment in which eligible customers are randomly assigned to an offer, a service message or no intervention. Outcomes include cancellation, revenue after three months, offer cost and customer complaints. Approach and decision policyThe company develops two analytical components. The first predicts baseline cancellation risk. The second estimates incremental retention attributable to each intervention. Counterfactual reasoning is used at the policy level: what is the expected outcome for similar customers under an offer compared with no offer? The retention policy prioritises customers with positive expected incremental value, not merely high predicted risk. Explanations are presented differently to different audiences. Customer-service staff receive concise local reasons and permitted intervention options. Managers receive aggregate patterns, subgroup performance and experiment results. Governance staff receive documentation of data sources, excluded variables, fairness checks and monitoring procedures. OutcomeThe illustrative evaluation finds that some high-risk customers are unlikely to respond because their payment method is no longer valid, while some medium-risk customers respond strongly to a service-quality intervention. The revised policy reduces indiscriminate discounting and directs different interventions to different segments. The company continues to monitor performance because customer behaviour, pricing and product features change over time. Learning pointsThe case demonstrates four principles. First, predictive importance is not causal importance. Second, a high-risk customer is not automatically a high-value intervention target. Third, local explanations help staff understand individual cases but do not replace experimental evidence. Fourth, decision quality depends on aligning features, explanations, interventions and evaluation with the actual business objective. Challenges, Risks and Common MistakesInterpretability methods can create false confidence. A visually appealing explanation may be unstable across samples, dependent on arbitrary baselines or based on unrealistic feature combinations. Analysts should test explanation stability through resampling, temporal validation and comparison of alternative methods. A second risk is explanation inconsistency. A model may produce explanations that vary substantially for nearly identical cases, especially near a decision threshold. This can undermine trust and create fairness concerns. Explanations should be evaluated for fidelity to the model, consistency, completeness, comprehensibility and actionability. A third risk is confusing explanation with justification. An explanation reports how a model arrived at an output; it does not establish that the output is correct, fair or desirable. A discriminatory model can be explained accurately. Similarly, a post hoc explanation may rationalise a decision without faithfully representing the actual computational process. Other common mistakes include interpreting coefficients without considering scaling and collinearity, treating feature importance as causal evidence, ignoring missingness mechanisms, using post-treatment variables, accepting impossible counterfactuals, and communicating probabilities without calibration or base-rate context. Data leakage is particularly damaging: a feature created after the decision point may appear highly informative during development but will not be available at deployment. Ethics, Governance and Professional ResponsibilityInterpretability is connected to accountability. Organisations should be able to identify who owns a model, what data it uses, which decisions it influences and how affected people can challenge an outcome. Documentation should include the intended use, population, training period, target definition, performance by subgroup, known limitations, explanation method and monitoring plan. Fairness assessment should examine both outcomes and explanations. A model may have similar aggregate accuracy across groups but produce different error patterns, calibration quality or access to favourable interventions. Proxy variables require special attention because location, language, device type or purchasing behaviour may encode protected characteristics. Removing an explicit sensitive attribute does not guarantee that its information has been removed. Privacy also matters. Detailed explanations may reveal sensitive information or expose characteristics that individuals did not realise were being inferred. Explanations should follow data-minimisation principles and be tailored to legitimate purposes. In high-impact domains, human review should be meaningful rather than a nominal approval step. Reviewers need authority, time and information to challenge automated recommendations. From a professional perspective, analysts should distinguish observed evidence from assumptions, report uncertainty and avoid overstating causal conclusions. Reproducible code, versioned data definitions and audit logs support later investigation. Model cards, data sheets and decision registers can complement technical documentation. Technology and Emerging DirectionsModern machine-learning platforms increasingly provide automated feature-importance reports, interactive dependence plots, local attribution dashboards and counterfactual generators. These tools accelerate analysis but do not eliminate the need for domain knowledge. Automated explanation systems may optimise for visual simplicity rather than causal validity or practical actionability. Emerging work is moving toward causal machine learning, heterogeneous treatment-effect estimation, invariant prediction and policy learning. These methods attempt to estimate which intervention works for which population and under what conditions. Integration with experiment platforms allows models to recommend actions while continuously comparing them with control groups. At the same time, large language models are being used to convert technical explanations into natural language. Such translations require verification because fluent language can conceal uncertainty or introduce unsupported claims. A mature analytics function is therefore likely to combine predictive models, causal studies, experimentation, monitoring and human judgement rather than treating interpretability as a final visualisation step. Best Practices for Defensible Decision AnalyticsA defensible practice follows several principles: - define the decision and intervention before selecting explanatory tools;
- separate predictive, descriptive and causal claims in reports;
- use global explanations to understand overall behaviour and local explanations to investigate individual cases;
- inspect correlated features, subgroup variation and feature-construction logic;
- prefer plausible, actionable counterfactuals and label them as model-based suggestions;
- validate explanations for fidelity, stability and sensitivity to the background data;
- use experiments or credible causal designs to evaluate interventions;
- report uncertainty, calibration, data limitations and performance drift;
- document governance responsibilities, appeal mechanisms and permitted uses; and
- monitor both model outcomes and the consequences of decisions made from those outcomes.
Chapter SummaryInterpretability explains how a model behaves globally or why it produced a particular prediction locally. Feature importance, permutation analysis, partial dependence, accumulated local effects and interaction analysis provide different views, each with assumptions and limitations. SHAP-style attribution decomposes an individual prediction relative to a baseline, while counterfactual explanations describe plausible changes that could alter an output. Neither method, by itself, proves that an intervention will change the real-world outcome. Prediction concerns likely outcomes, association concerns co-occurrence and causal analysis concerns the effect of intervention. Potential outcomes and causal diagrams help identify confounding, mediation and collider bias. Decision analytics combines evidence with objectives, costs, constraints and feasible actions. The illustrative retention case showed why targeting high-risk individuals is not equivalent to targeting those who will benefit from treatment. Ethical governance requires transparency, fairness assessment, privacy protection, documentation, human accountability and honest communication of uncertainty. Key TermsInterpretability: The degree to which a human can understand a model’s behaviour or output. Global explanation: An account of the model’s overall relationships and decision patterns. Local explanation: An account of why a model produced one particular prediction. Model-agnostic method: An interpretability method that can analyse models through their input-output behaviour rather than internal structure. Permutation importance: The reduction in predictive performance observed after disrupting a feature’s values. Partial dependence: The average predicted response as one or more features are varied across observations. Individual conditional expectation: Observation-level curves showing how predictions change as a feature varies. Accumulated local effects: A feature-effect method that aggregates local changes within regions supported by observed data. Interaction effect: A situation in which the influence of one feature depends on another feature. SHAP-style attribution: Additive allocation of a prediction’s difference from a baseline among input features. Baseline prediction: The reference output from which local feature contributions are measured. Counterfactual explanation: A plausible input change that would produce a specified alternative model output. Actionability: The extent to which a suggested change can realistically be implemented. Association: A statistical relationship or co-occurrence between variables. Causal effect: The change in an outcome produced by an intervention or treatment. Confounder: A variable affecting both an exposure and an outcome, creating a misleading association. Mediator: A variable on the causal pathway between an exposure and an outcome. Collider: A variable influenced by two other variables, whose conditioning can induce bias. Uplift modelling: Estimation of the incremental effect of an intervention for individuals or groups. Decision analytics: The use of analytical evidence, objectives, constraints and uncertainty to select actions. Review QuestionsShort Answer Questions1. Distinguish global interpretability from local interpretability. 2. Why can partial-dependence plots be misleading when features are strongly correlated? 3. What does a positive SHAP-style contribution indicate? 4. Why is a predictive relationship not necessarily a causal relationship? 5. State three desirable properties of a counterfactual explanation. Long Answer Questions1. Explain permutation importance, partial dependence and accumulated local effects, including the circumstances in which each may be unreliable. 2. Discuss how SHAP-style attribution can support professional decision-making and explain its limitations with correlated features. 3. Explain the potential-outcomes framework and the role of randomisation in causal inference. 4. Using a causal diagram conceptually, distinguish confounding, mediation and collider bias. 5. Describe a complete workflow for converting a predictive model into a defensible organisational decision policy. Analytical / Application Questions1. A bank’s model identifies previous collection calls as the strongest predictor of default. Explain why this finding should not immediately be interpreted as evidence that collection calls cause default, and propose a suitable evaluation strategy. 2. A counterfactual explanation tells an applicant to change age and residential location to obtain approval. Assess the explanation using validity, plausibility, actionability, fairness and governance criteria. 3. A retailer can offer discounts to only 5 percent of customers. Explain why selecting customers with the highest cancellation risk may be inferior to using an uplift or incremental-effect approach, and identify the evidence needed before deployment. 22. Multiple Choice Questions with AnswersQuestion 1Which statement best distinguishes prediction from causation? A. A predictive feature must always be a cause of the outcome. B. Prediction estimates likely outcomes, whereas causation concerns the effect of an intervention. C. Causal analysis never uses observational data. D. Prediction and causation are identical when model accuracy is high. Correct Answer: B. Prediction estimates likely outcomes, whereas causation concerns the effect of an intervention. Question 2What is a major limitation of a partial-dependence plot? A. It cannot be used with machine-learning models. B. It always provides individual-level explanations. C. It may evaluate unrealistic combinations of correlated feature values. D. It measures treatment effects directly. Correct Answer: C. It may evaluate unrealistic combinations of correlated feature values. Question 3In a SHAP-style explanation, a positive feature contribution generally means that the feature: A. moves the prediction above the selected baseline. B. is proven to cause the outcome. C. has the highest population frequency. D. must be changed by the decision-maker. Correct Answer: A. moves the prediction above the selected baseline. Question 4Which characteristic is most important for an actionable counterfactual? A. It changes as many features as possible. B. It proposes a realistic change that the relevant actor can implement. C. It uses only immutable variables. D. It ignores relationships among features. Correct Answer: B. It proposes a realistic change that the relevant actor can implement. Question 5A variable affects both treatment assignment and the outcome. In a causal analysis, this variable is a: A. Mediator B. Collider C. Confounder D. Baseline prediction Correct Answer: C. Confounder Question 6Why might targeting customers with the highest predicted churn fail to maximise retention benefits? A. High-risk customers cannot be predicted. B. High-risk customers may be unlikely to respond to the intervention. C. Risk models never use historical data. D. Uplift models ignore treatment effects. Correct Answer: B. High-risk customers may be unlikely to respond to the intervention. Question 7Which practice is most likely to create data leakage? A. Using only variables available at the decision time. B. Splitting data by time for validation. C. Including a feature recorded after the outcome has occurred. D. Checking calibration on a hold-out sample. Correct Answer: C. Including a feature recorded after the outcome has occurred. Question 8Conditioning on a collider can: A. remove all confounding automatically. B. create a spurious association between its causes. C. guarantee exchangeability. D. convert prediction into causation. Correct Answer: B. create a spurious association between its causes. Question 9What does permutation importance primarily measure? A. The causal effect of a feature intervention. B. The change in model performance after disrupting a feature. C. The fairness of a model across groups. D. The probability that a feature is immutable. Correct Answer: B. The change in model performance after disrupting a feature. Question 10Which combination best supports a defensible managerial action? A. High accuracy alone B. A local explanation without validation C. Predictive evidence, causal or experimental evidence where needed, feasibility and governance review D. A feature ranking produced by an automated tool Correct Answer: C. Predictive evidence, causal or experimental evidence where needed, feasibility and governance review
|