Model Evaluation, Causal Analytics and Explainability

Introduction

A machine-learning model is not useful merely because it produces predictions. A professional AI system must demonstrate that its predictions are accurate for the intended population, remain reliable under changing conditions, do not exploit information unavailable at decision time, and can be interpreted sufficiently for technical, managerial, ethical, or regulatory purposes. These requirements make evaluation broader than calculating a single accuracy score.

Model evaluation concerns the design of evidence about predictive performance. Causal analytics asks a different question: whether an intervention or exposure produces an outcome. Explainability concerns how model behaviour can be understood, communicated, challenged, and governed. These three areas are related but not interchangeable. A highly accurate model may provide no causal insight, while an interpretable model may be less accurate but more appropriate for a high-stakes decision.

This chapter develops an integrated framework for selecting metrics, designing validation experiments, diagnosing error and data leakage, distinguishing prediction from causation, and applying interpretable and post hoc explanation methods.

Learning Objectives

By the end of this chapter, a student should be able to:

1. Select evaluation metrics that correspond to the task, data distribution, and decision costs.

2. Design suitable train–validation–test and resampling strategies.

3. Diagnose error, subgroup disparities, distribution shift, drift, and leakage.

4. Distinguish predictive association from causal effect.

5. Explain the assumptions and limitations of causal methods.

6. Compare intrinsic interpretability with post hoc explainability.

7. Evaluate explanations for fidelity, stability, usefulness, and fairness.

8. Design an evidence-based evaluation plan for an AI-enabled decision system.

1. Evaluation as an Evidence-Design Problem

Evaluation begins by specifying the decision context. The same model can be acceptable in one setting and unsafe in another. For example, a false negative in disease screening may delay treatment, whereas a false positive may create unnecessary clinical workload. In fraud detection, the relative costs may be different, and in recommendation systems the relevant objective may involve ranking quality rather than classification accuracy.

A rigorous evaluation specification identifies:

• the prediction target and prediction horizon;
• the unit of analysis, such as a patient, transaction, or customer;
• the population to which results are intended to generalise;
• the information available at prediction time;
• the action taken from each prediction;
• the costs of false positives, false negatives, and delayed decisions;
• acceptable operational, fairness, latency, and interpretability constraints.

The target quantity must also be defined precisely. A model estimating whether a customer will default within 90 days is different from one estimating the probability of default at any time. Ambiguous labels create apparently technical problems that are actually problems of construct definition.

Terminology

A prediction is an estimate of an unknown outcome. A label is the observed target used for supervised learning. A score is a continuous model output, often converted into a class using a threshold. A threshold is a decision boundary applied to a score. Calibration concerns agreement between predicted probabilities and observed frequencies. Discrimination concerns the ability to order or separate cases with different outcomes. Generalisation is performance on relevant future or external data. Data drift is a change in the distribution of input data; concept drift is a change in the relationship between inputs and the target.

2. Metrics for Classification, Regression and Ranking

2.1 Confusion-matrix measures

For binary classification, predictions can be represented by true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN). From these counts:

Accuracy is

Accuracy = (TP + TN) / (TP + TN + FP + FN).

It is appropriate when classes and error costs are reasonably balanced. It can be misleading under class imbalance. Sensitivity, also called recall or true-positive rate, is

TPR = TP / (TP + FN).

Specificity, or true-negative rate, is

TNR = TN / (TN + FP).

Precision, also called positive predictive value, is

Precision = TP / (TP + FP).

The F1 score is the harmonic mean of precision and recall:

F1 = 2(Precision × Recall) / (Precision + Recall),

provided the denominator is non-zero. It is useful when both precision and recall matter, but it hides their separate values and does not directly encode business costs.

The false-positive rate is FP/(FP+TN), while the false-negative rate is FN/(TP+FN). In a risk-sensitive application, reporting these rates separately is generally more informative than reporting accuracy alone.

2.2 Thresholds and cost-sensitive decisions

Many classifiers produce scores or estimated probabilities rather than final decisions. A threshold determines the operational trade-off between false positives and false negatives. Threshold selection should be made using validation data and a documented decision objective, not by silently choosing 0.5.

If a decision has false-positive cost C_FP and false-negative cost C_FN, a simple expected-cost analysis can compare candidate thresholds. The costs must be defined in the same units, such as currency, staff hours, or estimated harm. In practice, prevalence, intervention costs, capacity constraints, and downstream effects should also be considered.

2.3 ROC and precision–recall analysis

The receiver operating characteristic (ROC) curve plots TPR against FPR across thresholds. The area under the ROC curve, commonly called ROC-AUC, measures ranking discrimination across thresholds. Under standard assumptions, it can be interpreted as the probability that a randomly selected positive receives a higher score than a randomly selected negative. It does not measure calibration and may appear strong when the positive class is rare.

The precision–recall curve plots precision against recall. It is often more informative for rare-event detection because precision directly reflects the proportion of selected cases that are positive. A baseline precision depends on prevalence, so comparisons should account for the class distribution.

2.4 Probability calibration

A model is calibrated if, among cases assigned probability p, approximately a proportion p experience the outcome. Calibration can be assessed using reliability diagrams, calibration intercept and slope, and proper scoring rules. The Brier score for binary outcomes is

Brier = (1/n) Σ(i=1 to n) (p_i − y_i)^2,

where p_i is the predicted probability, y_i is 0 or 1, and n is the number of observations. Lower values are better. The Brier score reflects both calibration and, to some extent, discrimination. Calibration methods such as Platt scaling and isotonic regression must be fitted on data separate from the final test set.

A model may rank cases well but be poorly calibrated. For example, predicted risks of 0.8 may correspond to an observed frequency of 0.6. In clinical or financial decisions, this difference can materially affect resource allocation.

2.5 Regression metrics

For continuous outcomes, mean absolute error is

MAE = (1/n) Σ |y_i − ŷ_i|,

where y_i is the observed value and ŷ_i is the prediction. MAE is expressed in the target's units and gives equal linear weight to errors.

Mean squared error is

MSE = (1/n) Σ(y_i − ŷ_i)^2,

and root mean squared error is RMSE = √MSE. Squaring gives greater influence to large errors. MAE is often easier to interpret, whereas RMSE is useful when large errors are particularly undesirable.

Mean absolute percentage error can be unstable or undefined when actual values are zero or close to zero. It should not be used automatically. For skewed outcomes, analysts may consider logarithmic transformations, quantile loss, or domain-specific measures.

2.6 Ranking and recommendation metrics

In information retrieval and recommendation, the order of results may matter more than exact probabilities. Precision@k measures the proportion of the top k retrieved items that are relevant. Recall@k measures the proportion of all relevant items retrieved among the top k. Normalised discounted cumulative gain (NDCG) gives greater weight to relevant items appearing near the top and is suitable when graded relevance is available. Evaluation must reflect the exposure and feedback mechanisms of the deployed system; offline ranking scores can be misleading when users only observe items selected by an earlier recommender.

3. Validation Strategies and Experimental Design

3.1 Holdout evaluation

A basic workflow divides data into training, validation, and test sets. The training set estimates model parameters, the validation set supports model selection and hyperparameter tuning, and the test set is reserved for final performance estimation. The test set must not influence feature selection, threshold selection, calibration, or repeated experimentation without adjustment.

Random splitting is unsuitable when observations are temporally ordered, grouped, or related. If the same patient appears in both training and test data, a medical model may benefit from patient-specific information rather than learning generalisable patterns. Grouped splitting prevents such contamination.

3.2 Cross-validation

In k-fold cross-validation, data are divided into k folds. The model is trained on k−1 folds and evaluated on the remaining fold, repeating until each fold has served as validation data. The mean provides an estimate of average performance and the variation across folds indicates uncertainty due to partitioning. Stratified folds preserve approximate class proportions for classification. Grouped and time-series cross-validation should be used when dependence structures require them.

Cross-validation does not eliminate bias caused by a non-representative dataset, label problems, or leakage. All preprocessing steps that learn from data, including imputation, scaling, feature selection, and dimensionality reduction, must be fitted within each training fold. A pipeline formalises this requirement.

3.3 Temporal and external validation

For changing environments, a temporal split trains on earlier observations and tests on later observations. This better approximates deployment and can reveal degradation. External validation evaluates the model on a different hospital, region, customer segment, device, or operating period. It is stronger evidence of transportability than another random split from the same source.

A final evaluation should report point estimates together with uncertainty, such as confidence intervals obtained through bootstrap resampling or appropriate statistical methods. Comparisons between models should use paired predictions on the same cases where possible. A small difference in mean score may not be practically or statistically persuasive.

3.4 Baselines and ablation experiments

A model should be compared with meaningful baselines: a majority-class predictor, historical average, simple linear or logistic model, existing business rule, or currently deployed system. An ablation study removes feature groups or components to determine what contributes to performance. Sensitivity analyses vary thresholds, preprocessing choices, missing-data assumptions, and subgroup definitions.

A strong experiment records the data version, code version, random seeds, evaluation protocol, metric definitions, and inclusion criteria. Reproducibility is part of evaluation rather than an optional documentation exercise.

4. Diagnosing Error, Leakage and Drift

4.1 Error analysis

Aggregate metrics conceal structure. Error analysis should examine false positives and false negatives, performance by subgroup, performance across score ranges, and representative examples. Confusion matrices can be stratified by region, language, age band, device type, or operational condition where legally and ethically appropriate.

Calibration should also be checked by subgroup. Equal overall accuracy does not imply equal error rates or equal predictive value. However, fairness metrics can conflict mathematically when base rates differ, so metric selection must follow the application, legal context, and harm assessment rather than a search for a universally optimal statistic.

4.2 Data leakage

Leakage occurs when information unavailable at prediction time enters model development. Direct leakage includes using a post-outcome field, such as a discharge code to predict hospital admission. Indirect leakage may occur when a feature is generated from a dataset containing future information or when preprocessing is performed before splitting.

Leakage can produce unusually high validation performance and catastrophic deployment failure. Prevention requires defining the prediction timestamp, auditing feature-generation pipelines, fitting transformations within folds, separating entities appropriately, and conducting a temporal or external test. Feature importance alone does not reliably identify leakage.

4.3 Distribution shift and drift monitoring

Let P_train(X,Y) denote the joint distribution during development and P_deploy(X,Y) the distribution after deployment. Covariate shift occurs when P(X) changes while the conditional relationship P(Y|X) remains approximately stable. Concept drift occurs when P(Y|X) changes. Label shift refers to changes in P(Y) with a relatively stable P(X|Y).

Monitoring may include input summaries, missingness rates, category frequencies, score distributions, calibration, delayed outcome performance, and subgroup metrics. Statistical tests can flag changes, but a statistically significant shift may be operationally trivial in a large sample, while a practically important shift may be difficult to detect in a small sample. Monitoring thresholds should therefore combine statistical evidence with domain impact.

5. Prediction Is Not Causation

Prediction asks, “What outcome is likely?” Causal analysis asks, “What would happen if an intervention or exposure were changed?” The distinction is fundamental. A feature can be highly predictive without being a useful intervention target. For example, a late-stage medical treatment may predict death because it is given to severely ill patients; prescribing that treatment to everyone would not necessarily reduce mortality.

5.1 Potential outcomes

For an individual i, let Y_i(1) denote the outcome if treated and Y_i(0) the outcome if untreated. The individual treatment effect is Y_i(1)−Y_i(0), but only one of these potential outcomes is observed for each individual. The average treatment effect (ATE) is

ATE = E[Y(1) − Y(0)] = E[Y(1)] − E[Y(0)].

The average treatment effect on the treated is

ATT = E[Y(1) − Y(0) | T=1],

where T indicates treatment. Estimating these quantities requires assumptions and a design that addresses confounding.

5.2 Confounding and causal graphs

A confounder affects both treatment and outcome. If socioeconomic status influences both access to a programme and employment, an observed association between programme participation and employment may partly reflect socioeconomic status. A causal directed acyclic graph (DAG) makes assumptions explicit. Back-door paths from treatment to outcome should be blocked by adjusting for appropriate pre-treatment confounders. Adjusting for a mediator can remove part of the effect being estimated; adjusting for a collider can create bias.

Three common identification assumptions are:

• Consistency: the observed outcome under the received treatment corresponds to the relevant potential outcome.
• Conditional exchangeability: after adjustment for measured covariates X, treatment assignment is independent of potential outcomes.
• Positivity: each relevant subgroup has a non-zero probability of receiving each treatment level.

These assumptions cannot generally be verified from observational data alone. Unmeasured confounding remains a central limitation.

5.3 Causal methods

Randomised controlled trials use random assignment to support exchangeability and are often the strongest design for estimating an intervention effect, although they may be expensive or ethically impossible.

In observational studies, regression adjustment, propensity-score methods, inverse probability weighting, matching, doubly robust estimators, instrumental variables, regression discontinuity, difference-in-differences, and synthetic controls may be appropriate under different assumptions. A propensity score is the probability of treatment conditional on observed covariates. It can help balance measured covariates but does not solve unmeasured confounding.

Difference-in-differences compares changes over time between treated and comparison groups. Its key identifying condition is a credible parallel-trends assumption: absent treatment, the groups would have followed similar trends. Instrumental-variable analysis requires relevance, exclusion, and additional assumptions that are often difficult to defend. Causal machine learning methods, including heterogeneous treatment-effect models, estimate which groups may benefit more, but their validity still depends on the underlying causal design.

6. Interpretability and Explainability

Interpretability is the extent to which a model's operation can be understood directly. Explainability often refers to methods that provide human-understandable accounts of a model's predictions. The distinction is not absolute, but it is useful.

6.1 Intrinsically interpretable models

Linear and logistic regression models can be interpretable when features are well-defined, appropriately transformed, and not excessively correlated. In logistic regression,

log(p/(1−p)) = β_0 + β_1x_1 + ... + β_kx_k,

where p is the estimated probability of the outcome and β_j is the change in log-odds associated with a one-unit increase in x_j, holding other included variables constant. The odds ratio for x_j is exp(β_j). This interpretation assumes the model specification is appropriate and does not imply causation.

Decision trees, scoring systems, generalised additive models, monotonic models, and sparse rule lists can provide transparent structure. Interpretability may decrease when there are many interactions, correlated variables, proxies, or complex preprocessing steps.

6.2 Post hoc explanation methods

Feature attribution methods assign portions of a prediction to input features. SHAP methods are based on Shapley-value ideas and can provide local or aggregated explanations under a specified background distribution. They are not automatically causal, and correlated features create ambiguity in how contribution is allocated.

LIME fits a simple local surrogate around a selected instance by perturbing inputs and weighting nearby samples. Its result can depend on the perturbation distribution, distance definition, and surrogate model. Counterfactual explanations identify changes to inputs that would alter a prediction, such as “the score would fall below the threshold if debt decreased by a specified amount.” A valid counterfactual should be actionable, plausible, consistent with causal relationships, and respectful of constraints.

Partial dependence plots estimate average predicted response as a feature varies, while averaging over other observations. Accumulated local effects can be preferable when features are correlated or extrapolation is a concern. Individual conditional expectation plots display the relationship for individual cases and can reveal heterogeneity hidden by averages.

6.3 Evaluating explanations

An explanation should be assessed rather than accepted because it is visually appealing. Fidelity asks whether the explanation accurately represents the model. Stability asks whether small irrelevant changes produce similar explanations. Completeness concerns whether important model behaviour is captured. Human usefulness asks whether the intended user can understand and appropriately act on it. Explanations should also be checked for privacy leakage, discriminatory framing, and misleading causal language.

Global explanations describe general model behaviour; local explanations describe one prediction. Neither establishes that changing a feature will change the outcome. A feature attributed strongly by SHAP is not necessarily a treatment target, and a counterfactual generated by a predictive model may be infeasible in the real world.

7. An Integrated Evaluation Workflow

A robust workflow can be organised as follows:

1. Define the decision, population, outcome, time horizon, action, and harms.

2. Establish a data-access and prediction-time specification.

3. Select metrics linked to operational costs, prevalence, calibration, discrimination, fairness, and latency.

4. Split data according to time, entity, and deployment structure.

5. Build a leakage-resistant pipeline containing preprocessing and model fitting.

6. Compare with simple and current-system baselines.

7. Tune on validation data and reserve a final test set.

8. Report uncertainty, subgroup results, calibration, and error examples.

9. Conduct ablations, sensitivity analyses, and, where relevant, causal identification analysis.

10. Select an explanation approach appropriate to the user and risk level.

11. Pilot under human oversight and monitor drift, outcomes, overrides, and harms.

12. Define retraining, rollback, incident response, and documentation procedures.

8. Illustrative Case: Hospital Readmission Decision Support

Background

A hospital develops a model to estimate whether a patient will be readmitted within 30 days of discharge. The purpose is to prioritise transitional-care support, not to deny discharge or treatment. This distinction affects the acceptable error costs and the interpretation of model outputs.

Problem

An initial random split produces an AUC of 0.86, apparently strong performance. However, several features are generated after discharge, including follow-up scheduling codes and post-discharge pharmacy claims. The model also performs poorly for patients transferred from a rural partner hospital.

Analysis

The evaluation team defines the prediction time as the planned discharge decision. All features generated after that time are removed. Because patients can have multiple admissions, splitting is performed by patient, and a temporal test set is created using later admissions. The team reports AUROC, area under the precision–recall curve, sensitivity at the available care-management capacity, calibration, and subgroup performance.

A baseline logistic model and the existing rule-based screening procedure are included. The model's overall AUC decreases to 0.74 after leakage removal, but the result is more credible. Calibration reveals that predicted risks are systematically too high for the rural subgroup. An external validation on the partner hospital shows further degradation due to different coding practices and service availability.

Approach and solution

The hospital recalibrates probabilities using a validation period from the partner site, documents the calibration population, and sets a threshold based on the number of care-management appointments available. A separate model audit examines missingness and coding differences. Explanations are delivered as a concise patient-level summary showing influential documented factors, uncertainty, and a warning that associations are predictive rather than causal. Care staff can override the recommendation and must record the reason.

A causal analysis is conducted separately to evaluate whether a transitional-care intervention reduces readmission. The prediction model is not treated as evidence that changing an influential feature will reduce risk. The intervention study uses a defined comparison design and examines confounding, treatment eligibility, and outcomes.

Outcome and learning points

The revised system has lower apparent predictive performance but better temporal validity, more reliable calibration, and clearer operational meaning. Monitoring detects drift when a coding-system change alters missingness patterns. The case demonstrates that removing leakage can reduce headline metrics while improving real-world validity. It also shows why prediction, explanation, and causal intervention evaluation must be designed as related but distinct activities.

9. Governance, Ethics and Contemporary Developments

Evaluation is inseparable from governance in high-impact applications. Documentation should identify intended use, prohibited use, data provenance, known limitations, subgroup performance, monitoring plans, and accountability. Privacy risks arise when explanations reveal sensitive attributes or information about other individuals. Security threats include model extraction, adversarial manipulation, and data poisoning.

Fairness assessment should consider allocation, error rates, calibration, accessibility, and the effects of human review. A mathematically balanced metric may still produce unequal real-world benefit if groups have different access to the intervention. Impact assessment should therefore extend beyond model outputs to downstream decisions.

Contemporary evaluation increasingly includes distributionally robust testing, uncertainty estimation, conformal prediction, causal representation learning, and human–AI team evaluation. Conformal methods can produce prediction sets or intervals with finite-sample coverage guarantees under exchangeability assumptions, but coverage may weaken under distribution shift. Large language models require additional evaluation for factuality, calibration, prompt sensitivity, retrieval contamination, privacy, and harmful or manipulative outputs. Automated explanations should be audited because fluent language can create unjustified confidence.

Best Practices

• Define the deployment decision before selecting metrics.
• Use time-, group-, or site-aware splits when random splitting is unrealistic.
• Keep preprocessing inside the validation pipeline.
• Reserve an untouched test set and report uncertainty.
• Report calibration and operational threshold performance, not only AUC.
• Examine subgroup and intersectional performance where sample sizes permit.
• Treat high performance with suspicion when features may encode future information.
• Use causal diagrams and explicit assumptions for intervention questions.
• Match explanation methods to users, risk, and model structure.
• Validate explanations for fidelity, stability, and actionability.
• Monitor both model statistics and real-world outcomes after deployment.
• Document limitations, human oversight, escalation, and rollback procedures.

Chapter Summary

Model evaluation is an evidence-design activity that connects metrics to decisions, costs, populations, and deployment conditions. Classification should be assessed through suitable combinations of discrimination, calibration, error rates, and operational utility; regression and ranking require task-specific measures. Validation must respect time, groups, entities, preprocessing, and external generalisation. Error analysis, leakage audits, and drift monitoring are essential because aggregate test performance can conceal serious failure.

Prediction and causation answer different questions. Potential outcomes, causal graphs, and identification assumptions clarify what is required to estimate intervention effects. Randomisation and well-designed observational methods can support causal claims, but predictive feature importance cannot establish causality. Interpretability may be intrinsic to a model, whereas explainability often uses post hoc approximations such as SHAP, LIME, partial dependence, or counterfactuals. Explanations must be evaluated for fidelity, stability, usefulness, feasibility, fairness, and privacy. A responsible AI evaluation programme combines quantitative testing, causal reasoning, human oversight, governance, and continuous monitoring.

Key Terms

Calibration: Agreement between predicted probabilities and observed outcome frequencies.

Causal effect: Change in an outcome attributable to an intervention or exposure.

Causal graph: Graphical representation of assumed causal relationships.

Concept drift: Change in the relationship between predictors and the target.

Confounder: Variable influencing both treatment and outcome.

Counterfactual explanation: Description of an input change that would alter a model prediction.

Cross-validation: Resampling procedure for estimating performance and tuning models.

Data leakage: Use of information during development that would be unavailable at prediction time.

Discrimination: Ability to distinguish or rank observations with different outcomes.

Distribution shift: Change between development and deployment data distributions.

External validation: Evaluation on a distinct site, population, or time period.

F1 score: Harmonic mean of precision and recall.

Fairness metric: Quantitative measure of performance or treatment across groups.

Interpretability: Degree to which model operation can be understood directly.

Potential outcome: Outcome that would occur under a specified treatment condition.

Precision: Proportion of predicted positives that are actual positives.

Recall: Proportion of actual positives correctly identified.

SHAP: Feature-attribution framework based on Shapley-value concepts.

Temporal validation: Testing on later data to approximate future deployment.

Treatment effect: Difference between outcomes under alternative interventions.

Review Questions

Short Answer Questions

1. Why can accuracy be misleading for a highly imbalanced classification problem?

2. Distinguish calibration from discrimination.

3. What is data leakage, and why does it inflate validation performance?

4. State the difference between prediction and causal inference.

5. What is the purpose of an external validation set?

Long Answer Questions

1. Explain how threshold selection should incorporate error costs, prevalence, capacity, and calibration.

2. Compare random holdout, cross-validation, temporal validation, grouped validation, and external validation.

3. Explain confounding, mediation, and colliders using causal-graph reasoning.

4. Compare intrinsically interpretable models with SHAP, LIME, and counterfactual explanations.

5. Develop a complete evaluation and monitoring plan for a high-stakes AI decision-support system.

Analytical / Application Questions

1. A fraud model has high ROC-AUC but very low precision because fraud prevalence is 0.2%. Which additional metrics and threshold analyses should be performed, and why?

2. A hospital model performs well under a random split but poorly when tested on the following year's data. Diagnose possible causes and propose an experiment to distinguish leakage, covariate shift, and concept drift.

3. A feature is the strongest SHAP contributor to a model predicting employee attrition. Explain why this does not prove that changing the feature will reduce attrition, and design a causal study for an appropriate intervention.

Multiple Choice Questions with Answers

Question 1

Which metric directly measures the proportion of predicted positive cases that are actually positive?

A. Recall

B. Precision

C. Specificity

D. Accuracy

Correct Answer: B. Precision

Question 2

Which validation strategy is most appropriate when the same individual contributes multiple records and individual-specific information could leak across splits?

A. Random splitting of all records

B. Grouped splitting by individual

C. Oversampling the minority class

D. Selecting the model with the highest training accuracy

Correct Answer: B. Grouped splitting by individual

Question 3

A model has an ROC-AUC of 0.90 but is poorly calibrated. What does this most directly indicate?

A. It cannot rank positive cases above negative cases

B. Its predicted probabilities do not reliably correspond to observed frequencies

C. Its false-negative rate must be zero

D. It has no discrimination ability

Correct Answer: B. Its predicted probabilities do not reliably correspond to observed frequencies

Question 4

Which situation is a clear example of data leakage?

A. Using age measured at prediction time

B. Using a discharge code created after the outcome to predict that outcome

C. Using a training-fold mean to impute a training-fold value

D. Evaluating a model on a later time period

Correct Answer: B. Using a discharge code created after the outcome to predict that outcome

Question 5

In the potential-outcomes framework, the average treatment effect is defined as:

A. E[Y(1)+Y(0)]

B. E[Y(1)−Y(0)]

C. E[Y(1)|T=1] only

D. P(T=1|X)

Correct Answer: B. E[Y(1)−Y(0)]

Question 6

Which assumption requires every relevant subgroup to have a non-zero probability of receiving each treatment level?

A. Consistency

B. Positivity

C. Stationarity

D. Monotonicity

Correct Answer: B. Positivity

Question 7

Which statement about SHAP feature attribution is correct?

A. It automatically identifies causal intervention targets

B. It explains model contributions under a specified attribution framework but does not by itself establish causation

C. It can only be used with linear regression

D. It guarantees equal performance across demographic groups

Correct Answer: B. It explains model contributions under a specified attribution framework but does not by itself establish causation

Question 8

Why is temporal validation often preferable to random validation for a changing operational environment?

A. It guarantees perfect calibration

B. It approximates prediction on future data and can reveal temporal degradation

C. It removes all confounding

D. It always produces a higher score

Correct Answer: B. It approximates prediction on future data and can reveal temporal degradation

Question 9

Which regression metric penalises large errors more strongly because errors are squared?

A. MAE

B. MSE

C. Median absolute error

D. Accuracy

Correct Answer: B. MSE

Question 10

A change in the distribution of inputs while the relationship between inputs and outcome remains stable is commonly called:

A. Covariate shift

B. Label leakage

C. Collider bias

D. Treatment effect heterogeneity

Correct Answer: A. Covariate shift