Statistical Computing, Responsible Analytics, and Emerging Practice

Introduction

Probability and statistics are increasingly practiced through computational systems rather than by hand calculation alone. Statistical software makes it possible to simulate complex processes, analyse large datasets, fit probabilistic models, and monitor predictions after deployment. However, software does not eliminate statistical reasoning. It can produce precise-looking results from invalid data, inappropriate assumptions, leakage, biased samples, or poorly specified models.

Contemporary analytics therefore requires two complementary capabilities. The first is computational competence: writing reproducible code, validating data, selecting methods, quantifying uncertainty, and monitoring models. The second is responsible statistical judgment: protecting privacy, examining fairness, communicating limitations, distinguishing association from causation, and establishing governance over automated decisions.

This chapter develops a practical framework for statistical computing in artificial intelligence and machine learning. It also introduces emerging practices such as probabilistic programming, conformal prediction, automated inference, and model observability.

Learning Objectives

By the end of this chapter, students should be able to:

1. Design a reproducible statistical workflow using software, version control, and documented environments.

2. Use simulation to understand sampling variation, uncertainty, and model behaviour.

3. Validate data and identify missingness, leakage, outliers, and distributional change.

4. Compare appropriate approaches to missing-data treatment.

5. Monitor deployed models using statistical and operational measures.

6. Communicate uncertainty accurately to technical and non-technical audiences.

7. Evaluate privacy, fairness, transparency, governance, and causal claims in analytics.

8. Explain the role of probabilistic AI, probabilistic programming, conformal prediction, and automated inference.

1. Statistical Computing as a Scientific Workflow

A statistical analysis is more than a model-fitting command. It is a sequence of decisions:

Question → data acquisition → data validation → exploratory analysis → design and modelling → uncertainty assessment → communication → monitoring and revision.

Each stage can introduce error. A sophisticated algorithm cannot correct a poorly defined target variable, a biased sampling process, or an outcome measured after the prediction decision.

Reproducibility and replicability

A reproducible analysis is one in which another analyst can use the same data, code, software specifications, and documented procedures to obtain the reported results. Replicability commonly refers to obtaining consistent findings from a new study or independently collected data, although terminology varies across disciplines.

A reproducible project should normally include:

• raw data preserved in read-only form where legally permitted;
• a data dictionary defining variables, units, coding, and provenance;
• scripts or notebooks that generate intermediate and final results;
• explicit random seeds for simulations and randomized algorithms;
• dependency and environment specifications;
• tests for data transformations and important functions;
• version control for code, documentation, and configuration;
• a clear separation between exploratory work and production analysis;
• a record of exclusions, missing-data decisions, and model versions.

A random seed makes a pseudorandom computation repeatable within a specified software environment. It does not make the result scientifically valid, and results may still differ across libraries, hardware, or algorithm implementations.

A useful design principle is to make the computational pipeline idempotent: rerunning it with the same inputs should produce the same outputs, except where deliberate nondeterminism is documented. Data lineage should allow an analyst to answer: Where did this value originate? Which transformation changed it? Which model used it?

Computational testing

Statistical code should be tested like other software. Unit tests can verify that a transformation preserves row counts, that probabilities remain between zero and one, or that a metric returns a known value on a small example. Invariant checks can detect impossible ages, negative quantities, duplicated identifiers, and dates outside the study period. Integration tests can ensure that data preparation and model prediction use compatible schemas.

A notebook is useful for communication and exploration, but a production workflow should not depend on manually executed cells. Automated pipelines reduce omitted steps and preserve an auditable sequence of operations.

2. Simulation for Understanding and Decision-Making

Simulation generates artificial data according to a specified probability model. It is valuable when analytical derivations are difficult, when assumptions need examination, or when students need to observe sampling variation.

Suppose a Bernoulli random variable X takes the value 1 with probability p and 0 otherwise. For n independent trials, the sample proportion is

p̂ = (1/n) Σᵢ₌₁ⁿ Xᵢ.

Here, n is the number of trials and p̂ is the observed proportion of successes. Under independence and a common success probability p,

E(p̂) = p, Var(p̂) = p(1 − p)/n.

The expectation and variance describe repeated samples from the same process; they do not claim that one observed p̂ equals p exactly.

Monte Carlo estimation

If a random quantity Y is simulated independently B times as Y₁, …, Yᴮ, the Monte Carlo estimate of its expectation is

μ̂ = (1/B) Σᵦ₌₁ᴮ Yᵦ.

If Y has finite variance σ², then the standard error of μ̂ is estimated by s/√B, where s is the sample standard deviation of the simulated values. The approximation assumes independent or suitably controlled simulation draws. Increasing B generally reduces Monte Carlo error, but it does not repair a misspecified data-generating model.

Simulation workflow

A defensible simulation includes:

1. Define the estimand or decision quantity.

2. Specify the data-generating assumptions.

3. Generate data using a documented random-number generator.

4. Apply the proposed analysis exactly as it would be applied to real data.

5. Repeat the experiment many times.

6. Summarize bias, variability, interval coverage, error rates, or decision loss.

7. Vary assumptions in sensitivity analyses.

Worked example: simulation and interval coverage

Assume a quality-control process has a true defect probability p = 0.10. In each simulated sample, n = 100 products are inspected. The sample proportion has standard deviation

√[p(1 − p)/n] = √(0.10 × 0.90/100) = 0.03.

Thus, before observing data, sample proportions commonly vary by roughly three percentage points around 0.10, although the exact distribution is discrete and not necessarily normal for every sample size. A simulation can generate thousands of samples, construct an interval using each sample, and calculate the proportion of intervals containing 0.10. This empirical proportion estimates the procedure’s coverage under the stated Bernoulli model.

The exercise demonstrates an important distinction: a confidence procedure has a long-run performance property, whereas a particular computed interval either contains or does not contain the fixed parameter.

3. Data Validation and Data Quality

Data validation examines whether data are suitable for the intended analysis. It should occur before model fitting and again at deployment.

Dimensions of data quality

Important dimensions include:

• Validity: values conform to definitions and permitted formats.
• Completeness: required fields are present.
• Consistency: related fields agree with one another.
• Accuracy: recorded values represent the underlying phenomenon sufficiently well.
• Timeliness: data are available at the appropriate time.
• Uniqueness: records are not unintentionally duplicated.
• Representativeness: the observed data support the target population and decision.

Validation rules should include both general checks and domain-specific checks. For example, a patient discharge date should not precede admission; a transaction amount may be zero but should not be negative unless refunds are encoded that way; and a model feature must be available before the prediction time.

Distributional and structural checks

Useful checks include frequency tables, quantiles, missingness patterns, duplicate rates, range checks, schema comparisons, and comparisons between training and operational data. Statistical tests can support investigation, but a statistically significant difference in a very large dataset may be practically negligible. Conversely, a meaningful shift may be missed with a small sample. Visualizations and subject-matter review remain important.

Data leakage

Data leakage occurs when information unavailable at prediction time enters model training. Examples include using a final diagnosis to predict diagnosis, calculating a feature with future observations, or splitting repeated records from one person randomly across training and test sets. Leakage produces optimistic evaluation and can be more damaging than ordinary measurement noise.

The remedy is temporal and procedural discipline: define the prediction timestamp, construct features using only information available then, and split data in a way that reflects deployment.

4. Missing Data and Its Treatment

Missingness is not merely a technical inconvenience. It may reflect cost, system failure, respondent choice, disease severity, or deliberate non-disclosure.

Three classical mechanisms are:

• MCAR (Missing Completely At Random): missingness is unrelated to observed and unobserved values.
• MAR (Missing At Random): conditional on observed variables, missingness is unrelated to the unobserved value.
• MNAR (Missing Not At Random): missingness depends on the unobserved value even after accounting for observed variables.

These are assumptions about a missingness mechanism, not labels that can usually be proven from the observed dataset alone.

Treatment strategies

Complete-case analysis uses only records with all required values. It is simple but may lose information and can be biased unless strong conditions hold.

Single imputation replaces missing values with a mean, median, category, or predicted value. It is easy to implement, but it often understates uncertainty because it treats an estimated value as known.

Multiple imputation creates several plausible completed datasets, analyses each one, and combines estimates while reflecting both within-imputation and between-imputation variation. It is generally preferable to single imputation when its modelling assumptions are reasonable.

Missingness indicators can sometimes be useful predictors, particularly when the fact that a value is missing carries information. They should not be used mechanically: an indicator may encode operational or social processes and create fairness concerns.

A responsible analysis reports the amount and pattern of missingness, explains the assumed mechanism, conducts sensitivity analyses, and distinguishes imputed values from observed values. Imputation must be performed within training folds during cross-validation; otherwise, information from validation data can leak into the model.

5. Model Evaluation and Monitoring

A model is not finished when it achieves a test score. Deployment changes the environment, data, incentives, and consequences.

Evaluation before deployment

The evaluation design must match the intended use. Classification measures include sensitivity, specificity, precision, recall, calibration, and discrimination measures such as area under the ROC curve. No single metric is universally best.

For a binary outcome, sensitivity is

Sensitivity = TP/(TP + FN),

where TP denotes true positives and FN false negatives. Specificity is

Specificity = TN/(TN + FP),

where TN denotes true negatives and FP false positives. These formulas require nonzero denominators and depend on a defined decision threshold.

Calibration concerns whether predicted probabilities correspond to observed frequencies. If among cases assigned a probability near 0.7 approximately 70% experience the event, predictions are well calibrated in that group. A model can discriminate well but be poorly calibrated, or be calibrated while offering little ranking ability.

For regression, common measures include mean absolute error (MAE),

MAE = (1/n) Σᵢ₌₁ⁿ |yᵢ − ŷᵢ|,

and mean squared error (MSE),

MSE = (1/n) Σᵢ₌₁ⁿ (yᵢ − ŷᵢ)².

MAE has the same units as the outcome. MSE has squared units and penalizes large errors more strongly. Both require paired observed outcomes yᵢ and predictions ŷᵢ.

Monitoring after deployment

Monitoring should cover:

• Data drift: changes in feature distributions.
• Concept drift: changes in the relationship between features and outcomes.
• Prediction drift: changes in predicted values or risk categories.
• Performance drift: changes in error, discrimination, or calibration.
• Operational drift: changes in latency, failure rates, or data availability.
• Impact drift: changes in outcomes for affected individuals or groups.

Monitoring thresholds should trigger investigation rather than automatic conclusions. A drift alarm may reflect a beneficial population change, a sensor replacement, a coding error, or genuine model deterioration. Outcome monitoring is delayed when labels become available later, so leading indicators and periodic retrospective validation are often needed.

6. Communicating Uncertainty

Uncertainty arises from sampling variation, measurement error, missing data, model assumptions, parameter estimation, and future change. It should be communicated in a form appropriate to the decision-maker.

A confidence interval is a property of a repeated-sampling procedure. For a parameter θ, a nominal 95% confidence procedure produces intervals that contain θ in approximately 95% of repeated samples under its assumptions. It is not technically correct to say that the fixed parameter has a 95% probability of lying in one frequentist interval, although informal communication often uses that wording.

A Bayesian credible interval has a different interpretation: given the model, prior distribution, and observed data, it assigns a stated posterior probability to a parameter interval. Neither interval is automatically superior; interpretation must match the inferential framework and assumptions.

Good communication should:

• state the estimand and population;
• provide an effect size, not only a p-value;
• report uncertainty and relevant assumptions;
• distinguish statistical from practical significance;
• explain base rates and denominators;
• avoid excessive decimal precision;
• describe what the analysis cannot establish;
• use visual intervals, scenarios, and plain language where useful.

For risk predictions, communicate both relative and absolute effects. A doubling of risk from 1% to 2% is a 100% relative increase but a one-percentage-point absolute increase.

7. Responsible Analytics

Privacy and data protection

Statistical data can identify people even when obvious identifiers are removed. Re-identification may occur through linkage with external datasets, rare combinations of attributes, or repeated release of related statistics.

Privacy practices include data minimization, purpose limitation, access controls, encryption, retention limits, secure computation, aggregation, and documented legal bases for processing. Differential privacy provides a formal framework in which the output distribution changes only slightly when one individual’s record is added or removed. In one common formulation, a randomized mechanism M is ε-differentially private if for neighboring datasets D and D′ and every output set S,

P(M(D) ∈ S) ≤ e^ε P(M(D′) ∈ S).

Here ε is a privacy-loss parameter; smaller ε generally implies stronger privacy but often more noise. The guarantee depends on the mechanism, neighboring-dataset definition, and composition across multiple releases. Differential privacy protects contribution-level information, not every possible attribute inference or all risks from the model itself.

Fairness

Fairness is not one universal metric. Depending on context, analysts may examine selection rates, false-positive rates, false-negative rates, calibration, equal opportunity, or group benefit. Some criteria cannot generally be satisfied simultaneously when groups have different base rates and predictions are imperfect.

Fairness assessment should therefore begin with the decision’s purpose, affected groups, harms, benefits, legal requirements, and available remedies. Intersectional groups and small groups require special attention because aggregate metrics can conceal disparities. Statistical parity in an output does not guarantee fairness if measurement, access, or treatment differs earlier in the process.

Transparency and explainability

Transparency includes documentation of data sources, variables, objectives, limitations, evaluation populations, and governance responsibilities. Explainability methods can describe model behaviour globally or provide local explanations for individual predictions. An explanation is not necessarily a causal account and should not be presented as proof that a feature caused an outcome.

Model cards, datasheets, decision logs, and explanation records support accountability. Human oversight must be meaningful: a person should have sufficient information, authority, competence, and time to challenge or override a system.

Governance

A governance program assigns responsibility across the model life cycle. It should define approval criteria, risk classification, documentation, access permissions, incident response, audit schedules, change control, and retirement conditions. High-impact applications require stronger validation, monitoring, appeal mechanisms, and independent review than low-risk recommendation systems.

8. Association, Causation, and Automated Inference

Prediction and causal inference answer different questions. A predictive model estimates an outcome accurately from available information. A causal analysis asks what would happen under an intervention or alternative exposure.

Correlation may arise from confounding, selection, reverse causation, measurement processes, or chance. A causal claim requires a design or set of assumptions that identify the relevant counterfactual quantity. Randomized experiments can support causal interpretation when treatment is properly assigned, interference is controlled, adherence and missing outcomes are addressed, and the target population is clearly defined. Observational methods may use adjustment, matching, instrumental variables, regression discontinuity, or difference-in-differences, but each relies on assumptions that must be defended.

Automated inference creates additional risks. Large-scale model searches can produce apparently impressive findings through multiple comparisons, researcher degrees of freedom, or selective reporting. Cross-validation protects predictive evaluation but does not automatically establish causal validity. Automated feature selection may choose proxies for protected characteristics or variables caused by the outcome.

9. Emerging Probabilistic AI Practice

Probabilistic programming

Probabilistic programming languages allow analysts to specify generative models and use algorithms such as Markov chain Monte Carlo or variational inference to estimate unknown quantities. This encourages explicit representation of uncertainty and hierarchical structure. It also introduces computational diagnostics: convergence, effective sample size, posterior predictive checks, sensitivity to priors, and approximation error must be assessed.

Conformal prediction

Conformal methods construct prediction sets or intervals using a calibration procedure. Under exchangeability—roughly, the assumption that calibration and future examples are symmetrically generated—a conformal procedure can provide finite-sample marginal coverage such as

P(Yₙ₊₁ ∈ C(Xₙ₊₁)) ≥ 1 − α,

where C(Xₙ₊₁) is the prediction set and α is the target miscoverage level. The guarantee concerns the specified population and validity assumptions. It does not automatically guarantee conditional coverage for every subgroup, independence under distribution shift, or useful narrow intervals.

Automated machine learning and automated inference

Automated machine learning can search preprocessing, models, and hyperparameters. It improves efficiency but expands the risk of overfitting the validation process, hidden leakage, opaque choices, and excessive computational cost. A fixed holdout set, nested validation where appropriate, experiment tracking, and human review are safeguards.

Automated inference systems should expose uncertainty, abstain when evidence is weak, and record the data and model version behind each decision. A system that always returns an answer can be less reliable than one that sometimes defers to a human.

Foundation and generative models

Generative models can synthesize text, images, tabular records, or probabilistic forecasts. Synthetic data may support development and sharing, but it can reproduce biases, memorize sensitive information, or fail to preserve rare cases and dependencies. Synthetic data must be validated against the intended analytical use and should not automatically be treated as anonymous or representative.

10. Illustrative Case: Monitoring an AI-Assisted Loan Triage System

Background

A fictional regional lender introduces a model that estimates the probability that an applicant will repay within twelve months. The model is used to prioritize applications for manual review, not to make final approvals automatically.

Problem

After six months, the overall ranking metric remains stable, but community organizations report that applicants from one neighborhood are receiving more manual reviews and longer processing times. At the same time, a new document-processing vendor has changed the format of income records.

Analysis

The analytics team first freezes the current model version and defines the prediction timestamp. It validates schema, missingness, duplicate applications, and feature availability. It discovers that income values from the new vendor are missing more often for self-employed applicants. A feature derived from document completeness therefore acts as a proxy for employment type and neighborhood.

The team compares performance by group, calibration, referral rates, false-positive rates, and processing delay. It does not rely on one fairness metric. It also checks whether the model’s apparent performance is being sustained by applicants with complete documentation only.

Approach and solution

The lender removes a post-submission administrative variable that was not consistently available at decision time, improves the document-ingestion process, and adds a missingness category with explicit review. Multiple imputation is evaluated within training folds, while cases with unresolved critical data are routed to a human reviewer. The model is recalibrated on recent data, and a monitoring dashboard tracks data quality, group-specific performance, calibration, referral rates, and delay.

An independent governance panel reviews the change. Applicants receive an explanation of the review process and a route to correct inaccurate data. The lender documents that the model predicts repayment risk; it does not establish that any applicant is personally responsible for an observed financial outcome.

Outcome

Overall ranking performance changes little, but disparity in referral rates decreases and processing delays become more similar across groups. The lender retains manual review because uncertainty and data-quality failures remain material.

Learning points

The case illustrates that model performance is multidimensional, data quality can create fairness problems, missingness can be informative, and monitoring must include operational and impact outcomes. Governance is not a final approval step; it is a continuing process.

Challenges and Limitations

Statistical computing cannot remove uncertainty or value judgments. Simulation results are conditional on the data-generating model. Monitoring can detect symptoms without identifying causes. Fairness metrics may conflict, and privacy protection can reduce statistical utility. Interpretability tools can be unstable or misleading. Reproducibility may be limited by proprietary data, changing software dependencies, or undocumented human decisions. Finally, a well-calibrated model can still be inappropriate if the decision itself is unjustified or if affected people lack recourse.

Best-Practice Checklist

Before releasing an analysis or model, ask:

1. Is the question and estimand explicit?

2. Are data sources, units, timing, and exclusions documented?

3. Have leakage, duplicates, missingness, and distribution shifts been examined?

4. Is the evaluation design aligned with deployment?

5. Are uncertainty and sensitivity analyses reported?

6. Are subgroup performance, fairness, and privacy risks assessed?

7. Can another analyst reproduce the result?

8. Is there an owner, monitoring plan, incident process, and retirement criterion?

9. Are causal claims separated from predictive claims?

10. Can affected people understand, contest, or correct consequential decisions?

Chapter Summary

Statistical computing is a complete scientific workflow rather than a collection of software commands. Reproducibility requires controlled data, documented code, tested transformations, versioned environments, and explicit randomness. Simulation helps evaluate sampling variability, estimator behaviour, uncertainty procedures, and decision rules, but its conclusions depend on the assumptions used to generate data.

Data validation must address validity, completeness, timing, duplication, representativeness, and leakage. Missing-data treatment should be based on a defensible mechanism and should preserve uncertainty, with multiple imputation and sensitivity analysis often preferable to simplistic replacement. Deployed models require monitoring for data drift, concept drift, calibration, performance, operational failures, and unequal impacts.

Responsible analytics integrates privacy, fairness, transparency, governance, and meaningful human oversight. Prediction is not causation, and automated inference can amplify selection and reporting errors. Probabilistic programming, conformal prediction, automated machine learning, and generative models offer powerful capabilities but require diagnostic checks and careful interpretation. The professional analyst combines computational skill with statistical skepticism, ethical judgment, and accountability.

Key Terms

Reproducibility: Ability to regenerate an analysis from the same data, code, and documented environment.

Replicability: Consistency of findings when an analysis or study is independently repeated.

Monte Carlo simulation: Repeated random sampling used to approximate probabilities, expectations, or operating characteristics.

Data leakage: Use of information in modelling that would not be available at prediction time.

Data drift: Change in the distribution of input data over time.

Concept drift: Change in the relationship between inputs and the target outcome.

Calibration: Agreement between predicted probabilities and observed event frequencies.

MCAR: Missingness unrelated to both observed and unobserved values.

MAR: Missingness conditionally unrelated to an unobserved value given observed information.

MNAR: Missingness dependent on the unobserved value after accounting for observed information.

Multiple imputation: Creation and combination of several plausible completed datasets.

Differential privacy: A formal privacy guarantee limiting the effect of one individual’s data on released outputs.

Fairness metric: A quantitative criterion used to examine equality or equity of model outcomes or errors.

Model card: Documentation describing a model’s purpose, performance, limitations, and appropriate use.

Causal inference: Analysis concerned with effects of interventions or exposures.

Probabilistic programming: Specification and computation of models that represent uncertainty through probability distributions.

Conformal prediction: A distribution-free framework for constructing prediction sets or intervals under exchangeability assumptions.

Exchangeability: A symmetry condition supporting certain finite-sample conformal guarantees.

Model observability: Ongoing visibility into data, predictions, performance, operations, and impacts.

Governance: Structures and processes assigning accountability across the model life cycle.

Review Questions

Short Answer Questions

1. What is the difference between reproducibility and replicability?

2. Why does setting a random seed not guarantee a valid statistical conclusion?

3. Define data leakage and give one example.

4. Distinguish MCAR, MAR, and MNAR missingness.

5. What is calibration in probabilistic prediction?

Long Answer Questions

1. Explain how a reproducible statistical-computing workflow should be designed from data acquisition through reporting.

2. Describe a simulation study for evaluating the coverage of an interval procedure, including its assumptions and outputs.

3. Compare complete-case analysis, single imputation, and multiple imputation.

4. Discuss the roles of privacy, fairness, transparency, and governance in high-impact automated decisions.

5. Explain why predictive accuracy does not establish a causal relationship.

Analytical / Application Questions

1. A deployed fraud model has stable overall accuracy but a sharp increase in missing values for one customer group. Design a diagnostic and monitoring plan.

2. A hospital model predicts readmission risk using a variable recorded after discharge. Identify the methodological problem and propose a valid redesign.

3. A lender’s model has equal calibration across two groups but unequal false-positive rates. Explain why this may occur and recommend how the organization should evaluate and respond to the disparity.

Multiple Choice Questions with Answers

Question 1

Which practice most directly improves computational reproducibility?

A. Increasing the number of model parameters

B. Recording code, data transformations, dependencies, and random seeds

C. Removing all exploratory analysis

D. Using a more complex visualization

Correct Answer: B. Recording code, data transformations, dependencies, and random seeds

Question 2

For independent Bernoulli trials with p = 0.10 and n = 100, what is the standard deviation of the sample proportion?

A. 0.003

B. 0.010

C. 0.030

D. 0.090

Correct Answer: C. 0.030

Question 3

Which situation is an example of data leakage?

A. Using age recorded at application time

B. Splitting data by calendar time

C. Using a final diagnosis to predict that diagnosis before it is made

D. Reporting a confidence interval

Correct Answer: C. Using a final diagnosis to predict that diagnosis before it is made

Question 4

Under MAR, missingness is assumed to be independent of the missing value conditional on:

A. Only the sample size

B. Observed information

C. The random seed

D. The model’s test accuracy

Correct Answer: B. Observed information

Question 5

Which metric is defined as TP/(TP + FN)?

A. Specificity

B. Precision

C. Sensitivity

D. Calibration

Correct Answer: C. Sensitivity

Question 6

What does a well-calibrated probability model imply?

A. Every individual prediction is correct

B. Predicted probabilities correspond to observed frequencies in comparable groups

C. The model has zero false positives

D. All groups have identical base rates

Correct Answer: B. Predicted probabilities correspond to observed frequencies in comparable groups

Question 7

What is the main purpose of differential privacy?

A. To guarantee perfect prediction accuracy

B. To limit how much one individual’s data affects released outputs

C. To eliminate all missing values

D. To prove a causal relationship

Correct Answer: B. To limit how much one individual’s data affects released outputs

Question 8

Which statement about causal inference is correct?

A. High predictive accuracy alone proves causation

B. Correlation always results from randomization

C. Causal claims require an appropriate design or identifying assumptions

D. A p-value automatically identifies a treatment effect

Correct Answer: C. Causal claims require an appropriate design or identifying assumptions

Question 9

A conformal prediction coverage guarantee commonly relies on which condition?

A. Exchangeability between calibration and future examples

B. All predictions being identical

C. Zero measurement error

D. Equal sample sizes in every subgroup

Correct Answer: A. Exchangeability between calibration and future examples

Question 10

Why should model monitoring include operational measures as well as predictive accuracy?

A. Operational failures can make a statistically accurate model unusable

B. Operational measures replace all outcome labels

C. Accuracy is never useful

D. Operational measures prove fairness

Correct Answer: A. Operational failures can make a statistically accurate model unusable