Responsible Mathematical Modelling and Emerging Developments

Introduction and context

Mathematical modelling is not merely the selection of an algorithm. It is the construction of a formal representation of a phenomenon, together with assumptions about measurement, causation, uncertainty, human behaviour and acceptable error. In data science, a model can be computationally accurate yet scientifically invalid, socially discriminatory, privacy-invasive or impossible to reproduce.

Responsible modelling therefore asks four related questions:

1. Is the model mathematically and empirically sound?
2. Does it answer the intended question rather than a convenient substitute?
3. Are its errors and impacts distributed acceptably across people and groups?
4. Can its decisions be audited, reproduced, challenged and governed?

This chapter develops a critical framework for assumptions, identifiability, robustness, fairness, privacy, causal limitations, reproducibility and model risk. It then examines federated learning, differential privacy, optimal transport, geometric deep learning and AI governance.

Learning objectives

By the end of this chapter, a student should be able to:

• identify and test assumptions in statistical and machine-learning models;
• distinguish identifiability from estimation accuracy;
• analyse sensitivity, robustness and distribution shift;
• compare mathematical fairness criteria and explain their incompatibilities;
• quantify privacy using differential privacy;
• distinguish prediction from causal inference;
• design a reproducible modelling workflow and risk-control process;
• explain the mathematical foundations and limitations of federated learning, optimal transport and geometric deep learning;
• connect technical controls to organisational AI governance.

1. Assumptions: the hidden architecture of a model

Every model simplifies reality. Let observed data be [[M:(X_i,Y_i)_{i=1}^n]], where [[M:X_i]] is a vector of predictors and [[M:Y_i]] is an outcome. A supervised-learning procedure estimates a function [[M:\hat f]] from data, often by minimising empirical risk:

Here, [[M:L]] is a loss function, [[M:F]] is the candidate function class, [[M:Ω(f)]] is a regularisation penalty, and [[M:λ≥0]] controls the penalty. This expression assumes that the variables are measured meaningfully, that the loss represents the decision objective, and that training data provide relevant information about future cases.

Common assumptions include:

• Representativeness: the data-generating process is sufficiently related to the deployment population.
• Independence: observations are independent, or dependence is explicitly modelled.
• Stable measurement: a feature has comparable meaning across time and groups.
• Correct specification: the model class can approximate the relevant relationship.
• Positivity or overlap: relevant subgroups have observations across decision conditions.
• Missingness assumptions: missing values are missing completely at random, at random conditional on observed variables, or not at random under an explicitly modelled mechanism.
• Stationarity: the relationship between inputs and outcomes does not change materially after deployment.

An assumption is not established merely because a model performs well on a test set. For example, random train-test splitting can produce optimistic results when records from the same patient, household or device occur in both sets. Grouped or temporal splitting is then more appropriate.

A useful practice is an assumption register. For each assumption, document its rationale, diagnostic test, likely violation, consequence and mitigation. This converts implicit judgement into an auditable modelling artefact.

2. Identifiability and the limits of inference

A parameter is identifiable when distinct parameter values imply distinct distributions of observable data. Formally, a parameter [[M:θ]] is identifiable in a model family [[M:{P_θ}]] if

If different parameter values produce exactly the same observable distribution, no amount of additional data from the same observational design can distinguish them. This is a structural limitation, not a small-sample problem.

Example: confounding

Suppose a treatment [[M:A]] and outcome [[M:Y]] are both influenced by health status [[M:U]]. An observational association between [[M:A]] and [[M:Y]] may combine the treatment effect with differences in [[M:U]]. Even if the association is estimated with negligible standard error, the causal effect is not identified without additional assumptions, measurements, randomisation or a valid design.

In a linear regression,

[[M:β_1]] has a causal interpretation only under appropriate design and assumptions, including correct adjustment for relevant confounders, no problematic measurement error and a suitable error structure. If [[M:U]] is unobserved, the coefficient on [[M:A]] generally need not equal the causal effect.

Identifiability also occurs in mixture models, latent-variable models, inverse problems and neural networks. Some parameters may be identifiable only up to label permutation or other transformations. A model can therefore have excellent predictive performance while its internal parameters lack a unique scientific interpretation.

3. Robustness, uncertainty and model risk

A model is robust when conclusions remain acceptably stable under plausible changes in data, assumptions or operating conditions. Robustness has several dimensions:

• Statistical robustness: resistance to outliers, contamination and sampling variation.
• Distributional robustness: acceptable performance under shifts from training to deployment populations.
• Adversarial robustness: resistance to deliberately manipulated inputs.
• Operational robustness: resilience to missing data, latency, software faults and changing interfaces.
• Decision robustness: stability of the recommended action when uncertainty is considered.

For a parameter estimate [[M:\hat θ]], sensitivity can be assessed by perturbing inputs, samples or assumptions and observing changes in outputs. In optimisation, local sensitivity is often represented by a derivative or gradient. If a scalar output [[M:g(x)]] changes under a small perturbation [[M:∆x]], then

This first-order approximation is useful only when [[M:g]] is sufficiently smooth and the perturbation is small. It does not replace stress testing for discontinuous decision rules or large shifts.

A simple robustness quantity is the worst-case risk over an uncertainty set [[M:U]]:

[[M:Q]] denotes a possible deployment distribution. The formulation is meaningful only when [[M:U]] is justified; an excessively broad set can produce unusably conservative decisions, whereas a narrow or unrealistic set creates false confidence.

Model risk is the risk of adverse consequences caused by incorrect, misused or misunderstood models. Controls include independent validation, challenger models, threshold review, monitoring, rollback procedures, documented limitations and human escalation. Validation should examine conceptual soundness, data quality, implementation correctness, performance, calibration and use within approved scope.

4. Fairness as a mathematical and normative problem

Let [[M:A]] denote a protected attribute, [[M:\hat Y]] a prediction and [[M:Y]] the outcome. Fairness cannot be reduced to one universal formula because different criteria express different ethical and institutional priorities.

Common group criteria include:

• Demographic parity:

• Equal opportunity: equal true-positive rates,

• Equalised odds: equal true-positive and false-positive rates conditional on outcome.
• Predictive parity: equal positive predictive value,

• Individual fairness: similar individuals should receive similar treatment, requiring a defensible similarity metric.

These criteria can conflict. If groups have different base rates and the predictor is not perfect, equalised error rates and equal predictive values generally cannot both hold. This is not a mathematical defect; it reflects competing definitions of justice and different costs of errors. Consequently, fairness analysis must state the decision context, affected parties, relevant harms, legal requirements and chosen trade-offs.

Fairness metrics should be reported with uncertainty, not as exact rankings. For a rate estimated from [[M:n]] relevant cases with observed proportion [[M:\hat p]], a rough standard error is [[M:√(\hat p(1-\hat p)/n)]] under independent Bernoulli sampling. Small subgroup samples can make apparent disparities unstable. Confidence intervals, stratified error analysis and intersectional evaluation are therefore essential.

Fairness interventions may occur before modelling (data collection and labels), during modelling (constraints or reweighting), or after modelling (thresholds and human review). Removing a protected attribute does not guarantee fairness because proxy variables may preserve group information.

5. Privacy mathematics

Data protection involves legal, organisational and mathematical controls. Differential privacy provides a formal guarantee about the influence of one individual record. A randomised mechanism [[M:M]] is [[M:ε]]-differentially private if, for all neighbouring datasets [[M:D,D']] differing in one individual and all output sets [[M:S]],

The parameter [[M:ε≥0]] is the privacy loss budget; smaller values generally provide stronger privacy but may reduce utility. The definition concerns distributions of outputs, not absolute secrecy or protection against all inference.

For a numeric query [[M:q(D)]], its global sensitivity is

over neighbouring datasets. The Laplace mechanism releases

and is [[M:ε]]-differentially private under the stated adjacency and sensitivity assumptions. For a count query, [[M:∆q=1]]; for an unclipped sum, sensitivity can be unbounded, so clipping or bounded contributions are required.

Important principles are post-processing (further processing of a private output does not weaken the guarantee), sequential composition (privacy losses accumulate across repeated releases), and parallel composition (disjoint subsets can sometimes be analysed separately). Privacy accounting becomes especially important in iterative machine learning, where gradients are clipped and noise is added over many steps.

Privacy can reduce accuracy, especially for rare groups, and privacy protection itself must be evaluated for disparate impact. Secure aggregation, access controls, minimisation, retention limits and governance complement differential privacy.

6. Prediction is not causation

Prediction estimates outcomes; causal analysis estimates changes under interventions. A causal effect may be defined using potential outcomes [[M:Y(1)]] and [[M:Y(0)]]. The average treatment effect is

The two potential outcomes for the same person are not simultaneously observed. Identification therefore requires assumptions such as consistency, treatment exchangeability conditional on measured covariates and positivity:

for relevant values of [[M:x]]. These assumptions are substantive claims, not properties automatically delivered by a flexible algorithm.

Conditioning can also create bias. Adjusting for a mediator may block part of a treatment effect; conditioning on a collider can induce an association between otherwise unrelated causes. Directed acyclic graphs are useful for making these relationships explicit, but a graph itself depends on domain knowledge.

Causal limitations matter in policy. A model predicting who is likely to default may identify risk, but changing an interest rate can alter behaviour and the data-generating process. Interventions require attention to treatment effects, feedback, equilibrium responses and unintended consequences.

7. Reproducibility and scientific accountability

Reproducibility means that an independent party can recreate results using the stated data, code, environment and procedure. Replicability more broadly concerns obtaining consistent findings in a new study or sample; terminology varies by discipline, so definitions should be declared.

A reproducible pipeline should record:

• data provenance, inclusion and exclusion rules;
• versions of code, libraries and model files;
• random seeds and hardware settings where relevant;
• preprocessing fitted only on training data;
• experiment configurations and hyperparameter searches;
• evaluation splits, metrics and uncertainty estimates;
• known failures, negative results and deviations from the protocol.

Reproducibility does not guarantee validity. A reproducibly biased model remains biased. Conversely, privacy and confidentiality may limit data sharing; controlled-access repositories, synthetic data with utility assessment and executable documentation can help.

8. Emerging mathematical developments

8.1 Federated learning

In federated learning, data remain on client devices or institutional servers. A central coordinator sends a model, clients compute local updates and the coordinator aggregates them. A basic weighted averaging rule is

where [[M:w_t]] is the global parameter vector, [[M:w_{t+1}^{(k)}]] is client [[M:k]]'s updated vector and [[M:n_k]] is its sample count. This rule is appropriate only under a specified aggregation protocol and does not itself guarantee privacy.

Federated systems face non-identically distributed data, unreliable participation, communication costs, poisoning attacks and leakage from updates. Secure aggregation can hide individual updates from the coordinator, while differential privacy can limit individual influence. These protections introduce utility and convergence trade-offs.

8.2 Optimal transport

Optimal transport compares probability distributions by finding a low-cost coupling. Given distributions [[M:μ]] and [[M:ν]] and cost [[M:c(x,y)]], the Kantorovich formulation is

Here, [[M:Π(μ,ν)]] is the set of couplings with marginals [[M:μ]] and [[M:ν]]. When [[M:c(x,y)=∥x-y∥^p]] and the [[M:1/p]] power is taken, this yields the Wasserstein distance of order [[M:p≥1]]. Optimal transport supports dataset alignment, domain adaptation, distribution shift analysis and generative modelling. Results depend strongly on the ground metric and can encode questionable assumptions about which individuals or features are comparable.

8.3 Geometric deep learning

Many data are structured by graphs, sets, manifolds or symmetries. A model should ideally respect transformations that leave the problem unchanged. A function is invariant to a transformation group [[M:G]] if

It is equivariant if

Convolutional networks exploit translation structure; graph neural networks use neighbourhood aggregation. For a graph with node features [[M:h_v]], a generic message-passing layer has the form

where [[M:N(v)]] is the neighbourhood of node [[M:v]], [[M:e_{uv}]] is an edge feature, and [[M:AGG]] should generally be permutation-invariant. Architectural symmetry improves inductive bias but does not guarantee fairness, causal validity or robustness to graph manipulation.

9. AI governance and model lifecycle

Technical quality must be embedded in governance. A proportionate lifecycle includes:

1. Purpose definition: specify the decision, affected people, lawful basis and unacceptable uses.
2. Impact assessment: assess discrimination, privacy, safety, security, accessibility and environmental cost.
3. Data and model documentation: maintain datasheets, model cards, assumption registers and risk classifications.
4. Validation and approval: use independent review, representative testing and documented acceptance thresholds.
5. Deployment controls: define access, human oversight, escalation, logging and rollback.
6. Continuous monitoring: track drift, calibration, subgroup outcomes, incidents and changes in context.
7. Retirement: remove models whose purpose, data or performance is no longer defensible.

Governance frameworks commonly emphasise validity, safety, security, accountability, transparency, explainability, privacy and fairness. Legal obligations vary by jurisdiction, but mathematical documentation strengthens compliance by making claims testable. Explainability should be matched to the audience and purpose: a feature-attribution plot may explain a local prediction but not establish causality or justify the underlying decision.

Applied case: illustrative hospital readmission model

Background

A regional hospital consortium develops a model to identify patients at high risk of readmission within 30 days. The model uses diagnoses, prior admissions, medication records and utilisation data from three hospitals.

Problem

Initial evaluation reports an area under the receiver operating characteristic curve of 0.82 using a random record-level split. After deployment, performance falls. A review also finds lower sensitivity for a minority population and uncertainty about whether a utilisation variable reflects access to care rather than health need.

Analysis

The random split allowed records from the same patients and time periods to appear in training and testing, creating leakage. The hospitals also differ in coding practices and patient populations, violating the assumption that deployment data follow the training distribution. The utilisation variable may be a proxy for socioeconomic access. Moreover, the model predicts readmission but does not estimate whether an intervention would prevent it.

Approach and solution

The consortium should:

• use patient-level and temporal splits, followed by external hospital validation;
• document missingness, coding changes and inclusion criteria;
• compare calibration and sensitivity across demographic and intersectional groups;
• report confidence intervals and clinically meaningful thresholds;
• examine whether the score remains useful after removing or redesigning access-sensitive variables;
• conduct a causal analysis of available interventions rather than treating risk as treatment benefit;
• apply privacy-preserving access, retention limits and role-based controls;
• establish human review, an appeal process and monitoring for drift;
• preregister the evaluation protocol and preserve versioned code and data lineage.

Outcome

In this illustrative case, the revised model has a lower headline score but more realistic external performance, better calibration and a transparent intervention pathway. The organisation decides not to use the score for denial of services. It is used only to invite eligible patients to a human-reviewed support programme.

Learning points

A lower apparent accuracy can represent an improvement in scientific validity. Fairness is assessed at the decision point, not only at the model-output level. Privacy, causal usefulness and operational governance must be designed together rather than added after deployment.

Challenges and limitations

Responsible modelling cannot eliminate uncertainty. Fairness metrics may conflict; privacy may reduce utility; robust optimisation may be conservative; federated learning may still leak information; optimal transport depends on a chosen geometry; and geometric inductive biases may be wrong for the domain. Monitoring can detect problems but cannot always explain their causes. Human oversight can improve accountability, yet poorly designed oversight may merely legitimise automated decisions.

The central professional obligation is therefore disciplined humility: distinguish what the data identify from what the modeller assumes, quantify uncertainty, disclose limitations and avoid using a model beyond its validated purpose.

Chapter Summary

Responsible mathematical modelling treats assumptions, data, algorithms and consequences as one system. Identifiability determines whether a target can be learned from the available design; estimation precision cannot solve structural non-identifiability. Robustness requires sensitivity analysis, shift testing and explicit uncertainty sets. Fairness is multidimensional, and mathematically incompatible criteria may represent legitimate normative choices. Differential privacy formalises limits on individual influence but requires sensitivity analysis and privacy accounting. Causal conclusions require intervention-oriented assumptions that prediction alone does not supply. Reproducibility demands traceable data, code, environments and evaluation procedures.

Federated learning keeps data distributed but introduces heterogeneity, communication and attack risks. Optimal transport compares distributions through a cost geometry, while geometric deep learning encodes invariance and equivariance for structured data. AI governance connects these technical ideas to impact assessment, documentation, validation, monitoring, human oversight and retirement. Responsible practice is not a final compliance step; it is a mathematical, empirical and institutional design process.

Key Terms

Assumption register: A documented list of model assumptions, diagnostics, risks and mitigations.

Identifiability: The property that observable data uniquely determine a parameter or target.

Model risk: The possibility of harm caused by an incorrect, misused or misunderstood model.

Distribution shift: A change between training and deployment data distributions.

Robustness: Stability of conclusions or performance under plausible perturbations.

Demographic parity: Equality of prediction rates across protected groups.

Equal opportunity: Equality of true-positive rates across groups.

Predictive parity: Equality of positive predictive values across groups.

Differential privacy: A formal bound on how much one individual can affect an algorithm’s output distribution.

Sensitivity: The maximum change in a query’s output between neighbouring datasets.

Composition: The accumulation of privacy loss across multiple releases.

Causal effect: Change in an outcome caused by an intervention.

Exchangeability: A causal assumption allowing treatment groups to be compared after conditioning on appropriate variables.

Reproducibility: Ability to recreate a result using documented materials and procedures.

Federated learning: Collaborative model training in which raw data remain at distributed clients.

Secure aggregation: A protocol that reveals an aggregate update without revealing each individual update.

Optimal transport: A framework for comparing distributions through minimum-cost couplings.

Equivariance: Transformation of a model output in correspondence with a transformation of its input.

AI governance: Policies, controls and accountability mechanisms governing AI throughout its lifecycle.

Review Questions

Short Answer Questions

1. What is the difference between a model assumption and a model parameter?
2. Define identifiability and explain why more data cannot solve structural non-identifiability.
3. State the mathematical definition of differential privacy.
4. Why can demographic parity and predictive parity conflict?
5. Distinguish prediction from causal inference.

Long Answer Questions

1. Explain how an assumption register can improve the validity and governance of a data-science project.
2. Compare demographic parity, equal opportunity and predictive parity, including conditions under which they may conflict.
3. Derive and interpret the Laplace mechanism for a count query with sensitivity one.
4. Discuss the benefits and limitations of federated learning in a cross-institutional healthcare system.
5. Explain how reproducibility, independent validation and continuous monitoring jointly control model risk.

Analytical / Application Questions

1. A classifier has equal true-positive rates across two groups but different positive predictive values. Using base rates and confusion-matrix reasoning, explain how this can occur and recommend what additional evidence should be collected.
2. Design a validation study for a hospital readmission model whose training data span five years and three hospitals. Specify the splitting strategy, metrics, subgroup analyses and monitoring plan.
3. A consortium wants to train a graph model across organisations without sharing raw records. Propose a governance and technical architecture combining federated learning, secure aggregation and differential privacy, and analyse at least three resulting trade-offs.

Multiple Choice Questions with Answers

Question 1

Which statement best describes structural non-identifiability?

A. The estimate has a large standard error because the sample is small.

B. Different parameter values generate the same observable data distribution.

C. The model has low predictive accuracy on a test set.

D. The training data contain a few outliers.

Correct Answer: B. Different parameter values generate the same observable data distribution.

Question 2

Which fairness criterion requires equal true-positive rates across groups?

A. Demographic parity

B. Predictive parity

C. Equal opportunity

D. Individual privacy

Correct Answer: C. Equal opportunity

Question 3

For a count query with neighbouring datasets differing by one record, the usual global sensitivity is:

A. 0

B. 1

C. ε

D. n

Correct Answer: B. 1

Question 4

In differential privacy, a smaller ε generally indicates:

A. Stronger privacy and potentially lower utility

B. Weaker privacy and higher noise

C. No privacy guarantee

D. A larger query sensitivity

Correct Answer: A. Stronger privacy and potentially lower utility

Question 5

What is a principal limitation of random record-level splitting in longitudinal healthcare data?

A. It always increases privacy.

B. It can place records from the same patient in both training and test sets.

C. It prevents calculation of calibration.

D. It guarantees external validity.

Correct Answer: B. It can place records from the same patient in both training and test sets.

Question 6

Which expression represents the average treatment effect?

A. E[Y(1)+Y(0)]

B. E[Y(1)Y(0)]

C. E[Y(1)-Y(0)]

D. P(Y=1|A=1)

Correct Answer: C. E[Y(1)-Y(0)]

Question 7

In basic federated averaging, client updates are commonly weighted by:

A. The alphabetical order of clients

B. The number of samples held by each client

C. The privacy parameter only

D. The model’s number of layers

Correct Answer: B. The number of samples held by each client

Question 8

Optimal transport requires, among other elements, a cost function that specifies:

A. How dissimilarity between transported mass locations is measured

B. The protected attribute of every individual

C. The exact causal graph

D. The neural network activation function

Correct Answer: A. How dissimilarity between transported mass locations is measured

Question 9

A function satisfying [[M:f(g·x)=f(x)]] is described as:

A. Equivariant

B. Invariant

C. Non-identifiable

D. Differentially private

Correct Answer: B. Invariant

Question 10

Which is the strongest reason that removing a protected attribute may not eliminate unfairness?

A. Protected attributes are never present in real data.

B. Proxy variables can retain information about the protected attribute.

C. Removing variables always causes perfect calibration.

D. Fairness metrics do not use predictions.

Correct Answer: B. Proxy variables can retain information about the protected attribute.