Explainable, Ethical and Responsible Machine Learning

Introduction and context

Machine-learning systems increasingly influence decisions about credit, employment, education, healthcare, insurance, public services and online information. Their value depends not only on predictive accuracy but also on whether people can understand, challenge and govern their decisions. A model that is accurate but systematically disadvantages a group, exposes sensitive information, or cannot be audited may be professionally and socially unacceptable.

Explainable, ethical and responsible machine learning therefore treats a model as part of a wider sociotechnical system. The system includes data collection, labelling, feature engineering, model development, deployment, human decisions, organisational policies and affected communities. Responsibility must be considered across the entire model lifecycle rather than added as a final compliance check.

This chapter focuses on practical foundations for interpreting models and assessing fairness, privacy, transparency, accountability, security and human oversight.

Learning objectives

By the end of this chapter, you should be able to:

1. Distinguish interpretability, explainability, transparency and accountability.
2. Interpret global and local model explanations, including feature importance and partial dependence.
3. Calculate and interpret selected classification fairness measures.
4. Identify privacy, security, transparency and human-oversight risks.
5. Apply a responsible-machine-learning lifecycle to a practical case.
6. Recognise the limitations of explanations and avoid treating them as proof of fairness or causality.

1. Foundations of responsible machine learning

Interpretability and explainability

Interpretability describes how directly a human can understand the relationship between inputs and outputs. A small decision tree or a linear model may be intrinsically interpretable because its structure is visible. Explainability is broader: it includes techniques that provide understandable accounts of a model's behaviour, even when the underlying model is complex.

A useful distinction is:

• Global explanation: describes the overall behaviour of a model across many observations.
• Local explanation: describes why a particular prediction was produced.
• Model-specific explanation: uses the internal structure of a model, such as coefficients or split rules.
• Model-agnostic explanation: treats the model as a black box and studies its inputs and outputs.

An explanation should answer a defined question. For example, a loan applicant may need to know why an application was declined, while a model developer may need to know which variables create instability across time. These are different explanation requirements.

Transparency, accountability and contestability

Transparency concerns the availability of information about the system: its purpose, data sources, limitations, performance, operating conditions and responsible organisation. Transparency does not necessarily mean publishing source code. Excessive technical detail can be useless to affected people, while insufficient information prevents meaningful scrutiny.

Accountability means that identifiable people or organisations are responsible for decisions, monitoring, remediation and governance. An algorithm cannot accept an appeal or compensate a person; an institution must do so.

Contestability is the ability of an affected person to question, appeal or correct an automated decision. A responsible system should provide an understandable reason, a route for human review and a method for correcting inaccurate data.

Core principles

Responsible machine learning commonly incorporates the following principles:

These principles can conflict. Collecting demographic information may create privacy concerns, but the same information may be necessary to audit fairness. A responsible design makes such trade-offs explicit and proportionate.

2. Understanding model behaviour

Feature importance in linear models

For a linear prediction model, a numerical output may be represented as:

Here, ŷ is the predicted outcome, β₀ is the intercept, βⱼ is the coefficient for feature xⱼ, and p is the number of features. The coefficient gives the expected change in the prediction for a one-unit increase in xⱼ, holding the other included features constant. This interpretation assumes the model form is appropriate and that the feature's units are understood.

Coefficient magnitude cannot always be compared directly. If income is measured in dollars and age in years, their coefficients use different scales. Standardising a feature to a mean of zero and standard deviation one can make coefficient magnitudes more comparable, but it changes the interpretation to a one-standard-deviation change. Correlated features also make individual coefficients unstable because several variables may share predictive information.

For logistic regression, the model commonly represents the log-odds of a positive class:

where p is the predicted probability of the positive class. Holding other variables constant, increasing xⱼ by one unit multiplies the odds by e^βⱼ. This is an odds interpretation, not a guaranteed causal effect.

Tree-based importance

Decision trees and ensembles often report impurity-based importance. A feature receives importance when splits using it reduce impurity. For classification, one impurity measure is Gini impurity:

where pᵢ is the proportion of observations belonging to class i in a node. Gini impurity is zero when a node contains only one class. In a tree, the importance of a split depends on the weighted reduction in impurity, commonly calculated as the parent impurity minus the weighted child impurities.

Impurity importance can be biased toward continuous variables or categorical variables with many possible values. It also distributes importance unreliably among correlated features. Therefore, it should be treated as a diagnostic rather than a complete explanation.

Permutation importance

Permutation importance measures how much performance decreases when a feature's values are randomly shuffled in a validation dataset. If shuffling feature xⱼ substantially reduces performance, the model relied on information associated with that feature.

A generic form is:

where S₀ is the original validation score and Sⱼ^perm is the score after permuting feature j. The score may be accuracy, F1 score, area under the ROC curve or an appropriate loss; higher Iⱼ indicates greater reliance under that test.

Permutation importance is model-agnostic, but correlated features can mask one another. If two features contain similar information, shuffling one may have little effect because the model can still use the other. Permutation must be performed on data not used to fit the model, otherwise importance can be misleading.

Partial dependence and individual conditional expectation

A partial-dependence plot estimates the average predicted response as one feature varies while other observed features are averaged over. For a prediction function f and feature xⱼ, the partial dependence at value z is conceptually:

X₋ⱼ denotes the remaining features. This is useful for identifying non-linear relationships, thresholds and diminishing returns. However, when features are strongly correlated, the plot may evaluate unrealistic combinations, such as a person's age and work history that rarely occur together.

An individual conditional expectation plot displays a separate prediction curve for each observation. It can reveal heterogeneous effects that are hidden by an average curve. Both methods describe model behaviour, not necessarily real-world causal effects.

Local explanations and Shapley-based methods

A local explanation attributes a particular prediction to contributions from features relative to a baseline prediction. Shapley-based methods originate in cooperative game theory, where the prediction is treated as a value produced by a coalition of features. Contributions are averaged over possible feature orderings. In simplified form:

Here, f(x) is the model prediction for observation x, φ₀ is the baseline output, and φⱼ is the contribution attributed to feature j. The contributions sum to the prediction under the chosen output scale and background dataset.

These explanations can be useful for investigating individual decisions. Nevertheless, they depend on choices about the background data, feature dependence and whether the output is expressed as probability, log-odds or another quantity. A plausible-looking explanation is not automatically faithful, causal or fair.

Worked example: interpreting a local explanation

Suppose a binary classifier predicts the probability that a customer will miss a payment. The baseline probability is 0.20. For one customer, the explanation reports contributions of +0.15 for high debt ratio, +0.08 for recent arrears and −0.04 for stable employment. The contributions are expressed on the probability scale and sum as follows:

0.20 + 0.15 + 0.08 − 0.04 = 0.39.

The predicted probability is therefore 0.39. The explanation suggests that debt ratio and recent arrears increased the prediction relative to the baseline, while stable employment reduced it. It does not establish that changing employment status would cause the probability to fall by exactly 0.04, nor does it show whether the model behaves similarly for other demographic groups.

3. Fairness assessment

Fairness is not one universal statistical property. The appropriate assessment depends on the decision, legal context, harm, affected groups and acceptable trade-offs.

Assume a binary classifier produces a predicted positive label and that a protected or monitored attribute divides people into groups A and B. A confusion matrix contains true positives (TP), false positives (FP), true negatives (TN) and false negatives (FN).

Important measures include:

TPR is the true-positive rate, also called sensitivity or recall. FPR is the false-positive rate. PPV is positive predictive value or precision. Each denominator must be non-zero, and the measure must be interpreted in relation to the decision context.

Common group-based criteria are:

• Demographic parity: the rate of positive predictions is similar across groups.
• Equal opportunity: TPR is similar across groups.
• Equalised odds: both TPR and FPR are similar across groups.
• Predictive parity: PPV is similar across groups.
• Calibration: among people assigned the same predicted risk, observed outcome rates are similar across groups.

These criteria are not generally simultaneously achievable when groups have different base rates and the classifier is not perfect. Consequently, selecting a fairness criterion is a substantive policy decision, not merely a technical optimisation.

Worked fairness example

Suppose a screening system produces the following results:

For group A:

• TPR = 80 ÷ (80 + 20) = 0.80, or 80%.
• FPR = 10 ÷ (10 + 90) = 0.10, or 10%.

For group B:

• TPR = 60 ÷ (60 + 40) = 0.60, or 60%.
• FPR = 20 ÷ (20 + 80) = 0.20, or 20%.

The system has unequal opportunity and unequal false-positive rates: members of group B are both less likely to be correctly identified and more likely to receive a false positive. The analysis does not by itself explain the cause. Possible causes include sampling, label bias, measurement differences, feature quality, threshold selection or historical inequality.

Fairness analysis should examine intersectional groups, uncertainty intervals, outcome definitions and operational consequences. Small samples can make apparent differences unstable. Aggregate equality can also hide harms to subgroups.

4. Privacy and data protection

Machine-learning projects can create privacy risks even when names are removed. Quasi-identifiers, such as age, location and occupation, may permit re-identification when combined with other datasets. Sensitive inferences can also be generated from apparently ordinary data.

A responsible privacy approach includes:

1. Purpose limitation: collect and use data for specified legitimate purposes.
2. Data minimisation: collect only what is necessary.
3. Access control: restrict data and model access according to role.
4. Retention control: delete or securely archive data when no longer needed.
5. Encryption: protect data in transit and at rest.
6. Auditability: record access, changes, approvals and incidents.
7. User rights and notice: communicate relevant processing and provide appropriate access, correction or appeal mechanisms.

Anonymisation is difficult and should not be assumed merely because direct identifiers are removed. Pseudonymisation reduces direct exposure but does not eliminate the possibility of re-identification.

Privacy-preserving techniques include aggregation, differential privacy, secure computation and federated learning. Differential privacy provides a formal way to limit how much the output changes when one person's record is added or removed. Its privacy protection is controlled by a budget parameter, commonly denoted ε; smaller values generally provide stronger privacy but may reduce statistical utility. The choice of privacy budget is a governance decision requiring expertise rather than a universal fixed setting.

5. Security and robustness

Responsible systems must address both ordinary failures and deliberate attacks. Threats include data poisoning, evasion through manipulated inputs, model extraction, membership inference, prompt or interface abuse, credential theft and insecure deployment.

Security controls should include threat modelling, secure software development, dependency management, input validation, access controls, rate limiting, logging, adversarial testing, backup procedures and incident response. A model may be mathematically accurate in a laboratory but unsafe if an attacker can alter its inputs or obtain sensitive outputs.

Robustness also concerns distribution shift. If training data represent one population or period and deployment conditions change, performance and fairness may deteriorate. Monitoring should therefore track data drift, prediction drift, error rates, subgroup performance and unusual access patterns.

6. Human oversight and accountability across the lifecycle

Human oversight is meaningful only when people have the authority, time, competence and information to intervene. A human who merely approves every machine recommendation without reviewing evidence provides nominal rather than substantive oversight.

A practical lifecycle is:

Useful governance artefacts include a data sheet, model card, risk register, impact assessment, evaluation report, change log and incident log. Documentation should record not only what the system does, but also where it should not be used.

7. Case study: illustrative loan-screening system

Background

A regional lender develops a model to prioritise applications for manual review. The model uses income, existing debt, repayment history, employment duration and geographic indicators. It is intended to improve processing speed, not to make an irreversible lending decision.

Problem

After deployment, the lender observes that applications from one neighbourhood are referred for review more often. A feature-importance report ranks the geographic indicator highly. Staff initially conclude that the feature is simply identifying financial risk.

Analysis

The responsible team examines data provenance and discovers that historical lending decisions were used as labels. Those decisions reflected earlier policies and unequal access to credit. The geographic feature is also correlated with race and local economic disadvantage. Removing the feature alone does not solve the problem because other variables act as proxies.

The team compares confusion matrices and calibration by group, tests models with and without geographic variables, examines local explanations, and reviews false-positive cases with lending specialists. It finds that a debt measure was calculated inconsistently for applicants with irregular income. The error disproportionately affected one group.

Approach and solution

The lender takes several actions:

• corrects the debt calculation and improves data-quality checks;
• changes the objective from automatic rejection to prioritisation for human review;
• excludes geographic variables from the decision model while retaining carefully governed group information for fairness auditing;
• establishes group-specific monitoring for error rates and review outcomes;
• provides applicants with a plain-language reason and an appeal route;
• requires documented human review for adverse decisions;
• conducts a privacy and security assessment;
• schedules periodic independent validation and a rollback process.

Outcome

Processing time improves while the most serious disparity decreases. The lender does not claim that the revised model is perfectly fair. Instead, it treats fairness as an ongoing monitoring and governance obligation. The model card states its intended use, known limitations, evaluation populations and conditions requiring suspension.

Learning points

First, historical labels can encode institutional decisions rather than objective truth. Second, removing a sensitive feature does not remove proxy discrimination. Third, explanation tools help locate suspicious behaviour but do not determine whether a decision is ethically justified. Finally, changing the decision process and adding appeal mechanisms may reduce harm more effectively than changing the algorithm alone.

8. Limitations and professional judgement

Explanations can be incomplete, unstable or misleading. Different explanation methods may produce different stories. A local explanation may be accurate near one observation but irrelevant elsewhere. Feature importance can reflect correlation rather than causation. A simple explanation can also create false confidence by hiding uncertainty.

Fairness metrics depend on labels, thresholds and group definitions. Privacy protections may reduce utility. Human review can introduce inconsistency or automation bias. Documentation can become outdated after retraining. Compliance with a regulation or standard is important, but formal compliance does not guarantee ethical acceptability.

Professionals should therefore triangulate evidence: inspect data, compare several explanation methods, test subgroup outcomes, consult domain experts and affected stakeholders, and document unresolved uncertainty. When risks cannot be controlled adequately, the responsible decision may be to narrow the use case or not deploy the system.

9. Best-practice checklist

Before deployment, a data-science team should be able to answer:

• What decision is being supported, and who may be harmed?
• What alternatives to automation were considered?
• Are data sources, labels and missing values documented?
• Which global and local explanations are appropriate?
• Have explanations been checked for stability and stakeholder usefulness?
• Which fairness criteria fit the decision, and what trade-offs remain?
• Are privacy, retention, access and re-identification risks controlled?
• Can the system resist foreseeable attacks and distribution shift?
• Who has authority to intervene, override, appeal and stop deployment?
• What metrics, thresholds and incident procedures will be monitored?

Chapter Summary

Responsible machine learning combines technical performance with fairness, privacy, transparency, accountability, security and meaningful human oversight. Global methods such as coefficients, permutation importance and partial-dependence plots describe general model behaviour, while local methods explain individual predictions. All explanations depend on assumptions and should not be interpreted automatically as causal evidence.

Fairness is assessed through context-dependent criteria such as demographic parity, equal opportunity, equalised odds, predictive parity and calibration. These criteria may conflict, particularly when groups have different base rates. Privacy requires minimisation, access control, secure retention and careful treatment of re-identification and inference risks. Security and robustness require protection against attacks, data drift and changing deployment conditions.

The central professional lesson is that responsible machine learning is a lifecycle activity. Careful problem definition, documentation, independent validation, stakeholder involvement, monitoring, appeals and accountable human decisions are as important as model selection.

Key Terms

Interpretability: The degree to which a model's behaviour can be understood directly.

Explainability: Methods and information used to communicate why a model behaves as it does.

Global explanation: An account of model behaviour across a population or dataset.

Local explanation: An account of a single prediction or small local region.

Feature importance: An estimate of how much a feature contributes to model performance or predictions.

Permutation importance: Performance change after randomly shuffling a feature.

Partial dependence: Average model response as a feature is varied.

Shapley value: A contribution assigned using the principles of cooperative game theory.

Transparency: Availability of information about a system's purpose, operation and limitations.

Accountability: Assignment of responsibility for decisions, monitoring and remediation.

Contestability: The ability to question, appeal or correct an automated decision.

Protected attribute: A characteristic relevant to discrimination or legal protection.

Demographic parity: Similar positive-prediction rates across groups.

Equal opportunity: Similar true-positive rates across groups.

Equalised odds: Similar true-positive and false-positive rates across groups.

Calibration: Agreement between predicted risk and observed outcome rates across groups.

Data minimisation: Collection and use of only the data necessary for a defined purpose.

Distribution shift: Change between training conditions and deployment conditions.

Human oversight: Competent, empowered human involvement in review and intervention.

Model card: Documentation describing a model's intended use, performance and limitations.

Review Questions

Short Answer Questions

1. Distinguish global explanations from local explanations.
2. Why can coefficient magnitudes be difficult to compare across unstandardised features?
3. Define equal opportunity in classification.
4. Why does removing a protected attribute not necessarily remove discrimination?
5. State two activities that should occur during post-deployment monitoring.

Long Answer Questions

1. Explain permutation importance, including its strengths and limitations with correlated features.
2. Compare demographic parity, equal opportunity, equalised odds and calibration.
3. Discuss why transparency and accountability are different requirements.
4. Explain how privacy and fairness auditing can conflict and how an organisation might manage that conflict.
5. Describe a responsible-machine-learning lifecycle from problem definition to retirement.

Analytical / Application Questions

1. Using the case study, identify three sources of bias and propose a test or control for each.
2. A model has equal accuracy across two groups but a substantially higher false-positive rate for one group. Explain why aggregate accuracy is insufficient and recommend additional analyses.
3. Design a human-oversight and monitoring plan for a model that prioritises applicants for manual review, including escalation and rollback conditions.

Multiple Choice Questions with Answers

Question 1

Which method is primarily a local explanation method?

A. A confusion matrix aggregated by group

B. A Shapley-based explanation for one prediction

C. A dataset retention schedule

D. A global permutation-importance ranking

Correct Answer: B. A Shapley-based explanation for one prediction

Question 2

In a linear model, what does a coefficient generally represent, holding other included variables constant?

A. The number of training records

B. The expected change in prediction for a one-unit feature increase

C. The model's fairness level

D. The probability that the feature is causal

Correct Answer: B. The expected change in prediction for a one-unit feature increase

Question 3

A group has TP = 60 and FN = 40. What is its true-positive rate?

A. 0.40

B. 0.50

C. 0.60

D. 0.67

Correct Answer: C. 0.60

Question 4

Which fairness criterion requires similar true-positive and false-positive rates across groups?

A. Demographic parity

B. Equal opportunity

C. Predictive parity

D. Equalised odds

Correct Answer: D. Equalised odds

Question 5

What is a major limitation of impurity-based tree feature importance?

A. It can never be calculated for classification

B. It may favour continuous or high-cardinality variables

C. It requires protected attributes to be removed

D. It measures only privacy risk

Correct Answer: B. It may favour continuous or high-cardinality variables

Question 6

Why can a partial-dependence plot be misleading with strongly correlated features?

A. It always reports training accuracy

B. It may evaluate unrealistic combinations of feature values

C. It cannot display non-linear relationships

D. It automatically proves causality

Correct Answer: B. It may evaluate unrealistic combinations of feature values

Question 7

Which practice best reflects data minimisation?

A. Collecting every potentially useful variable indefinitely

B. Collecting only information necessary for a defined purpose

C. Publishing all raw training data

D. Removing access logs to reduce administration

Correct Answer: B. Collecting only information necessary for a defined purpose

Question 8

What is the main purpose of a model card?

A. To encrypt model parameters

B. To document intended use, performance and limitations

C. To guarantee equalised odds

D. To replace human oversight

Correct Answer: B. To document intended use, performance and limitations

Question 9

Why may removing a geographic feature fail to eliminate discrimination?

A. Geographic variables are never predictive

B. Other features may act as correlated proxies

C. The model will always become random

D. Privacy law requires geographic features

Correct Answer: B. Other features may act as correlated proxies

Question 10

Which condition makes human oversight meaningful?

A. The reviewer must approve every output without question

B. The reviewer has authority, competence and sufficient information to intervene

C. The system hides uncertainty from the reviewer

D. The reviewer is responsible only after an incident

Correct Answer: B. The reviewer has authority, competence and sufficient information to intervene