Bias, Fairness, and Nondiscrimination

Introduction

Artificial intelligence systems increasingly influence employment, credit, education, health care, policing, insurance, housing, and access to public services. These systems can improve consistency and efficiency, but they can also reproduce or amplify inequalities. A model may be technically accurate overall while systematically disadvantaging a protected group. Responsible AI therefore requires more than measuring predictive performance: it requires identifying sources of bias, selecting defensible notions of fairness, evaluating trade-offs, and monitoring effects after deployment.

Bias is not limited to the algorithm. It can enter through historical institutions, data collection, measurement choices, labels, model design, interfaces, human decisions, and the social environment in which a system operates. Fairness is consequently a sociotechnical property rather than a property of code alone.

Learning Objectives

By the end of this chapter, students should be able to:

1. Distinguish historical, sampling, measurement, labeling, algorithmic, and deployment bias.

2. Explain group, individual, and counterfactual fairness.

3. Calculate and interpret demographic parity, equalized odds, predictive parity, and related metrics.

4. Explain why some fairness criteria cannot generally be satisfied simultaneously.

5. Select appropriate pre-processing, in-processing, and post-processing interventions.

6. Design an evaluation and monitoring process that includes governance, documentation, and stakeholder participation.

1. What Bias and Fairness Mean

1.1 Bias as systematic distortion

In statistics, bias often means a systematic difference between an estimator's expected value and the quantity it is intended to estimate. In AI ethics, the term is broader: it includes systematic patterns that produce unjustified disadvantages or advantages for people because of group membership, identity, or socially correlated characteristics.

A disparity is an observed difference in outcomes between groups. A disparity is not automatically proof of discrimination. It may reflect legitimate differences in relevant circumstances, unequal opportunity earlier in a causal process, measurement error, or discriminatory treatment. Conversely, the absence of a disparity on one metric does not establish fairness.

Protected attributes may include race, ethnicity, sex, gender identity, disability, age, religion, or other legally or ethically sensitive characteristics. Their legal treatment varies by jurisdiction. Proxy variables—such as postcode, language, school, browsing history, or names—may encode protected characteristics even when those attributes are omitted.

1.2 Fairness as a normative and technical concept

Fairness has no single universally correct mathematical definition. Relevant questions include:

• Are people treated equally at a decision threshold?
• Are errors distributed equally across groups?
• Do equally qualified individuals receive similar decisions?
• Would the decision change if only a protected attribute were changed?
• Are historically unequal opportunities being preserved or corrected?

These questions correspond to different fairness frameworks. A defensible choice must reflect the decision's purpose, the harms of errors, legal requirements, and the causal structure of the domain.

2. Where Bias Enters the AI Lifecycle

2.1 Historical and institutional bias

Historical bias occurs when the world represented by the data contains longstanding unequal treatment or unequal opportunity. For example, a hiring dataset based on past employees may reflect an organisation's exclusionary recruitment practices. A model trained to imitate those outcomes can learn that exclusion as if it were merit.

Historical bias is not necessarily removed by collecting more data. More observations of a discriminatory process can make the pattern statistically clearer. Corrective action may require changing labels, redesigning the target, collecting new evidence, or applying an explicit equity policy.

2.2 Sampling and representation bias

Sampling bias occurs when the training or evaluation sample does not represent the population or decision context in which the system will operate. Under-representation can increase error variance and make subgroup performance appear deceptively reliable. A facial-analysis system developed mainly with one skin-tone distribution, for example, may perform unevenly elsewhere.

Important forms include:

• Coverage bias: some relevant population segments are absent.
• Selection bias: inclusion depends on a process related to the outcome.
• Non-response bias: people who respond differ systematically from those who do not.
• Survivorship bias: only successful or surviving cases are observed.
• Temporal or geographic shift: training data differ from future or local populations.

Evaluation samples should be sufficiently large for subgroup estimates, and uncertainty intervals should be reported rather than treating small observed differences as precise facts.

2.3 Measurement bias

Measurement bias arises when recorded variables do not measure the intended construct equally well across groups. Arrest records, for instance, measure police activity as well as underlying offending. A health expenditure variable may measure access to care and ability to pay rather than illness severity.

Measurement bias can affect inputs, labels, and protected attributes. It may result from different sensor quality, language differences, inaccessible tests, culturally narrow instruments, or unequal opportunity to generate observable data.

2.4 Labeling and annotation bias

Labels are often judgments, not natural facts. Annotators may disagree, instructions may be unclear, and labels may encode stereotypes. A toxicity classifier can misclassify dialectal language if annotators interpret it through dominant cultural norms.

Labeling bias should be distinguished conceptually from historical bias, although they can coexist. If a label records a discriminatory historical decision, the problem has a historical source and a label-mediated representation. For diagnostic analysis, identify both the origin of the injustice and the mechanism by which it enters the dataset.

Useful practices include multiple annotators, adjudication rules, reliability analysis, uncertainty-aware labels, culturally diverse annotation teams, and auditing disagreement by subgroup.

2.5 Algorithmic and objective-function bias

Algorithmic bias may result from model architecture, feature selection, loss functions, thresholds, imbalanced optimization, or use of a proxy. Optimizing aggregate accuracy can sacrifice minority-group performance. Regularization, missing-data handling, and ranking choices can also produce unequal effects.

An algorithm may be mathematically neutral in its code while its objective encodes a value judgment. For example, minimizing false positives may be prioritised over false negatives, but that choice has different consequences in credit, medical diagnosis, and criminal justice.

2.6 Deployment and automation bias

Deployment bias occurs when system use differs from the assumptions under which it was validated. A risk score may be used as an automatic refusal rule even though it was designed only to support human review. Users may over-trust a model, ignore contrary evidence, or selectively override it for different groups.

Feedback loops are especially important. If a model directs police attention to selected locations, resulting records may make those locations appear still more risky. If a lending model rejects applicants from a community, reduced access to credit may later depress observed repayment history, reinforcing the model's assumptions.

3. Formalising Decisions and Groups

Let:

• \(A\) be a protected-group attribute;
• \(X\) be observed features;
• \(Y\in\{0,1\}\) be the relevant outcome or ground-truth label;
• \(\hat Y\in\{0,1\}\) be a binary model decision; and
• \(S\) be a score, where larger values indicate greater predicted likelihood of \(Y=1\).

For group \(g\), define:

\[

TPR_g=P(\hat Y=1\mid Y=1,A=g),\quad FPR_g=P(\hat Y=1\mid Y=0,A=g)

\]

\[

PPV_g=P(Y=1\mid \hat Y=1,A=g),\quad NPV_g=P(Y=0\mid \hat Y=0,A=g).

\]

The positive base rate is \(\pi_g=P(Y=1\mid A=g)\). These quantities must be calculated on comparable populations, using a clearly defined label and time period.

4. Major Fairness Criteria

4.1 Demographic parity

Demographic parity, or statistical parity, requires equal positive-decision rates:

\[

P(\hat Y=1\mid A=a)=P(\hat Y=1\mid A=b).

\]

It is relevant when groups should have comparable access to a benefit and when the decision should not reflect unequal historical opportunity. It can be inappropriate when the target represents a genuine qualification and group base rates differ for reasons that the policy accepts.

A commonly used ratio is the selection-rate ratio:

\[

\frac{P(\hat Y=1\mid A=a)}{P(\hat Y=1\mid A=b)}.

\]

A ratio of 1 indicates parity; it does not, by itself, prove fairness.

4.2 Equal opportunity and equalized odds

Equal opportunity requires equal true-positive rates:

\[

TPR_a=TPR_b.

\]

It focuses on people who truly qualify or have the relevant condition. Equalized odds is stronger and requires both equal true-positive and false-positive rates:

\[

TPR_a=TPR_b\quad\text{and}\quad FPR_a=FPR_b.

\]

These criteria are appropriate when errors of the same type should have comparable distribution across groups. They depend on the validity and fairness of \(Y\).

4.3 Predictive parity

Predictive parity requires equal positive predictive values:

\[

PPV_a=PPV_b.

\]

Thus, among people receiving a positive decision, the proportion truly positive is equal across groups. This may be important when a positive classification triggers a costly or stigmatizing intervention.

4.4 Individual fairness

Individual fairness is often expressed as: similar individuals should receive similar outcomes. A formal version specifies a task-relevant similarity metric \(d(x_i,x_j)\), and requires outcomes or score distributions to change no more than a permitted amount as \(d\) changes.

The difficulty is defining similarity without reproducing protected-group disadvantage. Two people may be similar on qualifications but differ in access to education because of structural inequality. Individual fairness therefore requires substantive domain reasoning, not merely a mathematical distance.

4.5 Counterfactual fairness

A decision is counterfactually fair for an individual when it would remain unchanged in appropriate counterfactual worlds in which that individual's protected attribute were different, while relevant background factors are handled according to a causal model. In a structural causal model, this requires comparing the decision under interventions such as \(do(A=a)\) and \(do(A=a')\), with careful treatment of descendants of \(A\).

Counterfactual fairness is powerful but model-dependent. A causal graph can be incomplete, and excluding every descendant of a protected attribute may remove legitimate information or prevent corrective policies. It is not equivalent to simply deleting \(A\) from the feature table.

5. Worked Metric Example

Consider 100 applicants in each of two groups. The following confusion matrices are observed:

| Group | TP | FP | FN | TN |

|---|---:|---:|---:|---:|

| A | 36 | 9 | 4 | 51 |

| B | 24 | 6 | 16 | 54 |

For group A, \(TPR_A=36/(36+4)=0.90\), \(FPR_A=9/(9+51)=0.15\), and \(PPV_A=36/(36+9)=0.80\). The positive rate is \((36+9)/100=0.45\).

For group B, \(TPR_B=24/(24+16)=0.60\), \(FPR_B=6/(6+54)=0.10\), and \(PPV_B=24/(24+6)=0.80\). The positive rate is \(30/100=0.30\).

Thus, predictive parity holds in this sample because both PPVs equal 0.80. Demographic parity does not hold because positive rates are 0.45 and 0.30. Equal opportunity does not hold because TPRs are 0.90 and 0.60, and equalized odds does not hold because neither TPR nor FPR is equal.

The observed base rates are \(40/100=0.40\) for A and \(40/100=0.40\) for B. This illustrates that equal base rates are not sufficient for other fairness criteria, and that one metric cannot stand in for a complete evaluation.

6. Why Fairness Criteria Can Conflict

Fairness criteria encode different aims and cannot generally all be imposed simultaneously. A particularly important result concerns calibrated or equally predictive scores, error-rate parity, and unequal base rates.

For a group \(g\), the relationship among positive predictive value, true-positive rate, false-positive rate, and base rate is:

\[

PPV_g=\frac{TPR_g\pi_g}{TPR_g\pi_g+FPR_g(1-\pi_g)}.

\]

This follows from Bayes' rule. If two groups have different base rates and both have nontrivial prediction errors, requiring equal \(TPR\) and \(FPR\) generally produces different \(PPV\) values. Conversely, requiring equal predictive value while base rates differ generally requires different error rates or group-specific thresholds.

The exact incompatibility statement must be made carefully:

• Perfect prediction alone does not imply demographic parity. If \(\hat Y=Y\) and \(\pi_a\ne\pi_b\), then the positive-decision rates equal the unequal base rates, so demographic parity fails.
• Under differing base rates, calibration/predictive parity and equalized odds cannot generally both hold when predictions are imperfect and non-degenerate (that is, when there are meaningful positive and negative predictions and nonzero errors).
• Degenerate or boundary cases—such as perfect prediction, a constant prediction, or special zero-error conditions—must be considered separately.

For example, suppose \(\pi_A=0.20\), \(\pi_B=0.40\), and both groups have \(TPR=0.80\), \(FPR=0.10\). Then:

\[

PPV_A=\frac{0.80(0.20)}{0.80(0.20)+0.10(0.80)}=\frac{0.16}{0.24}=0.667,

\]

whereas

\[

PPV_B=\frac{0.80(0.40)}{0.80(0.40)+0.10(0.60)}=\frac{0.32}{0.38}\approx0.842.

\]

Equalized odds holds, but predictive parity does not. This is not a reason to abandon fairness analysis; it is a reason to identify which harm and policy objective have priority and to disclose the resulting trade-off.

7. A Fairness Assessment Workflow

A reliable assessment should follow the decision lifecycle.

Step 1: Define the decision and harm

Specify who makes the decision, who is affected, what benefit or burden is allocated, and which errors are harmful. Document whether the model predicts, recommends, ranks, or decides.

Step 2: Define the construct and label

Ask whether the outcome is a valid representation of the desired construct. Separate observed behaviour from opportunity, institutional action, or access to resources. Record label uncertainty and possible historical contamination.

Step 3: Map the causal and data-generating process

Create a data sheet, data flow diagram, or causal diagram. Identify proxies, missingness, selection mechanisms, feedback loops, and variables that may be descendants of protected characteristics.

Step 4: Disaggregate evaluation

Report confusion matrices, calibration, score distributions, selection rates, and relevant fairness metrics by group and intersectional subgroup. Include confidence intervals or statistical uncertainty. Compare performance across time, geography, language, disability status, and other relevant contexts.

Step 5: Consult affected stakeholders

Affected communities, domain professionals, auditors, and frontline users can identify harms that aggregate metrics miss. Participation should influence the target, metric, threshold, and remedy—not merely validate a completed system.

Step 6: Decide and document trade-offs

Use a model card or algorithmic impact assessment to record the selected fairness concept, rejected alternatives, expected harms, legal constraints, and escalation process. Fairness thresholds should not be chosen solely because they are easy to meet.

8. Mitigation Methods

8.1 Pre-processing

Pre-processing changes data before model training. Methods include improved sampling, reweighting, stratified collection, removal or correction of unreliable labels, and carefully justified feature transformation. Reweighting can ensure that selected groups contribute differently to a loss function, but it does not repair a fundamentally invalid label.

8.2 In-processing

In-processing incorporates fairness into training. A constrained optimisation problem may be written as:

\[

\min_f L(f;D)\quad\text{subject to}\quad |TPR_a-TPR_b|\le \epsilon,

\]

where \(L\) is a defined predictive loss, \(D\) is the training data, and \(\epsilon\) is an allowed disparity. Alternatively, a penalty can be added to the objective. The constraint must be measurable and compatible with the intended fairness definition. Optimisation can trade accuracy, calibration, interpretability, and subgroup performance.

8.3 Post-processing

Post-processing changes thresholds or decisions after training. Group-specific thresholds can equalise a chosen metric, but they may raise legal, ethical, and transparency concerns. Randomised decisions may be necessary at a threshold boundary to attain exact rates. Post-processing cannot correct a target that is invalid or a deployment process that causes new harm.

8.4 Non-model interventions

Sometimes the appropriate remedy is not a better classifier. Alternatives include increasing access to services, changing institutional policy, providing human review, improving data collection, removing a high-risk use case, or refusing automation. A fairness intervention should reduce actual harm, not merely improve a dashboard metric.

9. Case Study: Illustrative Loan-Review System

Background

This fictional case concerns a bank developing a model to recommend whether applicants receive manual review for a small-business loan. The training label is repayment within two years. Group R has historically received fewer loans and therefore has fewer observed repayment outcomes.

Problem

The initial model has higher overall accuracy than human review, but its false-negative rate is substantially higher for Group R. Investigation shows that loan approval and repayment records are affected by unequal access to capital, branch locations, and prior banking relationships. Income volatility is measured less reliably for self-employed applicants, who are more common in Group R.

Analysis

The observed label is partly selected: repayment is visible mainly for people previously approved. It also reflects access to credit and economic conditions, not only business viability. The model uses postcode and account history, which are proxies for structural differences. A single accuracy figure conceals subgroup harm.

Approach

The bank first collects additional independent evidence for a representative sample, audits income measurement, and separates eligibility for review from an automatic rejection decision. It adopts equal opportunity as an interim monitoring criterion because missing a potentially viable applicant is the primary identified harm. It reports TPR, FPR, calibration, selection rates, and confidence intervals by group and intersectional categories. Borderline cases receive trained human review, with reasons recorded and audited.

Outcome

The disparity in false negatives falls, but overall accuracy decreases slightly and review workload increases. The bank accepts this trade-off because the former metric rewarded exclusion. It also begins monitoring approval rates, repayment outcomes, appeals, override patterns, and changes in applicant composition.

Learning points

The case demonstrates that fairness analysis must examine labels, selection, measurement, proxies, and institutional remedies together. A model cannot be declared fair merely because it excludes a protected attribute or satisfies one numerical criterion.

10. Governance, Regulation, and Professional Practice

Anti-discrimination law may regulate disparate treatment, disparate impact, reasonable accommodation, transparency, and documentation, depending on jurisdiction and sector. Technical fairness metrics do not replace legal advice or impact assessments. Organisations should define accountability, maintain audit logs, control access to sensitive attributes, protect privacy, and provide notice, explanation, appeal, and human escalation where appropriate.

Sensitive attributes may be needed for auditing even when they are not used for prediction. Collection must have a lawful and ethical basis, with data minimisation, security, retention limits, and safeguards against misuse. Removing group membership can make auditing impossible and does not eliminate proxy discrimination.

Best practice includes:

• documenting data provenance, label construction, and known limitations;
• testing intersectional and small-population effects;
• using uncertainty estimates rather than rigid pass/fail claims;
• evaluating the complete workflow, including human overrides;
• conducting pre-deployment and post-deployment impact assessments;
• establishing complaint, appeal, rollback, and incident-response procedures;
• repeating assessment after distribution shift or policy change; and
• involving affected communities in governance.

11. Contemporary Developments and Limitations

Foundation models and generative AI introduce additional concerns: uneven representation in pretraining data, dialect and language disparities, differential refusal behaviour, content moderation errors, and opacity in training sources. Fairness evaluation must consider both generated content and downstream use. Synthetic data may improve representation, but it can also reproduce the correlations and errors of its source data.

Intersectionality is essential. A system may appear fair for women and fair for older people separately while disadvantaging older women. Small subgroup samples create uncertainty, but lack of statistical power is not evidence that no harm exists. Privacy-preserving evaluation, participatory research, and carefully designed qualitative studies can complement quantitative metrics.

Metrics also have limitations. Group parity can hide within-group inequality; individual fairness depends on a contested similarity metric; counterfactual fairness depends on a contested causal model; and calibration can coexist with unequal error burdens. Fairness is therefore an iterative process of measurement, justification, intervention, and accountability.

Chapter Summary

Bias can arise from historical institutions, sampling, measurement, labels, algorithms, and deployment practices. Fairness must be defined in relation to the decision, the relevant harm, the validity of the outcome label, and the social context. Demographic parity compares positive-decision rates; equal opportunity compares true-positive rates; equalized odds compares both true- and false-positive rates; predictive parity compares positive predictive values; individual fairness concerns similar treatment of similar people; and counterfactual fairness uses a causal comparison across protected-attribute interventions.

These criteria express different values and can conflict. In particular, with unequal base rates and nontrivial prediction errors, equalized odds and predictive parity generally cannot both hold. Perfect prediction does not itself guarantee demographic parity when base rates differ. Responsible practice therefore requires explicit trade-offs, subgroup and intersectional evaluation, stakeholder participation, mitigation across the lifecycle, and continuous monitoring after deployment.

Key Terms

Bias: Systematic distortion or disadvantage in data, modelling, or decisions.

Protected attribute: A characteristic receiving legal or ethical protection against discrimination.

Proxy variable: A feature correlated with a protected attribute.

Historical bias: Bias inherited from unequal social or institutional practices.

Sampling bias: Distortion caused by non-representative data selection.

Measurement bias: Unequal validity or accuracy in measuring a construct.

Labeling bias: Systematic distortion introduced when outcomes are annotated or defined.

Deployment bias: Harm caused by using a system outside its validated context.

Base rate: The prevalence of a positive outcome in a group.

Demographic parity: Equality of positive-decision rates across groups.

True-positive rate: The proportion of actual positives correctly identified.

False-positive rate: The proportion of actual negatives incorrectly identified.

Predictive parity: Equality of positive predictive value across groups.

Equal opportunity: Equality of true-positive rates across groups.

Equalized odds: Equality of true-positive and false-positive rates across groups.

Individual fairness: Similar treatment for individuals who are similar on relevant dimensions.

Counterfactual fairness: Invariance of a decision under a justified counterfactual change in protected status.

Calibration: Agreement between predicted scores or probabilities and observed outcome frequencies.

Intersectionality: Analysis of overlapping social identities and their combined effects.

Feedback loop: A process in which system outputs alter later data or conditions, reinforcing or changing predictions.

Review Questions

Short Answer Questions

1. Distinguish sampling bias from measurement bias.

2. What does equalized odds require?

3. Why can removing a protected attribute fail to remove discrimination?

4. Define counterfactual fairness.

5. Why should fairness metrics be reported by intersectional subgroups?

Long Answer Questions

1. Explain how historical, labeling, and deployment bias can interact in an automated decision system.

2. Compare demographic parity, equal opportunity, equalized odds, and predictive parity, including contexts in which each may be relevant.

3. Explain the relationship among base rates, error rates, and predictive parity.

4. Compare pre-processing, in-processing, post-processing, and non-model mitigation.

5. Design a governance process for a high-impact AI system from data collection through post-deployment monitoring.

Analytical / Application Questions

1. Using the confusion matrices in this chapter, calculate the selection-rate ratio for Group B relative to Group A and interpret it.

2. A classifier has equal TPR and FPR across two groups but their base rates differ. Use the predictive-value equation to explain what can happen to PPV.

3. For a proposed university admissions model, identify two possible labels, three sources of bias, two appropriate fairness metrics, and one mitigation strategy. Justify each choice.

Multiple Choice Questions with Answers

Question 1

Which example most directly illustrates sampling bias?

A. A label records a supervisor's discriminatory judgment

B. A training sample excludes most rural applicants

C. A threshold is selected to reduce false positives

D. A deployed model is used for automatic rejection although it was validated only for ranking

Correct Answer: B. A training sample excludes most rural applicants

Question 2

In the chapter's worked metric example, what is the true-positive rate for Group B?

A. 0.10

B. 0.30

C. 0.60

D. 0.80

Correct Answer: C. 0.60

Question 3

Which condition defines demographic parity for two groups?

A. Equal false-positive rates

B. Equal positive-decision rates

C. Equal positive predictive values

D. Equal base rates

Correct Answer: B. Equal positive-decision rates

Question 4

Equalized odds requires equality of which pair of quantities across groups?

A. Base rate and selection rate

B. Positive predictive value and negative predictive value

C. True-positive rate and false-positive rate

D. Accuracy and calibration

Correct Answer: C. True-positive rate and false-positive rate

Question 5

If two groups have unequal base rates and a classifier has equal TPR and FPR with nonzero errors, what generally differs?

A. The protected attribute

B. The positive predictive value

C. The definition of true positive

D. The sample size by mathematical necessity

Correct Answer: B. The positive predictive value

Question 6

Why is excluding a protected attribute from model inputs insufficient to guarantee nondiscrimination?

A. Models cannot operate without protected attributes

B. All models necessarily produce equal outcomes

C. Other variables may act as proxies

D. Fairness metrics do not use predictions

Correct Answer: C. Other variables may act as proxies

Question 7

Which mitigation changes the training objective by imposing a fairness constraint?

A. Pre-processing

B. In-processing

C. Post-processing

D. Manual data deletion after deployment

Correct Answer: B. In-processing

Question 8

What is the main focus of equal opportunity?

A. Equal true-positive rates

B. Equal positive-decision rates regardless of outcomes

C. Equal training-set sizes

D. Equal model coefficients

Correct Answer: A. Equal true-positive rates

Question 9

In the loan-review case, why was the repayment label potentially problematic?

A. Repayment was observed only for many previously approved applicants

B. Repayment is always unrelated to access to credit

C. The label contained no historical information

D. The model had no features at all

Correct Answer: A. Repayment was observed only for many previously approved applicants

Question 10

Which statement about perfect prediction and demographic parity is correct when group base rates differ?

A. Perfect prediction always guarantees demographic parity

B. Perfect prediction guarantees parity only if thresholds differ

C. Perfect prediction can fail demographic parity because positive rates reproduce unequal base rates

D. Demographic parity and prediction accuracy are mathematically identical

Correct Answer: C. Perfect prediction can fail demographic parity because positive rates reproduce unequal base rates