Data Ethics, Privacy, and Consent

Introduction and Context

Artificial intelligence systems learn from data, make predictions about people and institutions, and increasingly influence access to employment, education, credit, healthcare, public services, and information. The ethical quality of an AI system therefore depends not only on its algorithm but also on how its data were collected, described, stored, combined, used, and eventually deleted.

Data ethics examines the moral and social implications of data practices. It asks whether data collection is justified, whether individuals understand and control the use of their information, whether datasets represent relevant populations, and whether data-driven decisions create unfair surveillance or discrimination. Privacy is not merely secrecy. It includes appropriate control over personal information, freedom from unjustified monitoring, protection of contextual boundaries, and the ability to participate in society without excessive profiling.

For AI professionals, responsible data practice is both an ethical obligation and an engineering requirement. Poor provenance can make a model legally or scientifically unreliable. Inadequate consent can undermine legitimacy. Biased or incomplete data can produce unequal outcomes. Weak privacy protection can expose individuals to identity theft, manipulation, discrimination, or physical harm.

Learning Objectives

By the end of this chapter, students should be able to:

1. Explain data provenance, ownership, quality, representativeness, and secondary use.

2. Distinguish privacy, confidentiality, anonymity, pseudonymity, and security.

3. Evaluate informed consent and identify when consent is inadequate.

4. Analyse surveillance and profiling risks in AI systems.

5. Apply data minimization, purpose limitation, anonymization, pseudonymization, and differential privacy.

6. Design a privacy-by-design process for an AI application.

7. Recognize relevant professional, governance, and regulatory responsibilities.

1. The Data Lifecycle and Ethical Responsibility

A dataset has a lifecycle: planning, collection, acquisition, cleaning, labeling, storage, sharing, analysis, deployment, retention, and deletion. Ethical responsibility applies at every stage. A team cannot justify questionable collection by claiming that a later model is accurate, nor can it treat deletion as an afterthought.

Data provenance

Data provenance is the documented history of data: where they originated, who collected them, under what conditions, what transformations occurred, and who accessed or modified them. A provenance record may include collection dates, instruments, sampling procedures, consent terms, licensing conditions, preprocessing code, labeling instructions, and known limitations.

Provenance supports:

• accountability and auditability;
• reproducibility of research and model development;
• identification of unlawful or inappropriate sources;
• detection of data drift and contamination; and
• communication of limitations to users.

For example, a facial-image dataset assembled from public websites may appear accessible, but public availability does not automatically grant ethical permission for biometric analysis. Provenance should record whether subjects were informed, whether images were scraped automatically, whether licenses permit machine learning, and whether the intended use differs from the original context.

Ownership, access, and control

Ownership is not a single concept. A person may have interests in information about them even when an organization legally holds a database. Relevant dimensions include:

• Legal rights: rights created by privacy, copyright, database, employment, or sector-specific law.
• Control: practical ability to decide how data are collected, corrected, shared, or deleted.
• Benefit and burden: who gains value and who bears risks.
• Collective and cultural interests: communities may have legitimate interests in data about their members, land, language, or cultural heritage.

An organization that purchases data does not acquire unlimited ethical authority. Contracts, professional duties, consent conditions, and reasonable expectations may restrict use. Data governance should therefore identify data subjects, custodians, processors, users, and affected third parties.

Data quality and representativeness

Data quality is multidimensional. Important dimensions include accuracy, completeness, consistency, timeliness, validity, relevance, and traceability. A dataset may be accurate in recording past decisions but still encode discriminatory institutional practices. Thus, technical quality is not equivalent to ethical quality.

Representativeness concerns whether the data adequately reflect the population and situations for which a system will be used. Underrepresentation can arise from unequal access to technology, selective participation, historical exclusion, missing labels, or measurement practices that work better for some groups than others. A model trained on urban speech may perform poorly for rural speakers; a health dataset drawn from one hospital may not generalize to other populations.

Representativeness should be assessed relative to the intended task, not treated as an abstract requirement. Teams should document the target population, sampling frame, inclusion and exclusion criteria, subgroup coverage, missingness patterns, and known distribution shifts. Performance should be evaluated across relevant groups, but group averages alone may conceal intersectional failures.

2. Consent, Purpose, and Secondary Use

Informed consent

Informed consent is an ethical process, not merely a checkbox or lengthy notice. Valid consent generally requires:

• adequate information about the purpose, methods, risks, benefits, recipients, and retention period;
• comprehension by the person giving consent;
• voluntariness, without coercion or unreasonable pressure; and
• a meaningful opportunity to refuse or withdraw where feasible.

Consent is especially problematic when people have little bargaining power, such as employees, students, patients, prisoners, or recipients of essential public services. A nominal choice may not be genuinely voluntary if refusal causes unfair exclusion.

Consent should be specific enough to support reasonable expectations, yet research and AI development often involve uncertain future uses. Broad consent may be ethically acceptable in some regulated research settings when governance, oversight, security, and withdrawal mechanisms are strong. It should not be used as a universal substitute for explaining foreseeable uses.

Purpose limitation and secondary use

Purpose limitation means that data collected for one justified purpose should not automatically be repurposed for an unrelated purpose. Secondary use can create a contextual integrity problem: information shared in one relationship may be inappropriate in another. A customer may provide a phone number for delivery but not expect it to be used for facial recognition or political profiling.

Before secondary use, organizations should ask:

1. Is the new purpose compatible with the original context?

2. Is a new legal or ethical basis required?

3. Can the objective be achieved with less data?

4. Were affected people informed or given a choice?

5. Are risks to vulnerable groups acceptable?

6. Are retention, access, and deletion controls adequate?

Anonymization may reduce risk but does not automatically legitimize every secondary use. Re-identification, inference, and group harms remain possible.

3. Privacy, Surveillance, and Profiling

Privacy includes several related interests:

• Confidentiality: preventing unauthorized disclosure.
• Anonymity: inability to link information to an identifiable person.
• Pseudonymity: use of a substitute identifier while a separate key may permit re-linking.
• Security: protection against unauthorized access, alteration, loss, or disruption.
• Contextual integrity: appropriate information flows according to social context, roles, and expectations.

Privacy and security overlap but are not identical. Encryption may protect a database from outsiders, while an organization may still collect excessive information lawfully but intrusively.

Surveillance

Surveillance is systematic observation, collection, or analysis of information about people or groups. AI expands surveillance through computer vision, location analytics, biometric recognition, sentiment analysis, online tracking, and predictive systems. Risks include chilling effects, function creep, mistaken identification, unequal targeting, and inability to escape monitoring.

A proportionate surveillance practice should have a defined legitimate aim, clear authority, necessity, limited scope, transparent governance, accuracy safeguards, independent oversight, retention limits, and accessible remedies. The fact that a technology is technically capable does not establish that deployment is justified.

Profiling and inference

Profiling involves using personal or inferred attributes to evaluate, classify, predict, or influence individuals or groups. AI can infer sensitive characteristics from seemingly ordinary data, such as location, browsing behavior, voice, or purchasing patterns. An individual may never have disclosed a health condition, yet a model may infer it.

Profiling can support beneficial personalization, fraud detection, and service delivery, but it may also produce discrimination, manipulation, price differentiation, or denial of opportunity. Ethical assessment should consider not only input attributes but also proxy variables, feedback loops, downstream decisions, and the possibility that individuals cannot contest an inference.

4. Privacy-Preserving Techniques

No technique provides absolute privacy. Controls should be selected according to threat models, data sensitivity, utility requirements, and the consequences of failure.

Data minimization

Data minimization means collecting, retaining, and processing only information necessary for a specified objective. It has three practical dimensions:

1. Collection minimization: do not gather unnecessary fields.

2. Processing minimization: restrict access and computation to what is needed.

3. Retention minimization: delete or irreversibly de-identify data when the purpose ends.

For example, a transit system predicting station crowding may need aggregated passenger counts but not individual names or continuous journey histories.

Pseudonymization and anonymization

Pseudonymization replaces direct identifiers with codes. Because a separate key may permit re-identification, pseudonymized data remain personal data in many governance contexts and require strong access controls.

Anonymization aims to make identification reasonably infeasible, considering available auxiliary data, reasonable means, and time. Removing names alone is rarely sufficient. Quasi-identifiers such as age, postcode, occupation, and dates can be combined with external datasets to identify individuals.

A classic group-based approach is k-anonymity. A dataset is k-anonymous when each combination of selected quasi-identifiers occurs for at least k records. If k equals 5, each released quasi-identifier pattern is shared by at least five records. K-anonymity can reduce singling-out risk, but it does not necessarily protect against attribute disclosure: if all five people in a group share the same sensitive diagnosis, the diagnosis is still revealed. It also depends on which external information an attacker possesses.

Differential privacy

Differential privacy provides a formal way to limit how much the inclusion or exclusion of one person changes an output. A randomized mechanism M is commonly described as satisfying epsilon-differential privacy if, for neighboring datasets D and D' differing in one individual, and for every possible output set S:

P[M(D) in S] <= exp(epsilon) P[M(D') in S].

Here, epsilon is the privacy-loss parameter. Smaller epsilon generally gives stronger privacy but may reduce accuracy; larger epsilon generally permits more accurate results but weaker protection. The guarantee assumes a defined neighboring-dataset relationship and applies to the mechanism, not automatically to an entire organization or system.

For a count query with sensitivity 1, a Laplace mechanism may release:

M(D) = f(D) + Y,

where Y is random noise drawn from a Laplace distribution with scale 1/epsilon. The count sensitivity is 1 because adding or removing one person changes the count by at most one. In practice, privacy budgets must be managed across repeated queries: privacy loss can accumulate. Differential privacy also does not prevent all harms, such as harmful decisions made using a statistically private aggregate.

5. Privacy by Design and Governance

Privacy by design integrates privacy into system architecture rather than adding notices after deployment. A practical process includes:

1. Define the purpose and threat model. Identify legitimate objectives, affected people, attackers, sensitive attributes, and potential harms.

2. Map the data lifecycle. Record collection points, transformations, storage locations, transfers, users, retention, and deletion.

3. Minimize. Remove fields, reduce precision, aggregate where possible, and avoid collecting data merely because future use is imaginable.

4. Select controls. Use access control, encryption, key separation, secure logging, pseudonymization, aggregation, differential privacy, and human review as appropriate.

5. Assess impacts. Conduct privacy and algorithmic impact assessments, including subgroup and misuse analysis.

6. Test and monitor. Test re-identification risk, leakage, access violations, model memorization, and changes in data distribution.

7. Provide rights and remedies. Offer notice, access, correction, deletion where applicable, objection, explanation appropriate to the system, and a route for complaints.

8. Retire responsibly. Delete data and credentials, revoke access, document residual copies, and communicate system discontinuation.

Governance should assign accountable owners rather than treating privacy as everyone's and therefore no one's responsibility. Data protection officers, security specialists, domain experts, affected communities, and independent reviewers may all contribute. Documentation such as datasheets, data registers, consent records, model cards, risk assessments, and incident reports supports accountability.

Worked Example: A University Early-Warning System

A university proposes an AI system to identify students at risk of dropping out. It plans to combine attendance, learning-platform activity, assessment results, disability information, financial-aid status, counseling records, and social-media activity.

Analysis. The objective—offering timely support—may be legitimate, but the proposed data exceed what is obviously necessary. Counseling and disability information are highly sensitive. Social-media collection creates surveillance and contextual-integrity concerns. Historical withdrawal decisions may reflect unequal institutional support and could reproduce disadvantage. Students may not reasonably expect private counseling information to be used for automated risk scores.

Responsible approach. The university should begin with a data minimization exercise. It might use attendance trends and missed assessments, while excluding counseling notes and social-media data. It should inform students clearly, establish an alternative support route, assess whether participation is voluntary, limit access to trained advisors, and prohibit punitive use. Predictions should trigger supportive human contact rather than automatic exclusion. The university should evaluate error rates and support outcomes across relevant student groups, permit correction of inaccurate records, define retention limits, and conduct an independent privacy and equity assessment.

Outcome and lesson. The system becomes less intrusive and its purpose becomes clearer. It may be less predictive than a system using every available field, but it is more legitimate, contestable, and aligned with student welfare. Accuracy alone is not a sufficient success criterion; proportionality, trust, fairness, and beneficial outcomes matter.

Professional, Regulatory, and Contemporary Considerations

Regulatory requirements vary by jurisdiction, but common principles include lawfulness, fairness, transparency, purpose limitation, data minimization, accuracy, storage limitation, security, and accountability. Sector rules may impose additional duties for health, education, employment, finance, and biometric information. Organizations operating across borders must identify which laws apply and should not assume that transferring data is ethically neutral.

Contemporary developments include synthetic data, federated learning, confidential computing, privacy-enhancing computation, biometric regulation, and foundation models trained on large-scale web data. Synthetic data can reduce direct exposure but may reproduce bias or permit membership inference if generated from small sensitive datasets. Federated learning keeps some data local but still requires protection against model updates, inference attacks, and compromised participants. Foundation models may memorize or expose training information and raise difficult questions about consent, licensing, provenance, and deletion.

Important limitations remain. De-identification is not a guarantee. Consent can be impractical at scale and may not address group harms. Differential privacy involves technical trade-offs and requires careful budget management. Privacy assessments can become paperwork unless connected to design decisions and monitoring. Finally, individual privacy does not exhaust data ethics: collective harms, power asymmetries, environmental costs, and unequal distribution of benefits must also be considered.

Best-Practice Checklist

Before approving an AI data project, a team should be able to answer yes to the following questions:

• Is the purpose specific, legitimate, and understandable?
• Is provenance documented and the source authorized?
• Are the population, sampling limitations, and quality issues known?
• Is each data field necessary?
• Are consent and expectations appropriate to the context?
• Has secondary use been justified and communicated?
• Are sensitive attributes and proxy risks assessed?
• Are retention, access, deletion, and breach procedures defined?
• Has the least intrusive effective privacy technique been selected?
• Can individuals correct, challenge, or seek remedy for harmful outcomes?
• Are there accountable owners and independent review?
• Will the system be monitored after deployment and retired responsibly?

Chapter Summary

Data ethics concerns the full lifecycle of data used in AI. Provenance establishes where data came from and how they changed; ownership and control clarify who may use data and under what conditions. Data quality must include accuracy, completeness, relevance, and representativeness, while recognizing that historically generated data may encode institutional bias.

Informed consent requires information, comprehension, voluntariness, and meaningful choice. Purpose limitation and contextual integrity warn against treating one-time disclosure as permission for unlimited secondary use. Privacy includes confidentiality, anonymity, pseudonymity, security, and appropriate information flows. Surveillance and profiling can create chilling effects, discrimination, manipulation, and inference of sensitive traits.

Data minimization, pseudonymization, anonymization, k-anonymity, and differential privacy reduce risk but have different assumptions and limitations. Differential privacy formally bounds the effect of one individual on an output, with a privacy parameter epsilon that represents a trade-off between privacy and utility. Privacy by design embeds purpose definition, threat modeling, minimization, technical controls, impact assessment, monitoring, rights, and deletion into the system lifecycle. Responsible AI requires evaluating legitimacy, proportionality, fairness, contestability, and social impact—not accuracy alone.

Key Terms

Data provenance: Documented history and origin of data and their transformations.

Data minimization: Limiting collection, processing, and retention to what is necessary.

Representativeness: Degree to which data reflect the population and conditions of intended use.

Informed consent: Voluntary agreement based on adequate information and understanding.

Purpose limitation: Restriction of data use to specified and compatible purposes.

Secondary use: Use of data for a purpose different from the original collection purpose.

Privacy: Appropriate control and protection of information and personal boundaries.

Confidentiality: Protection against unauthorized disclosure.

Anonymity: Condition in which data cannot reasonably be linked to an identifiable person.

Pseudonymization: Replacement of identifying information with codes or substitutes.

Anonymization: Processing intended to make re-identification reasonably infeasible.

K-anonymity: A property in which each quasi-identifier pattern is shared by at least k records.

Differential privacy: A formal guarantee limiting the effect of one person’s data on an output.

Privacy budget: Allocated cumulative privacy loss, often represented through epsilon.

Surveillance: Systematic observation or analysis of people or groups.

Profiling: Automated or structured evaluation, classification, prediction, or influence of people.

Contextual integrity: Appropriateness of information flows within a social context.

Privacy by design: Integrating privacy protections into system architecture and lifecycle.

Data quality: Fitness of data in dimensions such as accuracy, completeness, validity, and timeliness.

Algorithmic impact assessment: Structured evaluation of risks, affected groups, safeguards, and accountability.

Review Questions

Short Answer Questions

1. What information should a data provenance record contain?

2. Why is public availability not equivalent to ethical permission to use data?

3. Distinguish pseudonymization from anonymization.

4. What are the four central conditions of informed consent?

5. Why does data minimization reduce both privacy and governance risk?

Long Answer Questions

1. Explain how data quality and representativeness affect the ethical reliability of an AI system.

2. Discuss purpose limitation and contextual integrity in relation to secondary data use.

3. Compare surveillance risks with potential benefits in an AI-enabled public service.

4. Explain differential privacy, including the role of epsilon, sensitivity, and privacy-budget accumulation.

5. Describe a complete privacy-by-design process for a machine-learning application.

Analytical / Application Questions

1. A retailer wants to use loyalty-card data collected for discounts to infer customers’ health conditions and sell targeted advertisements. Analyse the consent, purpose-limitation, profiling, and privacy issues, and recommend safeguards.

2. A hospital has a dataset containing age, postcode, admission date, and diagnosis. Explain why removing names may not anonymize it, and propose a release strategy using minimization, aggregation, or formal privacy methods.

3. Design an ethical review plan for an AI system that predicts university dropout risk. Specify the data to include and exclude, consent and notice requirements, subgroup evaluations, human oversight, retention policy, and remedies.

Multiple Choice Questions with Answers

Question 1

Which statement best describes data provenance?

A. The percentage of missing values in a dataset

B. The documented history, origin, and transformations of data

C. The encryption algorithm used to store data

D. The accuracy of a machine-learning model

Correct Answer: B. The documented history, origin, and transformations of data

Question 2

Why is public availability of personal data not automatically sufficient ethical permission for AI use?

A. Public data are always inaccurate

B. Public data cannot be stored digitally

C. The original context, expectations, rights, and intended purpose may not support the new use

D. AI systems cannot process public data

Correct Answer: C. The original context, expectations, rights, and intended purpose may not support the new use

Question 3

Which practice most directly implements data minimization?

A. Collecting every potentially useful field for future purposes

B. Retaining raw data indefinitely

C. Collecting only fields necessary for a specified objective

D. Publishing all collected fields openly

Correct Answer: C. Collecting only fields necessary for a specified objective

Question 4

What is the main distinction between pseudonymization and anonymization?

A. Pseudonymization always removes all privacy risk

B. Pseudonymized data may be re-linked using a separate key

C. Anonymization requires storing names separately

D. Pseudonymization applies only to financial data

Correct Answer: B. Pseudonymized data may be re-linked using a separate key

Question 5

A dataset satisfies 5-anonymity when each selected quasi-identifier pattern is shared by at least:

A. One record

B. Two records

C. Five records

D. Ten records

Correct Answer: C. Five records

Question 6

In differential privacy, a smaller epsilon generally indicates:

A. Stronger privacy and potentially lower utility

B. Weaker privacy and higher utility in every case

C. No randomization

D. Guaranteed elimination of all bias

Correct Answer: A. Stronger privacy and potentially lower utility

Question 7

Which is an example of secondary use?

A. Using a collected variable for the purpose stated at collection

B. Deleting data at the end of its retention period

C. Using delivery telephone numbers to develop political profiles

D. Correcting an inaccurate address

Correct Answer: C. Using delivery telephone numbers to develop political profiles

Question 8

Which feature is most consistent with informed consent?

A. A hidden purpose disclosed only after participation

B. Voluntary agreement based on understandable information about relevant risks and uses

C. Mandatory agreement with no opportunity to refuse

D. A technical document that participants cannot understand

Correct Answer: B. Voluntary agreement based on understandable information about relevant risks and uses

Question 9

Why can a model using counseling records to predict student dropout be ethically problematic?

A. Counseling records are never digital

B. They may be sensitive data collected in a context where students did not expect predictive profiling

C. They always improve prediction accuracy

D. They cannot be protected by access controls

Correct Answer: B. They may be sensitive data collected in a context where students did not expect predictive profiling

Question 10

Which activity is a core element of privacy by design?

A. Adding a privacy notice only after deployment

B. Collecting data first and defining the purpose later

C. Integrating threat modeling, minimization, controls, assessment, and monitoring into the lifecycle

D. Maximizing data retention to preserve future options

Correct Answer: C. Integrating threat modeling, minimization, controls, assessment, and monitoring into the lifecycle