Privacy, Ethics and Responsible Data Use

Introduction and Context

Data science creates value by transforming observations into decisions, predictions and automated actions. The same capability can also expose individuals to surveillance, discrimination, manipulation, financial loss and loss of autonomy. Privacy, therefore, is not merely a legal compliance function or a requirement to encrypt databases. It is a socio-technical discipline concerned with how information is collected, interpreted, shared, retained and used to affect people.

Responsible data use requires alignment among four dimensions:

1. Law—compliance with applicable privacy, data-protection and sectoral rules.
2. Ethics—respect for autonomy, dignity, fairness and accountability, including harms not explicitly prohibited by law.
3. Engineering—controls embedded in architectures, algorithms, interfaces and operational processes.
4. Governance—institutional responsibility, oversight, documentation and routes for remedy.

A sound privacy program asks not only whether data can be collected, but whether it should be collected, whether the stated purpose is legitimate, whether individuals understand the processing, and whether the resulting benefits justify the risks.

Learning Objectives

By the end of this chapter, students should be able to:

• explain privacy-by-design and privacy-by-default;
• distinguish consent from other lawful bases for processing;
• apply purpose limitation, data minimisation and retention controls;
• describe data-subject rights under GDPR and comparable rights under CCPA/CPRA;
• evaluate anonymisation, pseudonymisation and re-identification risk;
• identify surveillance, bias and secondary-use risks;
• design a responsible data-governance process for analytics and AI systems; and
• critically evaluate privacy decisions in a realistic organisational case.

1. Privacy as a Data-Science Responsibility

Privacy is often defined as control over personal information, but contemporary scholarship treats it more broadly. Privacy includes informational self-determination, freedom from unjustified surveillance, contextual integrity and protection from harmful inference. A dataset may contain no obvious name or address yet still reveal health status, political affiliation, location patterns or financial vulnerability.

Important terminology

• Personal data/information: Information relating to an identified or identifiable person. Definitions vary by jurisdiction, but online identifiers, location data, device identifiers and inferred attributes may qualify.
• Sensitive or special-category data: Data requiring heightened protection, such as health, biometric, genetic, racial or ethnic information under particular laws.
• Data subject/consumer: The person to whom personal data relates.
• Controller/business: The organisation deciding why and how data is processed, although legal terminology differs.
• Processor/service provider: An organisation processing data on behalf of another organisation.
• Processing: Broadly, collection, storage, use, disclosure, alteration, analysis or deletion.
• Pseudonymisation: Replacing direct identifiers with codes while retaining a method of re-linking.
• Anonymisation: Processing intended to make identification no longer reasonably possible, taking account of available means and context.
• De-identification: A broad term covering techniques that reduce the link between data and individuals; it does not always imply irreversible anonymisation.
• Inference: A conclusion generated from data, including a conclusion not explicitly supplied by the individual.

Privacy risk concerns both data about a person and decisions made about a person. A model may not disclose a record yet still produce an unfair insurance price or deny employment. Consequently, privacy and fairness are related but distinct: privacy concerns information and power over information, whereas fairness concerns whether treatment or outcomes are unjustifiably unequal.

2. Privacy-by-Design

Privacy-by-design means incorporating privacy protections into the conception, architecture, development and operation of a system rather than adding them after deployment. The approach is associated with principles such as being proactive, making privacy the default, embedding privacy into design, maintaining full functionality, protecting data throughout its lifecycle, and making practices visible and user-centred.

2.1 Privacy-by-default

A privacy-by-default system should initially collect, expose and retain only what is necessary for the specified service. Examples include:

• location collection disabled unless needed;
• the least detailed location resolution sufficient for the task;
• sharing restricted to the intended audience;
• short default retention periods;
• optional personalisation separated from essential service operation; and
• clear refusal choices rather than preselected consent boxes.

2.2 The privacy engineering lifecycle

A practical lifecycle is:

1. Define the purpose. State the operational or research objective precisely.
2. Map data flows. Identify sources, transformations, recipients, storage locations and deletion points.
3. Assess necessity and proportionality. Ask whether each field and each processing step is necessary for the purpose.
4. Identify threats and harms. Consider disclosure, inference, misuse, exclusion, surveillance and re-identification.
5. Select controls. Use minimisation, access control, encryption, aggregation, privacy-enhancing technologies and governance controls.
6. Test and monitor. Conduct privacy impact assessments, red-team exercises, audits and user research.
7. Retire responsibly. Delete, anonymise or return data when the purpose ends.

A privacy impact assessment (PIA), or data protection impact assessment (DPIA) where required, should document the processing, necessity, risks, safeguards and residual risk. It is not simply a form: it is an argument that can be challenged by security, legal, ethics and affected stakeholders.

Figure: Privacy-by-design is a continuous lifecycle rather than a one-time approval.

3. Consent, Lawful Basis and Purpose Limitation

Consent is an important mechanism but is frequently misunderstood. Valid consent generally requires a meaningful, specific, informed and unambiguous indication of agreement. It should be distinguishable from unrelated terms, as easy to withdraw as to give, and not obtained through coercion or deceptive interface design. Silence, inactivity and preselected options are generally weak evidence of affirmative consent.

Consent is not always the appropriate lawful basis. Under GDPR, processing may rely on bases including consent, contract, legal obligation, vital interests, public task or legitimate interests, subject to conditions and balancing requirements. An organisation should select and document the basis before processing begins. It should not collect data on the pretext of consent and later assume that consent covers unrelated purposes.

The CCPA, as amended by the CPRA, uses a different structure. It grants California residents rights concerning access, deletion, correction, opting out of sale or sharing, and limiting certain uses of sensitive personal information, subject to scope and exceptions. It is not simply a consent-based equivalent of GDPR. Organisations must therefore conduct jurisdiction-specific analysis rather than applying one generic privacy notice.

Purpose limitation

Purpose limitation has two parts:

• Specification: The purpose is stated clearly enough that a person and an auditor can understand it.
• Compatibility: Later use is assessed against the original context, expectations, sensitivity, relationship, safeguards and consequences.

“Improve services” is usually too vague as a standalone purpose. “Detect fraudulent card transactions during payment authorisation” is more specific. A purpose statement should identify whether data will be used for service delivery, safety, research, marketing, personalisation, model training or disclosure to partners.

Secondary use

Secondary use occurs when data collected for one purpose is subsequently used for another. It can be beneficial—for example, using hospital data to improve clinical quality—but it can also violate expectations. Before secondary use, an organisation should ask:

• Is the new purpose compatible with the original purpose?
• Is a new notice or consent required?
• Can the purpose be achieved with less data?
• Are individuals exposed to new decisions or risks?
• Is the dataset sufficiently de-identified for the new use?
• Are contractual, ethical and legal restrictions satisfied?

4. Data-Subject Rights and Organisational Duties

A mature data program treats rights requests as part of system design. Relevant rights vary by law, but commonly include:

Implementation is difficult because information is distributed across warehouses, feature stores, logs, model training sets, backups and vendor systems. A rights-management architecture should therefore maintain a data inventory, identity-resolution process, lineage records, retention rules and processor communication procedures.

The principle of accountability means that an organisation must be able to demonstrate compliance, not merely assert it. Evidence may include processing records, DPIAs, consent logs, access reviews, retention reports, model cards, incident records and training documentation.

5. De-Identification and Re-Identification Risk

Removing names is not equivalent to anonymisation. A dataset containing age, postcode and event date may be linked with publicly available information. Even when direct linkage is difficult, unique behavioural patterns can identify individuals.

5.1 Common techniques

• Suppression: Removing fields or records.
• Generalisation: Replacing precise values with categories, such as age bands.
• Aggregation: Reporting groups rather than individuals.
• Perturbation: Adding controlled noise or modifying values.
• Pseudonymisation: Replacing identifiers with tokens stored separately.
• Synthetic data: Generating artificial records intended to preserve useful statistical structure.
• Differential privacy: Adding mathematically calibrated randomness so that the output changes little when one person’s data is added or removed.

Pseudonymisation reduces exposure but preserves linkability and is generally still personal data where re-identification remains reasonably possible. Anonymisation is contextual: a dataset may be considered safe against one attacker but vulnerable to another with auxiliary data.

5.2 k-anonymity and its limits

A release satisfies k-anonymity with respect to selected quasi-identifiers if every combination of those identifiers appears in at least k records. For example, if k = 5, each released combination of age band, region and occupation must occur in at least five records.

K-anonymity reduces simple linkage attacks, but it does not guarantee privacy. If all five members of an equivalence class have the same disease, an attacker may learn the disease even without identifying one exact record. This is a homogeneity attack. Background knowledge can also defeat the protection. Extensions such as l-diversity and t-closeness address particular weaknesses but introduce assumptions and trade-offs.

5.3 Differential privacy

Differential privacy provides a formal guarantee about the effect of one individual’s participation on an algorithm’s output. A mechanism M is ε-differentially private if, for neighbouring datasets D and D′ differing in one individual, and for every possible output set S:

Here, ε (epsilon) is the privacy-loss parameter; smaller ε generally provides stronger privacy, Pr denotes probability, and S is any set of possible outputs. The guarantee assumes a clearly defined adjacency relation and a correctly implemented mechanism. It protects participation, not necessarily every inference about a person, and utility decreases as privacy loss is restricted. Repeated analyses consume a privacy budget, so composition must be managed.

For a counting query with sensitivity 1, the Laplace mechanism may release:

where f(D) is the true count and the Laplace noise scale is 1/ε. This example assumes one person can change the count by at most one and that the sensitivity has been correctly determined. Differential privacy is not a substitute for access control, secure computation or ethical review.

6. Surveillance and Inference Risks

Data systems can create surveillance even when each individual data point appears harmless. Risks increase when collection is continuous, invisible, unavoidable or linked across contexts. Examples include workplace productivity monitoring, facial recognition in public places, location tracking, behavioural advertising and monitoring of students or patients.

A surveillance assessment should examine:

• Visibility: Do people know they are monitored?
• Choice: Can they refuse without losing essential services?
• Proportionality: Is monitoring necessary for a legitimate objective?
• Function creep: Could data later be used for discipline, policing or marketing?
• Power imbalance: Can individuals realistically challenge the organisation?
• Accuracy: Can errors trigger harmful interventions?
• Group effects: Are vulnerable communities monitored more intensely?

Inference risk is particularly important in machine learning. A model may infer pregnancy, illness, credit distress or political preference from proxies. Even if the inferred attribute is not stored, the decision based on it may have equivalent consequences. Privacy reviews should therefore analyse features, labels, embeddings, model outputs and downstream actions.

7. Bias, Fairness and Ethical Responsibility

Bias can enter through sampling, measurement, labelling, historical decisions, missingness, proxy variables and deployment context. Privacy protections can also have unequal effects. For example, strict identity verification may exclude people lacking conventional documents; facial recognition may perform differently across demographic groups; and opt-out processes may be harder for people with limited digital access.

Fairness metrics must be interpreted in context. For a binary outcome, a group’s positive predictive value is:

where TP is true positives and FP is false positives. Equal predictive value across groups may conflict with equal false-positive rates when base rates differ. No single metric resolves the ethical question. Teams must define the harm, affected population, decision context and acceptable trade-offs with stakeholder participation.

Ethical review should consider:

• autonomy and informed participation;
• beneficence and reduction of foreseeable harm;
• non-maleficence and avoidance of preventable injury;
• justice and distribution of benefits and burdens;
• explainability and contestability;
• accountability for human and automated decisions; and
• respect for communities, including collective and cultural interests.

8. Responsible Data Governance for Analytics and AI

A governance operating model should assign responsibility at several levels:

For AI systems, governance should extend across the model lifecycle: problem formulation, data acquisition, training, validation, deployment, monitoring and retirement. Documentation should record intended use, prohibited use, data provenance, known limitations, performance by relevant groups, privacy risks and human oversight.

Technical controls include encryption, tokenisation, role-based or attribute-based access, query monitoring, secure enclaves, federated learning, aggregation, differential privacy and retention automation. None is universally sufficient. Federated learning, for example, keeps raw data distributed but can still leak information through model updates; secure aggregation and differential privacy may be needed.

9. Illustrative Case: Transit Analytics Platform

Background

MetroPulse, a fictional metropolitan transit authority, plans to analyse mobile-ticket and Wi-Fi connection data to reduce congestion and improve service frequency. The proposed system would record device identifiers, station entry and exit times, route history and approximate location. A vendor proposes retaining raw records indefinitely to support future “innovation.”

Problem

The authority faces four competing objectives: operational improvement, individual privacy, vendor accountability and public trust. Even if device identifiers are hashed, repeated travel patterns may reveal home and workplace locations. The data could also be attractive for law-enforcement requests, commercial advertising or employee monitoring.

Analysis

The initial purpose—aggregate transit planning—does not require indefinitely retaining person-level journeys. The authority should distinguish essential operational data from optional research data. It should also assess whether Wi-Fi data is necessary when ticket aggregates provide adequate information.

A DPIA identifies risks:

• re-identification from regular travel patterns;
• secondary use by the vendor;
• chilling effects if passengers believe movements are monitored;
• unequal impact on communities with fewer transportation alternatives; and
• breach consequences involving sensitive routines.

Approach and solution

The authority adopts the following controls:

1. Collect station-level counts by default rather than persistent device histories.
2. Use short-lived rotating tokens when journey linkage is genuinely necessary.
3. Aggregate outputs only when group sizes meet a documented threshold.
4. Apply a formal privacy budget to repeated public dashboards.
5. Prohibit advertising, individual profiling and law-enforcement access except under a defined legal process.
6. Retain detailed records for 30 days for service validation, then delete or transform them for approved aggregate analysis.
7. Publish a plain-language notice and conduct public consultation.
8. Contractually prohibit vendor reuse and require deletion certification.
9. Test re-identification using realistic auxiliary datasets.
10. Establish an access, correction and complaint channel where applicable.

Outcome

The authority obtains sufficient information for route planning while reducing persistent tracking. Some research questions become impossible at individual level, but the loss is accepted as a proportionality trade-off. The governance committee reviews the system quarterly and reassesses retention, accuracy and public expectations.

Learning points

The case demonstrates that privacy is not achieved by hashing alone. Purpose specification, minimisation, retention, contractual controls, transparency and technical safeguards must work together. It also shows that a privacy-preserving design may deliberately reduce analytical capability; this is often an ethical success rather than a technical failure.

10. Challenges and Limitations

10.1 Anonymity is difficult to prove

External datasets, repeated releases and machine-learning inference can undermine de-identification. Claims should be bounded by an explicit threat model, attacker capability and time horizon.

10.2 Consent fatigue

Long notices and repeated prompts can produce formal compliance without meaningful understanding. Layered notices, just-in-time explanations and usable controls are preferable.

10.3 Regulatory fragmentation

GDPR, CCPA/CPRA, sectoral laws, contractual duties and local rules may differ in definitions, thresholds, rights and deadlines. Organisations need a jurisdictional matrix and professional legal review; this chapter does not provide legal advice.

10.4 Utility and privacy trade-offs

Suppressing detail can reduce model accuracy or operational value. Decisions should compare residual privacy risk with measurable benefit and consider whether a less intrusive method can achieve the same objective.

10.5 Organisational incentives

Teams may be rewarded for growth, model performance or rapid deployment rather than restraint. Governance must therefore include escalation rights, independent challenge, executive accountability and consequences for misuse.

10.6 Privacy in foundation and generative models

Large models may memorise or reproduce training data, expose personal information through prompts, or infer sensitive attributes from seemingly benign inputs. Controls include data-provenance review, filtering, memorisation testing, access restrictions, retention limits for prompts and outputs, human review and incident response. Removing a record from a source database may not remove its influence from a trained model; deletion and machine-unlearning obligations require careful technical and legal analysis.

11. Best-Practice Decision Framework

Before approving a data-science project, ask:

1. What precise problem is being solved?
2. What is the minimum data needed?
3. What is the lawful and ethical basis?
4. What would a reasonable affected person expect?
5. Which people or communities bear the risks?
6. What inferences and downstream decisions are possible?
7. Can the data be aggregated, pseudonymised or protected with differential privacy?
8. How will access, retention, deletion and vendor use be controlled?
9. How can an individual challenge an error or harmful outcome?
10. What evidence will demonstrate accountability after deployment?

A project should be paused when the purpose is vague, consent is coerced, risks are disproportionate, affected groups have not been considered, or the organisation cannot explain who is accountable.

Chapter Summary

Responsible data use integrates privacy law, ethics, engineering and governance. Privacy-by-design places purpose definition, minimisation, transparency and safeguards at the beginning of the system lifecycle. Consent must be meaningful and is only one possible lawful basis; it does not legitimise unrelated secondary use. Purpose limitation requires clear original purposes and a disciplined compatibility assessment.

Data-subject rights require operational capabilities spanning inventories, lineage, identity resolution, processors, backups and model systems. De-identification reduces risk but does not automatically create anonymity. K-anonymity addresses certain linkage attacks but has important weaknesses, while differential privacy offers a formal participation guarantee under explicit assumptions and privacy-budget management.

Surveillance and inference risks extend beyond direct disclosure. Bias, unequal access, inaccurate predictions and automated decisions can create harms even when data is technically protected. Effective governance assigns responsibility, documents decisions, tests systems, provides remedy and continuously reassesses proportionality. The central professional principle is that analytical possibility is not sufficient justification: data use must be necessary, fair, understandable, secure and accountable.

Key Terms

Privacy-by-design: Integrating privacy protections throughout system design and operation.

Privacy-by-default: Configuring a system to collect, expose and retain only what is necessary unless a person actively chooses otherwise.

Personal data: Information relating to an identified or identifiable person.

Sensitive data: Personal information requiring heightened protection because of its nature or potential impact.

Consent: A voluntary, specific, informed and unambiguous indication of agreement to processing.

Purpose limitation: Restricting collection and use to specified, explicit and legitimate purposes.

Data minimisation: Limiting data to what is adequate, relevant and necessary.

Secondary use: Use of data for a purpose different from the original collection purpose.

Data subject: The person to whom personal data relates.

Controller: The organisation determining why and how personal data is processed.

Processor: An organisation processing data on behalf of a controller.

Pseudonymisation: Replacing identifiers with codes while retaining possible re-linkage.

Anonymisation: Processing intended to prevent identification using reasonably available means.

Re-identification: Linking de-identified information to a person.

K-anonymity: A property in which each quasi-identifier combination appears in at least k records.

Differential privacy: A formal privacy guarantee limiting the effect of one person’s participation on an output.

Privacy impact assessment: A structured evaluation of processing necessity, risks and safeguards.

Data minimisation: Reduction of collected or retained information to the amount necessary for the purpose.

Function creep: Expansion of data use beyond the original or reasonably expected purpose.

Accountability: The duty to comply with privacy responsibilities and demonstrate that compliance.

Review Questions

Short Answer Questions

1. What is the difference between privacy-by-design and privacy-by-default?
2. Why is a hashed identifier not necessarily anonymous?
3. Name four common data-subject rights.
4. What is purpose limitation?
5. What does epsilon represent in differential privacy?

Long Answer Questions

1. Explain how consent, lawful basis and purpose limitation interact under a privacy-governance program.
2. Compare pseudonymisation, k-anonymity and differential privacy, including their principal limitations.
3. Describe how a data protection impact assessment should be conducted for a high-risk analytics system.
4. Discuss why privacy, fairness and security should be governed together but not treated as identical concepts.
5. Explain the organisational and technical controls required to support deletion and access requests in modern data platforms.

Analytical / Application Questions

1. A retailer collects precise location data for store-navigation functionality and later proposes using it to infer health conditions for targeted advertising. Evaluate the proposal using purpose limitation, consent, proportionality and inference-risk principles.
2. Design a privacy-preserving architecture for a public dashboard reporting neighbourhood crime trends. Specify aggregation, retention, access, re-identification testing and transparency controls.
3. A hiring model produces similar overall accuracy for two demographic groups but substantially different false-positive rates. Analyse the possible causes, ethical implications and governance steps before deployment.

Multiple Choice Questions with Answers

Question 1

Which practice best represents privacy-by-default?

A. Collecting all available fields and deleting them if users complain

B. Enabling precise location tracking for every user automatically

C. Starting with the least data collection and exposure necessary for the service

D. Allowing vendors to retain raw data indefinitely

Correct Answer: C. Starting with the least data collection and exposure necessary for the service

Question 2

Which statement about consent is most accurate?

A. Consent is valid even when refusal prevents access to an unrelated essential service

B. Silence and inactivity are always sufficient affirmative consent

C. Consent should be specific, informed, voluntary and reasonably easy to withdraw

D. Consent automatically authorises all future purposes

Correct Answer: C. Consent should be specific, informed, voluntary and reasonably easy to withdraw

Question 3

Why may pseudonymised data remain personal data?

A. Pseudonymisation always increases identifiability

B. A retained method or available information may permit re-linkage to an individual

C. Pseudonymisation removes all security controls

D. Pseudonymised data cannot be processed analytically

Correct Answer: B. A retained method or available information may permit re-linkage to an individual

Question 4

What is the principal weakness illustrated by a homogeneous k-anonymous group?

A. Every record has a different quasi-identifier

B. The group contains too many direct identifiers

C. An attacker may learn a sensitive attribute even without identifying one exact record

D. The data cannot be stored in a database

Correct Answer: C. An attacker may learn a sensitive attribute even without identifying one exact record

Question 5

In the differential-privacy expression Pr[M(D) ∈ S] ≤ e^ε × Pr[M(D′) ∈ S], what does ε primarily control?

A. The number of database columns

B. The privacy-loss parameter

C. The number of authorised users

D. The encryption key length

Correct Answer: B. The privacy-loss parameter

Question 6

Which is an example of secondary use?

A. Using collected payment data to authorise the same payment

B. Deleting data at the end of its retention period

C. Using transit journey records collected for planning to target passengers with advertising

D. Restricting database access to authorised staff

Correct Answer: C. Using transit journey records collected for planning to target passengers with advertising

Question 7

Which control most directly addresses vendor reuse risk?

A. A contract prohibiting unrelated use and requiring deletion certification

B. Publishing the vendor’s logo

C. Increasing model complexity

D. Removing the organisation’s privacy notice

Correct Answer: A. A contract prohibiting unrelated use and requiring deletion certification

Question 8

Why should privacy reviews examine model inferences?

A. Inferences can never affect individuals

B. An inferred sensitive attribute can produce harmful consequences even if it was not directly collected

C. Inference is unrelated to data processing

D. Models cannot use proxy variables

Correct Answer: B. An inferred sensitive attribute can produce harmful consequences even if it was not directly collected

Question 9

Which item is most characteristic of accountability?

A. Assuming compliance without records

B. Demonstrating compliance through documentation, assessments and monitoring

C. Collecting data before defining a purpose

D. Delegating all responsibility to an algorithm

Correct Answer: B. Demonstrating compliance through documentation, assessments and monitoring

Question 10

What is the main purpose of a privacy impact assessment?

A. To guarantee that a project has zero privacy risk

B. To replace all security testing

C. To evaluate necessity, risks, safeguards and residual privacy impacts

D. To maximise the amount of data collected

Correct Answer: C. To evaluate necessity, risks, safeguards and residual privacy impacts