Ethics, Privacy, Security and Data Governance

Introduction and Context

Data science creates value by converting data into descriptions, predictions, recommendations and automated decisions. The same processes can also cause harm. A dataset may reveal health conditions, financial difficulty, political preferences or a person’s location. A predictive model may systematically disadvantage a group, while a poorly secured database may expose millions of records. Ethical and professional data science therefore requires more than technical accuracy: it requires justified purposes, lawful processing, respect for people, protection against misuse, explainable decisions and accountable institutions.

Ethics concerns what ought to be done. Privacy concerns people’s control over information about themselves and protection from unjustified observation or intrusion. Security concerns protection against unauthorised access, alteration, loss and disruption. Data governance is the organisational system that assigns authority and responsibility for managing data throughout its life cycle. These areas overlap, but they are not identical. Encryption can improve security without making an unfair model ethical; legal compliance can be achieved while an organisation still violates reasonable expectations of privacy.

This chapter uses a practical perspective suitable for data-science work in organisations. Laws differ across jurisdictions, so legal specialists must be consulted for specific projects. Nevertheless, the principles discussed here are broadly applicable.

Learning Objectives

By the end of this chapter, a learner should be able to:

1. Explain consent, purpose limitation, privacy, confidentiality, security and fairness.
2. Identify risks involving personal data, sensitive data, re-identification and intellectual property.
3. Apply principles of transparency, accountability and responsible AI to a data-science project.
4. Describe governance roles, data stewardship, access controls and data-life-cycle management.
5. Distinguish major regulatory ideas, including lawful processing, data-subject rights and breach obligations.
6. Evaluate a data project and recommend proportionate safeguards.

1. Ethical Foundations of Data Science

1.1 Values and professional responsibility

A data scientist has responsibilities to several stakeholders: individuals represented in the data, customers, employees, communities, the commissioning organisation and society. A narrow objective such as maximising predictive accuracy may conflict with other values. For example, a fraud-detection system that flags many legitimate customers may reduce financial loss but impose unfair burdens on innocent people.

Important ethical principles include:

• Respect for persons: individuals should not be treated merely as sources of exploitable data.
• Beneficence: a project should provide a genuine benefit or legitimate organisational purpose.
• Non-maleficence: foreseeable harms should be identified and reduced.
• Justice and fairness: benefits and burdens should not be distributed discriminatorily.
• Autonomy: people should have meaningful information and, where appropriate, choices about data use.
• Accountability: identifiable people and institutions must be answerable for decisions and consequences.

Professional responsibility continues after deployment. Models drift, data sources change and social effects may differ from predictions made during development. Monitoring and review are therefore ethical duties, not optional administrative tasks.

1.2 Consent and lawful purpose

Consent is a communication through which a person agrees to a specified use of personal data. Meaningful consent is normally informed, specific, freely given and capable of being withdrawn. A pre-ticked box, confusing notice or forced agreement to unnecessary processing is weak evidence of meaningful consent.

Consent is not always the only lawful basis for processing. Depending on the jurisdiction, alternatives may include performance of a contract, compliance with a legal obligation, protection of vital interests, a task in the public interest or a carefully balanced legitimate interest. The correct basis must be established before collection, documented and communicated. An organisation must not collect data first and search for a justification later.

Consent also has limits. It does not automatically legitimise harmful processing, unlimited secondary uses or inadequate security. Where children or people with limited decision-making capacity are involved, stronger safeguards may be required.

1.3 Purpose limitation and data minimisation

Purpose limitation means collecting and using data for specified, explicit and legitimate purposes. Data minimisation means using only data that is adequate, relevant and necessary for those purposes. These principles challenge the assumption that more data always produces better analytics.

A practical test is: if a variable cannot be linked to a documented decision, service or research purpose, its collection should be questioned. Minimisation reduces privacy exposure, storage costs, attack impact and opportunities for discriminatory use. It does not mean that every dataset must contain very few variables; it means that each variable requires a defensible rationale.

2. Privacy, Confidentiality and Re-identification

2.1 Personal and sensitive data

Personal data is information relating to an identified or identifiable person. Direct identifiers include names, identity numbers, email addresses and telephone numbers. Indirect identifiers, such as date of birth, postcode, occupation and rare events, may identify a person when combined.

Sensitive or special-category data commonly includes health information, biometric identifiers, genetic information, racial or ethnic origin, religious beliefs, political opinions and sexual orientation. Laws define these categories differently, but such data generally requires a higher level of justification and protection.

Privacy includes more than secrecy. It can involve control over collection, appropriate use, freedom from unjustified surveillance, contextual integrity and the ability to challenge decisions. Information supplied to a hospital for treatment, for example, may not reasonably be expected to support targeted advertising.

2.2 Anonymisation, pseudonymisation and aggregation

Pseudonymisation replaces direct identifiers with codes or tokens, while a separate key allows re-linking. It reduces risk but remains personal data where re-identification is reasonably possible. Anonymisation aims to make identification irreversible or sufficiently impracticable, considering available technology and other information.

Removing names is not necessarily anonymisation. A famous demonstration showed that supposedly anonymous records could be linked with public information using combinations of quasi-identifiers. Small groups, unusual occupations, exact timestamps and geographic precision increase this risk.

Common safeguards include aggregation, generalisation, suppression of rare records, perturbation, access restrictions and differential privacy. Differential privacy provides a formal way to limit how much the output of an analysis changes when one person’s record is added or removed. A common mechanism adds calibrated random noise; its quality depends on the privacy budget and the query sensitivity. It is not a guarantee that every disclosure risk disappears, and it may reduce analytical accuracy.

2.3 Privacy by design

Privacy by design integrates privacy into architecture rather than adding it after implementation. Typical practices include:

1. define the purpose and lawful basis;
2. map data flows and recipients;
3. minimise collection;
4. separate identifiers from analytical attributes;
5. apply retention and deletion rules;
6. provide access, correction and objection mechanisms;
7. test re-identification and misuse risks;
8. document decisions and review changes.

A privacy impact assessment is useful when processing is large-scale, systematic, innovative or likely to create significant risk. It should record the processing purpose, data categories, affected people, threats, controls, residual risks and approval decision.

3. Fairness, Bias and Transparency

3.1 Sources of bias

Bias can enter through:

• Historical bias: past decisions reflect unequal treatment.
• Sampling bias: some groups are missing or under-represented.
• Measurement bias: a recorded variable is an inaccurate proxy for the concept of interest.
• Label bias: outcomes used as labels reflect previous human judgments.
• Aggregation bias: one model is imposed on groups with different relationships.
• Deployment bias: a technically valid model is used in an unsuitable context.

Removing a protected attribute does not remove unfairness. Other variables may act as proxies. Conversely, protected attributes may be needed during audit to measure disparities, subject to lawful and secure handling.

3.2 Fairness is contextual

There is no single fairness metric that is correct for every application. A classification system may be assessed using a confusion matrix:

Here TP is true positive, TN true negative, FP false positive and FN false negative. For group g, common measures include:

where TPR is the true-positive rate or recall, and

where FPR is the false-positive rate. Equal opportunity seeks similar TPR values across relevant groups. Equalised odds seeks similar TPR and FPR values. Demographic parity compares the proportion receiving a positive outcome, while predictive parity compares precision across groups.

These criteria can conflict, particularly when groups have different base rates or score distributions. Fairness assessment therefore requires a clearly defined harm, affected population, decision context and justification for the chosen metric. Statistical parity alone may conceal poor service quality; equal error rates may be inappropriate when the costs of false positives and false negatives differ.

3.3 Transparency and explainability

Transparency means making relevant information understandable and available. It can include the purpose of a system, data categories, performance measures, known limitations, responsible organisation, retention period and routes for appeal. Explainability concerns why a particular model or decision produced an output.

A simple model may be intrinsically interpretable, whereas a complex model may require post-hoc explanations such as feature importance or local counterfactuals. Explanations should not be treated as proof that a model is fair or causally correct. A feature can be influential without being a legitimate basis for decision-making.

A useful individual explanation answers questions such as: What information was used? Which factors materially affected the result? What could change the result? How reliable is the prediction? Who can review it? High-impact decisions should generally include human oversight and a meaningful appeal process.

4. Security of Data and Analytical Systems

Security protects the confidentiality, integrity and availability of information—the CIA triad.

• Confidentiality: only authorised parties can view data.
• Integrity: data and models are accurate and protected from unauthorised alteration.
• Availability: authorised users can access systems when required.

Threats include phishing, stolen credentials, malware, ransomware, insecure application programming interfaces, insider misuse, cloud misconfiguration, supply-chain compromise and accidental publication. Machine-learning systems also face data poisoning, model extraction, prompt injection and adversarial manipulation.

4.1 Access control

Access should follow least privilege: a user receives only the permissions necessary for an approved task. Role-based access control assigns permissions to roles rather than individuals; attribute-based access control evaluates properties such as user role, location, device security and data classification.

Strong controls include unique accounts, multi-factor authentication, periodic access reviews, separation of duties, privileged-access management, encryption in transit and at rest, network segmentation, secure backups and immutable audit logs. Encryption protects confidentiality when keys are managed properly; it does not prevent an authorised but malicious user from misusing decrypted data.

4.2 Secure development and incident response

Security should be integrated into the data life cycle. Code and infrastructure require review, dependency management, vulnerability scanning and controlled deployment. Production and development data should be separated, and real personal data should not be copied into test environments without a defensible reason and safeguards.

An incident-response plan should identify detection channels, escalation contacts, containment actions, evidence preservation, recovery procedures, communication duties and post-incident learning. Breach notification timelines and thresholds depend on applicable law. Concealing a breach can increase harm and regulatory consequences.

5. Data Governance and Stewardship

Data governance is the framework of decision rights, policies, standards, processes and controls used to manage data. It answers: who may decide, who is responsible, what rules apply and how compliance is demonstrated?

5.1 Common governance roles

Titles differ between organisations. What matters is that responsibilities are explicit and not left to an undefined “data team.”

5.2 Data life-cycle governance

A governed life cycle normally includes collection, ingestion, storage, preparation, analysis, sharing, publication, archival and deletion. Each stage should have controls for quality, access, lineage, retention and review.

A data catalogue records datasets, owners, descriptions, sensitivity and usage conditions. A business glossary defines terms such as “active customer” or “successful completion.” Data lineage shows where data originated, how it was transformed and which reports or models depend on it. These artefacts support reproducibility, impact analysis and auditability.

Data quality should be assessed against the intended use. Common dimensions are accuracy, completeness, consistency, timeliness, validity and uniqueness. A dataset may be accurate but incomplete, or complete but systematically biased. Quality thresholds should therefore be documented rather than assumed.

5.3 Classification and retention

A practical classification scheme may distinguish public, internal, confidential and restricted data. Classification should consider sensitivity, legal requirements, business impact and likely harm if disclosed or altered. Retention schedules specify how long records are kept, why they are retained and how secure deletion occurs. Indefinite retention increases risk and should not be the default.

6. Regulatory and Intellectual-Property Considerations

6.1 Regulatory principles

Modern data-protection regimes commonly require a lawful basis, purpose limitation, minimisation, accuracy, storage limitation, security and accountability. They may give individuals rights to access, correct, delete, restrict or object to processing, and sometimes rights concerning automated decisions and data portability.

Organisations may need to document processing activities, appoint responsible officers, conduct impact assessments, control international transfers and notify certain breaches. Sector-specific rules may impose additional duties for health, education, finance, employment or children’s data. Compliance is jurisdiction-specific: a project involving several countries must map applicable obligations rather than assume one national rule applies everywhere.

6.2 Intellectual property and licensing

Data can be subject to copyright, database rights, trade-secret protection, contractual restrictions or licences. Public availability does not automatically mean unrestricted reuse. A dataset may contain copyrighted text, licensed images or personal data even when downloadable without payment.

Before using external data, check provenance, licence terms, permitted purposes, attribution requirements, commercial-use restrictions and redistribution conditions. Training a model on protected material can create uncertain legal and ethical issues, especially where outputs reproduce expressive content. Organisations should maintain source records, licence evidence and procedures for responding to takedown or rights-holder claims.

7. Responsible AI in Practice

Responsible AI extends governance to models and automated decisions. A proportionate model-governance process includes:

1. Problem definition: establish a legitimate objective, affected people, decision authority and unacceptable uses.
2. Data assessment: document provenance, consent or lawful basis, representativeness, labels, missingness and sensitive attributes.
3. Risk assessment: identify physical, financial, social, privacy, security and environmental harms.
4. Validation: test performance overall and across relevant groups; assess robustness, calibration and error costs.
5. Human oversight: define when a person must review, override or halt a system.
6. Deployment controls: restrict access, record versions, monitor inputs and outputs, and communicate limitations.
7. Monitoring: detect drift, disparate impact, unexpected feedback loops and misuse.
8. Retirement: remove or replace systems that no longer meet purpose, performance or safety requirements.

A model card or system factsheet can record intended use, excluded use, training data, evaluation results, limitations and contact details. A decision log should record material choices and approvals. These practices convert abstract accountability into evidence.

8. Applied Case: Illustrative University Scholarship Screening System

Background

A university proposes a model to prioritise scholarship applications. It plans to use academic results, household income, attendance, postcode and historical scholarship decisions. The objective is to allocate limited funds to students with high financial need and strong educational potential.

Problem

Historical decisions were made when some groups had less access to advanced courses and support. Postcode may act as a proxy for ethnicity or socioeconomic disadvantage. Attendance may reflect disability, caring responsibilities or unreliable transport. Historical award decisions may reproduce earlier human bias. The university also wishes to obtain information from external data brokers without informing applicants.

Analysis

The proposed purpose is legitimate, but the design creates fairness, privacy and accountability risks. Using historical decisions as labels risks learning past preferences rather than educational potential. External data collection may violate reasonable expectations and data-minimisation principles. A high score could be mistaken for an objective entitlement, even though the model only estimates a prioritisation rule.

The university should define the decision as assistance to, not replacement of, a trained review panel. It should consult students and relevant support groups, conduct a privacy and equality impact assessment, and identify necessary variables. Protected attributes may be securely collected for fairness evaluation even if not used for prediction.

Approach and solution

The university should:

• publish the purpose, data categories, retention period and review process;
• avoid broker data unless a clear lawful basis and necessity are established;
• compare alternative labels, including independently assessed financial need and educational barriers;
• test selection rates, false exclusions and error patterns across relevant groups;
• provide applicants with an understandable explanation and an appeal route;
• restrict access using role-based permissions and retain an audit log;
• require human review for borderline or adverse outcomes;
• monitor outcomes each funding cycle and suspend the model if harms emerge.

Outcome

In this illustrative case, the university rejects fully automated ranking. It adopts a decision-support tool using a smaller set of justified variables, conducts annual fairness audits and gives applicants a manual review route. The system is retained only while it demonstrates improved consistency without unacceptable disparities.

Learning points

Accuracy does not establish legitimacy. Historical data may encode structural inequality; proxy variables can create discrimination; transparency must include contestability; and governance must assign named responsibility. The most ethical solution may be to redesign or limit a data-science system rather than deploy it at maximum scale.

9. Implementation Checklist for Data-Science Projects

Before collecting or using data, ask:

• What specific problem and public or organisational benefit justify the project?
• Who is affected, including people not directly represented as customers?
• What lawful basis, consent process and notice are required?
• Are all variables necessary, proportionate and appropriately sourced?
• Could records be re-identified or linked with external information?
• Who owns the data and model, and who can approve access?
• What are the plausible security threats and incident procedures?
• Which fairness risks and group-level outcomes must be tested?
• How will explanations, corrections, appeals and human intervention work?
• How long will data and model outputs be retained?
• What licences and intellectual-property restrictions apply?
• What evidence will demonstrate compliance and accountability?

Challenges and Limitations

Ethical governance cannot eliminate all risk. Fairness metrics may conflict, privacy safeguards may reduce analytical utility, and transparency may be difficult for complex models. Consent can be formal rather than genuinely informed. Organisations may lack representative data for auditing, while collecting protected attributes for evaluation introduces its own risks. Small organisations may struggle with specialist resources, and global projects face inconsistent laws and norms.

Governance can also become bureaucratic if it focuses on forms rather than decisions and outcomes. Conversely, rapid innovation can bypass review. Effective practice is proportionate, risk-based and iterative. It combines legal advice, technical testing, domain expertise and participation by affected communities.

Contemporary Developments and Best Practices

Current practice increasingly emphasises algorithmic impact assessments, model inventories, privacy-enhancing technologies, synthetic or de-identified test data, confidential computing, secure multiparty analysis and continuous monitoring. Generative AI introduces additional risks involving memorisation, prompt data leakage, fabricated outputs, copyright and unclear accountability. Organisations should prohibit sensitive data in unapproved tools, verify outputs, record model versions and define human responsibility for generated content.

Best practice is to treat ethics, privacy and security as design requirements; use the least data and least privilege compatible with the purpose; test systems across relevant populations; involve affected stakeholders; maintain clear documentation; and provide a credible route for correction and remedy. Responsible data science is not a single approval event. It is an ongoing professional discipline.

Chapter Summary

Ethical data science requires justified purposes, respect for persons, minimisation, fairness, transparency and accountability. Consent must be meaningful, and other lawful bases must be documented where applicable. Privacy is broader than secrecy; pseudonymisation reduces risk but does not necessarily create anonymous data. Security protects confidentiality, integrity and availability through controls such as least privilege, multi-factor authentication, encryption, secure development and incident response.

Data governance assigns decision rights and responsibilities to owners, stewards, custodians, privacy and security officers, model owners and oversight bodies. Catalogues, glossaries, lineage, classification, quality standards and retention schedules support trustworthy life-cycle management. Fairness is contextual, and metrics such as true-positive and false-positive rates must be interpreted alongside social consequences. Responsible AI requires problem definition, data assessment, validation, human oversight, monitoring and retirement. Intellectual-property licences, regulatory duties and data-subject rights must be considered before deployment. The central professional lesson is that a technically effective system is not necessarily lawful, secure, fair or socially beneficial.

Key Terms

Consent: A person’s informed, specific and freely given agreement to a defined use of personal data.

Data governance: Policies, roles, decision rights and controls for managing data.

Data minimisation: Using only data that is adequate, relevant and necessary for a stated purpose.

Data owner: Person accountable for decisions concerning a data domain.

Data steward: Person responsible for operational quality, metadata and appropriate use.

Data lineage: A record of data origins, transformations and downstream uses.

Differential privacy: A formal privacy approach that limits the effect of one record on an analytical output.

Fairness: The context-dependent requirement that people and groups are not subjected to unjustified disadvantage.

Integrity: Protection of data and systems against unauthorised or improper alteration.

Least privilege: Granting only the access necessary for an approved task.

Personal data: Information relating to an identified or identifiable person.

Pseudonymisation: Replacing identifiers with codes while retaining a possible re-linking method.

Purpose limitation: Using data only for specified, explicit and legitimate purposes.

Re-identification: Linking records thought to be anonymous to an identifiable person.

Security: Protection against unauthorised access, alteration, loss or disruption.

Transparency: Providing understandable information about data processing and decisions.

True-positive rate: The proportion of actual positive cases correctly classified as positive.

Model drift: Change over time in data patterns, relationships or model performance.

Review Questions

5 Short Answer Questions

1. Distinguish privacy from security.
2. What makes consent meaningful?
3. Explain why pseudonymisation is not necessarily anonymisation.
4. State the three elements of the CIA security triad.
5. What is the main purpose of data lineage?

5 Long Answer Questions

1. Explain how purpose limitation and data minimisation reduce ethical and operational risk.
2. Discuss the sources of bias in a predictive model and describe suitable mitigation measures.
3. Compare the responsibilities of a data owner, data steward and data custodian.
4. Explain why fairness metrics can conflict and why context matters in selecting one.
5. Describe a responsible-AI life cycle from problem definition to system retirement.

3 Analytical / Application Questions

1. A retailer wants to use loyalty-card data collected for discounts to assess customers’ creditworthiness. Analyse the consent, purpose-limitation, fairness and transparency issues, and recommend safeguards.
2. A hospital model has equal overall accuracy but a much higher false-positive rate for one demographic group. Explain why this matters and design an investigation plan.
3. A small organisation discovers that an employee uploaded customer records to an external generative-AI tool. Propose an immediate incident-response plan and longer-term governance controls.

Multiple Choice Questions with Answers

Question 1

Which principle requires an organisation to collect only data necessary for a stated purpose?

A. Data minimisation

B. Model drift

C. Availability

D. Predictive parity

Correct Answer: A. Data minimisation

Question 2

Why does pseudonymisation not always make data anonymous?

A. It prevents all statistical analysis

B. A separate key or other information may permit re-identification

C. It always publishes names publicly

D. It removes the need for access controls

Correct Answer: B. A separate key or other information may permit re-identification

Question 3

Which element of the CIA triad concerns protection against unauthorised alteration?

A. Confidentiality

B. Availability

C. Integrity

D. Consent

Correct Answer: C. Integrity

Question 4

What does least privilege mean?

A. Every employee receives administrator access

B. Users receive only the permissions necessary for an approved task

C. Access is granted permanently after authentication

D. Data is stored without encryption

Correct Answer: B. Users receive only the permissions necessary for an approved task

Question 5

A true-positive rate is calculated as which expression?

A. TP ÷ (TP + FN)

B. FP ÷ (FP + TN)

C. TP ÷ (TP + FP)

D. TN ÷ (TN + FN)

Correct Answer: A. TP ÷ (TP + FN)

Question 6

Which role is primarily accountable for a data domain and access decisions?

A. Data owner

B. Data custodian

C. Data subject

D. External auditor

Correct Answer: A. Data owner

Question 7

Why can removing a protected attribute fail to eliminate unfairness?

A. Models cannot use any variables without it

B. Other variables may act as proxies

C. It guarantees equal outcomes

D. It prevents data security testing

Correct Answer: B. Other variables may act as proxies

Question 8

Which practice best supports accountability for an automated decision?

A. Deleting all model documentation after deployment

B. Allowing unrestricted access to training data

C. Maintaining decision records, named responsibility and an appeal route

D. Using the most complex model available

Correct Answer: C. Maintaining decision records, named responsibility and an appeal route

Question 9

What should be checked before using an external dataset?

A. Only its file size

B. Its colour scheme and visual format

C. Its provenance, licence, permitted purposes and restrictions

D. Whether it contains the maximum possible number of variables

Correct Answer: C. Its provenance, licence, permitted purposes and restrictions

Question 10

In responsible AI, why is monitoring required after deployment?

A. Data and model performance can change over time

B. Validation is impossible before deployment

C. Monitoring removes the need for human oversight

D. It guarantees that no breach can occur

Correct Answer: A. Data and model performance can change over time