14. Ethics, Security and Data/AI Governance

Introduction and Context

Programming for data science is often presented as a technical activity involving data structures, statistical models, visualisation and software libraries. Yet every analytical program also makes choices about people: whose data is collected, what is inferred, who can access it, how decisions are automated and who bears the consequences of error. A technically accurate model can still be unacceptable if it violates privacy, reproduces discrimination, exposes confidential information or cannot be explained to affected individuals.

Ethics, security and governance therefore extend beyond compliance departments. They are part of the professional responsibilities of data scientists, engineers, analysts, managers and institutional leaders. Governance provides the rules, roles and controls through which data and artificial intelligence are managed. Ethics supplies principles for judging whether an action is legitimate, fair and socially responsible. Security protects data, systems and models from unauthorised access, alteration, loss and misuse.

These concerns apply throughout the data and model lifecycle: planning, collection, storage, preparation, analysis, development, deployment, monitoring, reuse and retirement. A Python or R script may implement only one stage, but its assumptions can affect the entire lifecycle. This chapter develops a strategic and practical framework for responsible data science, including privacy, consent, intellectual property, cybersecurity, bias, fairness, transparency, accountability, quality, lineage, access governance and model documentation.

Learning Objectives

By the end of this chapter, students should be able to:

1. Distinguish ethics, privacy, security, governance and compliance in data science.

2. Evaluate consent, lawful use, intellectual property and data minimisation decisions.

3. Explain common cybersecurity threats affecting data pipelines, notebooks, APIs and models.

4. Identify sources of algorithmic bias and select appropriate fairness and quality controls.

5. Apply data lineage, access governance, model documentation and accountability mechanisms.

6. Use responsible AI and risk-management frameworks to structure organisational decisions.

7. Analyse a data or AI project and recommend proportionate safeguards across its lifecycle.

The Governance Problem: Data as an Asset and a Responsibility

Data governance is the coordinated exercise of authority and control over data assets. It defines ownership, permitted uses, quality expectations, retention periods, access conditions and accountability. AI governance extends these concerns to models and automated decisions, including their objectives, training data, performance, explainability, human oversight and social impact.

Governance is not the same as data management. Data management concerns the operational handling of data, such as storage, integration and backup. Governance establishes why these activities are undertaken, who may authorise them and what constraints apply. Nor is governance identical to legal compliance. Law establishes minimum obligations in a jurisdiction; governance translates those obligations, organisational values and risk tolerances into operational controls. An organisation may comply formally with a rule while still making an ethically poor decision.

A useful distinction is between three questions. The first is “May we?”—a legal and policy question about authority and permission. The second is “Can we?”—a technical question about feasibility. The third is “Should we?”—an ethical and social question about legitimacy, proportionality and consequences. Responsible practice requires all three questions to be considered.

Important Concepts and Terminology

Privacy concerns appropriate control over information about identifiable or identifiable individuals. Confidentiality means preventing unauthorised disclosure. Integrity means protecting data and systems from unauthorised or inappropriate alteration. Availability means ensuring that authorised users can access systems and information when needed. These three properties form the familiar confidentiality–integrity–availability model of information security.

Personal data is information relating to an identified or identifiable person. Sensitive or special-category data may include health, biometric, financial, racial, political or other information subject to stronger safeguards under applicable law. Pseudonymisation replaces direct identifiers with codes but does not necessarily prevent re-identification. Anonymisation aims to make identification sufficiently unlikely, although it is difficult to guarantee when datasets can be linked.

Consent is a voluntary, informed and specific agreement to a defined processing activity. It is not always the only lawful basis for processing, and it is not valid when participation is coerced or the purpose is unclear. Purpose limitation requires data to be used for specified and legitimate purposes rather than unlimited future exploitation. Data minimisation means collecting and retaining only what is necessary. Retention defines how long data is kept; deletion and disposal must address copies, backups and derived artefacts where feasible.

Data lineage records where data originated, how it was transformed, which datasets were combined and where outputs were used. Provenance is closely related and emphasises the history and evidence supporting a data item or result. A data steward manages data quality and use within a domain, while a data owner is accountable for authorising access and defining business responsibility.

Algorithmic bias is a systematic pattern of disadvantage associated with data, design, implementation or use. Fairness is not one universal statistical property; it is a family of normative and measurement choices concerning comparable treatment, opportunity and outcomes. Transparency means making relevant information about data, methods, limitations and decisions available. Explainability concerns the ability to provide meaningful reasons for a model output. Accountability means that identifiable people or bodies can answer for decisions, correct errors and provide remedy.

Ethical Reasoning Across the Data Lifecycle

Ethical assessment should begin before data collection. Teams should identify affected groups, expected benefits, foreseeable harms, alternatives and power imbalances. A data protection or ethical impact assessment can make these questions explicit. High-risk uses deserve greater scrutiny, particularly where decisions affect employment, education, housing, credit, healthcare, public benefits or criminal justice.

During collection, individuals should receive understandable information about purpose, categories of data, retention, sharing, automated decision-making and their available rights. Consent interfaces should not hide important conditions in dense legal language or use manipulative defaults. Where consent is inappropriate or impossible, the organisation must document another legitimate basis and ensure that the use remains proportionate.

During preparation and analysis, analysts should question whether variables are necessary, whether proxies encode protected characteristics and whether missingness is socially patterned. Removing a protected attribute does not eliminate bias: postcode, language, name, purchasing behaviour or education may act as proxies. Analysts should inspect distributions across relevant groups, document exclusions and avoid presenting correlations as causal conclusions.

At deployment, users should understand the system’s purpose, limitations and escalation route. Human review must be meaningful rather than a ceremonial approval step. A reviewer needs adequate time, information, authority and training to challenge an automated recommendation. After deployment, monitoring should examine performance drift, data drift, subgroup outcomes, security incidents, complaints and unexpected uses. A system should be modified, suspended or retired when its risks are no longer acceptable.

Privacy, Consent and Intellectual Property

Privacy protection is a design requirement, not merely a process of removing names. Re-identification can arise from combinations of quasi-identifiers, linkage with public datasets, rare records or model outputs. Techniques such as aggregation, suppression, generalisation, pseudonymisation, differential privacy and secure multiparty computation may reduce exposure, but each involves trade-offs in utility, complexity and residual risk.

Differential privacy, for example, adds calibrated randomness so that the inclusion or exclusion of one person has limited influence on a published result. Its protection depends on parameters, repeated queries and implementation quality. It does not make a fundamentally inappropriate data collection purpose acceptable. Privacy-enhancing technology should therefore support, rather than replace, purpose limitation, access control and organisational accountability.

Intellectual property issues arise in datasets, source code, documentation, images, text, pretrained models and generated outputs. A dataset being accessible online does not automatically make it free to copy, combine or use commercially. Licences may restrict redistribution, attribution, modification or machine-learning use. Open-source software also carries licence obligations. Teams should maintain an inventory of data and software licences, preserve attribution and obtain legal review when rights are uncertain. Copyright, database rights, trade secrets and contractual confidentiality may apply differently across jurisdictions.

Security as an Engineering and Governance Discipline

Security threats may enter through credentials, dependencies, notebooks, cloud storage, APIs, data exports or model interfaces. Common risks include phishing, ransomware, insecure secrets, excessive privileges, injection attacks, malicious packages, data poisoning, model extraction and prompt or input manipulation. A data scientist who stores a database password in a notebook or commits an access token to a public repository creates a governance failure as well as a technical vulnerability.

A proportionate security programme uses defence in depth. Identity should be verified through strong authentication, preferably with multifactor authentication, and authorisation should follow least privilege. Role-based or attribute-based access can limit use according to job function, data sensitivity, project and context. Encryption should protect data in transit and at rest, while keys and secrets must be managed separately from application code. Logging, monitoring, vulnerability management, secure backups and incident-response procedures are essential.

Secure development includes code review, dependency scanning, input validation, environment separation, reproducible builds and testing with synthetic or suitably masked data. Production data should not be copied casually into personal laptops or development notebooks. Access should be time-limited where possible, reviewed periodically and revoked when roles change. Security controls must preserve availability and usability; a control that prevents legitimate emergency access without an alternative may create a different risk.

Incident response normally involves preparation, detection, containment, eradication, recovery and lessons learned. Organisations must preserve evidence, assess affected data and notify relevant parties where required by applicable regulation. Concealing an incident often increases harm and undermines trust.

Fairness, Bias and Responsible Model Development

Bias can enter at every stage. Historical data may reflect unequal opportunity; sampling may exclude minority groups; labels may encode subjective or institutional judgments; measurement may be less accurate for some populations; and deployment may change behaviour. A model can have strong aggregate accuracy while producing unacceptable error rates for a smaller group.

Fairness analysis begins with defining the decision, affected groups and relevant harms. Common measures include demographic parity, equal opportunity, equalised odds, subgroup calibration and error-rate comparisons. These measures are not generally all satisfiable simultaneously when base rates differ, so selecting one requires a normative and organisational justification. Metrics should be accompanied by qualitative investigation, stakeholder participation and examination of actual consequences.

Possible interventions include improving representation, changing labels, reweighting observations, adjusting thresholds, constraining optimisation, providing human review and redesigning the decision itself. Removing protected variables is rarely sufficient. Nor should fairness be reduced to a single numerical audit performed after deployment. Teams should ask whether the outcome is necessary, whether the system should be used at all and whether affected people have a meaningful appeal mechanism.

Transparency is audience-specific. Technical documentation may describe features, training data, validation design and uncertainty. A public notice may explain purpose, data categories and rights. An affected person may need an understandable reason for an individual decision and a route to correction. Full disclosure of every implementation detail can create security or intellectual-property problems, so transparency should be meaningful, relevant and proportionate rather than merely maximal.

Frameworks for Responsible AI and Governance

A practical governance framework assigns responsibility at several levels. The board or executive committee establishes risk appetite. A governance or ethics committee reviews high-impact uses. Product owners define purpose and success criteria. Data owners authorise use. Data stewards manage quality and definitions. Security and privacy specialists assess controls. Model developers build and test systems. Independent reviewers challenge assumptions. Operational users monitor results and escalate concerns. The allocation should be recorded in a responsibility matrix rather than assumed.

The widely used NIST AI Risk Management Framework organises responsible AI work around governing, mapping, measuring and managing risk. The EU AI Act introduces a risk-based regulatory approach for certain AI systems, with stronger obligations for high-risk applications and restrictions on unacceptable practices; its exact application depends on role, system and jurisdiction. Data protection laws such as the European Union General Data Protection Regulation express principles including lawfulness, fairness, transparency, purpose limitation, data minimisation, accuracy, storage limitation, integrity and confidentiality. Other jurisdictions use different terminology and requirements. Students and practitioners must verify current local law rather than treating any framework as universal legal advice.

The OECD AI principles, ISO/IEC 27001 for information-security management and ISO/IEC 42001 for AI management systems are examples of standards or principles that can guide organisational controls. Frameworks do not make a system responsible automatically. Their value lies in converting broad values into evidence: assessments, approvals, test results, logs, documentation, monitoring records and remediation decisions.

Data Quality, Lineage and Model Documentation

Responsible decisions depend on trustworthy data. Quality dimensions commonly include accuracy, completeness, consistency, timeliness, validity, uniqueness and fitness for purpose. Quality is contextual: a dataset may be sufficiently accurate for exploratory analysis but unsuitable for an eligibility decision. Automated checks can test schema, ranges, referential integrity, duplicates, missingness and distribution shifts. Exceptions should be investigated rather than silently deleted.

A lineage record should identify source systems, collection context, transformations, joins, filters, feature engineering, versions, responsible persons and downstream uses. In Python or R, this can be supported through version control, environment files, pipeline orchestration, metadata catalogues and experiment-tracking tools. Reproducibility is valuable, but a reproducible analysis of unauthorised data remains unacceptable.

Model documentation should explain intended use, out-of-scope use, training data, features, labels, algorithms, assumptions, evaluation design, subgroup performance, uncertainty, known limitations, security considerations, human oversight and change history. A model card, system card or equivalent record can provide this information. Documentation should be updated when the model, data, threshold or operating context changes.

An Illustrative Case: Automated Scholarship Screening

Background

Consider an illustrative university case in which a data science team develops a model to prioritise scholarship applications for human review. The institution has five years of historical decisions, applicant demographics, academic records, household information and engagement data.

Situation and Analysis

The proposed model achieves high overall predictive performance. However, historical awards were influenced by unequal access to mentoring and application support. The “number of completed enrichment activities” variable is strongly associated with household resources. Some fields are incomplete for international and first-generation applicants. The team also discovers that a vendor’s data licence permits internal reporting but is unclear about use in automated ranking.

A privacy review identifies that the original notice did not explain predictive profiling. Security testing finds that the development notebook contains an unprotected export. Fairness analysis shows different false-negative rates across applicant groups. The model’s ranking would not itself make the final decision, but reviewers are likely to rely heavily on it.

Approach and Strategy

The university pauses deployment and defines the legitimate purpose: prioritising review, not determining eligibility. It consults students, financial-aid staff and legal and privacy officers. It removes unnecessary vendor fields, confirms licence rights, encrypts the approved dataset and applies role-based access. A lineage register records sources and transformations.

The team tests alternative labels and models, examines subgroup performance, and rejects enrichment activity as a direct predictor because it reflects opportunity rather than merit relevant to need. It introduces a review protocol requiring staff to consider the full application, record reasons for disagreement and provide an appeal route. A model card describes limitations and prohibits use for automatic rejection. Performance and subgroup outcomes are monitored during a pilot, with a predefined suspension threshold.

Outcome

The model is used only as a bounded triage aid during the pilot. Reviewers receive training, applicants receive an explanation of the process, and an independent committee reviews complaints and monitoring results. The university finds that a simpler model, combined with improved application guidance, is more defensible than the original high-performing system.

Learning Points

The case demonstrates that governance can change the objective, variables and deployment design—not merely document a finished model. Accuracy alone did not resolve questions about consent, fairness, intellectual property, security or human reliance. A responsible outcome may involve limiting, redesigning or declining automation.

Organisational Implementation and Common Failures

A mature programme begins with an inventory of datasets, models, vendors and processing purposes. Each asset should have an owner, classification, retention rule, access policy, lineage record and review date. Risk-tiering helps allocate effort: a low-impact internal forecast does not require the same controls as a system influencing medical treatment or access to housing.

Common failures include treating de-identification as permanent anonymity, collecting data “in case it becomes useful,” assuming open data has no licence restrictions, relying on aggregate accuracy, hiding automated processing in legal notices, approving access indefinitely, failing to test subgroup performance, and documenting a model only after an incident. Another failure is ethics washing: adopting impressive principles without resources, authority, monitoring or consequences for non-compliance.

Trade-offs are unavoidable. Stronger privacy may reduce analytical granularity; stricter access may slow collaboration; fairness constraints may alter accuracy; explainability may conflict with complex models; and retention may support auditability while increasing exposure. These are not reasons to abandon controls. They require explicit, evidence-based decisions, documented residual risk and appropriate approval.

Emerging Developments

Synthetic data, federated learning, confidential computing and privacy-enhancing computation may reduce the need to centralise sensitive information, but they introduce new assumptions and attack surfaces. Generative AI creates additional issues involving training-data provenance, memorisation, fabricated outputs, prompt leakage, copyright and accountability for human reliance. Continuous monitoring is becoming more important as models are updated, connected to tools and used in changing environments.

Regulation is also moving toward lifecycle and risk-based oversight. Organisations increasingly need evidence that systems were assessed before deployment and monitored afterwards, rather than relying on a one-time compliance statement. Data scientists should therefore develop not only programming skill but also documentation, communication, threat modelling, impact assessment and governance literacy.

Best Practices for Data Science Teams

1. Define purpose, affected groups, foreseeable harms and success criteria before collecting data.

2. Minimise data, verify lawful authority and document consent or another legitimate basis.

3. Classify data and models; apply least privilege, encryption, secret management and multifactor authentication.

4. Maintain versioned lineage, quality checks, reproducible environments and change records.

5. Test data and model performance overall and across relevant groups, including missingness and uncertainty.

6. Document intended use, limitations, human oversight, security risks and appeal mechanisms.

7. Use independent review for high-impact systems and involve affected stakeholders.

8. Monitor drift, incidents, complaints and subgroup outcomes after deployment.

9. Provide correction, appeal, withdrawal or remedy where appropriate.

10. Retire systems whose risks, performance or purpose are no longer acceptable.

Chapter Summary

Ethics, security and governance are lifecycle responsibilities rather than final approval steps. Privacy requires purpose limitation, minimisation, lawful authority, meaningful information and protection against re-identification. Security protects confidentiality, integrity and availability through layered technical and organisational controls. Fairness requires attention to historical data, representation, labels, proxies, subgroup outcomes and real-world harm. Transparency must provide meaningful information to the relevant audience, while accountability requires ownership, review, remedy and evidence.

Data quality controls, lineage, access governance and model documentation convert principles into operational practice. Frameworks such as the NIST AI Risk Management Framework, data-protection principles, ISO management standards and risk-based AI regulation help organisations structure decisions, but they do not replace judgement. The central professional lesson is that a technically successful model may still be inappropriate. Responsible data science asks not only whether a system works, but whether its purpose is legitimate, its risks are controlled and affected people are treated fairly.

Key Terms

Privacy: Appropriate control over information relating to individuals.

Personal data: Information relating to an identified or identifiable person.

Consent: Voluntary, informed and specific agreement to defined processing.

Purpose limitation: Restricting data use to specified legitimate purposes.

Data minimisation: Collecting and retaining only necessary data.

Pseudonymisation: Replacing identifiers with codes while retaining possible re-identification risk.

Anonymisation: Processing intended to make identification sufficiently unlikely.

Data lineage: A record of data origins, transformations and downstream uses.

Data provenance: Evidence describing the history and source of data or results.

Confidentiality: Protection against unauthorised disclosure.

Integrity: Protection against unauthorised or inappropriate alteration.

Availability: Accessibility of authorised data and systems when required.

Least privilege: Giving users only the access necessary for their duties.

Algorithmic bias: Systematic disadvantage arising from data, design or deployment.

Fairness: Normative and measurable principles concerning equitable treatment and outcomes.

Explainability: Ability to provide meaningful reasons for a model output.

Accountability: Responsibility for decisions, correction and remedy.

Model card: Documentation describing a model’s purpose, performance, limitations and use.

Data steward: Person responsible for data quality, definitions and appropriate use in a domain.

Risk-based governance: Allocating controls according to potential impact and likelihood of harm.

Review Questions

Short Answer Questions

1. Distinguish data governance from data management.

2. Why does removing protected attributes not necessarily remove algorithmic bias?

3. Define pseudonymisation and explain its privacy limitation.

4. What are the three properties of the confidentiality–integrity–availability model?

5. What information should a model card contain?

Long Answer Questions

1. Explain how privacy, consent, purpose limitation and data minimisation should be applied across the data lifecycle.

2. Discuss the sources of algorithmic bias and evaluate methods for assessing and mitigating fairness risks.

3. Describe a defence-in-depth security strategy for a data science environment.

4. Explain the roles of data owners, data stewards, model developers, reviewers and operational users in AI governance.

5. Compare the contributions and limitations of responsible AI frameworks and legal compliance requirements.

Analytical / Application Questions

1. A company wants to use employee communication metadata to predict resignation risk. Analyse the privacy, consent, purpose, fairness and security issues, and recommend governance controls.

2. A credit model has equal overall accuracy but a substantially higher false-negative rate for one protected group. What further analysis and organisational decisions are required before deployment?

3. During an audit, an analyst discovers that a production model was trained on data with unclear licensing, undocumented transformations and unrestricted notebook access. Develop a prioritised remediation plan.

22. Multiple Choice Questions with Answers

Question 1

Which action best demonstrates purpose limitation?

A. Collecting all available attributes for possible future analysis

B. Using data only for the clearly specified and legitimate purpose communicated to individuals

C. Removing names while retaining unlimited secondary uses

D. Allowing every analyst to decide independently how data may be reused

Correct Answer: B. Using data only for the clearly specified and legitimate purpose communicated to individuals

Question 2

Why is pseudonymised data not necessarily anonymous?

A. Pseudonymisation always increases data accuracy

B. Pseudonymised data cannot be encrypted

C. Codes or linked datasets may permit re-identification

D. Pseudonymised data contains no personal information

Correct Answer: C. Codes or linked datasets may permit re-identification

Question 3

Which control most directly follows the principle of least privilege?

A. Giving every project member administrator access

B. Providing access only to the data and functions required for a person’s role

C. Storing passwords in a shared notebook

D. Disabling all audit logs

Correct Answer: B. Providing access only to the data and functions required for a person’s role

Question 4

A model has high aggregate accuracy but much higher false-negative rates for one group. What is the most appropriate conclusion?

A. The model is fair because aggregate accuracy is high

B. The group should be removed from evaluation

C. Subgroup fairness and the consequences of errors require further investigation

D. The protected attribute should simply be deleted from the dataset

Correct Answer: C. Subgroup fairness and the consequences of errors require further investigation

Question 5

Which item is primarily a data-lineage record?

A. A list of employee salaries

B. A record of sources, joins, transformations and downstream uses

C. A marketing description of a model

D. A password rotation schedule

Correct Answer: B. A record of sources, joins, transformations and downstream uses

Question 6

In the scholarship case, why was the enrichment-activity variable problematic?

A. It was impossible to store in a database

B. It was unrelated to all applicant outcomes

C. It could reflect unequal access to opportunities rather than relevant merit or need

D. It guaranteed perfect predictions

Correct Answer: C. It could reflect unequal access to opportunities rather than relevant merit or need

Question 7

Which statement best describes meaningful human oversight?

A. A human automatically approves every model output without review

B. A reviewer has time, information and authority to challenge or override the system

C. The model is hidden from operational staff

D. Human involvement is added only after a complaint

Correct Answer: B. A reviewer has time, information and authority to challenge or override the system

Question 8

What is the main purpose of a model card?

A. To replace all security controls

B. To document intended use, performance, limitations and responsible-use conditions

C. To guarantee that a model is legally compliant everywhere

D. To disclose every confidential training record publicly

Correct Answer: B. To document intended use, performance, limitations and responsible-use conditions

Question 9

Which response is most consistent with defence in depth?

A. Relying only on a single password

B. Combining authentication, least privilege, encryption, logging, secure development and incident response

C. Deleting all monitoring records

D. Allowing production data in public repositories

Correct Answer: B. Combining authentication, least privilege, encryption, logging, secure development and incident response

Question 10

Why can responsible governance lead to rejecting a technically accurate model?

A. Accuracy is never useful in data science

B. Technical performance does not by itself establish legitimate purpose, fairness, privacy or acceptable risk

C. Governance prohibits all automated systems

D. Documentation always reduces model performance

Correct Answer: B. Technical performance does not by itself establish legitimate purpose, fairness, privacy or acceptable risk