Security, Privacy, Ethics and Governance

Introduction

Big data systems create strategic value by combining high-volume, high-velocity and high-variety information from operational systems, sensors, applications, social platforms and external data providers. The same characteristics that enable discovery also increase risk. Data may identify individuals, reveal sensitive behaviour, influence decisions, or be copied across jurisdictions and organizations. A security breach can therefore cause financial loss, regulatory penalties, operational disruption and lasting damage to public trust.

Security, privacy, ethics and governance are related but distinct. Security protects data and systems from unauthorized access, alteration, disclosure and destruction. Privacy concerns appropriate control over information about identifiable or identifiable people. Ethics asks whether a practice is fair, justified and socially responsible, even when it is lawful. Governance establishes authority, accountability, policies, standards and processes for managing data throughout its life cycle.

A mature big data program treats these concerns as design requirements rather than as controls added after deployment.

Learning Objectives

After studying this chapter, you should be able to:

1. Explain identity management, encryption, access control and auditing in distributed data environments.
2. Apply privacy principles, anonymization techniques, retention rules and regulatory reasoning.
3. Design governance arrangements involving ownership, catalogs, lineage and data-quality controls.
4. Evaluate surveillance, discrimination, consent and environmental effects of big data.
5. Recommend proportionate controls for a practical big data case.

1. A Risk-Based View of Big Data

Big data risk is shaped by four interacting properties:

• Scale: millions or billions of records increase the impact of compromise.
• Linkability: apparently harmless datasets can become identifying when combined.
• Velocity: automated collection and real-time decisions reduce opportunities for human review.
• Opacity: complex pipelines and machine-learning models may make decisions difficult to explain.

A useful governance principle is proportionality: controls should reflect the sensitivity of the data, the consequences of misuse, the likelihood of threats and the purpose of processing. Public weather observations do not require the same controls as health records, biometric identifiers or precise location histories.

A simplified risk model is:

Likelihood estimates how probable an adverse event is; impact estimates its consequences. This is a prioritization aid, not a precise physical law. Organizations should document their rating scales and include legal, human and environmental consequences, not only financial loss.

The data life cycle normally includes collection, ingestion, storage, processing, sharing, use, archival and deletion. Security and governance controls should operate at every stage.

2. Identity and Access Management

2.1 Identity management

Identity management establishes who or what is requesting access. Subjects include employees, contractors, customers, applications, service accounts, devices and automated agents. Core functions are:

• Identification: claiming an identity, such as a user name or service identity.
• Authentication: proving control of that identity.
• Authorization: determining permitted actions.
• Accountability: recording actions so they can be traced to an identity.
• Lifecycle management: creating, modifying, reviewing and disabling identities.

Authentication factors are commonly categorized as something known, such as a password; something possessed, such as a security key; and something inherent, such as a biometric characteristic. Multi-factor authentication combines factors from different categories. A second password is not genuinely multi-factor because both factors belong to the knowledge category.

Federated identity allows a trusted identity provider to authenticate a user for several services. In a distributed data platform, federation can reduce password duplication, but trust relationships must be documented and revoked promptly when employment or authorization changes.

2.2 Authorization models

Role-based access control (RBAC) assigns permissions to roles such as data engineer, analyst or auditor. Users receive roles according to job responsibilities. RBAC is understandable and manageable, but role proliferation can occur in large organizations.

Attribute-based access control (ABAC) evaluates attributes of the subject, resource, action and context. A policy might permit a clinician to view records for patients in the clinician’s department during an approved shift, while denying bulk export. ABAC supports fine-grained decisions but requires reliable and current attributes.

The principle of least privilege grants only the access necessary for an approved task. Separation of duties prevents one person from controlling incompatible activities, such as approving and executing a sensitive data export. Default deny means access is refused unless an explicit policy permits it.

Data platforms should also distinguish data-plane access from control-plane access. Reading a table is different from changing retention rules, encryption keys or cluster configuration. Administrative privileges require stronger authentication, monitoring and independent review.

3. Encryption and Key Management

Encryption transforms readable plaintext into ciphertext using a key. Its purpose depends on where it is applied:

• Encryption in transit protects network communication, commonly through authenticated transport protocols.
• Encryption at rest protects stored files, database pages, backups and snapshots.
• Application-level or field-level encryption protects selected values even from some infrastructure administrators.
• End-to-end encryption aims to ensure that only communicating endpoints can read content.

Encryption does not solve every security problem. An authorized application can decrypt data; stolen credentials can therefore defeat otherwise strong cryptography. Metadata, traffic patterns and exposed keys may also remain vulnerable.

Key management includes generation, distribution, storage, rotation, backup, revocation and destruction. Keys should be separated from the data they protect, access should be logged, and recovery procedures should be tested. A key hierarchy often uses a highly protected key-encryption key to protect data-encryption keys. Loss of a key may make data unrecoverable, so availability and confidentiality must be balanced.

Hashing is not encryption. A cryptographic hash is a one-way transformation useful for integrity checks and password verification when combined with a unique salt and an appropriate password-hashing scheme. Digital signatures provide integrity, origin authentication and non-repudiation properties under stated cryptographic assumptions; they do not by themselves provide confidentiality.

4. Auditing, Monitoring and Incident Response

Auditing creates evidence about what happened, who acted, when, from where and against which resource. Important events include authentication attempts, privilege changes, data reads, exports, schema changes, key operations and policy exceptions.

A useful audit record generally contains a time synchronized to a trusted source, actor identity, action, object, result, source context and correlation identifier. Logs should be protected against alteration, retained for a defined period and restricted because they may themselves contain sensitive information.

Monitoring should distinguish unusual activity from legitimate high-volume workloads. Examples of warning signals include impossible travel, repeated failed authentication, access outside normal duties, unusually large exports and queries against unrelated populations. Automated detection should support investigation rather than automatically punish users without safeguards.

Incident response commonly involves preparation, detection and analysis, containment, eradication, recovery and lessons learned. Notification duties depend on the applicable law, sector and jurisdiction. A post-incident review should examine technical causes, governance failures, incentives and whether affected individuals were treated fairly.

5. Privacy Principles and Regulatory Reasoning

Privacy is not equivalent to secrecy. People may reasonably expect information to be used only for appropriate purposes, with transparency, security, accuracy and meaningful control. Common privacy principles include:

• lawfulness, fairness and transparency;
• purpose limitation;
• data minimization;
• accuracy;
• storage limitation;
• integrity and confidentiality; and
• accountability.

Regulatory requirements vary by jurisdiction. Examples include comprehensive data-protection laws, sectoral health and financial rules, breach-notification laws, employment regulations and cross-border transfer requirements. A university student should not memorize one statute as a universal answer. Instead, identify the data subjects, controller or responsible organization, processors, purposes, locations, legal basis, rights, retention period and security obligations.

A privacy impact assessment is a structured process for identifying risks to people and selecting mitigations before processing begins. It should describe data flows, necessity, proportionality, affected groups, potential harms, controls and residual risk. Privacy by design incorporates safeguards into architecture, such as collecting approximate rather than exact location when exact location is unnecessary.

Consent is meaningful only when it is informed, specific, freely given and capable of withdrawal where the legal framework requires those conditions. A long notice that users cannot realistically understand does not automatically establish ethical or legal legitimacy. In employment, education or essential services, power imbalances may make consent a weak basis for some processing.

6. Anonymization, Pseudonymization and Re-identification

Pseudonymization replaces direct identifiers with codes while retaining a separate means of linking records. It reduces exposure but does not necessarily make data anonymous, because re-identification may remain possible.

Anonymization aims to make identification impossible or sufficiently impracticable under the relevant circumstances. It is difficult to guarantee because auxiliary datasets, rare attributes and repeated releases can enable linkage attacks.

Common techniques include:

• suppression or removal of fields;
• generalization, such as replacing exact age with an age range;
• aggregation, such as publishing counts rather than individual records;
• perturbation or controlled noise; and
• tokenization, which is generally pseudonymization unless the re-identification possibility is eliminated.

A classic disclosure model is k-anonymity: each released record should be indistinguishable from at least k−1 others with respect to selected quasi-identifiers. If k = 5, each equivalence class contains at least five records. K-anonymity can still permit attribute disclosure when all members of a group share a sensitive value, and it may be vulnerable to background knowledge. Differential privacy offers a formal approach by limiting how much the output changes when one person’s record is added or removed. Its protection depends on a privacy budget and the composition of repeated queries; it does not make every release automatically safe.

Before release, organizations should test uniqueness, suppression, linkage risk, membership inference and utility loss. The safest dataset is not necessarily the most useful, so the purpose and risk tolerance must be explicit.

7. Retention, Deletion and Data Life-Cycle Control

Retention should be based on business need, legal obligations, archival value and risk. “Keep everything indefinitely” is not a neutral policy: it expands the attack surface, increases discovery obligations and permits future uses that people may not expect.

A retention schedule specifies the record class, owner, active period, archival period, deletion trigger, legal hold rule and evidence of disposal. Deletion must cover primary stores, replicas, caches, indexes, backups and derived datasets where feasible. Some systems cannot immediately erase immutable backups; in that case, access should be restricted and expiry documented.

A legal hold suspends routine deletion when litigation, investigation or regulatory review is reasonably anticipated. Holds must be communicated to responsible custodians and released formally. Retention also applies to models, feature stores, prompts, logs and model outputs when they contain personal or regulated information.

8. Governance: Ownership, Catalogs, Lineage and Quality

Governance defines decision rights. Typical roles include:

• Data owner: accountable for a domain, permissible uses and risk acceptance.
• Data steward: maintains definitions, quality rules and operational coordination.
• Custodian: operates storage, security and technical controls.
• Privacy or compliance officer: advises on obligations and monitors conformity.
• Data subject or customer: a person whose information is processed and whose rights must be considered.

A data catalog records datasets, business definitions, owners, sensitivity classifications, access conditions, refresh frequency and quality indicators. A catalog turns an uncontrolled data lake into a discoverable but governed information environment.

Data lineage documents the origin, transformations, transfers and destinations of data. It supports impact analysis, debugging, auditability and explanations of analytical results. Lineage should cover batch and streaming pipelines, manual interventions, model features and published reports.

Data quality is multidimensional. Common dimensions are accuracy, completeness, consistency, timeliness, validity, uniqueness and integrity. A quality rule must be operationally testable. For example, completeness for a field can be defined as:

The formula assumes that expected values and required fields are clearly defined and that null and missing values are consistently represented. If 9,700 of 10,000 expected customer records contain a required email value, completeness is 97%. This does not prove that the emails are correct or that all customers should have email addresses; it measures only the stated dimension.

Quality controls include schema validation, referential-integrity checks, duplicate detection, anomaly thresholds, reconciliation to source systems and issue-management workflows. Quality is fitness for an intended use, not an abstract property independent of context.

9. Ethics: Surveillance, Discrimination and Social Impact

Surveillance

Large-scale collection can create pervasive monitoring even when each individual data source appears harmless. Location histories, workplace telemetry, facial recognition and behavioural profiles may chill lawful activity. Ethical assessment should ask whether collection is necessary, proportionate, transparent, contestable and limited to a legitimate purpose.

Discrimination and fairness

A model can discriminate through explicit use of protected attributes, proxy variables, biased historical labels, unequal data coverage or deployment practices. Removing a protected attribute does not guarantee fairness because correlated variables may preserve its effect.

Fairness criteria can conflict. For a binary classifier, demographic parity compares positive-decision rates across groups, while equal opportunity compares true-positive rates. When base rates differ, it may be mathematically impossible to satisfy several fairness criteria simultaneously except in special cases. Consequently, fairness is a sociotechnical decision requiring domain judgment, not merely a metric-selection exercise.

Organizations should evaluate subgroup error rates, calibration, data representativeness, accessibility and the consequences of false positives and false negatives. People affected by high-impact decisions should receive understandable explanations, avenues for correction and human review where appropriate.

Consent, autonomy and power

Ethical data practice respects the ability of people to understand and influence processing. Dark patterns, bundled consent, secondary use and opaque data brokerage weaken autonomy. Children and vulnerable groups require stronger safeguards. Public benefit does not automatically justify unrestricted collection.

Environmental impact

Big data systems consume electricity and hardware resources through data centers, networking, storage, model training and repeated computation. Governance should consider data minimization, efficient architectures, retention limits, workload scheduling, hardware life cycles and the carbon intensity of energy. Environmental impact must be assessed across the system life cycle, including manufacturing and disposal, not only server runtime.

10. A Governance Architecture

An effective architecture combines preventive, detective and corrective controls. Preventive controls include classification, least privilege, encryption, validation and purpose restrictions. Detective controls include logging, alerts, quality dashboards and access reviews. Corrective controls include incident response, data correction, revocation, deletion and disciplinary or regulatory action.

Figure: Governance controls operate across the complete big data life cycle.

11. Applied Case: Hospital Readmission Analytics

Background

This illustrative case concerns a hospital network that wants to predict readmission risk using clinical records, laboratory results, demographics and appointment history. The goal is to offer additional support after discharge.

Problem

The network must combine records from several hospitals. Risks include unauthorized access, inaccurate records, discrimination against underserved groups, excessive retention and use of data for purposes unrelated to care.

Analysis

The program classifies health information as highly sensitive. A data owner is appointed for the clinical domain, while a steward defines terms such as “readmission within 30 days.” A privacy impact assessment maps collection, feature engineering, model scoring, clinician access and patient communication.

The design uses federated identity and multi-factor authentication for staff, service identities for pipelines, role- and attribute-based authorization, encryption in transit and at rest, and separate key-management responsibilities. Direct identifiers are separated from analytical identifiers. Access to identifiable records is logged and reviewed. The model feature store has a documented retention period, and backups are covered by deletion and legal-hold procedures.

Quality testing checks missing laboratory values, duplicate patients, inconsistent coding and temporal leakage. Model evaluation reports sensitivity, specificity, calibration and error rates by relevant demographic and clinical groups. A high-risk score is treated as a prompt for professional review, not as an automatic denial of services. Patients receive an understandable explanation of the program and a route to correct inaccurate information.

Outcome

The network deploys a limited pilot, monitors subgroup outcomes and establishes a review committee including clinicians, security staff, privacy professionals and patient representatives. The model is withdrawn from a use case when evaluation shows that its training labels reflect unequal access to follow-up care rather than underlying clinical need.

Learning Points

The case demonstrates that governance is not a document placed beside the system. Ownership, lineage, quality, security, privacy, fairness and clinical accountability must be connected in the architecture and operating process. A technically accurate model can still be inappropriate if its purpose, data or decision consequences are unjustified.

12. Best Practices and Contemporary Developments

Organizations should adopt data classification, standardized identity federation, strong authentication, least privilege, encrypted communication, centralized audit collection, tested backups and incident exercises. They should maintain a catalog, lineage records, quality metrics, retention schedules and documented risk acceptances.

Modern platforms increasingly use policy-as-code, confidential computing, privacy-enhancing computation, synthetic data and automated governance checks. These tools are promising but not magic solutions. Synthetic data may reproduce source biases or permit membership inference; confidential computing depends on hardware and deployment assumptions; automated policy can enforce an incorrect rule consistently. Generative AI introduces additional concerns involving prompt data, training provenance, model memorization, hallucinated sensitive information and unclear accountability. Human oversight, evaluation and documented limitations remain essential.

Professional practice requires evidence. Teams should be able to show who approved a purpose, which data were used, how a transformation occurred, what controls were applied, whether quality thresholds were met and how affected people can challenge an outcome.

Chapter Summary

Big data security protects systems and information through identity management, authentication, authorization, encryption, key management, auditing and incident response. Privacy requires appropriate purpose, transparency, minimization, accuracy, retention and accountability. Pseudonymization reduces exposure but does not guarantee anonymity; re-identification risk must be assessed in context.

Governance assigns ownership and decision rights, while catalogs make data discoverable, lineage makes transformations traceable and quality controls make fitness for use measurable. Retention and deletion should be deliberate and should include replicas, backups and derived datasets. Ethical analysis extends beyond legal compliance to surveillance, autonomy, discrimination, explainability, contestability and environmental impact. Effective programs integrate preventive, detective and corrective controls throughout the data life cycle.

Key Terms

Identity management: Processes for creating, authenticating, authorizing and retiring identities.

Authentication: Verification that a subject controls a claimed identity.

Authorization: Determination of which actions an authenticated subject may perform.

Least privilege: Granting only the access required for an approved task.

Role-based access control: Authorization based on assigned organizational roles.

Attribute-based access control: Authorization based on subject, resource, action and contextual attributes.

Encryption: Transformation of plaintext into protected ciphertext using a key.

Key management: Generation, protection, distribution, rotation and destruction of cryptographic keys.

Audit trail: Protected record of actions and events in a system.

Privacy by design: Incorporating privacy safeguards into systems from the beginning.

Purpose limitation: Restricting data use to specified and legitimate purposes.

Pseudonymization: Replacing identifiers with codes while retaining possible linkage.

Anonymization: Processing intended to prevent identification under relevant circumstances.

Re-identification: Linking released data to an identifiable person.

Data catalog: Governed inventory of datasets, definitions, owners and conditions.

Data lineage: Record of data origins, transformations and destinations.

Data quality: Fitness of data for a defined use, measured through dimensions such as accuracy and completeness.

Data owner: Person accountable for a data domain and its permissible uses.

Legal hold: Suspension of routine deletion because of anticipated or active proceedings.

Differential privacy: A formal privacy approach limiting the effect of one person’s data on an output.

Review Questions

Short Answer Questions

1. Distinguish authentication from authorization.
2. Why is pseudonymization not necessarily anonymization?
3. State four dimensions of data quality.
4. What is the purpose of a data catalog?
5. Why should retention policies include backups and replicas?

Long Answer Questions

1. Explain how RBAC and ABAC differ, and discuss situations in which each is appropriate.
2. Describe a complete security and privacy control strategy for a distributed data lake.
3. Explain why encryption, key management and auditing must be designed together.
4. Discuss how data lineage supports compliance, quality management and incident investigation.
5. Evaluate the ethical risks of large-scale workplace monitoring.

Analytical / Application Questions

1. A retailer wants to combine loyalty-card, location and purchase data to predict customer behaviour. Design a privacy impact assessment, including purpose, minimization, anonymization, retention and individual rights.
2. A public-sector model approves housing assistance. Its error rate is substantially higher for one demographic group. Propose an investigation and remediation plan that considers data quality, fairness metrics, human review and governance.
3. An organization discovers that a service account exported ten million records outside normal hours. Describe the immediate response, evidence to preserve, access-control changes and post-incident governance review.

Multiple Choice Questions with Answers

Question 1

Which control determines what an authenticated user is permitted to do?

A. Identification

B. Authorization

C. Encryption

D. Hashing

Correct Answer: B. Authorization

Question 2

Which statement best describes pseudonymization?

A. It permanently destroys all identifiers and linkage possibilities.

B. It replaces identifiers while retaining a possible means of linkage.

C. It encrypts every network connection.

D. It guarantees compliance with every privacy law.

Correct Answer: B. It replaces identifiers while retaining a possible means of linkage.

Question 3

If 9,700 of 10,000 expected values are non-null, what is completeness?

A. 7%

B. 9.7%

C. 97%

D. 103%

Correct Answer: C. 97%

Question 4

Which principle grants only the access necessary for an approved task?

A. Least privilege

B. Open access

C. Data duplication

D. Purpose expansion

Correct Answer: A. Least privilege

Question 5

What is the primary purpose of data lineage?

A. To increase password length

B. To document origins and transformations of data

C. To replace encryption keys

D. To eliminate all missing values

Correct Answer: B. To document origins and transformations of data

Question 6

Which statement about hashing is correct?

A. Hashing is identical to reversible encryption.

B. Hashing is primarily a one-way transformation used for integrity or verification.

C. Hashing automatically anonymizes a dataset.

D. Hashing prevents authorized applications from reading data.

Correct Answer: B. Hashing is primarily a one-way transformation used for integrity or verification.

Question 7

Why can removing a protected attribute fail to eliminate discrimination?

A. Encryption always reintroduces the attribute.

B. Correlated proxy variables may preserve its effect.

C. Audit logs erase group differences.

D. Data catalogs create discrimination.

Correct Answer: B. Correlated proxy variables may preserve its effect.

Question 8

What does a legal hold generally do?

A. Accelerates routine deletion

B. Suspends routine deletion for relevant information

C. Makes all data public

D. Disables authentication

Correct Answer: B. Suspends routine deletion for relevant information

Question 9

Which activity is most directly associated with a data catalog?

A. Recording dataset definitions, owners and sensitivity

B. Generating random passwords only

C. Replacing all human review

D. Destroying audit logs

Correct Answer: A. Recording dataset definitions, owners and sensitivity

Question 10

Why is consent not automatically meaningful merely because a notice exists?

A. Consent must be informed, specific and freely given where applicable.

B. Consent is valid only when data are encrypted.

C. Consent eliminates the need for data quality.

D. Consent always permits indefinite retention.

Correct Answer: A. Consent must be informed, specific and freely given where applicable.