9. Cryptography, Data Protection and Privacy Engineering

Introduction and Context

Software systems routinely collect, transmit, process, and retain information whose compromise may harm individuals, organizations, and society. Passwords, health records, payment details, source code, identity documents, business strategies, and location data all require protection, but not necessarily the same kind of protection. Cryptography provides mathematical mechanisms for confidentiality, integrity, authentication, accountability, and privacy. It does not, by itself, determine what data should be collected, who may access it, how long it should be retained, or whether a processing activity is ethically justified. Those questions belong to data protection and privacy engineering.

For secure software developers, the central challenge is not simply to “use encryption.” It is to select appropriate cryptographic primitives, compose them correctly, protect keys throughout their lifecycle, avoid implementation misuse, and align technical controls with legal, organizational, and privacy requirements. A system may use strong encryption and still expose sensitive information through logs, backups, metadata, error messages, analytics platforms, or excessive retention.

This chapter treats cryptography as part of a broader protection architecture. It examines encryption, hashing, message authentication, digital signatures, certificates, public key infrastructure, randomness, key management, privacy-enhancing controls, retention, governance, crypto-agility, and post-quantum readiness.

Learning Objectives

By the end of this chapter, students should be able to:

1. Distinguish confidentiality, integrity, authentication, non-repudiation, and privacy objectives.

2. Select appropriate encryption, hashing, MAC, signature, and password-protection mechanisms.

3. Explain certificates, PKI, trust chains, revocation, and key lifecycle management.

4. Identify common cryptographic misuse and evaluate its consequences.

5. Design data protection controls across collection, use, transmission, storage, sharing, archival, and deletion.

6. Apply privacy engineering principles such as minimization, purpose limitation, pseudonymization, and differential privacy.

7. Evaluate retention obligations, governance responsibilities, crypto-agility, and post-quantum migration risks.

Cryptographic Protection as a Set of Security Services

Cryptography transforms information using algorithms and keys so that particular security properties can be achieved. The security objective must be defined before an algorithm is selected.

Confidentiality prevents unauthorized parties from learning plaintext. Symmetric encryption, such as AES or ChaCha20, is generally efficient for protecting large amounts of data. Integrity provides confidence that data has not been altered. Hash functions, MACs, authenticated encryption, and signatures can contribute to integrity, but they provide different assurances. Authentication establishes the identity of a communicating party or the origin of a message. Non-repudiation, in the limited technical and evidentiary sense, concerns the ability to associate a signature with a key and a signing event; it is never guaranteed by mathematics alone because legal rules, key custody, and operational evidence also matter.

Privacy is related but not identical to confidentiality. Encryption may conceal content from an unauthorized observer while still revealing metadata such as timing, volume, location, or the existence of a relationship. Privacy engineering therefore considers collection, inference, identifiability, secondary use, user expectations, and institutional power, not merely secrecy.

Symmetric Encryption and Authenticated Encryption

Symmetric encryption uses the same secret key, or closely related secret keys, for encryption and decryption. It is computationally efficient and is commonly used for database fields, files, backups, and network sessions. However, the key must be shared or made available to every legitimate decrypting component, which creates a distribution and access-control problem.

Modern applications should normally prefer authenticated encryption with associated data (AEAD). An AEAD scheme protects plaintext confidentiality and produces an authentication tag that detects alteration. Associated data, such as a record identifier or protocol version, is authenticated but not encrypted. AES-GCM and ChaCha20-Poly1305 are widely used examples. A nonce or initialization vector must be generated and used according to the algorithm’s requirements. In particular, reusing a nonce with the same key in schemes such as GCM can seriously compromise security. A nonce need not always be secret, but it must be unique where the construction requires uniqueness.

Developers should use a reputable, reviewed library and its high-level API rather than implementing an algorithm. Ciphertext should normally be stored with the algorithm identifier, version, nonce, authentication tag, and key reference needed for controlled decryption. This supports migration and key rotation without confusing old and new formats.

Hash Functions and Password Protection

A cryptographic hash function maps input data to a fixed-length digest. A secure hash should make it computationally difficult to find a preimage, a second preimage, or two different inputs with the same digest. SHA-256 and SHA-3 are examples of general-purpose cryptographic hash functions. Hashes are useful for file integrity, content addressing, signatures, and protocol constructions, but a plain hash is not a suitable password-storage mechanism.

Passwords have low and variable entropy and are vulnerable to offline guessing if a database is exposed. Password verifiers should use a password-specific, deliberately expensive, memory-hard or computationally costly function such as Argon2id, scrypt, or bcrypt, with a unique random salt for every password. A pepper, stored separately from the password database, may provide an additional defense. Password policies should be complemented by multifactor authentication, breached-password screening, rate limiting, and secure recovery processes. Passwords should not be encrypted for later recovery unless there is an exceptional and justified requirement; ordinarily, the system needs to verify them rather than retrieve them.

Message Authentication Codes

A MAC uses a shared secret key to authenticate a message and detect modification. HMAC, constructed from a secure hash function, remains widely used. A MAC proves that a party possessing the shared secret generated the tag, but it does not identify which party when several parties share the key. MAC verification must be performed using constant-time comparison where timing leakage could reveal information. The system should also define what is covered by the MAC; authenticating only the message body while leaving critical headers or parameters unauthenticated creates substitution opportunities.

Digital Signatures

Digital signatures use a private key to sign data and a corresponding public key to verify it. They support origin authentication, integrity, and evidence that a particular key was used. Unlike a MAC, a signature can be verified by multiple parties without giving them the signing capability. EdDSA, ECDSA, and RSA-based signature schemes are examples, although algorithm selection must consider library support, deployment constraints, and future migration.

Signatures do not automatically solve replay, canonicalization, or key-compromise problems. A signed request may be copied and replayed unless it contains an expiry time, nonce, sequence number, or transaction identifier. Structured data must have an unambiguous canonical representation before signing. A private signing key requires strong protection because compromise may allow fraudulent signatures until trust is revoked or the key expires.

Public Key Infrastructure and Trust Decisions

Public key cryptography reduces the need for pre-sharing a secret with every communicating party. In a PKI, a certificate binds a public key to an identity or domain through a digital signature by a certificate authority (CA). A relying party validates the certificate chain, validity period, name or identity binding, key usage, signature algorithms, and trust anchor. TLS certificates are one familiar application, but certificates may also support device identity, code signing, document signing, and enterprise authentication.

Trust is a policy decision implemented through technical validation. A valid signature does not prove that the asserted identity is appropriate for the business transaction. Certificate validation must include hostname or identity checks, chain validation, critical extension processing, and appropriate algorithm and key-size policies. Disabling certificate validation to resolve a development error converts an authentication failure into a silent impersonation vulnerability.

Certificate revocation mechanisms, including certificate revocation lists and online status protocols, have operational limitations such as availability, caching, and privacy concerns. Short-lived certificates, automated renewal, key rotation, and monitoring can reduce reliance on revocation. Organizations should maintain inventories of certificates, owners, expiry dates, trust anchors, and dependent services.

Randomness, Nonces, and Secrets

Cryptographic security depends heavily on unpredictable values. Keys, password salts, session identifiers, reset tokens, nonces, and initialization vectors should be generated with a cryptographically secure pseudorandom number generator (CSPRNG) seeded by adequate system entropy. General-purpose random functions used for simulations or interface effects are not suitable for security tokens.

A random value can be public while still needing unpredictability at creation; a nonce may need uniqueness rather than secrecy. Confusing these requirements can lead to either unnecessary complexity or catastrophic reuse. Tokens should have sufficient length, limited lifetime, narrow scope, and safe storage. Secrets should not appear in source code, container images, version-control history, logs, exception messages, or client-side bundles.

Key Management as an Operational Discipline

The effective strength of encryption is often determined less by the cipher than by key management. A key lifecycle includes generation, registration, distribution, activation, use, rotation, backup, recovery, revocation, archival, and destruction. Each key should have an owner, purpose, algorithm, permitted operations, environment, creation date, and retirement criteria.

Key-encryption keys and data-encryption keys can be separated through envelope encryption. A data-encryption key protects a file or data object, while a key-encryption key in a hardware security module or cloud key-management service protects the data key. This arrangement supports scalable storage and centralized policy without requiring the master key to encrypt every record directly. Access should be granted to workloads through narrowly scoped identities, and decryption should be logged and monitored.

Key rotation can limit exposure but does not repair compromise automatically. A rotation plan must address old ciphertext, backups, replicas, caches, exports, and legal holds. Destruction must be verifiable where deletion is required. Recovery procedures must be tested: a key that is perfectly protected but irretrievable can cause an availability incident.

Data Protection Across the Information Lifecycle

Data protection begins before data enters the system. A data inventory should identify categories, subjects, sources, purposes, sensitivity, locations, recipients, retention periods, and applicable obligations. Classification schemes may distinguish public, internal, confidential, restricted, and highly restricted information, but labels are useful only when connected to enforceable controls.

During collection, systems should minimize fields, explain purpose, obtain valid consent where required, and avoid collecting information merely because it may become useful later. During transmission, use authenticated encrypted channels and prevent sensitive data from entering URLs or unprotected telemetry. During processing, enforce least privilege, separate duties, and control privileged queries. During storage, use encryption at rest, access controls, backups, tenant isolation, and integrity monitoring. During sharing, document recipients, purposes, legal bases, contractual protections, and transfer mechanisms.

During archival, data should be moved to controlled storage only when there is a continuing operational, legal, or historical justification. During deletion, remove primary copies, replicas, indexes, temporary files, caches, and accessible backups according to a defined schedule. Secure deletion is complicated by distributed systems and immutable storage; therefore, design should include deletion capability from the outset. Where immediate physical deletion is impossible, strong cryptographic erasure, access revocation, and documented backup expiry may reduce residual risk, subject to applicable requirements.

Retention is a governance decision, not merely a storage setting. Keeping data indefinitely increases breach impact, discovery burden, misuse potential, and the number of systems requiring protection. Deleting it too early may violate statutory, contractual, accounting, safety, or litigation-preservation duties. Retention schedules should specify trigger events, responsible owners, exceptions, review dates, and defensible deletion evidence.

Privacy Engineering in Software Design

Privacy engineering translates privacy principles into system architecture and development practice. Data minimization limits collection and precision. Purpose limitation prevents unapproved secondary use. Pseudonymization replaces direct identifiers with tokens, but the data remains personal if re-identification information exists. Anonymization seeks to make re-identification reasonably infeasible in the relevant context; it is difficult to guarantee, particularly when datasets can be linked.

Access control, encryption, and auditability support confidentiality, but privacy also requires user control and institutional accountability. Interfaces should provide meaningful notices, consent choices where appropriate, access and correction mechanisms, and understandable explanations of automated processing. Privacy threat modeling can examine linkability, identifiability, detectability, information disclosure, non-repudiation, and non-compliance. A data protection impact assessment may be appropriate where processing is likely to create high risks for individuals.

Privacy-enhancing technologies include secure multi-party computation, homomorphic encryption, trusted execution environments, federated learning, private set intersection, and differential privacy. These technologies have different costs and threat assumptions. Differential privacy adds calibrated randomness so that the contribution of an individual has limited influence on published results. Its privacy budget must be governed across repeated queries; simply adding noise without accounting for composition does not create a defensible privacy program.

Common Cryptographic Misuse

Frequent failures include using obsolete algorithms, hard-coded keys, predictable tokens, reused nonces, unauthenticated encryption, home-grown cryptography, weak password hashing, incorrect certificate validation, excessive key permissions, and logging plaintext secrets. Another error is encrypting data without planning for search, indexing, recovery, rotation, or lawful access. Developers may also assume that HTTPS protects data after it reaches application memory, logs, databases, or third-party processors.

Security review should therefore ask: What is the asset and threat? What must be confidential, authentic, or unlinkable? Who holds each key? What happens when a key is compromised? How are old versions decoded? How is deletion verified? What information leaks through metadata? Are cryptographic failures fail-safe, observable, and testable? These questions reveal design weaknesses that algorithm selection alone cannot address.

Illustrative Case Study: A Health Analytics Platform

Background

MedInsight, an illustrative health analytics company, receives laboratory records from hospitals and produces population-level dashboards. It stores identifiable patient data, pseudonymous research datasets, model outputs, and audit records in cloud services.

Situation and Problem

An internal review found that the platform encrypted database disks but stored export files in a shared object-storage location. Service accounts had broad decryption permissions, password reset tokens were generated with a general-purpose random function, and data was retained indefinitely “for research.” Analysts could download row-level data even when aggregate results would have met their needs.

Analysis

Disk encryption reduced risk from lost physical media but did not address compromised credentials, excessive analyst access, exports, or retention. The token-generation flaw created a direct account-takeover risk. Indefinite retention increased the consequences of a future breach and conflicted with purpose limitation. The platform also lacked a clear distinction between pseudonymized data and anonymous data; the re-identification mapping remained available to operational teams.

Approach and Solution

The organization introduced a data inventory and classification scheme, then separated operational identity data from research datasets. Each dataset received a distinct data-encryption key protected by a managed key-encryption key. Workload identities were granted narrowly scoped decrypt permissions, and sensitive decrypt operations generated alerts. Export files were encrypted with authenticated encryption and automatically expired.

Password verifiers were migrated to Argon2id, reset tokens to a CSPRNG, and all external connections to validated TLS. Aggregate dashboards became the default, while row-level access required documented approval and was limited by purpose. A retention schedule defined deletion or review events, with exceptions for legal holds. Privacy reviews assessed re-identification risk and the cumulative privacy budget for statistical releases. The cryptographic data format included version and key identifiers to support future rotation.

Outcome

The revised design reduced the number of people and services able to decrypt identifiable data, improved detection of unusual access, and made deletion and migration operationally feasible. It did not eliminate all risk: authorized analysts could still misuse data, and statistical disclosure remained possible. These residual risks were addressed through governance, training, monitoring, and periodic reassessment.

Learning Points

The case demonstrates that encryption at rest is only one control in a lifecycle architecture. Effective protection combines minimization, access governance, authenticated encryption, secure randomness, key management, retention, privacy analysis, and operational monitoring.

Governance, Ethics, and Professional Responsibility

Cryptographic decisions must be governed through security policies, architecture standards, supplier requirements, incident procedures, and documented ownership. Legal obligations vary by jurisdiction and sector, but recurring themes include lawful processing, transparency, individual rights, breach response, cross-border transfers, records management, and protection of children or other vulnerable groups. Compliance is not equivalent to security: a formally complete notice cannot justify insecure implementation, and a technically secure system may still process data unfairly or without proper authority.

Professionals must consider proportionality. Stronger surveillance or indefinite logging may improve forensic capability while undermining privacy. Exceptional access mechanisms and “backdoors” create concentrated targets and should not be treated as cost-free solutions. Where access to encrypted information is operationally necessary, it should be narrowly authorized, auditable, legally reviewed, and designed to limit abuse.

Crypto-Agility and Post-Quantum Readiness

Crypto-agility is the ability to replace algorithms, key sizes, certificates, providers, and protocols without redesigning the entire system. It requires algorithm identifiers, versioned formats, inventory of cryptographic dependencies, automated certificate management, separation of cryptographic policy from application logic, and tested migration procedures. Agility does not mean enabling many algorithms indiscriminately; it means being able to retire weak mechanisms and adopt approved alternatives safely.

Quantum computers sufficiently capable of running relevant cryptanalytic algorithms could threaten widely used public-key systems based on integer factorization and discrete logarithms. Symmetric cryptography is affected differently, generally requiring attention to key sizes and security margins. The “harvest now, decrypt later” concern is particularly relevant to information that remains sensitive for many years. Organizations should inventory where public-key cryptography is used, identify long-lived data and certificates, consult evolving standards, test hybrid or post-quantum mechanisms where appropriate, and avoid embedding unchangeable cryptographic assumptions in protocols.

Post-quantum readiness is not achieved by replacing one library call. It involves supply-chain assessment, performance and message-size testing, certificate and protocol support, hardware capacity, interoperability, and migration of archived data. A risk-based roadmap should prioritize data with long confidentiality lifetimes and systems that are difficult to update.

Best-Practice Design Principles

A defensible cryptographic and privacy architecture generally follows these principles:

• Define security and privacy objectives before selecting mechanisms.
• Prefer reviewed, maintained libraries and standard protocols.
• Use AEAD for encryption and password-specific hashing for password verifiers.
• Generate keys, salts, tokens, and nonces with appropriate secure randomness.
• Separate key-encryption keys, data keys, identities, environments, and duties.
• Authenticate all security-relevant fields and prevent replay where necessary.
• Inventory certificates, algorithms, keys, data stores, processors, and retention rules.
• Minimize collection, access, precision, retention, and data movement.
• Test rotation, recovery, revocation, deletion, incident response, and migration.
• Treat logs, backups, analytics, caches, and exports as part of the protected data environment.
• Build crypto-agility and post-quantum migration into architecture decisions.

Chapter Summary

Cryptography provides distinct services: encryption protects confidentiality, hashes support integrity-related functions, MACs authenticate shared-secret messages, and digital signatures support public verification of origin and integrity. Correct use depends on secure randomness, nonce discipline, certificate validation, PKI trust decisions, and disciplined key lifecycle management. Passwords require specialized slow hashing rather than ordinary encryption or hashing.

Data protection extends across collection, processing, transmission, storage, sharing, archival, and deletion. Privacy engineering adds minimization, purpose limitation, pseudonymization, anonymization analysis, user control, and privacy-enhancing computation. Retention must balance legal and operational needs against breach and misuse risks. Finally, crypto-agility and post-quantum planning ensure that protection can evolve as algorithms, threats, standards, and technologies change.

Key Terms

Cryptography: The study and application of techniques for protecting information using mathematical transformations.

Authenticated encryption: Encryption that provides confidentiality and detects unauthorized modification.

AEAD: Authenticated encryption with associated data; it authenticates selected unencrypted data along with ciphertext.

Hash function: A function that maps input to a fixed-length digest designed to resist specified attacks.

Salt: A unique value added to password input before password-specific hashing.

MAC: A keyed integrity and authentication code based on a shared secret.

Digital signature: A public-key mechanism used to verify message origin and integrity.

Nonce: A value intended for one-time or unique use in a cryptographic operation.

CSPRNG: A cryptographically secure pseudorandom number generator.

PKI: The systems, policies, certificates, authorities, and processes supporting public-key trust.

Certificate authority: An entity that issues digitally signed certificates binding keys to identities.

Key rotation: Replacing a cryptographic key according to a defined schedule or event.

Envelope encryption: Protecting data keys with a separate key-encryption key.

Pseudonymization: Replacing identifiers while retaining a possibility of controlled re-identification.

Anonymization: Processing intended to make identification reasonably infeasible in context.

Differential privacy: A formal method limiting the influence of one individual on released information.

Data minimization: Limiting data collection and precision to what is necessary for a defined purpose.

Crypto-agility: The ability to change cryptographic mechanisms without extensive system redesign.

Post-quantum cryptography: Cryptographic techniques designed to resist attacks from capable quantum computers.

Review Questions

Short Answer Questions

1. Why is authenticated encryption generally preferable to encryption without integrity protection?

2. What is the difference between a password salt and a password pepper?

3. What does a certificate authority contribute to PKI trust?

4. Why is pseudonymized data not necessarily anonymous?

5. What is the purpose of envelope encryption?

Long Answer Questions

1. Explain the lifecycle of a cryptographic key and discuss the organizational controls required at each stage.

2. Compare symmetric encryption, MACs, and digital signatures in terms of keys, scalability, and security services.

3. Describe how data should be protected from collection through deletion, including retention and backup considerations.

4. Explain common cryptographic implementation failures and propose controls to prevent them.

5. Discuss crypto-agility and develop a risk-based strategy for post-quantum readiness.

Analytical / Application Questions

1. A development team encrypts customer records with AES-GCM but uses a counter that resets whenever the service restarts. Analyze the risk and recommend a safer design.

2. An organization wants to publish detailed mobility data after removing names and identifiers. Evaluate whether this is sufficient for privacy and propose additional controls.

3. A company must retain encrypted historical records for seven years, but the key-management service is being replaced. Design a migration plan that preserves confidentiality, availability, auditability, and future crypto-agility.

22. Multiple Choice Questions with Answers

Question 1

Which mechanism is most appropriate for protecting a message while also detecting its modification?

A. Plain SHA-256 hashing

B. Authenticated encryption such as AES-GCM

C. Base64 encoding

D. A random number generator

Correct Answer: B. Authenticated encryption such as AES-GCM

Question 2

Why should passwords normally be processed with Argon2id, scrypt, or bcrypt rather than a plain SHA-256 hash?

A. These functions are designed to make offline guessing more expensive

B. They allow administrators to recover the original password

C. They eliminate the need for unique salts

D. They provide public-key authentication

Correct Answer: A. These functions are designed to make offline guessing more expensive

Question 3

What is the primary danger of reusing a nonce with the same key in a nonce-sensitive AEAD scheme such as GCM?

A. The ciphertext becomes larger than the plaintext

B. The certificate chain becomes invalid

C. Confidentiality and authentication guarantees can be seriously compromised

D. The key is automatically rotated

Correct Answer: C. Confidentiality and authentication guarantees can be seriously compromised

Question 4

Which statement best distinguishes a MAC from a digital signature?

A. A MAC uses a shared secret, while a signature uses a private/public key pair

B. A MAC provides confidentiality, while a signature cannot provide integrity

C. A MAC requires a certificate authority, while a signature never uses keys

D. A MAC is always publicly verifiable by anyone

Correct Answer: A. A MAC uses a shared secret, while a signature uses a private/public key pair

Question 5

Which practice best supports secure certificate validation?

A. Accepting any certificate issued during development

B. Checking only whether the certificate is unexpired

C. Validating the chain, identity binding, key usage, and trust policy

D. Disabling hostname verification for internal services

Correct Answer: C. Validating the chain, identity binding, key usage, and trust policy

Question 6

What is the main purpose of envelope encryption?

A. To replace all access controls with encryption

B. To use a key-encryption key to protect data-encryption keys

C. To make plaintext searchable without disclosure

D. To avoid the need for key rotation

Correct Answer: B. To use a key-encryption key to protect data-encryption keys

Question 7

Which statement about pseudonymization is correct?

A. It always makes data anonymous

B. It removes the need for access control

C. It replaces identifiers but may permit re-identification using additional information

D. It is identical to encryption of a database disk

Correct Answer: C. It replaces identifiers but may permit re-identification using additional information

Question 8

A system stores sensitive data in encrypted backups but retains the decryption key indefinitely after the data retention period. What is the most relevant concern?

A. The data may remain recoverable despite apparent deletion

B. The backup will automatically become anonymous

C. The encryption algorithm will become a hash function

D. The key will prevent all future cryptographic migration

Correct Answer: A. The data may remain recoverable despite apparent deletion

Question 9

Which capability is central to crypto-agility?

A. Using as many algorithms as possible simultaneously

B. Hard-coding one algorithm into every application component

C. Replacing algorithms, keys, and protocols through versioned and tested migration processes

D. Avoiding certificates and public-key cryptography entirely

Correct Answer: C. Replacing algorithms, keys, and protocols through versioned and tested migration processes

Question 10

Why should organizations begin post-quantum planning before large-scale quantum computers exist?

A. All symmetric encryption is already broken

B. Long-lived sensitive data may be collected now and decrypted later

C. Quantum computers eliminate the need for key management

D. Post-quantum algorithms require no interoperability testing

Correct Answer: B. Long-lived sensitive data may be collected now and decrypted later