10. Cryptography, Secrets, Data Protection and Privacy

Introduction and Context

Web applications routinely handle information whose disclosure, alteration, or unavailability can harm individuals and organisations. Passwords, payment details, health records, identity documents, access tokens, business plans, source code, and confidential communications all require protection. Cryptography provides the mathematical and technical mechanisms for that protection, but cryptography alone does not create security. A strong algorithm used with a leaked key, an encrypted database exposed through a vulnerable application, or a correctly hashed password stored beside its recovery key can still result in compromise.

This chapter examines cryptography as part of a broader information-protection system. It distinguishes encryption from hashing, authentication from confidentiality, digital signatures from encryption, and certificates from keys. It also considers secrets management, key life-cycle controls, data classification, retention, privacy engineering, and common implementation failures. These topics are especially important for full-stack developers because security decisions are distributed across browsers, mobile clients, APIs, databases, cloud services, deployment pipelines, and third-party providers.

The central principle is that protection must be matched to the security objective and the data life cycle. Data should be collected only when justified, classified according to its sensitivity, protected in transit and at rest, retained only as long as necessary, and securely destroyed when its purpose ends.

Learning Objectives

After studying this chapter, a student should be able to:

1. Explain confidentiality, integrity, authenticity, non-repudiation, and availability as security objectives supported by cryptographic controls.

2. Select appropriate uses for symmetric encryption, public-key cryptography, hashing, password hashing, message authentication codes, digital signatures, and certificates.

3. Design practical key and secrets-management life cycles.

4. Apply data classification, minimisation, retention, and privacy-by-design principles.

5. Identify cryptographic misuse, including hard-coded keys, weak randomness, insecure modes, poor password storage, and unsafe certificate validation.

6. Evaluate cryptographic choices in application architecture and incident-response scenarios.

Cryptography as a Set of Security Services

Cryptography transforms information using algorithms and secret or public values called keys. Its purpose is not simply to make data unreadable. A well-designed cryptographic system can provide several distinct services.

Confidentiality prevents unauthorised parties from learning the content of data. Encryption is the principal mechanism. Integrity provides evidence that data has not been changed. Hashes, message authentication codes, and digital signatures can support integrity. Authenticity helps establish that a message, user, service, or key is genuinely associated with the claimed source. Non-repudiation, in a limited technical and legal sense, is supported by digital signatures when the signing key is controlled and the surrounding governance is credible. Cryptography does not, by itself, prove that a person intended an action or that a key was not stolen.

These services should not be confused. Encryption without authentication may conceal a message while allowing an attacker to modify it. A hash may detect accidental change but does not authenticate who produced the data. A digital signature proves possession of a private key at signing time, but it does not necessarily provide confidentiality.

Symmetric Encryption

Symmetric encryption uses the same secret key, or closely related secret material, to encrypt and decrypt data. It is computationally efficient and is therefore suitable for database fields, files, backups, network sessions, and large volumes of application data. Modern authenticated encryption modes, such as AES-GCM and ChaCha20-Poly1305, combine confidentiality with integrity and authenticity through an authentication tag.

A nonce or initialization value is commonly required. It normally need not be secret, but it must be used according to the algorithm’s rules. In particular, nonce reuse with the same key can seriously compromise some authenticated-encryption schemes. A developer should use a trusted library that generates and validates nonces correctly rather than designing a format independently.

Encryption at rest protects stored data against particular threats, such as unauthorised access to disks or backups. It does not protect data from an attacker who has obtained application-level access and can ask the application to decrypt it. Field-level encryption may reduce exposure, but it complicates searching, indexing, recovery, and key rotation. The security objective and threat model should therefore determine whether storage encryption, application encryption, tokenisation, or a combination is appropriate.

Asymmetric Cryptography

Public-key cryptography uses a mathematically related public key and private key. A public key may be distributed widely, while the private key must remain controlled. Data encrypted with a public key can generally be decrypted only with the corresponding private key. More commonly in web systems, public-key cryptography establishes identities or negotiates a temporary symmetric key, after which symmetric encryption protects the session.

Public-key algorithms are slower and require careful key validation. They are used in TLS certificates, secure email, key exchange, digital signatures, software signing, and identity systems. The distinction between encryption keys and signing keys should be maintained because they serve different functions and may have different compromise consequences.

Hashing and Password Hashing

A cryptographic hash function maps input data to a fixed-length digest. A secure hash is designed to make it computationally infeasible to recover the input from the digest, find two different inputs with the same digest, or produce a chosen input with a particular digest. SHA-256 and SHA-3 are examples of general-purpose hash functions.

Hashing is appropriate for integrity checks, content addressing, fingerprints, and certain protocol constructions. It is not reversible encryption. Hashing a password once with a fast algorithm is inadequate because attackers can test enormous numbers of guesses. Passwords should be processed using a password-specific, deliberately expensive and memory-intensive function such as Argon2id, scrypt, or bcrypt, with a unique randomly generated salt for each password.

A salt is not a secret. It prevents identical passwords from producing identical stored values and makes precomputed lookup tables less useful. It does not make a weak password strong. Work factors should be selected through performance testing and reviewed as hardware changes. Password verification should use the password-hashing library’s verification function and should avoid revealing whether a username or password was the failing factor.

Message Authentication Codes

A message authentication code, or MAC, is generated from a message and a shared secret. It allows a party possessing the secret to verify that a message was produced by someone with that secret and was not modified. HMAC using a secure hash is a common construction. MACs are valuable for API request signing, webhook verification, and internally protected messages.

A MAC does not provide non-repudiation because both communicating parties possess the same secret. It also requires secure distribution and storage of the shared key. Where independent verification by many parties is required, digital signatures may be more appropriate.

Digital Signatures

A digital signature is created with a private key and verified using the corresponding public key. It supports integrity, origin authentication, and evidence that the private key was used. Signatures are used for signed tokens, software packages, certificates, legal workflows, and document approval systems.

A signature does not automatically encrypt the signed content. Nor does it guarantee that the signer is trustworthy. Verification must include the correct algorithm, public-key binding, certificate or trust-chain checks where relevant, scope, expiry, and protection against replay. A signed message may still be reused in a different context unless it contains a purpose, audience, timestamp, nonce, or transaction identifier.

Certificates and Trust in Web Applications

A digital certificate binds an identity or domain name to a public key through a trusted issuer. In HTTPS, a browser or client validates a certificate chain leading to a trusted root, checks validity dates and permitted names, and applies rules concerning key usage and revocation or status. Transport Layer Security then negotiates authenticated encryption for the connection.

The certificate is not the private key. It may be publicly disclosed; the private key must be protected. Certificate management includes issuance, renewal, deployment, monitoring, revocation, and emergency replacement. Expired certificates cause outages, while stolen private keys can enable impersonation until trust is withdrawn or the key is replaced.

A common programming error is disabling certificate validation to solve a development problem. This converts a temporary configuration issue into vulnerability to interception. Development systems should use locally trusted test certificates, suitable test authorities, or controlled environments—not production-style clients that accept any certificate.

Key Management: The Operational Core

The strength of an algorithm is irrelevant if its keys are exposed, misused, or permanently inaccessible. Key management covers generation, distribution, storage, use, rotation, backup, recovery, revocation, archival, and destruction.

Keys should be generated using a cryptographically secure random-number generator. They should have an explicit purpose, owner, algorithm, environment, creation date, and status. Development, testing, and production keys must be separated. A key used to encrypt customer data should not also sign software or authenticate administrators. Least privilege applies to keys: services should receive access only to the keys and operations they require.

A practical architecture often uses a key-encryption key held in a key-management service or hardware security module to protect data-encryption keys. Applications request controlled cryptographic operations through an authenticated service identity. Envelope encryption allows large data to be encrypted locally with a data key while the data key is protected by a higher-level key. This reduces exposure and supports rotation without re-encrypting every record immediately.

Rotation does not mean simply generating a new key and abandoning old data. A system must define how new data is encrypted, how old data is read and re-encrypted, how key versions are identified, and what happens to backups. Destruction must be deliberate: if a key is securely destroyed, data encrypted solely under it may become unrecoverable. Recovery keys and backup copies therefore require strong access control and documented procedures.

Secrets in the Software Delivery Life Cycle

Secrets include passwords, API tokens, private keys, database credentials, signing keys, encryption keys, and service-account credentials. They should not be embedded in source code, container images, mobile applications, client-side JavaScript, issue trackers, or ordinary log files. Anything delivered to a browser is effectively available to the user and potentially to malicious extensions or compromised endpoints.

A secrets-management system should provide controlled storage, identity-based retrieval, auditing, expiry, rotation, and emergency revocation. Configuration should refer to a secret by name or identifier rather than duplicating its value. Continuous integration pipelines should use short-lived, narrowly scoped credentials and should prevent secret values from appearing in build output. Secret scanning is useful but cannot replace good architecture; once a secret has entered source control, it should be considered exposed and revoked.

Logs deserve special attention. Tokens, session identifiers, authorisation headers, passwords, full payment details, and personal identifiers should be excluded or masked. Debugging mechanisms must not create a second, less protected copy of sensitive data.

Protecting Data Throughout Its Life Cycle

Data protection begins before collection. Teams should identify the purpose of processing, determine whether the data is necessary, and classify it according to potential harm. A simple classification scheme may include public, internal, confidential, and restricted information. Classification should influence access, encryption, retention, monitoring, transfer, and disposal requirements.

Data minimisation means collecting and retaining only what is necessary. For example, an application that needs to verify age may not need to retain a complete identity document. Pseudonymisation replaces direct identifiers with tokens or codes, but the data remains personal if a re-identification mechanism exists. Anonymisation aims to prevent re-identification, though combinations of seemingly harmless attributes can undermine that goal. Tokenisation substitutes a value with a token, with the sensitive mapping held separately; it is often useful for payment or account references.

Data in transit should be protected with current TLS configurations and correct endpoint validation. Data at rest should be protected through platform encryption, database controls, application-level encryption, or appropriate combinations. Data in use is exposed to the running application and its authorised operators, so access control, isolation, monitoring, and minimisation remain important.

Retention should be based on business, legal, contractual, and security requirements rather than indefinite convenience. A retention schedule should state the data category, purpose, retention period, owner, storage location, archival rule, and disposal method. Deletion should address primary databases, replicas, caches, search indexes, object storage, exports, and backups according to the organisation’s recovery design. Legal holds may temporarily suspend ordinary deletion, but they should be documented and limited.

Privacy Engineering and Governance

Privacy is concerned not only with secrecy but also with appropriate collection, use, sharing, transparency, choice, accountability, and individual rights. A privacy-respecting application explains why information is needed, uses it consistently with that purpose, restricts secondary use, and provides appropriate mechanisms for access, correction, deletion, or portability where required by applicable law.

Privacy-by-design incorporates privacy controls into architecture rather than adding a policy after implementation. Useful techniques include purpose limitation, default-private settings, granular consent where consent is the lawful basis, separation of identity from behavioural data, access logging, redaction, aggregation, and privacy impact assessments for high-risk processing.

Governance assigns responsibility. Product owners determine business purpose; security teams define protection requirements; developers implement controls; privacy or legal specialists interpret obligations; data owners approve access and retention; and auditors examine evidence. A data inventory and processing register help an organisation understand where sensitive information flows, including to cloud providers and third parties.

Encryption can reduce exposure but does not eliminate privacy obligations. An organisation may still be responsible for collecting excessive information, sharing it without a valid basis, retaining it too long, or failing to respond to a breach. Decisions should consider applicable data-protection laws, sectoral rules, contractual requirements, and cross-border transfer restrictions.

Common Cryptographic Misuse

Several recurring mistakes are especially important in secure coding:

• Using outdated algorithms, custom cryptography, or unexplained proprietary formats instead of well-reviewed libraries.
• Using encryption without authentication, such as unauthenticated CBC or electronic codebook mode, or ignoring authentication failures.
• Reusing nonces with authenticated encryption.
• Using predictable random values for keys, salts, tokens, or reset links.
• Hard-coding keys or placing secrets in client-side code.
• Treating Base64 or encoding as encryption.
• Encrypting passwords instead of using password hashing.
• Comparing hashes, signatures, or MACs with unsafe ordinary comparisons where timing differences matter.
• Trusting a user-supplied key, algorithm identifier, issuer, or certificate without policy validation.
• Accepting expired, self-signed, mismatched, or otherwise invalid certificates in production.
• Designing tokens without expiry, audience, issuer, purpose, or replay controls.
• Logging plaintext secrets or decrypted personal information.
• Rotating keys without a versioning and recovery plan.

The remedy is not to memorise a larger collection of algorithms. It is to define the security property, use a maintained library and approved protocol, minimise exposure, validate failures, and subject the design to review and testing.

A Decision Framework for Selecting Controls

A useful design sequence begins with the question: what must be protected, from whom, and for how long? Next identify whether the requirement is confidentiality, integrity, authenticity, accountability, or several of these. Then determine the data life-cycle points at which exposure can occur.

For a password, use password hashing rather than encryption. For a large confidential file, use authenticated symmetric encryption and protect the key through a key-management service. For proving that a webhook came from a partner, use a MAC or signature according to the trust relationship. For secure browser-to-server communication, use TLS with proper certificate validation. For proving that software came from a publisher, use a digital signature and a trusted distribution process.

The design should document algorithms, modes, key sizes as approved by current organisational standards, nonce generation, key storage, rotation, error handling, recovery, and testing. Threat modelling should ask what happens if a database is copied, an application server is compromised, a key is leaked, a certificate expires, an employee misuses access, or a backup is restored in an insecure environment.

Applied Case Study: Protecting a Multi-Tenant Learning Platform

Background

This illustrative case concerns LearnSphere, a multi-tenant platform used by universities and professional-training organisations. It stores student profiles, assessment records, invoices, instructor materials, access tokens, and audit logs. Its first version used a shared database password in an environment file, encrypted backups with a key stored beside the backup, logged bearer tokens during troubleshooting, and stored passwords as SHA-256 digests without salts.

Situation and Problem

A contractor’s development laptop was compromised. The attacker obtained a copy of a repository containing an old database credential and found exported logs containing live session tokens. The production database was not directly accessed, but the organisation could not determine which tokens remained valid or whether historical backups were protected. An internal review also found that tenants could accidentally query records belonging to another tenant because tenant identifiers were enforced inconsistently in application code.

Analysis

The immediate issue was not a failure of a particular cryptographic algorithm. It was a failure of secret life-cycle management, data minimisation, access control, and architectural isolation. The password hashes were vulnerable to efficient offline guessing. The logs converted operational data into authentication credentials. The backup key arrangement meant that possession of the backup could potentially lead to possession of the decryption key. Finally, encryption would not have corrected the missing tenant-authorisation checks.

Approach and Solution

LearnSphere first revoked all exposed credentials and invalidated active sessions. It replaced passwords using Argon2id with unique salts and a tested work factor. Database credentials were moved to a managed secrets service, scoped by service identity, audited, and rotated. Backups were protected with envelope encryption; key versions and restoration procedures were documented, and backup access was separated from routine application access.

Bearer tokens were removed from logs, and existing tokens were replaced with short-lived tokens and controlled refresh mechanisms. TLS certificate validation was enforced in all production and staging clients. Sensitive fields were classified, and the team decided that complete identity documents were unnecessary for ordinary course enrolment. Retention rules were created for assessment records, invoices, logs, and support exports.

The team also introduced central tenant-scoping controls, authorisation tests, repository secret scanning, deployment checks, and incident playbooks. Key and secret access generated audit events, while monitoring detected unusual retrieval volume. A privacy review documented data purposes, third-party processors, and deletion workflows.

Outcome

The platform became more resistant to database-copy attacks, credential leakage, and accidental over-retention. Recovery remained possible because key backup and restoration were tested under controlled procedures. The organisation also gained evidence about who accessed secrets and why. The case demonstrates that cryptography is most effective when integrated with identity, authorisation, logging, privacy governance, and operational discipline.

Learning Points

1. A cryptographic primitive cannot compensate for exposed keys or poor access control.

2. Incident response must include revocation, rotation, token invalidation, and evidence preservation.

3. Privacy and security decisions begin with reducing unnecessary data.

4. Multi-tenant isolation must be enforced systematically, not assumed from individual queries.

5. Recovery and deletion are both part of responsible key and data management.

Professional and Emerging Considerations

Modern cloud platforms provide managed key-management services, secret vaults, certificate automation, confidential-computing features, data-loss-prevention tools, and automated dependency scanning. These services can reduce operational burden, but they introduce configuration and vendor-dependency risks. A managed key service is not automatically secure if an overly broad identity policy permits every service to decrypt every dataset.

Automation can enforce minimum TLS settings, scan repositories and container layers for secrets, identify unencrypted storage, detect unusual key usage, and generate retention reports. Machine-learning systems may assist with classification or anomaly detection, but classification errors can expose sensitive information, and automated decisions require human review and governance. Sensitive prompts and training data also create new privacy concerns when generative-AI services are used.

Post-quantum cryptography is an emerging area driven by concern that sufficiently capable quantum computers could threaten some widely used public-key systems. Organisations with long-lived confidential data should inventory cryptographic dependencies and follow standards-based migration guidance. Immediate replacement with unreviewed algorithms is not appropriate; transition planning, crypto-agility, and vendor support are more reliable strategies.

Best-Practice Checklist

A professional implementation should, at minimum:

• Use established, maintained cryptographic libraries and approved protocols.
• Prefer authenticated encryption for confidential application data.
• Use password-specific hashing with unique salts and reviewed work factors.
• Generate keys, nonces, salts, and tokens with a cryptographically secure random source.
• Store secrets in a managed system with identity-based access, audit records, expiry, and rotation.
• Separate keys by purpose, tenant or environment where appropriate, and privilege level.
• Validate certificates, signatures, issuers, audiences, expiry, and algorithm policy.
• Design key versioning, backup, revocation, recovery, and destruction before deployment.
• Classify data, minimise collection, define retention, and securely dispose of copies.
• Prevent secrets and personal data from entering logs, source control, client code, and build artefacts.
• Test failure paths, including invalid tags, expired certificates, unavailable key services, and restoration from backup.
• Review cryptographic architecture through threat modelling, code review, penetration testing, and periodic reassessment.

Chapter Summary

Cryptography supplies mechanisms for confidentiality, integrity, authenticity, and evidence, but each mechanism has a distinct purpose. Symmetric encryption is efficient for bulk data; public-key cryptography supports identity, key exchange, and signatures; hashes support integrity; password-hashing functions resist offline guessing; MACs authenticate messages between parties sharing a secret; and digital signatures enable public verification. Certificates bind public keys to identities through a trust system.

Effective protection depends on key and secrets management, not merely algorithm selection. Keys require controlled generation, storage, use, rotation, recovery, revocation, and destruction. Sensitive data should be classified, minimised, protected throughout its life cycle, retained only when justified, and securely disposed of. Privacy engineering extends beyond secrecy to purpose limitation, transparency, accountability, and responsible data use. Secure coding requires avoiding hard-coded secrets, weak randomness, unauthenticated encryption, unsafe certificate validation, poor password storage, excessive logging, and unplanned key rotation. The strongest designs integrate cryptography with identity, authorisation, monitoring, governance, and incident response.

Key Terms

Authenticated encryption: Encryption that provides confidentiality together with integrity and authenticity verification.

Certificate: A digitally signed statement binding an identity or domain to a public key.

Confidentiality: Protection against unauthorised disclosure of information.

Cryptographic hash: A fixed-length digest generated by a one-way cryptographic function.

Data minimisation: Collecting and retaining only data necessary for a defined purpose.

Digital signature: A private-key-generated value that can be verified with a public key.

Encryption: Transformation of readable data into a protected form using an algorithm and key.

Envelope encryption: Protecting data-encryption keys with a higher-level key-encryption key.

Hashing: Applying a one-way function to produce a digest, rather than reversible ciphertext.

Key management: Governance and technical control of keys throughout their life cycle.

MAC: A keyed value used to verify message integrity and origin between parties sharing a secret.

Nonce: A value used once or according to strict uniqueness rules in a cryptographic operation.

Pseudonymisation: Replacing identifiers while retaining a means of possible re-identification.

Retention schedule: A documented rule specifying how long categories of data are kept and disposed of.

Salt: A unique non-secret random value added to password processing.

Secret: Sensitive authentication or cryptographic material requiring controlled access.

Symmetric encryption: Encryption using a shared secret key for protection and recovery.

Tokenisation: Replacing sensitive data with a token whose original value is held separately.

Trust chain: A sequence of certificate signatures leading from an end certificate to a trusted root.

Zero-knowledge password storage: Storing a password-derived verifier so the original password is not recoverable by the application.

Review Questions

Short Answer Questions

1. Distinguish encryption, hashing, and password hashing.

2. Why must authenticated encryption validate its authentication tag?

3. What is the purpose of a salt in password storage?

4. Explain the difference between a certificate and a private key.

5. Define data minimisation and state one practical example.

Long Answer Questions

1. Explain the roles of symmetric and asymmetric cryptography in a typical HTTPS application.

2. Describe the complete life cycle of a cryptographic key, from generation to destruction.

3. Discuss how privacy-by-design influences data collection, storage, access, and retention.

4. Analyse the security and operational risks of storing secrets in source code and ordinary configuration files.

5. Explain how digital signatures, MACs, and authenticated encryption differ in their security properties and suitable applications.

Analytical / Application Questions

1. An application encrypts customer records with AES but stores the encryption key in the same database. Analyse the weaknesses and propose a safer architecture.

2. A webhook receiver accepts a signed request but does not check its timestamp, audience, or transaction identifier. What attack is possible, and what controls should be added?

3. A university wants to retain identity documents indefinitely “in case they are useful later.” Evaluate this decision using classification, minimisation, privacy, retention, and incident-risk principles.

22. Multiple Choice Questions with Answers

Question 1

A system must protect a large file and detect any modification to it. Which choice is most appropriate?

A. A fast unsalted hash alone

B. Authenticated symmetric encryption

C. Base64 encoding

D. A public key without a private key

Correct Answer: B. Authenticated symmetric encryption

Question 2

Why is a unique salt used when storing passwords?

A. It makes the password reversible only by administrators

B. It replaces the need for a strong password-hashing function

C. It prevents identical passwords from producing identical stored values

D. It encrypts the password during network transmission

Correct Answer: C. It prevents identical passwords from producing identical stored values

Question 3

Which statement best describes a digital signature?

A. It provides confidentiality without requiring verification

B. It is created with a private key and verified with a public key

C. It is a reversible hash protected by a salt

D. It is always produced by a certificate authority

Correct Answer: B. It is created with a private key and verified with a public key

Question 4

A developer disables TLS certificate validation because a staging certificate has expired. What is the best response?

A. Keep validation disabled in all environments

B. Replace TLS with Base64 encoding

C. Use a properly trusted staging certificate or test trust configuration

D. Accept any certificate only when transmitting passwords

Correct Answer: C. Use a properly trusted staging certificate or test trust configuration

Question 5

Which practice most directly reduces the impact of a leaked application secret?

A. Using the same secret in all environments

B. Using short-lived, narrowly scoped credentials with rotation

C. Printing the secret in deployment logs for auditing

D. Embedding the secret in client-side JavaScript

Correct Answer: B. Using short-lived, narrowly scoped credentials with rotation

Question 6

A MAC differs from a digital signature because a MAC:

A. Uses a shared secret and does not provide public verifiability

B. Can only protect data at rest

C. Always provides non-repudiation

D. Requires no key management

Correct Answer: A. Uses a shared secret and does not provide public verifiability

Question 7

Which design best represents envelope encryption?

A. Encrypting a database password with Base64

B. Encrypting data with a data key and protecting that key with a key-encryption key

C. Publishing the encryption key beside the ciphertext

D. Hashing the encryption key before every read

Correct Answer: B. Encrypting data with a data key and protecting that key with a key-encryption key

Question 8

Which statement about pseudonymisation is correct?

A. It always makes data anonymous

B. It removes the need for access control

C. The data may remain personal if re-identification is possible

D. It is identical to deleting the original data

Correct Answer: C. The data may remain personal if re-identification is possible

Question 9

An organisation destroys a key without checking backups or recovery requirements. What is the primary risk?

A. The ciphertext may become permanently unrecoverable

B. The plaintext becomes publicly searchable

C. The hash automatically becomes reversible

D. The certificate chain becomes longer

Correct Answer: A. The ciphertext may become permanently unrecoverable

Question 10

Which control most directly addresses replay of a valid signed webhook?

A. Removing the signature

B. Adding an expiry time, audience, and unique transaction identifier

C. Using a longer URL

D. Storing the webhook in a public log

Correct Answer: B. Adding an expiry time, audience, and unique transaction identifier