Secure, Ethical, and Governed Data Handling

Introduction

Python programs routinely read files, process uploads, call APIs, parse configuration, and store results. These operations create security and governance responsibilities. A program may be technically correct yet still expose confidential information, permit unauthorised access, retain data unnecessarily, or process information in ways that violate organisational policy or law.

Secure data handling therefore combines defensive programming with ethical judgement and governance. The central principle is that data should be collected, accessed, transformed, stored, shared, and deleted only for legitimate purposes and under controlled conditions. This chapter applies that principle to Python file I/O, modules, exceptions, and application design.

Learning Objectives

By the end of this chapter, you should be able to:

• prevent path traversal and unsafe file selection;
• explain why deserialising untrusted data can execute harmful behaviour;
• protect data against leakage through logs, exceptions, temporary files, and outputs;
• reduce injection risks when constructing queries, commands, and file paths;
• apply secure permissions, least privilege, and controlled temporary storage;
• design retention, deletion, audit, and incident-handling procedures;
• evaluate data processing using privacy, ethical, and governance principles; and
• implement a small Python component that validates input and records safe audit events.

1. A Security and Governance Model for Data

A useful model treats every data item as moving through a lifecycle:

1. Collection — obtain only information needed for a defined purpose.
2. Use — process it within an authorised business or educational purpose.
3. Storage — protect it through access control, encryption where appropriate, and safe permissions.
4. Sharing — disclose it only to approved recipients and systems.
5. Retention — keep it for a justified period.
6. Disposal — delete or anonymise it in a verifiable manner.
7. Accountability — retain sufficient evidence to determine what happened without logging unnecessary sensitive content.

The security objectives of confidentiality, integrity, and availability are often called the CIA triad. Confidentiality prevents unauthorised disclosure; integrity prevents unauthorised or accidental alteration; availability ensures that authorised users can obtain data when needed. Privacy adds questions about people: Was collection justified? Was the person informed? Is the use proportionate? Can an individual exercise applicable rights?

Governance translates these principles into rules, roles, controls, and evidence. A policy may specify who may access payroll data, how long it is retained, which encryption standard is approved, and how incidents are reported. A Python module implements only part of that system; it does not replace organisational policy, legal advice, risk assessment, or human accountability.

Figure: Data lifecycle stages and the governance controls that apply throughout them.

2. Preventing Path Traversal and Unsafe File Access

Path traversal occurs when an attacker manipulates a filename so that a program reads or writes outside its intended directory. Inputs such as ../../secrets.txt, absolute paths, Windows drive paths, or symbolic links may defeat a naïve check. A web application that stores user uploads in uploads/ must not allow a submitted name to select /etc/passwd or a confidential application file.

String prefix checks are unreliable. For example, /srv/app/uploads_evil/file begins with the characters in /srv/app/uploads but is not inside that directory. Normalisation also requires care because symbolic links can point elsewhere. A safer approach is to resolve both the permitted directory and candidate path, then compare them using path-aware operations.

from pathlib import Path

BASE_DIR = Path("/srv/app/uploads").resolve()


def safe_upload_path(untrusted_name: str) -> Path:
    # Reject names that are not simple file names for this policy.
    candidate_name = Path(untrusted_name)
    if candidate_name.name != untrusted_name or untrusted_name in {"", ".", ".."}:
        raise ValueError("invalid file name")

    candidate = (BASE_DIR / candidate_name).resolve()
    try:
        candidate.relative_to(BASE_DIR)
    except ValueError as exc:
        raise ValueError("path is outside upload directory") from exc
    return candidate

This function addresses ordinary traversal, but production designs must also consider symlink races, mount points, permissions, archive extraction, and concurrent attackers. When creating a new file, use restrictive creation modes and avoid following links where the platform and API permit. For archive extraction, inspect every member path before writing; an archive entry such as ../../config.py is itself a traversal attempt.

A stronger design often generates a server-side identifier rather than preserving the user-supplied filename. The original name can be stored as metadata after validation, while the actual storage name is produced with secrets.token_hex() or uuid.uuid4(). File type should not be trusted solely from an extension or client-provided MIME type; inspect size, content, and business rules.

3. Unsafe Deserialisation

Serialisation converts an object or data structure into a storable or transferable representation. Deserialisation reconstructs it. Formats such as JSON and CSV represent data, whereas Python pickle can represent executable object behaviour. Loading a malicious pickle can invoke code during reconstruction. Consequently, never unpickle data received from an untrusted user, network, email, or unknown storage location.

# Safer for ordinary structured data
import json

with open("settings.json", encoding="utf-8") as stream:
    settings = json.load(stream)

if not isinstance(settings, dict):
    raise ValueError("settings must be a JSON object")

JSON is not automatically safe: a parser may be exposed to denial-of-service inputs, excessively large numbers, deeply nested structures, or harmful values in later processing. Validate a schema, enforce size and complexity limits, and treat all parsed values as untrusted. YAML loaders require particular caution; use a safe, restricted loader rather than a general object-construction loader.

Deserialisation controls include:

• prefer plain formats such as JSON, CSV, or a documented binary protocol;
• authenticate and integrity-check data where tampering is a concern;
• validate types, ranges, lengths, and permitted fields;
• reject unknown fields when the application requires a strict schema;
• apply resource limits before parsing; and
• isolate legacy conversion in a restricted environment, never as a casual convenience.

Cryptographic signatures or message authentication codes can establish integrity and origin only when keys are managed correctly. Encryption alone does not prove that the sender is trusted, and encoding such as Base64 provides no confidentiality.

4. Data Leakage Through Files, Logs, and Exceptions

Data leakage is the unintended disclosure of information. Common sources include debug logs containing passwords, exception messages containing full paths or personal records, temporary files readable by other users, core dumps, analytics events, and output files left in shared directories.

Logging should support diagnosis and accountability without becoming a second database of secrets. Log event type, timestamp, actor or service identity, correlation identifier, and outcome. Avoid access tokens, passwords, session cookies, full payment-card numbers, health details, and unnecessary personal identifiers. If an identifier is needed for correlation, use a controlled internal identifier or carefully designed masking; do not assume that a simple hash is anonymous, because low-entropy values may be guessed.

import logging

logger = logging.getLogger(__name__)

try:
    record = load_record(record_id)
except FileNotFoundError:
    logger.warning("record unavailable", extra={"record_id": record_id})
    raise LookupError("record is unavailable") from None

The user receives a general message, while the internal log records a controlled event. Exception chaining is useful for developers, but a production boundary should not expose tracebacks to untrusted users. Configure log access, retention, redaction, rotation, and alerting. Review logs because they themselves may contain regulated information.

Use tempfile.NamedTemporaryFile or TemporaryDirectory rather than predictable names. Ensure temporary data is closed and removed, and do not place confidential files in globally writable directories. Deletion does not guarantee that copies, backups, caches, or filesystem snapshots have disappeared; disposal requirements must match the storage technology and policy.

5. Injection and Trust Boundaries

Injection occurs when untrusted data is interpreted as instructions. The classic examples are SQL injection, shell command injection, and template injection. The underlying error is mixing code and data in a single string.

For database work, use parameterised queries:

cursor.execute(
    "SELECT id, name FROM users WHERE email = ?",
    (email_address,)
)

The placeholder syntax differs among database drivers, but the principle is constant: the driver binds the value as data. Do not construct SQL by concatenating user input. Parameterisation does not automatically protect table or column names; those should come from a fixed allow-list mapped by the application.

Avoid shell commands when a Python library can perform the operation. If a command is genuinely required, use an argument list and avoid shell=True:

import subprocess

subprocess.run(["tar", "-tf", archive_path], check=True,
               capture_output=True, text=True)

The path must still be validated, and the subprocess should run with limited privileges and a controlled environment. Shell escaping is not a substitute for eliminating the shell. Similar separation is needed in HTML templates, spreadsheets, regular expressions, and generated configuration. Context determines the correct escaping or validation strategy.

6. Permissions, Authentication, and Least Privilege

Least privilege means granting a subject only the permissions required for its defined task, for only as long as necessary. A file-processing service may need read access to an input directory and write access to an output directory, but it should not run as an administrator or read an entire home directory.

Controls include:

• separate operating-system users and service accounts;
• restrictive file and directory permissions;
• explicit access checks at the application and storage layers;
• short-lived credentials stored outside source code;
• role-based or attribute-based authorisation;
• periodic access reviews and immediate removal of obsolete access; and
• separation of duties for sensitive operations.

Authentication establishes who or what is requesting access; authorisation decides what that subject may do. A logged-in user is not automatically authorised to access every record. Check ownership or permitted role on every relevant request, not only on a user interface screen.

Python code should not embed passwords or API keys. Use a secrets manager or protected environment configuration, restrict configuration-file permissions, and prevent secrets from entering version control. Rotate credentials and design applications to handle rotation without emergency code changes.

7. Privacy, Ethics, and Responsible Data Use

Security asks whether unauthorised parties can access data; privacy and ethics also ask whether the processing should occur. A responsible design follows several principles:

• Purpose limitation: define why data is collected and do not silently reuse it for incompatible purposes.
• Data minimisation: collect the smallest amount that supports the purpose.
• Accuracy: provide ways to correct important errors.
• Transparency: communicate relevant processing in understandable language.
• Fairness and non-discrimination: test whether data or models disadvantage groups.
• User agency: obtain valid consent where consent is the basis, and provide appropriate choices.
• Accountability: document decisions, controls, owners, and evidence.

Privacy laws differ by jurisdiction and sector. Depending on context, obligations may address lawful basis, special-category information, breach notification, cross-border transfers, individual access, correction, deletion, or automated decision-making. A developer should escalate uncertainty to the designated privacy, security, or legal function rather than making unsupported legal claims.

Anonymisation aims to prevent re-identification, whereas pseudonymisation replaces direct identifiers but may remain reversible with additional information. Combining supposedly harmless fields—such as location, date, and occupation—can enable re-identification. Therefore, “remove the name” is not a complete anonymisation assessment.

8. Retention, Auditability, and Incident Response

A retention schedule identifies the data owner, purpose, storage location, retention period, legal or operational justification, and disposal method. Retention should not be indefinite merely because storage is inexpensive. At the same time, deletion may need to be suspended under a legal hold or active investigation. Backups require a documented expiry and restoration policy.

Audit logs should answer: who acted, what resource was involved, what operation occurred, when it occurred, whether it succeeded, and which request or system component initiated it. They should be protected against alteration, access-controlled, time-synchronised where feasible, and retained according to policy. Do not log the complete sensitive payload simply to make an audit trail look detailed.

An incident response process normally includes preparation, detection, containment, eradication, recovery, and lessons learned. A developer who discovers exposed credentials should not quietly delete evidence; they should follow the reporting route, preserve relevant evidence, revoke or rotate credentials, and cooperate with investigation. Speed matters, but uncontrolled activity can destroy evidence or widen the incident.

9. A Governed Python File-Processing Pattern

The following illustrative design combines several controls. It validates the path, restricts file size, reads with an explicit encoding, handles expected errors, and records a minimal audit event. It does not by itself establish complete authorisation, malware scanning, encryption, or legal compliance.

from pathlib import Path
import logging

logger = logging.getLogger(__name__)
MAX_BYTES = 1_000_000


def read_text_document(base: Path, supplied_name: str, actor: str) -> str:
    root = base.resolve()
    name = Path(supplied_name)
    if name.name != supplied_name or supplied_name in {"", ".", ".."}:
        raise ValueError("invalid document name")

    target = (root / name).resolve()
    try:
        target.relative_to(root)
    except ValueError as exc:
        raise ValueError("document is outside permitted directory") from exc

    if not target.is_file():
        raise FileNotFoundError("document is unavailable")
    if target.stat().st_size > MAX_BYTES:
        raise ValueError("document exceeds permitted size")

    try:
        content = target.read_text(encoding="utf-8")
    except UnicodeDecodeError as exc:
        raise ValueError("document is not valid UTF-8") from exc
    except OSError as exc:
        logger.error("document read failed", extra={"actor": actor,
                                                     "operation": "read"})
        raise RuntimeError("document could not be read") from exc

    logger.info("document read", extra={"actor": actor, "operation": "read"})
    return content

A production version should define how actor is authenticated, authorise that actor for the specific document, defend against symlink races, scan or constrain content where appropriate, and ensure that the logging system does not expose identifiers. The example demonstrates an important design habit: validate at the boundary, use safe library interfaces, catch only errors that can be handled meaningfully, and preserve internal causes without disclosing them externally.

10. Illustrative Case: Student Submission Portal

Background

A college operates a Python portal through which students submit programming assignments. The original system accepted a filename, saved the upload under that name, used pickle to store session objects, and logged request parameters for debugging.

Problem

A security review found four weaknesses. A crafted filename could write outside the submission directory. A malicious session file could execute code when loaded. Logs contained email addresses and session tokens. The service account could read the entire server filesystem.

Analysis

The weaknesses crossed technical and governance boundaries. Path traversal threatened integrity and confidentiality. Unsafe deserialisation created a code-execution risk. Logs violated data minimisation and increased the impact of a breach. Excessive service permissions violated least privilege. The portal also lacked a retention schedule and could not reliably show who accessed a submission.

Approach and Solution

The college adopted the following controls:

1. Uploaded content received a generated identifier; the displayed filename was stored only as validated metadata.
2. Resolved-path checks, size limits, type checks, and controlled temporary storage were introduced.
3. Session data moved from pickle to a constrained, schema-validated representation.
4. Logs recorded an internal student identifier, operation, timestamp, result, and correlation ID, but not tokens or assignment contents.
5. The service ran under a dedicated account with read/write access only to designated directories.
6. Database access used parameterised queries, and credentials moved to a managed secret store.
7. A retention schedule deleted submissions after the academic and appeals periods, subject to approved holds.
8. Access reviews, incident reporting, and an audit dashboard were assigned to named owners.

Outcome

The redesign removed the immediate code-execution and traversal paths, reduced the quantity of exposed data, and improved evidence for investigations. It did not make the portal risk-free: uploaded programs still required sandboxing, dependency management, malware controls, and resource limits before execution.

Learning Points

Security controls work as a system. A safe path function cannot compensate for an over-privileged account; encryption cannot compensate for unjustified collection; and an audit log cannot compensate for absent authorisation. Governance makes ownership, purpose, retention, and review explicit.

11. Challenges and Contemporary Practice

Secure data handling is difficult because controls can conflict with usability, performance, and diagnostic detail. Strict validation may reject legitimate formats; aggressive deletion may remove useful evidence; extensive logging may improve troubleshooting while increasing privacy risk. Decisions should therefore be risk-based, documented, tested, and reviewed.

Modern developments include software supply-chain scanning, secret detection in repositories, policy-as-code, confidential computing, data-loss-prevention tooling, and privacy-enhancing techniques such as aggregation and differential privacy. AI systems add further concerns: prompts and uploaded documents may contain personal or proprietary information, training or retrieval pipelines may retain data beyond its purpose, and generated code may include insecure patterns. Use approved services, minimise prompts, remove unnecessary identifiers, review generated code, and never treat an AI system as an authority on confidentiality or legal compliance.

A practical review checklist asks:

• What data is processed, and how sensitive is it?
• What is the defined purpose and lawful or organisational basis?
• Which inputs cross a trust boundary?
• Can paths, queries, commands, and templates interpret input as code?
• Who can read, modify, export, or delete the data?
• What is logged, and could the log itself leak information?
• How long is each copy retained, including backups?
• How are incidents detected, reported, contained, and learned from?
• Has the design been tested with hostile input and reviewed by the appropriate owner?

Chapter Summary

Secure and ethical Python data handling is a lifecycle discipline rather than a collection of isolated tricks. Path traversal is reduced through path-aware validation, safe naming, and controlled storage. Untrusted serialised data should use constrained formats and schema validation, never unsafe object reconstruction. Parameterised database queries, argument-based subprocess calls, and context-appropriate output encoding prevent injection. Least privilege, protected secrets, restrictive permissions, and controlled temporary files limit the consequences of mistakes.

Privacy requires purpose limitation, minimisation, transparency, fairness, and accountability. Retention schedules, safe disposal, auditability, and incident response convert these principles into operational governance. Effective solutions combine secure code with policy, ownership, testing, monitoring, and responsible professional judgement.

Key Terms

Path traversal: Manipulation of a path to access a location outside the intended directory.

Deserialisation: Reconstruction of data or objects from a stored or transmitted representation.

Unsafe deserialisation: Deserialisation that permits untrusted input to execute behaviour or create dangerous objects.

Injection: Treating untrusted data as instructions in a language or interpreter.

Trust boundary: A point where data or control crosses from one level of trust to another.

Least privilege: Granting only the access required for an authorised task.

Authentication: Establishing the identity of a user, service, or device.

Authorisation: Deciding which operations an authenticated subject may perform.

Confidentiality: Protection against unauthorised disclosure.

Integrity: Protection against unauthorised or accidental alteration.

Availability: Ensuring authorised access when needed.

Data minimisation: Collecting and processing no more data than the purpose requires.

Purpose limitation: Restricting processing to defined and compatible purposes.

Pseudonymisation: Replacing direct identifiers while retaining a controlled possibility of linkage.

Anonymisation: Processing intended to prevent identification of individuals.

Retention schedule: A documented rule specifying how long data is kept and how it is disposed of.

Auditability: The ability to reconstruct relevant actions and decisions from reliable evidence.

Incident response: Coordinated activities for detecting, containing, recovering from, and learning from incidents.

Secrets manager: A controlled system for storing, accessing, and rotating credentials or keys.

Review Questions

Short Answer Questions

1. What is path traversal, and why is a string prefix check insufficient?
2. Why should untrusted data not be loaded with Python pickle?
3. Distinguish authentication from authorisation.
4. State two examples of sensitive information that should normally be excluded from logs.
5. What is the purpose of a retention schedule?

Long Answer Questions

1. Explain how a Python file-upload service can defend against traversal, unsafe filenames, excessive file sizes, and insecure permissions.
2. Compare JSON-based validated data exchange with general-purpose object deserialisation.
3. Explain how least privilege, parameterised queries, and safe subprocess APIs reduce security risk.
4. Discuss the relationship between privacy, security, ethics, and governance in a student-record system.
5. Describe an incident-response process for exposed application credentials and explain why evidence preservation matters.

Analytical / Application Questions

1. A service accepts an archive and extracts each member by joining its name to an upload directory. Analyse the risks and design a safer extraction procedure.
2. A developer proposes logging full request bodies to improve debugging. Evaluate this proposal and design a lower-risk diagnostic approach.
3. A document-processing service runs as an administrator, stores files using user-provided names, and retains all uploads indefinitely. Prioritise the risks and propose a staged remediation plan.

Multiple Choice Questions with Answers

Question 1

Which approach most directly prevents ordinary path traversal when selecting a file inside a permitted directory?

A. Check whether the raw string contains the word 'uploads'

B. Resolve the candidate path and verify it is relative to the resolved permitted directory

C. Convert the filename to uppercase

D. Store the filename in a log before opening it

Correct Answer: B. Resolve the candidate path and verify it is relative to the resolved permitted directory

Question 2

Why is loading an untrusted Python pickle unsafe?

A. Pickle supports only text files

B. Pickle always corrupts binary data

C. Object reconstruction can invoke harmful behaviour or code

D. Pickle cannot represent dictionaries

Correct Answer: C. Object reconstruction can invoke harmful behaviour or code

Question 3

Which database practice is the appropriate defence against SQL injection in a value supplied by a user?

A. Concatenate the value into the SQL string

B. Remove all spaces from the value

C. Use a parameterised query and bind the value separately

D. Convert the value to a filename

Correct Answer: C. Use a parameterised query and bind the value separately

Question 4

What does least privilege require?

A. Every service runs with administrator access

B. Users receive only the access necessary for authorised tasks

C. All files are publicly readable

D. Permissions are never reviewed

Correct Answer: B. Users receive only the access necessary for authorised tasks

Question 5

Which item should normally not be placed in an application log?

A. Operation name

B. Correlation identifier

C. Password or session token

D. Success or failure status

Correct Answer: C. Password or session token

Question 6

Which statement correctly distinguishes authentication and authorisation?

A. Authentication determines permissions; authorisation proves identity

B. Authentication proves identity; authorisation determines permitted actions

C. They are two names for encryption

D. Authorisation is required only for anonymous users

Correct Answer: B. Authentication proves identity; authorisation determines permitted actions

Question 7

Which is an example of data minimisation?

A. Collecting every available attribute in case it is useful later

B. Keeping all records forever because storage is inexpensive

C. Collecting only the fields required for a defined purpose

D. Logging complete documents for convenience

Correct Answer: C. Collecting only the fields required for a defined purpose

Question 8

Which subprocess design is generally safer when an external command is necessary?

A. Pass a list of arguments and avoid shell interpretation

B. Build one command string from user input and use shell=True

C. Put the command in a log and execute the log

D. Run the command as administrator

Correct Answer: A. Pass a list of arguments and avoid shell interpretation

Question 9

What is the primary purpose of an audit trail?

A. To store every secret used by the application

B. To provide reliable evidence of relevant actions and outcomes

C. To replace authorisation checks

D. To guarantee that no incident can occur

Correct Answer: B. To provide reliable evidence of relevant actions and outcomes

Question 10

Which statement about pseudonymisation is correct?

A. It always makes re-identification impossible

B. It replaces direct identifiers but may remain linkable with additional information

C. It is identical to public disclosure

D. It removes the need for access controls

Correct Answer: B. It replaces direct identifiers but may remain linkable with additional information