Files, Serialization and Exception Handling

Introduction and Context

Python programs rarely operate only on values held in memory. Real applications read configuration files, process datasets, store results, communicate with other systems, and recover from failures. These activities require three closely related capabilities: file and path management, representation of data for storage or transmission, and controlled handling of exceptional conditions.

A file-processing program must answer several questions. What resource is being accessed? Is it text or binary data? How is the data structured? What should happen if the file is missing, malformed, inaccessible, or unexpectedly large? Reliable software treats these questions as design concerns rather than afterthoughts.

This chapter develops Python techniques for text, binary, CSV, and JSON data; introduces serialization and deserialization; explains paths and resource management; and shows how exception hierarchies support fault-tolerant input/output workflows. The examples use the standard library and are applicable to data engineering, machine learning pipelines, automation, and general application development.

Learning Objectives

By the end of this chapter, students should be able to:

1. Open, read, write, append, and update text and binary files safely.

2. Use `pathlib` to construct portable and inspectable file paths.

3. Explain character encoding, newline handling, buffering, and file modes.

4. Process CSV and JSON data using appropriate representations.

5. Distinguish serialization from deserialization and select suitable formats.

6. Use context managers to guarantee resource cleanup.

7. Design exception handling with specific exception types and meaningful recovery actions.

8. Create custom exception hierarchies and validate input/output workflows.

9. Evaluate security, portability, data-integrity, and governance issues in stored data.

1. Files as External State

A file is a named sequence of bytes stored by an operating system or another storage service. A Python program commonly interacts with a file through a file object. The file object maintains a position, or cursor, indicating where the next read or write occurs.

The built-in `open()` function has the general form:

```python

open(file, mode="r", encoding=None, newline=None)

```

The `file` argument may be a string or a path-like object. Common modes are:

• `r`: read; the file must already exist.
• `w`: write; creates a file or truncates an existing file.
• `a`: append; creates a file if necessary and writes at its end.
• `x`: exclusive creation; fails if the file exists.
• `b`: binary mode, used with another mode such as `rb` or `wb`.
• `t`: text mode, which is the default.
• `+`: update mode, allowing both reading and writing.

The distinction between `w` and `a` is operationally important. Opening a file with `w` immediately discards its previous contents, whereas `a` preserves them. The mode `x` is useful when overwriting an existing file would be dangerous.

Text and Encoding

Text in Python is represented as Unicode strings. Files, however, contain bytes. An encoding defines how characters are mapped to bytes. UTF-8 is a widely used variable-length encoding and is generally a strong default for interoperable text files:

```python

from pathlib import Path

path = Path("notes.txt")

path.write_text("Café\nMachine learning", encoding="utf-8")

content = path.read_text(encoding="utf-8")

print(content)

```

When an encoding is not specified, Python may use a platform-dependent default. Explicit encoding reduces portability failures. `UnicodeDecodeError` can occur when bytes are decoded with an incompatible encoding, while `UnicodeEncodeError` can occur when a character cannot be represented by a selected encoding.

Newline translation also differs between platforms. In ordinary text use, Python can translate line endings. For structured formats such as CSV, specifying `newline=""` when opening the file is recommended so that the CSV module can manage newlines correctly.

Reading and Writing Patterns

```python

with open("report.txt", "w", encoding="utf-8") as file:

file.write("Status: complete\n")

file.write("Rows: 120\n")

with open("report.txt", encoding="utf-8") as file:

for line in file:

print(line.rstrip("\n"))

```

Iteration is usually preferable to `readlines()` for large files because it processes one line at a time. `read()` is convenient for small files, but loading a very large file into memory may cause resource exhaustion.

The `flush()` method transfers buffered data to the underlying operating-system interface, but it does not necessarily guarantee that data has been physically committed to storage. Applications requiring strong durability may need operating-system-specific synchronization and transactional design.

2. Paths and Resource Management

A path identifies a location in a filesystem. Hard-coded path strings such as `"C:\\data\\file.csv"` are fragile because path separators differ across operating systems and because the current working directory may change.

The `pathlib` module provides an object-oriented and portable interface:

```python

from pathlib import Path

base = Path.home() / "project" / "data"

base.mkdir(parents=True, exist_ok=True)

input_path = base / "records.csv"

print(input_path.exists())

print(input_path.suffix)

print(input_path.name)

```

Useful operations include `exists()`, `is_file()`, `is_dir()`, `mkdir()`, `glob()`, `iterdir()`, `read_text()`, and `write_text()`. `Path.resolve()` can produce an absolute path, although applications should be cautious when resolving paths supplied by untrusted users.

A path is not automatically safe merely because it is syntactically valid. A web application accepting a filename must defend against path traversal, in which an input such as `../../secret.txt` attempts to escape an intended directory. Security controls include validating names, restricting operations to an approved root, and checking the resolved path before access.

Context Managers

A file is an external resource. It should be closed even when an exception occurs. The `with` statement creates a context manager that performs cleanup automatically:

```python

with open("data.txt", encoding="utf-8") as file:

first_line = file.readline()

```

When execution leaves the block normally or because of an exception, the file is closed. This is safer than manually calling `close()` because cleanup is not forgotten on an error path. The same principle applies to sockets, database connections, locks, and temporary resources.

3. Binary Files

Binary mode transfers bytes rather than decoded strings. It is appropriate for images, compressed archives, audio, model files, and formats whose structure is defined in bytes.

```python

from pathlib import Path

source = Path("photo.jpg")

backup = Path("photo.backup.jpg")

with source.open("rb") as input_file, backup.open("wb") as output_file:

while chunk := input_file.read(64 * 1024):

output_file.write(chunk)

```

The walrus operator assigns the chunk while testing whether it is nonempty. Chunked copying limits memory use. Binary files must not be decoded as text merely because they contain some readable characters. Conversely, writing text to a binary stream requires explicit encoding:

```python

data = "hello".encode("utf-8")

text = data.decode("utf-8")

```

The conversion is reversible only when the correct encoding and valid data are used.

4. CSV Data

Comma-separated values are common in business and scientific workflows. Despite the name, CSV is not simply “split each line on commas.” Values may contain commas, quotation marks, and embedded newlines. Python’s `csv` module handles these rules.

```python

import csv

rows = [

{"id": "A01", "score": "87", "active": "yes"},

{"id": "A02", "score": "92", "active": "no"},

]

with open("scores.csv", "w", newline="", encoding="utf-8") as file:

writer = csv.DictWriter(file, fieldnames=["id", "score", "active"])

writer.writeheader()

writer.writerows(rows)

with open("scores.csv", newline="", encoding="utf-8") as file:

reader = csv.DictReader(file)

for row in reader:

score = int(row["score"])

print(row["id"], score)

```

CSV has no universal type system. A value read from a CSV file is normally a string; the application must validate and convert it. For example, `int()` can reject a malformed score, while custom validation should check permitted ranges. Missing columns, duplicate headers, unexpected extra fields, and inconsistent quoting should be treated as data-quality concerns.

For large CSV files, process rows incrementally rather than constructing a complete list. For sensitive workflows, preserve the original file, record validation errors, and produce an audit report rather than silently discarding invalid rows.

5. JSON Data

JSON represents objects, arrays, strings, numbers, booleans, and null values. Python’s standard `json` module maps JSON objects to dictionaries, arrays to lists, strings to strings, numbers generally to `int` or `float`, booleans to `bool`, and null to `None`.

```python

import json

configuration = {

"model": "classifier",

"threshold": 0.75,

"features": ["age", "income"],

"enabled": True

}

with open("config.json", "w", encoding="utf-8") as file:

json.dump(configuration, file, indent=2)

with open("config.json", encoding="utf-8") as file:

loaded = json.load(file)

```

`json.dumps()` and `json.loads()` operate on strings, while `json.dump()` and `json.load()` operate on file-like objects. JSON is human-readable and widely supported, making it useful for configuration files, APIs, and moderate-sized records.

JSON does not directly represent Python sets, tuples, dates, decimals, or arbitrary class instances. Such values require conversion, a custom encoder, or a different format. Floating-point values can also lose decimal precision. Financial applications should consider storing decimal values as strings or using a format with explicit decimal semantics.

A JSON document may be syntactically valid but semantically invalid. For example, a configuration can contain a string where a positive integer is required. Syntax parsing and domain validation are separate stages.

6. Serialization and Deserialization

Serialization converts an in-memory object into a representation suitable for storage or transmission. Deserialization reconstructs an in-memory representation from stored data. The process is often illustrated as:

`object → serialized representation → object`

The reconstructed object need not be identical in every implementation detail. A tuple may become a JSON array, and a custom class may become a dictionary containing its attributes. A robust design specifies the schema and version expectations.

JSON Serialization

JSON is language-independent and comparatively safe to exchange, but it supports a limited set of data types. A useful practice is schema-oriented serialization:

```python

record = {

"schema_version": 1,

"user_id": 42,

"tags": ["python", "data"]

}

```

A version field enables future code to migrate older records. Applications should validate required fields, types, ranges, and relationships after loading.

Pickle Serialization

Python’s `pickle` module can serialize many Python-specific objects:

```python

import pickle

with open("object.pkl", "wb") as file:

pickle.dump(configuration, file)

with open("object.pkl", "rb") as file:

restored = pickle.load(file)

```

Pickle is convenient for trusted internal workflows but is not a safe interchange format for untrusted data. Unpickling can execute arbitrary code because pickle protocols can represent object-construction behavior. Therefore, never unpickle data received from an unknown user, public download, or untrusted network source. Prefer JSON or another explicitly designed data format when trust cannot be established.

Serialization also raises compatibility concerns. A Python class can change between releases, dependencies can disappear, and serialized data can become unreadable. Long-lived systems should document schemas, test backward compatibility, and separate durable data formats from short-term caches.

7. Exception Handling as Control-Flow Design

An exception signals an abnormal condition that interrupts ordinary control flow. Python exceptions form an inheritance hierarchy rooted in `BaseException`; application errors normally derive from `Exception`. Common examples include `FileNotFoundError`, `PermissionError`, `IsADirectoryError`, `UnicodeDecodeError`, `json.JSONDecodeError`, `csv.Error`, `ValueError`, and `TypeError`.

The standard structure is:

```python

try:

with open("config.json", encoding="utf-8") as file:

settings = json.load(file)

except FileNotFoundError:

print("Configuration file is missing.")

except json.JSONDecodeError:

print("Configuration is not valid JSON.")

else:

print("Configuration loaded.")

finally:

print("Loading attempt finished.")

```

`try` contains operations that may fail. `except` handles selected failures. `else` runs only when no exception occurs in the `try` block. `finally` runs whether or not an exception occurs and is appropriate for cleanup or final logging.

Catch specific exceptions before broader ones. Catching `Exception` indiscriminately can hide programming defects, interrupt signals, or produce misleading recovery. A handler should either recover, add useful context and re-raise, or terminate with a clear user-facing message.

Raising and Chaining Exceptions

```python

def positive_integer(value, field_name):

try:

number = int(value)

except (TypeError, ValueError) as error:

raise ValueError(f"{field_name} must be an integer") from error

if number <= 0:

raise ValueError(f"{field_name} must be positive")

return number

```

The phrase `from error` preserves exception chaining. The low-level cause remains available for diagnosis while the new exception communicates domain meaning.

Custom Exception Hierarchies

Custom exceptions make an application’s failure vocabulary explicit:

```python

class DataPipelineError(Exception):

"""Base class for expected pipeline failures."""

class InputFormatError(DataPipelineError):

pass

class RecordValidationError(DataPipelineError):

pass

```

A caller can handle all expected pipeline failures with `except DataPipelineError`, or handle a particular category with a more specific class. Exceptions should represent exceptional conditions, not ordinary branching such as “search found no result” when absence is expected and can be represented by `None` or an empty collection.

8. Building a Fault-Tolerant Input Workflow

A dependable workflow separates stages:

1. Locate and open the resource.

2. Parse its representation.

3. Validate structure and domain rules.

4. Transform valid data.

5. Persist results safely.

6. Report failures with sufficient context.

Consider a simple JSON record loader:

```python

import json

from pathlib import Path

class DataPipelineError(Exception):

pass

class InputFormatError(DataPipelineError):

pass

class RecordValidationError(DataPipelineError):

pass

def load_records(path: Path) -> list[dict]:

try:

with path.open(encoding="utf-8") as file:

data = json.load(file)

except FileNotFoundError as error:

raise InputFormatError(f"Missing input: {path}") from error

except json.JSONDecodeError as error:

raise InputFormatError(f"Invalid JSON in {path}") from error

except OSError as error:

raise InputFormatError(f"Cannot read {path}: {error}") from error

if not isinstance(data, list):

raise RecordValidationError("The top-level JSON value must be a list")

for index, record in enumerate(data):

if not isinstance(record, dict):

raise RecordValidationError(f"Record {index} is not an object")

if "id" not in record or "value" not in record:

raise RecordValidationError(f"Record {index} lacks a required field")

return data

```

This function does not catch every possible error. A `KeyboardInterrupt` is not normally an input-format problem, and a programming error should not be silently converted into a data error. The function also distinguishes access/parsing failure from validation failure, allowing callers to choose different responses.

Atomic Output

Writing directly to a destination can leave a partial file if the process crashes. A common strategy is to write a temporary file in the same directory and replace the destination after successful completion:

```python

import os

import tempfile

from pathlib import Path

def atomic_write(path: Path, text: str) -> None:

with tempfile.NamedTemporaryFile(

mode="w", encoding="utf-8", dir=path.parent,

delete=False

) as temporary:

temporary.write(text)

temporary_name = temporary.name

try:

os.replace(temporary_name, path)

except Exception:

Path(temporary_name).unlink(missing_ok=True)

raise

```

`os.replace()` is designed to replace the destination as one filesystem operation when source and destination are on the same filesystem. Atomic replacement reduces the chance that readers observe an incomplete output, but it does not solve all durability, concurrency, or distributed-storage problems.

9. Worked Example: A Reliable CSV-to-JSON Pipeline

Suppose a training department receives a CSV file containing student identifiers and assessment scores. The required rules are: each record must contain a nonempty identifier, the score must be an integer from 0 through 100, and invalid rows must be reported without stopping valid rows from being processed.

```python

import csv

import json

from pathlib import Path

class RowError(Exception):

pass

def convert(input_path: Path, output_path: Path, error_path: Path) -> None:

valid = []

errors = []

try:

source = input_path.open("r", newline="", encoding="utf-8")

except OSError as error:

raise RowError(f"Unable to open input: {input_path}") from error

with source:

reader = csv.DictReader(source)

required = {"student_id", "score"}

if not required.issubset(reader.fieldnames or set()):

raise RowError("CSV is missing required columns")

for line_number, row in enumerate(reader, start=2):

try:

student_id = row["student_id"].strip()

score = int(row["score"])

if not student_id:

raise ValueError("empty student_id")

if not 0 <= score <= 100:

raise ValueError("score outside 0..100")

valid.append({"student_id": student_id, "score": score})

except (AttributeError, TypeError, ValueError) as error:

errors.append({"line": line_number, "reason": str(error)})

output_path.write_text(json.dumps(valid, indent=2), encoding="utf-8")

error_path.write_text(json.dumps(errors, indent=2), encoding="utf-8")

```

The design separates file-level failures from row-level failures. If the input cannot be opened or its required columns are absent, the operation fails clearly. If one score is malformed, the row is recorded as an error while other rows continue. In production, output writing should itself use temporary files, and permissions, disk capacity, and concurrent writers should be considered.

10. Case Study: Model Configuration in an AI Service

This is an illustrative case. A small image-classification service stores model configuration in JSON and receives periodic updates from an operations team.

Background. The service requires a model path, a positive batch size, and a threshold between 0 and 1. A malformed configuration can prevent startup or, worse, cause incorrect predictions.

Problem. An early implementation used `pickle` for convenience and caught all exceptions during loading, replacing a bad configuration with defaults. This concealed permission failures and allowed silent behavior changes.

Analysis. Pickle created an unnecessary security risk because configuration did not require arbitrary Python objects. Catching all exceptions made diagnosis difficult. The application also failed to distinguish syntactic JSON errors from invalid threshold values.

Approach. The team adopted UTF-8 JSON with a `schema_version` field, validated the top-level structure, raised custom `ConfigurationError` subclasses, and logged failures without exposing secrets. Updates were written to a temporary file and atomically replaced only after validation succeeded. The service retained the last known valid configuration and required explicit operator approval before rollback.

Outcome. Untrusted object deserialization was removed, malformed updates were rejected before activation, and operational logs identified whether a failure involved access, syntax, or domain validation. The design did not eliminate all risks: model files still required integrity checks, and concurrent deployment coordination remained necessary.

Learning points. Select formats according to trust and interoperability requirements; validate after parsing; do not hide failures with broad exception handlers; and treat configuration updates as controlled state transitions.

11. Limitations, Governance, and Best Practices

File-based systems are simple but can suffer from corruption, concurrent access, slow random access, unclear schemas, and difficult backups. Large-scale or transactional workloads may require databases or object stores. CSV is interoperable but weakly typed. JSON is flexible but can become verbose and ambiguous without schemas. Pickle is Python-specific and unsafe for untrusted input.

Data governance is also relevant. Files may contain personal, educational, financial, or health information. Apply least-privilege permissions, encryption where appropriate, retention rules, access logging, and secure deletion policies. Do not write passwords, access tokens, or unnecessary personal data into diagnostic logs. Validate filenames and content from untrusted sources, and consider denial-of-service risks from enormous files, deeply nested JSON, or decompression bombs.

Recommended practices include:

• Specify text encodings explicitly.
• Use `with` for resource management.
• Prefer `pathlib` over manual path concatenation.
• Stream large files and impose reasonable size limits.
• Parse and validate in separate stages.
• Catch the narrowest meaningful exception.
• Preserve causes with exception chaining.
• Use custom exceptions for domain-level failures.
• Never unpickle untrusted data.
• Version durable schemas.
• Test missing files, permissions, malformed data, empty files, duplicate records, partial writes, and interrupted operations.
• Report errors with context while avoiding sensitive information.

Contemporary Developments

Modern data systems increasingly use typed schemas, data contracts, object storage, columnar formats, and validation libraries. Formats such as Parquet can be more efficient than CSV for analytical workloads, while newline-delimited JSON supports incremental event processing. Cloud applications often replace local paths with authenticated object-storage APIs, introducing retries, eventual consistency, and network-specific exceptions. Machine-learning systems also serialize model weights, tokenizers, feature definitions, and experiment metadata; compatibility and provenance must be managed together.

Python’s standard library remains valuable because its interfaces are portable and auditable. External libraries may add schema validation, compression, encryption, or high-performance parsing, but they do not remove the need for careful resource management, validation, security, and exception design.

Chapter Summary

Files provide persistent external state, while serialization provides representations that can be stored or transmitted. Text files require deliberate encoding choices; binary files require byte-oriented operations. `pathlib` improves path portability, and context managers ensure that resources are closed reliably. CSV processing should use the `csv` module rather than naïve string splitting, and JSON parsing should be followed by domain validation because syntactic validity does not guarantee correct data.

Serialization formats differ in portability, type support, security, and durability. JSON is useful for interoperable structured data, whereas pickle is Python-specific and unsafe for untrusted input. Robust exception handling uses specific exception classes, custom domain hierarchies, exception chaining, and clear separation between recovery and diagnosis. Fault-tolerant workflows distinguish resource failures, parsing failures, validation failures, and output failures. Atomic output, schema versioning, testing, access control, and careful logging strengthen reliability in professional and AI/ML applications.

Key Terms

Encoding: A rule for converting characters to bytes and back.

Binary mode: File mode that reads and writes bytes without text decoding.

Buffer: Temporary memory used to improve input/output efficiency.

Path: An identifier for a filesystem location.

Context manager: An object controlling setup and cleanup around a code block.

CSV: A delimited text format commonly used for tabular data.

JSON: A language-independent format for structured data.

Serialization: Conversion of an object into a storable or transferable representation.

Deserialization: Reconstruction of data from a serialized representation.

Schema: A specification of expected structure, fields, and types.

Pickle: Python-specific serialization supporting many Python objects.

Exception: An object representing an abnormal or exceptional condition.

Exception hierarchy: Inheritance structure used to classify exceptions.

Exception chaining: Preservation of an original cause when raising a new exception.

Validation: Checking data against structural and domain rules.

Atomic replacement: Replacing a destination in one filesystem operation to reduce partial-output risk.

Path traversal: An attack that attempts to access files outside an intended directory.

Fault tolerance: The ability to continue or fail safely when errors occur.

Review Questions

Short Answer Questions

1. What is the difference between text mode and binary mode?

2. Why should `newline=""` commonly be used with Python’s CSV module?

3. What is the purpose of a context manager when working with files?

4. Why is unpickling untrusted data dangerous?

5. What is exception chaining, and why is it useful?

Long Answer Questions

1. Explain the major file modes in Python and discuss situations in which using `w` instead of `a` could cause data loss.

2. Compare CSV, JSON, and pickle in terms of structure, portability, type support, and security.

3. Describe a staged input workflow from opening a file through parsing, validation, transformation, and output.

4. Explain how custom exception hierarchies improve the design of a data-processing application.

5. Discuss how encoding, buffering, streaming, and atomic output affect the reliability of file-based systems.

Analytical / Application Questions

1. Design a secure procedure for loading a JSON configuration supplied by an external user. Identify validation, path, size, and exception-handling controls.

2. A CSV contains 100,000 rows, of which 2,000 have invalid scores. Propose a workflow that preserves valid records, reports invalid rows, limits memory use, and avoids partial output.

3. An application sometimes replaces a valid model configuration with defaults when a permission error occurs. Analyze the design flaw and propose a revised exception and recovery strategy.

Multiple Choice Questions with Answers

Question 1

Which file mode truncates an existing file before writing?

A. r

B. a

C. w

D. x

Correct Answer: C. w

Question 2

Why is an explicit encoding such as UTF-8 recommended when opening text files?

A. It guarantees the file is compressed

B. It improves portability across systems

C. It converts all values to integers

D. It prevents every possible I/O error

Correct Answer: B. It improves portability across systems

Question 3

Which module is designed to correctly handle quoted fields and embedded commas in CSV data?

A. csv

B. pickle

C. pathlib

D. math

Correct Answer: A. csv

Question 4

What does json.load() expect as its primary input?

A. A class definition

B. A binary image

C. A file-like object containing JSON

D. A list of exception types

Correct Answer: C. A file-like object containing JSON

Question 5

Why should untrusted data not be passed to pickle.load()?

A. Pickle can represent behavior that enables code execution

B. Pickle only supports plain text

C. Pickle cannot serialize dictionaries

D. Pickle always deletes the source file

Correct Answer: A. Pickle can represent behavior that enables code execution

Question 6

Which block executes whether or not an exception occurs?

A. else

B. try

C. except

D. finally

Correct Answer: D. finally

Question 7

What is the principal advantage of using pathlib over manually concatenating path strings?

A. It automatically encrypts files

B. It provides a more portable path abstraction

C. It eliminates the need for permissions

D. It converts CSV into JSON

Correct Answer: B. It provides a more portable path abstraction

Question 8

What should usually be caught first in an exception handler?

A. The most specific relevant exception

B. BaseException in every case

C. Only SyntaxError

D. Every exception using an empty handler

Correct Answer: A. The most specific relevant exception

Question 9

Which operation is most appropriate for copying a large binary file with bounded memory use?

A. Read the entire file as text

B. Read and write fixed-size byte chunks

C. Split the filename at commas

D. Deserialize it with json.load()

Correct Answer: B. Read and write fixed-size byte chunks

Question 10

What is the main purpose of validating JSON after it has been parsed successfully?

A. To ensure the data satisfies required structure and domain rules

B. To convert every string into a path

C. To make the file binary

D. To guarantee that no disk failure can occur

Correct Answer: A. To ensure the data satisfies required structure and domain rules