Structured Data and Serialization

Introduction

Most useful programs exchange data with other programs, store information between executions, or move information across networks. A Python application may read customer records from a CSV file, load settings from JSON, preserve a trained model, or communicate with a Java service. These activities depend on structured data: data organised according to identifiable fields, types, and relationships.

Serialization is the process of converting an in-memory object or data structure into a storable or transferable representation. Deserialization reverses that process. The representation may be text-based, such as CSV or JSON, or binary, such as a database page, protocol buffer, or Python pickle stream.

Choosing a representation is not merely a programming convenience. It affects interoperability, precision, storage size, performance, validation, long-term maintainability, and security. This chapter develops practical techniques using Python’s standard library and provides criteria for selecting an appropriate format.

Learning Objectives

By the end of this chapter, you should be able to:

• read and write CSV data reliably using csv;
• encode and decode JSON using json;
• design configuration files that are clear, validated, and maintainable;
• distinguish text serialization from binary serialization;
• explain the strengths and limitations of pickle;
• preserve schema consistency and handle missing, extra, or incorrectly typed fields;
• evaluate interoperability, precision, performance, and security risks;
• construct a robust data-processing workflow using standard Python libraries.

1. Data Representation and Serialization

An in-memory Python object has a type, structure, and behaviour. For example:

record = {
    "student_id": 1042,
    "name": "Amina Khan",
    "active": True,
    "modules": ["Python", "Databases"]
}

This dictionary is convenient inside Python, but another program cannot automatically access Python’s memory. Serialization creates an external representation, such as:

{"student_id": 1042, "name": "Amina Khan", "active": true, "modules": ["Python", "Databases"]}

A useful serialization design should answer five questions:

1. What is the schema? Which fields exist, and what types do they have?
2. Who will consume the data? Python, Java, a spreadsheet, a person, or a web service?
3. How durable must the representation be? Is it temporary or an archival format?
4. What precision is required? Are monetary values, dates, or scientific measurements involved?
5. What threats exist? Could an attacker control or modify the serialized data?

A format is not automatically superior because it is compact or convenient. A human-readable format may be preferable for configuration, while a binary format may be suitable for controlled internal caching.

2. CSV: Tabular Data for Interchange

CSV, or comma-separated values, represents a table as rows and columns. The first row often contains column headings. CSV is common because it is supported by spreadsheet applications, databases, statistical tools, and programming languages.

A simple file might be:

student_id,name,mark
101,Amina Khan,82
102,Jon Bell,74

CSV has no universal complete standard in practice. Variations include delimiters, quoting rules, line endings, character encodings, and whether a header exists. Python’s csv module handles these details more safely than manual string splitting.

Reading CSV with dictionaries

import csv

with open("marks.csv", "r", newline="", encoding="utf-8") as file:
    reader = csv.DictReader(file)
    for row in reader:
        student_id = int(row["student_id"])
        mark = float(row["mark"])
        print(student_id, row["name"], mark)

DictReader maps column headings to values. However, every value initially arrives as a string. Conversion must therefore be explicit and validated.

Writing CSV

import csv

records = [
    {"student_id": 101, "name": "Amina Khan", "mark": 82},
    {"student_id": 102, "name": "Jon Bell", "mark": 74},
]

with open("marks.csv", "w", newline="", encoding="utf-8") as file:
    fieldnames = ["student_id", "name", "mark"]
    writer = csv.DictWriter(file, fieldnames=fieldnames)
    writer.writeheader()
    writer.writerows(records)

The newline="" argument is recommended by the documentation for the csv module because it allows the module to manage newline translation correctly on different platforms.

Quoting and dialects

Fields containing commas, quotation marks, or line breaks must be quoted. The module handles this automatically:

import csv

with open("people.csv", "w", newline="", encoding="utf-8") as file:
    writer = csv.writer(file)
    writer.writerow(["name", "comment"])
    writer.writerow(["Lee", "Excellent, reliable work"])

For semicolon-separated files, specify a delimiter:

reader = csv.DictReader(file, delimiter=";")

CSV limitations

CSV does not intrinsically record types. The string 2025-04-01 may be a date, identifier, or ordinary text. It also has weak support for nested objects, null values, and relationships. Two systems may interpret an empty field differently: as missing, empty text, or zero.

CSV is therefore appropriate for flat tables and broad human interoperability, but not for complex hierarchical structures or strict self-describing data.

3. JSON: Structured, Interoperable Text

JSON (JavaScript Object Notation) represents objects, arrays, strings, numbers, Boolean values, and null. Python’s corresponding values are dictionaries, lists, strings, integers or floating-point numbers, True and False, and None.

Encoding and decoding

import json

record = {
    "student_id": 101,
    "name": "Amina Khan",
    "active": True,
    "modules": ["Python", "Databases"]
}

with open("student.json", "w", encoding="utf-8") as file:
    json.dump(record, file, indent=2)

with open("student.json", "r", encoding="utf-8") as file:
    loaded = json.load(file)

dump and load operate on files. dumps and loads operate on strings:

text = json.dumps(record)
copy = json.loads(text)

Indentation improves readability but increases file size. For machine-to-machine transfer, compact output may be preferable.

JSON and custom types

JSON does not natively represent Python sets, tuples, dates, decimals, or arbitrary class instances. A common solution is to define an explicit representation:

from datetime import date
import json

item = {
    "description": "Subscription",
    "start_date": date.today().isoformat()
}

text = json.dumps(item)

The ISO 8601 date representation is readable and widely interoperable. On reading, the application can validate and convert it back to a date object.

Validation and schema consistency

Successful JSON parsing does not prove that the data is acceptable. This parses successfully:

{"student_id": "unknown", "mark": -20}

A reliable application performs structural and semantic validation:

def validate_mark_record(record):
    required = {"student_id", "name", "mark"}
    missing = required - record.keys()
    if missing:
        raise ValueError(f"Missing fields: {sorted(missing)}")

    if not isinstance(record["student_id"], int):
        raise TypeError("student_id must be an integer")
    if not isinstance(record["name"], str) or not record["name"].strip():
        raise ValueError("name must be non-empty text")
    if not isinstance(record["mark"], (int, float)):
        raise TypeError("mark must be numeric")
    if not 0 <= record["mark"] <= 100:
        raise ValueError("mark must be between 0 and 100")

A production system should also decide whether unknown fields are rejected, ignored, or preserved. Rejecting them detects mistakes early; allowing them can support forward compatibility. The policy should be documented rather than accidental.

JSON numbers and precision

JSON specifies numbers but does not require every implementation to use the same numeric representation. Python normally decodes JSON numbers into int or float. Binary floating-point cannot represent every decimal fraction exactly. For example, a value such as 0.1 may be stored as the nearest representable binary fraction.

For financial values, storing a decimal amount as a string or as an integer number of minor units is often safer:

{"amount_minor": 1099, "currency": "GBP"}

Here, 1099 means 1,099 pence, provided the schema explicitly states the unit. Alternatively, Python’s decimal.Decimal can be used with a controlled conversion process. Precision requirements must be agreed by all systems, not solved only at the file-writing stage.

4. Configuration Files

Configuration separates changeable deployment choices from program code. Examples include database addresses, logging levels, feature flags, and file locations. JSON is suitable for many applications because it is standard, readable, and available without third-party packages.

Example configuration:

{
  "service": {
    "host": "localhost",
    "port": 8080
  },
  "logging": {
    "level": "INFO"
  },
  "features": {
    "new_report": false
  }
}

A robust configuration loader should distinguish at least three failures:

• FileNotFoundError: the configuration file is absent;
• json.JSONDecodeError: the file is not valid JSON;
• KeyError, TypeError, or ValueError: the structure or values are invalid.

import json

DEFAULTS = {"host": "localhost", "port": 8000, "log_level": "INFO"}


def load_config(path):
    try:
        with open(path, encoding="utf-8") as file:
            data = json.load(file)
    except FileNotFoundError:
        return DEFAULTS.copy()
    except json.JSONDecodeError as error:
        raise ValueError(f"Invalid configuration syntax: {error}") from error

    service = data.get("service", {})
    host = service.get("host", DEFAULTS["host"])
    port = service.get("port", DEFAULTS["port"])

    if not isinstance(host, str):
        raise TypeError("service.host must be text")
    if not isinstance(port, int) or not 1 <= port <= 65535:
        raise ValueError("service.port must be an integer from 1 to 65535")

    return {"host": host, "port": port}

Configuration should not normally contain passwords or private keys in a repository. Use environment variables, an operating-system secret store, or a dedicated secrets-management system. File permissions should restrict access to sensitive configuration.

5. Binary Data and the pickle Module

Binary data is represented as bytes rather than ordinary text. Images, compressed files, executables, and many network protocols are binary. Python provides the bytes type and functions such as open(..., "rb") and open(..., "wb").

with open("input.bin", "rb") as source:
    data = source.read()

with open("copy.bin", "wb") as destination:
    destination.write(data)

Pickle

pickle serializes many Python objects into a Python-specific binary stream:

import pickle

model_state = {"weights": [0.2, 0.8], "name": "demo"}

with open("state.pkl", "wb") as file:
    pickle.dump(model_state, file)

with open("state.pkl", "rb") as file:
    restored = pickle.load(file)

Pickle can preserve object graphs, shared references, and many Python-specific types. It is useful for controlled temporary storage, testing, and communication between trusted Python processes using compatible code versions.

However, unpickling is not merely data parsing. A malicious pickle can cause code execution during loading. Never unpickle data received from an untrusted user, public download, email attachment, or unverified service. Authentication, file permissions, and a trusted origin do not make arbitrary external pickle data automatically safe; the loader must still have a justified trust boundary.

Pickle is also Python-specific and can be fragile across versions, class changes, and package changes. It is usually unsuitable for public APIs, long-term archives, or interoperability with Java and non-Python systems. JSON, CSV, or a deliberately specified binary protocol is normally more appropriate for those purposes.

6. Serialization Choices: A Decision Framework

The following comparison supports practical selection.

A decision can be expressed as a sequence of questions:

1. Is the data a simple table intended for spreadsheets or many languages? Choose CSV.
2. Is the data hierarchical or intended for web and service integration? Choose JSON.
3. Must Python-specific object structure be retained, and is the source fully trusted? Pickle may be suitable.
4. Are extreme performance, compactness, or specialised types required? Define or adopt a documented binary protocol.

The final choice should also consider file size, processing speed, schema evolution, audit requirements, and the cost of recovering from corruption.

7. Robust File-Processing Patterns

Context managers and encoding

Always use with when opening files so that resources are closed even when an exception occurs. Specify the encoding for text files, usually UTF-8 when interoperating across systems.

Atomic replacement

Writing directly to an important configuration file can leave a partially written file if the process stops during writing. A safer pattern is to write a temporary file in the same directory, flush and close it, then replace the original using os.replace.

import json
import os
import tempfile


def save_json_atomically(path, value):
    directory = os.path.dirname(os.path.abspath(path))
    fd, temporary = tempfile.mkstemp(dir=directory, prefix=".tmp-")
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as file:
            json.dump(value, file, indent=2)
            file.write("\n")
        os.replace(temporary, path)
    except Exception:
        try:
            os.unlink(temporary)
        except FileNotFoundError:
            pass
        raise

os.replace provides a stronger update pattern on ordinary local filesystems than overwriting the target in place, although durability guarantees can vary by operating system and storage system.

Exception boundaries

Catch exceptions at the level where the program can respond meaningfully. Do not use a broad except Exception merely to hide failures. Report malformed input with context, preserve the original exception using raise ... from error, and avoid logging passwords or personal data.

Streaming large files

json.load reads a complete JSON document into memory. For very large datasets, line-oriented JSON objects, CSV iteration, or a database may be more appropriate. A CSV reader naturally supports streaming:

with open("large.csv", newline="", encoding="utf-8") as file:
    for row in csv.DictReader(file):
        process(row)

A standard JSON document is not generally streamable one object at a time without an agreed framing convention. One possible convention is one JSON object per line, commonly called JSON Lines, but the producer and consumer must explicitly agree on it.

8. Schema Evolution and Interoperability

A schema is a contract describing fields, types, constraints, units, and meanings. Schema evolution occurs when a contract changes. Adding an optional field is usually less disruptive than renaming or changing the meaning of an existing field.

Professional practices include:

• include a format or schema version;
• document required and optional fields;
• define units, time zones, and date formats;
• specify whether unknown fields are accepted;
• maintain backward-compatible readers where feasible;
• test representative old and new files;
• avoid relying on dictionary ordering or incidental formatting;
• use stable identifiers rather than display names.

Dates are a frequent interoperability problem. A date-time should specify its time zone or use UTC by agreement. A number should specify its unit and precision. A field named amount is ambiguous unless the schema states currency and scaling.

9. Applied Case: Student Assessment Exchange

The following is an illustrative case.

Background

A college stores assessment results in Python. The registry exports marks to a spreadsheet team, while a Java reporting service receives data through a shared directory. The original implementation writes comma-separated strings manually and occasionally produces incorrect columns when a student’s name contains a comma.

Problem

The college needs reliable transfer of student identifiers, module codes, marks, and assessment dates. It must preserve numeric meaning, reject invalid marks, and allow future fields such as moderation status.

Analysis

CSV is appropriate for the spreadsheet export because the data is tabular. JSON is more suitable for the Java service because it can represent a record with metadata and a list of assessment components. Pickle is rejected because Java cannot naturally consume it and because a shared directory does not establish a trusted pickle boundary.

The team defines the following JSON contract:

{
  "schema_version": 1,
  "student_id": "S1042",
  "module_code": "PY401",
  "mark": 82,
  "assessment_date": "2025-05-14",
  "currency": null
}

The currency field is shown only to emphasise that irrelevant fields should not be added casually; an actual assessment schema would omit it. The mark is an integer from 0 through 100, and the date uses ISO format. The identifier remains a string because leading zeroes may be meaningful.

Approach and solution

The exporter uses csv.DictWriter for the spreadsheet and json.dump for service records. Both outputs are generated from the same validated internal record. Validation occurs before serialization, so the two formats cannot silently disagree about a malformed mark. The service reader accepts version 1 and rejects unknown schema versions with a clear error.

Outcome

Names containing commas are correctly quoted in CSV. Java reads the JSON without depending on Python-specific types. Invalid marks are rejected before publication, and later schema changes can be handled through a versioned contract.

Learning points

The central design decision was not a choice between fashionable formats. It was matching each representation to its consumer while defining a shared semantic contract. Serialization is successful only when the receiving system interprets values in the intended way.

10. Security, Ethics, and Governance

Serialized data may contain personal, educational, financial, or operational information. Developers should apply data minimisation: store only what is needed. Access controls, encryption during transfer and at rest, retention limits, and audit trails may be required by organisational policy or data-protection law.

Parsing untrusted data can also create denial-of-service risks. Very large JSON documents, deeply nested structures, or intentionally repeated input may consume excessive memory or processing time. Applications should impose appropriate file-size, record-count, and nesting limits where the threat model requires them.

Validation is not only a correctness measure. It is a security boundary. Treat external data as untrusted, reject unexpected types, avoid constructing executable code from fields, and never confuse serialization with authentication. A signed file can provide integrity or origin evidence only when signatures and key management are implemented correctly.

11. Contemporary Practice

Modern systems increasingly use schema registries, API contracts, versioned event formats, and automated compatibility tests. JSON remains dominant for many web APIs, while specialised binary formats are used when compactness, speed, or strict schemas are important. Python’s standard library is sufficient for CSV, JSON, bytes, and pickle, but larger systems may adopt validated data models, databases, or organisation-wide serialization protocols.

Artificial intelligence systems also depend heavily on serialization. A model checkpoint may contain executable or framework-specific state and therefore requires a stronger trust policy than an ordinary JSON prediction record. Model files should be obtained from trusted sources, verified where possible, isolated during loading, and documented with framework and version metadata.

Best-Practice Checklist

• Define the schema before implementing the writer.
• Use csv rather than splitting lines manually.
• Use UTF-8 and explicit newline handling for text files.
• Validate both structure and meaning after parsing.
• Document dates, time zones, units, precision, and missing values.
• Version formats that will evolve.
• Use atomic replacement for important files.
• Stream large tabular files where practical.
• Keep secrets out of ordinary configuration files.
• Never unpickle untrusted data.
• Choose interoperability and longevity over convenience when data leaves Python.
• Test malformed, incomplete, extra-field, and legacy inputs.

Chapter Summary

Structured data allows applications to store and exchange information in a predictable form. CSV is effective for flat tables and broad tool compatibility, but it provides little type information. JSON supports hierarchical, readable, language-independent data, yet still requires explicit validation and careful treatment of numbers, dates, and custom types. Configuration files should be loaded with clear exception handling, defaults, validation, and secure handling of secrets.

Binary data is represented as bytes, while pickle preserves many Python-specific objects. Pickle is convenient only within a trusted and controlled Python environment; unpickling untrusted content can execute malicious code. Serialization choices should be evaluated against schema clarity, interoperability, precision, performance, evolution, and security. Professional implementations use context managers, explicit encodings, atomic updates, streaming where suitable, and documented versioned contracts.

Key Terms

Serialization: Conversion of an in-memory object into a storable or transferable representation.

Deserialization: Reconstruction of an in-memory value from serialized data.

Schema: A documented description of fields, types, constraints, units, and relationships.

CSV: A text format representing tabular rows and columns.

Delimiter: A character separating fields in a tabular text format.

Quoting: Enclosing field content so delimiters or line breaks are treated as data.

JSON: A text-based format for objects, arrays, strings, numbers, Booleans, and null.

Interoperability: The ability of different systems to exchange and interpret data.

Validation: Checking that input has acceptable structure, types, and values.

Schema evolution: Modification of a data contract over time.

Precision: The degree to which a stored value represents the intended value.

Binary data: Data represented as bytes rather than ordinary text.

Pickle: Python-specific binary serialization supporting many Python object types.

Trust boundary: A point at which data changes from trusted to untrusted or vice versa.

Atomic replacement: Updating a file by preparing a complete temporary file and replacing the target.

Encoding: The mapping between characters and bytes.

Streaming: Processing data incrementally rather than loading it all into memory.

Configuration: External settings controlling application behaviour.

JSON Lines: A convention in which each line contains a separate JSON value, usually an object.

Data minimisation: Storing or processing no more personal or sensitive data than necessary.

Review Questions

Short Answer Questions

1. Why should CSV values be explicitly converted after reading?
2. What is the difference between json.dump and json.dumps?
3. Why is pickle.load dangerous for untrusted input?
4. Give two examples of information that a schema should document besides field names.
5. What problem does atomic file replacement help prevent?

Long Answer Questions

1. Compare CSV and JSON for a data exchange between a spreadsheet user and a web service.
2. Explain how a robust configuration loader should handle missing files, invalid syntax, and invalid values.
3. Discuss the effects of floating-point representation on serialized numeric data and recommend a strategy for monetary amounts.
4. Explain schema evolution and describe practices that support backward compatibility.
5. Evaluate the advantages and limitations of pickle for temporary storage, public APIs, and long-term archives.

Analytical / Application Questions

1. Design a serialization strategy for a Python application that exports flat sales data to analysts and nested order data to a Java service. Justify the formats, validation rules, and versioning policy.
2. A JSON file contains a valid object but uses a string for an identifier, a negative quantity, an ambiguous local date, and an unknown field. Analyse which issues are syntactic and which are semantic, then propose a validation response.
3. An organisation receives model files from an external supplier. Assess whether loading them with pickle is appropriate and design a safer trust, verification, and isolation process.

Multiple Choice Questions with Answers

Question 1

Which Python class is most appropriate for reading CSV rows using column names?

A. csv.DictReader

B. json.JSONDecoder

C. pickle.Unpickler

D. io.BytesIO

Correct Answer: A. csv.DictReader

Question 2

What does json.dumps() return?

A. A Python dictionary

B. A JSON-formatted string

C. A binary pickle stream

D. A file object

Correct Answer: B. A JSON-formatted string

Question 3

Why must JSON data be validated after successful parsing?

A. Parsing automatically converts all values to integers

B. Valid JSON syntax does not guarantee correct types or business values

C. JSON cannot contain objects

D. Validation is required only for CSV

Correct Answer: B. Valid JSON syntax does not guarantee correct types or business values

Question 4

Which format is generally most suitable for a flat table intended for spreadsheet users?

A. CSV

B. Pickle

C. Python bytecode

D. An arbitrary memory dump

Correct Answer: A. CSV

Question 5

What is the principal security risk of loading an untrusted pickle file?

A. It always loses Unicode characters

B. It cannot represent dictionaries

C. It may execute malicious code during loading

D. It changes every integer into a float

Correct Answer: C. It may execute malicious code during loading

Question 6

Which technique best supports safe replacement of an important configuration file?

A. Overwrite it one character at a time

B. Write a temporary complete file and use os.replace

C. Convert it to CSV first

D. Load it with pickle

Correct Answer: B. Write a temporary complete file and use os.replace

Question 7

What does csv.DictReader initially return for ordinary field values?

A. Strings

B. Only integers

C. Python dates

D. Bytecode instructions

Correct Answer: A. Strings

Question 8

Which representation is usually best for a Python-specific object graph exchanged only between trusted compatible Python processes?

A. Pickle

B. CSV

C. Plain text with no schema

D. An HTML table

Correct Answer: A. Pickle

Question 9

Why should a monetary JSON value include a unit or currency definition?

A. JSON cannot store numbers

B. The meaning and scale of an amount are otherwise ambiguous

C. Currency makes JSON executable

D. It prevents all file corruption

Correct Answer: B. The meaning and scale of an amount are otherwise ambiguous

Question 10

Which practice most directly improves interoperability across programming languages?

A. Relying on private Python class names

B. Using a documented, versioned schema with explicit types and units

C. Serializing arbitrary memory addresses

D. Omitting field names to reduce size

Correct Answer: B. Using a documented, versioned schema with explicit types and units