Files, Data Formats, Exceptions, and Robustness

Introduction

Engineering software rarely operates only on values typed at a keyboard. It reads measurements from sensors, imports calibration tables, stores simulation results, exchanges configuration files, and produces reports for people and other systems. These activities are forms of data processing. A program that produces correct results for ideal input but fails when a file is missing, a value is malformed, or a device is disconnected is not a reliable engineering program.

This chapter develops practical foundations for handling text, binary, tabular, and structured data. It also explains exceptions, error codes, logging, validation, serialization, and defensive programming. The examples use Python because its file and exception facilities are concise, but the underlying principles apply to C, C++, Java, MATLAB, and other engineering languages.

A useful model is:

source → acquire → validate → transform → store/transmit → report
                         ↓
                 detect, record, recover

Every stage can introduce errors. Robust software detects errors early, preserves useful diagnostic information, avoids corrupting data, and fails in a controlled way.

Learning Objectives

By the end of this chapter, a student should be able to:

1. Distinguish text, binary, tabular, and structured files.
2. Open, read, write, append, and close files safely.
3. Select suitable data formats for engineering applications.
4. Use serialization and deserialization appropriately.
5. Design validation rules for values, records, and complete files.
6. Explain exceptions, error codes, and logging and select among them.
7. Develop defensive programs that handle malformed input and partial failure.
8. Construct a reliable data pipeline and evaluate its limitations.

1. Files as Persistent Data

A variable normally exists only while a program is running. A file provides persistent storage: a sequence of bytes associated with a name and managed by an operating system or storage device. The same bytes can represent human-readable characters, an image, floating-point values, or a structured document depending on the agreed format.

A file has at least three important properties:

• Name and location: its path in a file system.
• Content: the bytes stored in the file.
• Metadata: information such as size, timestamps, permissions, and sometimes ownership.

A path may be absolute, such as /data/logs/run01.csv, or relative to the program's current working directory. Programs should not assume that the current directory is fixed. Configuration or command-line arguments are generally safer than hard-coded paths.

Text and binary files

A text file stores characters encoded as bytes. UTF-8 is a widely used encoding because it represents ASCII characters directly and supports many international scripts. A binary file stores bytes according to an application-specific representation. Images, compressed archives, executable programs, and many instrument formats are binary.

Text is convenient for inspection, debugging, and interoperability. Binary data is often more compact and faster to process, but it requires a precise specification. A binary file must define byte order, field sizes, signedness, floating-point representation, and record layout. For example, four bytes may represent an integer, four ASCII characters, or part of a floating-point value; the bytes alone do not determine the meaning.

File modes and resource management

Common modes include r for reading, w for writing and replacing existing content, a for appending, and x for creating a new file only if it does not already exist. A mode can include b for binary operation and t for text operation. In Python, text mode performs decoding and newline handling; binary mode returns byte sequences.

Use a context manager so that the file is closed even if an exception occurs:

with open("measurements.csv", "r", encoding="utf-8") as file:
    for line in file:
        process(line)

Closing a file releases operating-system resources and ensures buffered output is flushed. Explicitly calling close() is possible, but it is easier to omit accidentally when a program has multiple exit paths.

Writing should be deliberate. Mode w can destroy valuable data. A safer pattern is to write a temporary file, flush and close it, and then replace the destination. In systems where power loss matters, an atomic replacement strategy and, where appropriate, an operating-system synchronization operation reduce the chance of leaving a partially written file.

2. Data Formats for Engineering Work

A format is a contract between the producer and consumer of data. It should specify field names, units, valid ranges, missing-value conventions, encoding, time representation, and version information.

Delimited tabular data

CSV and TSV files represent records as rows and fields as columns. They are suitable for measurements, laboratory results, and data exchange with spreadsheets. A correct CSV reader should handle quoted fields, embedded delimiters, and escaped quotation marks; splitting each line with str.split(',') is not a complete CSV parser.

import csv

with open("temperature.csv", newline="", encoding="utf-8") as file:
    reader = csv.DictReader(file)
    for row in reader:
        temperature = float(row["temperature_C"])
        print(row["time_s"], temperature)

A table should have a clear schema. For example:

CSV is simple but weakly typed. It does not inherently preserve whether 0012 is an identifier or the number 12, and it has no universal standard for units or missing values. These details must be documented or represented in a separate schema.

Structured text

JSON represents objects, arrays, strings, numbers, Boolean values, and null values. It is useful for configuration, web APIs, and moderately complex records.

{
  "device": "motor_07",
  "sample_rate_hz": 1000,
  "channels": ["current_A", "speed_rpm"],
  "enabled": true
}

YAML is also human-oriented and supports richer presentation, but parsers and security settings vary. XML is verbose but remains important where schemas, namespaces, and document-oriented interoperability are required.

JSON is not automatically safe simply because it is text. Input must still be validated, and applications should limit size and nesting depth when data may come from untrusted sources.

Binary formats

Binary formats are appropriate when compactness, speed, exact bit-level representation, or high-volume acquisition is important. Examples include images, audio, compressed scientific arrays, and proprietary instrument records. Python's struct module can encode and decode fixed binary layouts, provided the layout is specified.

import struct

# Little-endian unsigned 16-bit integer and 32-bit float
record = struct.pack("<Hf", 1200, 3.5)
count, value = struct.unpack("<Hf", record)

The format string establishes field widths and byte order. Without such an agreement, data exchanged between machines can be misinterpreted. Binary formats also require attention to corruption detection, often through a checksum or cryptographic hash. A checksum detects accidental changes; a cryptographic integrity mechanism is needed when deliberate tampering is a concern.

Serialization

Serialization converts an in-memory object into a storable or transferable representation. Deserialization reverses the operation. Serialization is not the same as validation: a serialized object may still contain impossible values.

Python's pickle can serialize many Python objects, but unpickling data from an untrusted source can execute malicious code. It should therefore not be used as a general interchange format across trust boundaries. JSON, CSV, or a deliberately designed binary format is usually safer for external data.

3. Reading and Writing Reliable Data

A robust reader separates acquisition from interpretation. First acquire bytes or text, then parse the format, then validate the resulting values, and only then perform engineering calculations.

from pathlib import Path

path = Path("settings.txt")

try:
    text = path.read_text(encoding="utf-8")
except FileNotFoundError:
    raise RuntimeError(f"Required configuration is missing: {path}")
except OSError as error:
    raise RuntimeError(f"Cannot read {path}: {error}") from error

The with statement and Path methods simplify resource handling, but they do not eliminate failures. A path may be missing, permissions may change, storage may be full, or a device may return incomplete data.

When writing records, define whether a partial result is acceptable. For a daily measurement report, appending one valid record at a time may be appropriate. For a configuration file, a partial file can be dangerous; temporary output followed by replacement is preferable.

Encoding and newline issues

Text decoding requires the correct encoding. UTF-8 is a sound default, but an instrument may produce another encoding. Replacing undecodable bytes silently can corrupt identifiers or measurements. If recovery is necessary, record that it occurred.

Newline conventions differ among operating systems. Language-specific text modes generally translate them, while newline="" is recommended when Python's CSV module should control newline handling.

4. Validation: Making Assumptions Explicit

Validation tests whether data satisfies stated requirements. It should be based on domain knowledge, not merely on whether conversion to a numeric type succeeds.

Important validation levels are:

1. Syntax validation: Is the file parseable? Is a field a valid integer or date?
2. Schema validation: Are required fields present and correctly typed?
3. Range validation: Is a voltage within the safe operating interval?
4. Relational validation: Is stop time after start time? Are channel lengths equal?
5. Semantic validation: Does the device identifier exist, and are the units appropriate?
6. Cross-record validation: Are timestamps ordered and unique where required?

For a measured quantity x with permitted interval [xmin, xmax], inclusive range validation is:

Here x, xmin, and xmax must have the same physical unit, and the rule applies only when the engineering specification defines inclusive limits. Some specifications require strict inequalities instead.

A value may be numerically valid but physically implausible. For example, a temperature of 25 °C is syntactically valid, but a sensor specified for −40 °C to 85 °C should reject 200 °C unless an over-range condition is explicitly represented.

Example validator:

def validate_measurement(row):
    required = {"time_s", "voltage_V"}
    missing = required - row.keys()
    if missing:
        return False, f"missing fields: {sorted(missing)}"

    try:
        time_s = float(row["time_s"])
        voltage_v = float(row["voltage_V"])
    except (TypeError, ValueError):
        return False, "time_s and voltage_V must be numeric"

    if time_s < 0:
        return False, "time_s must be nonnegative"
    if not -12.0 <= voltage_v <= 12.0:
        return False, "voltage_V is outside the permitted range"
    return True, "valid"

A useful system distinguishes rejected data, missing data, and uncertain data. Replacing all three with zero can silently produce false engineering conclusions.

5. Exceptions, Error Codes, and Recovery

An exception is an abnormal event represented by the language runtime or program. It interrupts the normal control flow and can carry diagnostic information. Typical examples include FileNotFoundError, PermissionError, ValueError, and TimeoutError.

The basic pattern is:

try:
    value = float(raw_value)
except ValueError:
    value = None
else:
    use(value)
finally:
    record_completion_status()

except handles selected failures, else runs only when no exception occurred, and finally runs whether or not an exception occurred. Catching Exception broadly can be appropriate at an application boundary for logging and controlled shutdown, but it is usually poor practice inside a function because it can hide programming defects.

Exceptions are appropriate when an operation cannot fulfill its contract and the caller must decide what to do. They should include useful context and preserve the original cause:

try:
    data = parse_file(path)
except ValueError as error:
    raise DataFormatError(f"Invalid record in {path}") from error

Error codes are explicit values returned by a function. They are useful in embedded systems, real-time loops, C interfaces, and situations where failure is expected and must be handled without stack unwinding. A code should have documented meaning; for example, 0 may indicate success, while distinct nonzero codes identify timeout, invalid input, or hardware failure.

Neither mechanism is universally superior. Exceptions support separation of normal logic and error logic, while error codes make failure part of the ordinary return interface. In either case, the caller must not ignore failure.

6. Logging and Diagnostics

A message shown to a user is not necessarily sufficient for maintenance. Logging creates a chronological record of events and failures. A useful log entry answers: what happened, when, where, to which component, and with what consequence?

Common levels are DEBUG, INFO, WARNING, ERROR, and CRITICAL. Logs should be structured where possible, using fields such as timestamp, device identifier, operation, severity, and correlation or run ID.

import logging

logging.basicConfig(level=logging.INFO,
                    format="%(asctime)s %(levelname)s %(message)s")
logger = logging.getLogger(__name__)

try:
    load_configuration()
except OSError:
    logger.exception("Configuration loading failed")
    raise

logger.exception includes a traceback when called inside an exception handler. Logs should not contain passwords, private keys, personal information, or unrestricted raw sensor payloads if those payloads are sensitive. Log rotation, access control, clock synchronization, and retention rules matter in production systems.

Logging is not a substitute for recovery. Excessive messages can obscure the important event, while insufficient context makes diagnosis impossible. A good design logs once at the layer that can decide the operational response and adds context as errors move upward.

7. Defensive Programming and Robust Pipelines

Defensive programming assumes that inputs, resources, and dependencies may behave unexpectedly. Its practices include:

• Validate at trust boundaries.
• Keep functions small and define their contracts.
• Use explicit units and names such as voltage_V and time_s.
• Avoid silently accepting malformed records.
• Set limits on file size, record count, and nesting depth.
• Use timeouts for network and hardware operations.
• Make retries bounded and use backoff where repeated attempts could overload a service.
• Preserve the original data before transformation when traceability matters.
• Make operations idempotent when possible, meaning repeating them does not create unintended additional effects.
• Test missing files, empty files, malformed fields, duplicate records, extreme values, and interrupted writes.

A pipeline should define what happens when one record fails. Possible policies include reject the whole batch, reject only the bad record, substitute a documented missing value, or stop and request human review. The correct policy depends on risk. Dropping one corrupted display sample may be acceptable; dropping one safety-critical measurement may not be.

Security and governance

File processing has security implications. Never construct shell commands from untrusted filenames without careful escaping. Restrict paths to approved directories, limit resource consumption, and treat uploaded files as hostile. Validate file type by content when necessary rather than trusting only a filename extension.

Engineering records may be subject to traceability, quality assurance, export controls, or privacy rules. Version schemas and calibration information. Record who or what generated a result, which software version was used, and whether data was modified. For safety-related decisions, a human-review path and immutable audit records may be required.

8. Worked Example: A Robust Measurement Importer

Suppose a laboratory produces a CSV file with columns timestamp_s, current_A, and temperature_C. The importer must reject missing fields, invalid numbers, out-of-range values, and non-increasing timestamps while reporting line numbers.

import csv
import math
from pathlib import Path

class DataFormatError(Exception):
    pass

def read_measurements(filename):
    records = []
    previous_time = None

    with Path(filename).open(newline="", encoding="utf-8") as file:
        reader = csv.DictReader(file)
        required = {"timestamp_s", "current_A", "temperature_C"}
        if not reader.fieldnames or not required.issubset(reader.fieldnames):
            raise DataFormatError("required columns are missing")

        for line_number, row in enumerate(reader, start=2):
            try:
                t = float(row["timestamp_s"])
                current = float(row["current_A"])
                temperature = float(row["temperature_C"])
            except (TypeError, ValueError) as error:
                raise DataFormatError(
                    f"line {line_number}: nonnumeric value") from error

            if not all(math.isfinite(v) for v in (t, current, temperature)):
                raise DataFormatError(f"line {line_number}: non-finite value")
            if t < 0 or not -10 <= current <= 10:
                raise DataFormatError(f"line {line_number}: value out of range")
            if not -40 <= temperature <= 125:
                raise DataFormatError(f"line {line_number}: temperature out of range")
            if previous_time is not None and t <= previous_time:
                raise DataFormatError(f"line {line_number}: timestamps not increasing")

            records.append((t, current, temperature))
            previous_time = t

    return records

The function uses a parser rather than manual splitting, checks the schema, converts values, rejects non-finite numbers, applies domain ranges, and enforces a cross-record rule. It reports the first invalid line with a meaningful cause. A production version might instead collect all errors, attach units and device metadata, and write a quarantine file for rejected records.

If the first and last timestamps are 0 s and 4 s and there are five records, the average sampling interval is not inferred merely from the number of records unless the samples are known to be uniformly spaced. For uniformly spaced records, the interval is:

where ∆t is the sampling interval in seconds, tfirst and tlast are endpoint timestamps in seconds, and N is the number of records with N > 1. For the stated values, ∆t = (4 − 0) ÷ (5 − 1) = 1 s. This formula is applicable only when the records are ordered and the interval is uniform; otherwise, individual differences should be examined.

9. Applied Case: Motor-Test Data Pipeline

Background

This illustrative case concerns a university motor laboratory. A data-acquisition computer records current, voltage, speed, and temperature. Students export CSV files, while the analysis program stores processed runs in JSON.

Problem

The original script assumed every file existed, split lines on commas, converted every field directly to float, and overwrote the output report. Quoted metadata containing commas caused parsing errors. A sensor disconnection produced blank fields that became zeros in a spreadsheet. Once, a power interruption left a truncated JSON report that appeared syntactically valid up to the point where it was read.

Analysis

The failures occurred at several boundaries: file access, CSV parsing, numeric conversion, physical validation, missing-value interpretation, and output durability. Treating all failures as conversion errors prevented diagnosis. Treating blank current as zero introduced a false low-current event.

Approach and solution

The revised pipeline:

1. Reads files through a context manager using an explicit encoding.
2. Uses a CSV parser and validates a versioned header.
3. Represents missing measurements as missing, not zero.
4. Applies ranges based on sensor specifications and checks timestamp order.
5. Logs the run ID, source file, software version, and rejected line numbers.
6. Writes results to a temporary file and replaces the destination only after successful completion.
7. Stores units and calibration identifiers in structured metadata.
8. Preserves the original input and places rejected records in a quarantine file.
9. Uses tests containing empty, malformed, duplicated, and extreme records.

Outcome

The revised system makes failures visible and prevents an incomplete report from replacing a complete one. Analysts can distinguish a real zero-current measurement from a disconnected sensor. The approach does not guarantee correct measurements—the sensors and calibration may still be wrong—but it makes assumptions explicit and improves traceability.

Learning points

Robustness is a system property, not merely a try statement. Correct parsing, domain validation, controlled recovery, durable output, logging, and governance must work together. In safety-related applications, the acceptable response may be to stop rather than continue with uncertain data.

10. Contemporary Practice and Design Guidance

Modern engineering pipelines increasingly use schema registries, typed data models, columnar storage, checksums, cloud object storage, and automated data-quality tests. These tools are valuable only when their contracts are understood. A schema validator can confirm that a field is a number, but it cannot determine whether the number has the correct calibration or physical meaning.

AI-assisted code generation can produce file-handling code quickly, but generated code must be reviewed for unsafe deserialization, broad exception handling, path traversal, silent coercion, and missing unit checks. Automated tools should support—not replace—engineering judgment, testing, and accountability.

A practical design checklist is:

Chapter Summary

Files provide persistent data but do not provide meaning; a format specification supplies that meaning. Text files are readable and interoperable, binary files can be compact and efficient, tabular formats represent records and fields, and structured formats represent nested data. Reliable programs use correct parsers, explicit encodings, safe resource management, and deliberate output strategies.

Validation should address syntax, schema, ranges, relationships, semantics, and cross-record consistency. Exceptions and error codes are alternative mechanisms for communicating failure, while logging supplies diagnostic history. Defensive programming validates trust boundaries, limits resources, preserves context, distinguishes missing from zero, and chooses recovery policies according to risk. Serialization must be separated from validation, and untrusted serialized objects must never be deserialized casually. A robust engineering pipeline makes assumptions explicit, records what happened, and fails safely when its assumptions are violated.

Key Terms

File: Persistent sequence of bytes managed by a storage system.

Text file: File whose bytes represent characters under an encoding.

Binary file: File whose bytes follow an application-specific representation.

Encoding: Rule mapping characters to bytes and bytes to characters.

Delimited file: Tabular file whose fields are separated by a delimiter.

Schema: Specification of fields, types, constraints, and relationships.

Serialization: Conversion of an in-memory representation into stored or transferable data.

Deserialization: Reconstruction of an in-memory representation from serialized data.

Validation: Testing data against stated syntactic, structural, or domain rules.

Exception: An abnormal event represented through program control flow.

Error code: Explicit status value communicating success or failure.

Logging: Recording time-stamped operational and diagnostic events.

Defensive programming: Designing software to anticipate invalid inputs and failures.

Idempotence: Property that repeating an operation does not create unintended additional effects.

Atomic replacement: Updating a destination so observers see either the old complete file or the new complete file.

Quarantine: Separate storage for rejected or suspicious records.

Traceability: Ability to associate a result with its source, processing steps, and software context.

Schema version: Identifier indicating the structure and interpretation rules of a data format.

Review Questions

Short Answer Questions

1. What is the difference between a text file and a binary file?
2. Why should a CSV parser generally be used instead of splitting each line on commas?
3. Distinguish serialization from validation.
4. What are the roles of try, except, else, and finally?
5. Why should missing sensor data not automatically be replaced with zero?

Long Answer Questions

1. Compare CSV, JSON, and a fixed binary format for a high-volume measurement system.
2. Explain a multi-level validation strategy for an engineering data pipeline.
3. Compare exceptions and error codes, including contexts in which each is appropriate.
4. Describe how logging, schema versioning, and metadata improve traceability.
5. Explain how temporary output and atomic replacement reduce the risk of corrupted reports.

Analytical / Application Questions

1. Design a failure policy for a batch containing 99 valid measurements and one malformed safety-critical measurement. Justify whether the batch should continue, stop, or quarantine the record.
2. A file contains timestamps 0, 1.0, 2.1, and 3.0 s. Determine whether uniform sampling may safely be assumed and propose validation tests.
3. Review a program that catches every error, substitutes zero for invalid values, and overwrites its input file. Identify at least four risks and propose specific improvements.

Multiple Choice Questions with Answers

Question 1

Which file mode appends data without replacing existing content?

A. r

B. w

C. a

D. x

Correct Answer: C. a

Question 2

What is the primary purpose of a schema?

A. To compress every file

B. To specify fields, types, and constraints

C. To encrypt passwords automatically

D. To replace all validation

Correct Answer: B. To specify fields, types, and constraints

Question 3

Why is untrusted Python pickle data dangerous to deserialize?

A. It cannot represent numbers

B. It always loses text encoding

C. Deserialization can execute malicious code

D. It only works with CSV files

Correct Answer: C. Deserialization can execute malicious code

Question 4

Which validation checks whether a voltage lies between specified engineering limits?

A. Range validation

B. Syntax validation

C. Encoding validation

D. Path validation

Correct Answer: A. Range validation

Question 5

What does a finally block generally provide?

A. Execution only when parsing succeeds

B. A guaranteed cleanup or completion path

C. Automatic correction of all errors

D. Conversion of text to binary

Correct Answer: B. A guaranteed cleanup or completion path

Question 6

For uniformly spaced records, what is the interval when t_first = 0 s, t_last = 8 s, and N = 5?

A. 0.625 s

B. 1.6 s

C. 2 s

D. 40 s

Correct Answer: C. 2 s

Question 7

Which practice best protects a configuration file from being replaced by a truncated result?

A. Write directly in append mode

B. Ignore write exceptions

C. Write a temporary file and replace after success

D. Convert all values to strings

Correct Answer: C. Write a temporary file and replace after success

Question 8

What is the main purpose of logging an exception with its traceback?

A. To hide the original cause

B. To provide diagnostic context for investigation

C. To make invalid data valid

D. To eliminate the need for testing

Correct Answer: B. To provide diagnostic context for investigation

Question 9

Which statement best describes defensive programming?

A. Assuming all inputs satisfy ideal conditions

B. Removing all error checks for speed

C. Anticipating invalid inputs and controlled failures

D. Using binary files exclusively

Correct Answer: C. Anticipating invalid inputs and controlled failures

Question 10

What does an atomic replacement strategy aim to ensure?

A. Every file becomes smaller

B. Observers see a complete old or complete new file

C. All malformed records are accepted

D. Files require no permissions

Correct Answer: B. Observers see a complete old or complete new file