Engineering Data Structures and File Processing

Introduction

Engineering software rarely operates on one value at a time. A temperature-monitoring program may receive thousands of readings; a design tool may store dimensions, materials, and tolerances; a test system may exchange results through CSV or JSON files. The quality of such software depends not only on calculations but also on how data are represented, organized, validated, and stored.

A data structure is a method of organizing data so that it can be accessed and modified efficiently. File processing is the controlled reading, writing, validation, and transformation of persistent data. In this chapter, Python is used because its built-in structures correspond closely to common engineering requirements. The principles, however, apply to other programming languages.

An engineering data structure should be selected according to four criteria: the meaning of the data, the operations required, computational efficiency, and the risk of invalid or ambiguous data.

Learning Objectives

By the end of this chapter, you should be able to:

• distinguish strings, arrays, lists, tuples, dictionaries, tables, and structured records;
• select a suitable structure for a stated engineering task;
• process measurements and tabular data safely;
• read and write CSV and JSON files;
• validate data types, units, ranges, and missing values;
• explain basic time and space efficiency;
• design a small data-processing workflow suitable for engineering practice;
• identify limitations, security risks, and reproducibility requirements.

1. Data Representation in Engineering

Engineering data have both values and interpretation. The number 25 may represent 25 °C, 25 mm, 25 kPa, or a categorical code. A reliable program therefore preserves metadata such as units, variable names, sensor identity, timestamp, and uncertainty where these affect interpretation.

A useful conceptual model is:

raw source → parsing → validation → structured storage → computation → output and archive

Parsing converts characters into program values. Validation checks whether those values are acceptable. Storage provides an organization appropriate to the intended operations. Computation should normally be separated from input and output so that it can be tested independently.

1.1 Scalar values and strings

A scalar is one value, such as an integer, real number, Boolean, or string. Strings are sequences of characters used for names, identifiers, labels, and textual fields.

sensor_id = "T-104"
material = "aluminium"
temperature = 23.6
active = True

Strings are indexed from zero and may be sliced:

code = "PUMP-07"
print(code[0])       # P
print(code[0:4])     # PUMP
print(code.lower())

Strings are immutable: an operation creates a new string rather than changing the existing one. This is useful for safety but important when processing large text files. Whitespace and capitalization should be handled deliberately, because "OK", "ok", and " OK " are different strings until normalized.

1.2 Lists

A list is an ordered, mutable collection. It can contain repeated values and values of different types, although homogeneous lists are usually clearer in engineering applications.

readings = [21.4, 21.7, 21.5, 21.9]
readings.append(22.1)
readings[1] = 21.8

Lists are suitable for a sequence whose length changes or whose elements must be updated. Typical operations include appending a new measurement, iterating over records, and sorting values.

A list search generally takes time proportional to the number of elements, written as O(n), because elements may need to be examined one by one. Access by index is usually O(1), meaning approximately constant time for a Python list. These are general implementation characteristics, not guarantees that every operation has identical physical execution time.

1.3 Tuples

A tuple is an ordered, immutable collection:

point = (12.5, 8.0, 3.2)
x, y, z = point

Tuples communicate that the number and meaning of positions should not change. They are useful for coordinates, fixed dimensions, or a function result containing several related values. A tuple is not automatically self-documenting: (12.5, 8.0, 3.2) is less clear than a record with named fields when the positions are not obvious.

1.4 Dictionaries

A dictionary stores key–value associations:

sensor = {
    "id": "T-104",
    "temperature_C": 23.6,
    "status": "valid"
}
print(sensor["temperature_C"])

Keys should be unique and generally immutable types such as strings or numbers. Dictionary lookup is typically O(1) on average, making dictionaries effective for direct retrieval by identifier. They are not a substitute for a table when operations involve columns, sorting, aggregation, or relationships among many records.

Safe access is useful when a field may be absent:

operator = sensor.get("operator", "unknown")

1.5 Nested and mixed structures

Engineering records commonly combine structures:

experiment = {
    "name": "fatigue_test_01",
    "specimen": {"material": "steel", "diameter_mm": 10.0},
    "forces_N": [1200, 1350, 1280],
    "completed": True
}

Such flexibility is powerful, but excessive nesting can make data difficult to inspect and validate. A design should be as simple as possible while preserving meaning.

2. Tables and Structured Records

A table represents observations in rows and variables in columns. For example:

A row may be represented as a dictionary:

row = {
    "time_s": 2.0,
    "temperature_C": 99.9,
    "pressure_kPa": 101.1,
    "status": "suspect"
}

A table can therefore be represented as a list of dictionaries:

table = [
    {"time_s": 0.0, "temperature_C": 22.4},
    {"time_s": 1.0, "temperature_C": 22.6}
]

This representation is readable and flexible. A column-oriented representation is often more convenient for numerical operations:

columns = {
    "time_s": [0.0, 1.0],
    "temperature_C": [22.4, 22.6]
}

The list-of-records model emphasizes individual observations; the dictionary-of-lists model emphasizes variables. For larger numerical datasets, specialized array or data-frame libraries can provide vectorized computation and memory efficiency, but their use does not remove the need for validation and unit control.

Structured records

A structured record has named fields and a defined schema. In Python, a dataclass provides a clear option:

from dataclasses import dataclass

@dataclass
class Measurement:
    time_s: float
    temperature_C: float
    sensor_id: str

m = Measurement(2.0, 22.6, "T-104")

The type annotations document intended types, but ordinary Python does not automatically enforce them at runtime. Explicit validation remains necessary.

3. Choosing a Data Structure

The following decision principles are useful:

The choice should also consider access patterns. If a program repeatedly asks for the pressure associated with sensor T-104, a dictionary indexed by sensor ID is preferable to scanning a list. If it calculates the mean of a temperature column, a column-oriented array is more natural.

Complexity and engineering judgment

Big-O notation describes how resource use grows as input size n increases. Common examples include:

• O(1): constant-time average dictionary lookup or list index access;
• O(n): scanning a list or processing every record once;
• O(n log n): comparison-based sorting in typical implementations;
• O(n²): comparing every pair of n items.

Complexity does not replace measurement. For small datasets, clarity may matter more than a theoretically faster method. For real-time systems, large test campaigns, or embedded devices, time and memory constraints become design requirements.

4. File Processing

A file is persistent data stored outside the running program. Reliable file processing has five stages:

1. identify the file format and schema;
2. open the file using an explicit encoding and mode;
3. parse its contents;
4. validate values and relationships;
5. close the file and report errors appropriately.

Python's with statement closes a file automatically:

with open("notes.txt", "r", encoding="utf-8") as file:
    text = file.read()

Common modes are "r" for reading, "w" for writing and replacing, and "a" for appending. Writing with "w" can destroy an existing file, so important data should be archived or written to a new file first.

4.1 Text files and parsing

A line-oriented measurement file might contain:

0.0,22.4,101.2
1.0,22.6,101.3

Manual splitting is possible for simple files, but it becomes unreliable when fields contain quoted commas, missing values, or embedded text. Format-aware libraries should be used for formal CSV data.

4.2 CSV files

CSV, or comma-separated values, is a tabular interchange format. It is human-readable and widely supported, but it does not inherently store data types, units, formulas, or relationships. A header and a data dictionary should accompany important engineering files.

import csv

with open("measurements.csv", newline="", encoding="utf-8") as file:
    reader = csv.DictReader(file)
    records = []
    for row in reader:
        record = {
            "time_s": float(row["time_s"]),
            "temperature_C": float(row["temperature_C"]),
            "sensor_id": row["sensor_id"].strip()
        }
        records.append(record)

The conversion to float is essential: values read from CSV are strings. A malformed value such as "--" raises a conversion error and should be handled according to the application's policy.

Writing CSV:

fields = ["time_s", "temperature_C", "sensor_id"]
with open("clean_measurements.csv", "w", newline="", encoding="utf-8") as file:
    writer = csv.DictWriter(file, fieldnames=fields)
    writer.writeheader()
    writer.writerows(records)

4.3 JSON files

JSON represents objects, arrays, strings, numbers, Boolean values, and null values. It is well suited to hierarchical configuration and self-describing records.

import json

configuration = {
    "sample_rate_Hz": 1000,
    "channels": ["force_N", "displacement_mm"],
    "calibrated": True
}

with open("config.json", "w", encoding="utf-8") as file:
    json.dump(configuration, file, indent=2)

with open("config.json", "r", encoding="utf-8") as file:
    loaded = json.load(file)

JSON does not natively represent arbitrary Python objects, tuples, dates, or measurement units. Dates and units should be encoded using an agreed convention, such as an ISO 8601 string and a field name ending in _mm or _Pa. For example, temperature_C is clearer than temperature.

5. Validation of Engineering Data

Validation should occur at the boundary where external data enter the program. Typical checks include:

• presence of required fields;
• numeric conversion;
• finite values rather than NaN or infinity;
• physically meaningful ranges;
• consistent units;
• monotonically increasing timestamps where required;
• unique identifiers;
• relationships among fields.

For a measurement x with an accepted interval from a to b, a range check can be expressed as:

Here a and b are lower and upper limits in the same units as x. The limits must come from engineering requirements, calibration specifications, or domain knowledge; they should not be invented merely to make data pass.

Example validation:

def validate_measurement(record):
    required = {"time_s", "temperature_C", "sensor_id"}
    missing = required - record.keys()
    if missing:
        raise ValueError(f"Missing fields: {missing}")

    if record["time_s"] < 0:
        raise ValueError("Time cannot be negative")
    if not (-80.0 <= record["temperature_C"] <= 200.0):
        raise ValueError("Temperature outside instrument range")
    if not record["sensor_id"].strip():
        raise ValueError("Sensor ID is empty")

A rejected record should not silently disappear. A professional system records the reason, preserves the original input when appropriate, and distinguishes invalid data from missing data and suspected outliers.

6. Worked Example: Processing a Measurement File

Suppose a laboratory records time, temperature, and pressure in run.csv. The objective is to load valid records and calculate the arithmetic mean temperature. For n valid temperatures x₁, x₂, …, xₙ, the arithmetic mean is:

The variable xᵢ is the i-th temperature, n is the number of valid observations, and x̄ has the same unit as xᵢ. The formula assumes that each included observation is appropriately comparable; it does not correct for sensor bias or unequal sampling importance.

import csv

valid = []
with open("run.csv", newline="", encoding="utf-8") as file:
    for row_number, row in enumerate(csv.DictReader(file), start=2):
        try:
            record = {
                "time_s": float(row["time_s"]),
                "temperature_C": float(row["temperature_C"]),
                "pressure_kPa": float(row["pressure_kPa"])
            }
            if record["time_s"] < 0:
                raise ValueError("negative time")
            if not -80 <= record["temperature_C"] <= 200:
                raise ValueError("temperature out of range")
            valid.append(record)
        except (KeyError, TypeError, ValueError) as error:
            print(f"Rejected row {row_number}: {error}")

if not valid:
    raise ValueError("No valid measurements")

mean_temperature = sum(r["temperature_C"] for r in valid) / len(valid)
print(f"Mean temperature: {mean_temperature:.2f} °C")

The program separates parsing, validation, and calculation. If the valid temperatures are 22.4 °C, 22.6 °C, and 22.5 °C, then the mean is (22.4 + 22.6 + 22.5) ÷ 3 = 22.5 °C. The calculation is meaningful only if all three values refer to the same measured quantity and unit.

7. Illustrative Case: Thermal Test Data Pipeline

Background

A university laboratory tests an aluminium specimen in a temperature-controlled chamber. A data logger produces one CSV file per test containing time, chamber temperature, specimen temperature, and sensor status. Students previously copied values manually into spreadsheets.

Problem

Manual copying caused unit confusion, omitted rows, and loss of rejected readings. A repeatable process was required for traceability.

Analysis

The team identified a row-oriented table as the primary structure. Each row became a dictionary, while a separate configuration dictionary stored the file schema and expected units. CSV was retained for compatibility with the logger; JSON was used for configuration and summary metadata.

Approach and solution

The new program:

1. reads the CSV with csv.DictReader;
2. converts numeric fields explicitly;
3. checks required columns and allowable ranges;
4. stores valid records in a list of dictionaries;
5. stores rejected row numbers and reasons;
6. calculates summary quantities only from valid records;
7. writes a cleaned CSV and a JSON report containing the test identifier, software version, units, count of valid records, and count of rejected records.

The process also preserves the original file as read-only evidence. A checksum can be recorded when strict data provenance is required.

Outcome

The workflow makes invalid data visible, reduces transcription, and permits the same test to be reprocessed. It does not prove that the sensors were correctly calibrated; calibration remains an experimental and quality-assurance responsibility.

Learning points

The best structure was not selected solely for speed. A list of named records was chosen because readability, auditability, and straightforward validation were more important than maximum numerical performance. The case also demonstrates that file formats, data structures, and engineering governance are connected design decisions.

8. Limitations, Security, and Professional Practice

CSV is simple but weakly typed. JSON is expressive but may become inconsistent without a schema. Very large files may exceed available memory if loaded in full; streaming line by line or processing chunks is then preferable. Floating-point values can contain small representation errors, so rounding should be applied for presentation, not used to conceal computational uncertainty.

Never treat external files as trustworthy. Validate paths, reject unexpected file types where appropriate, limit file sizes, and avoid executing content from data files. Do not deserialize untrusted formats that can execute arbitrary code. Protect personal, proprietary, or safety-critical data through access control and appropriate retention policies.

Engineering records should include units, coordinate conventions, time zones, sensor identifiers, calibration status, software version, and processing assumptions. Changes to processing code should be tracked. Reproducibility requires that another competent person can identify the source data, transformations, and parameter choices.

Artificial intelligence can assist with schema discovery, anomaly screening, and code generation, but it does not establish physical validity. AI-generated transformations must be reviewed, tested against known data, and checked for silent unit conversion or fabricated assumptions.

Best-Practice Checklist

• Define the schema before coding.
• Use names that encode units where practical, such as force_N.
• Convert external strings to explicit types.
• Validate at input boundaries.
• Separate input, processing, and output functions.
• Preserve raw data and document transformations.
• Use dictionaries for key lookup and records for named meaning.
• Select arrays or specialized tables for large numerical workloads.
• Handle missing, invalid, and suspicious values differently.
• Test empty files, malformed rows, duplicate identifiers, and unexpected units.
• Record provenance, software version, and assumptions.

Chapter Summary

Engineering programs require data structures that preserve both values and meaning. Strings represent textual information; lists provide mutable ordered sequences; tuples represent fixed groups; dictionaries support key-based lookup; tables organize observations and variables; and structured records make schemas explicit. The appropriate choice depends on required operations, clarity, scale, and reliability.

File processing consists of parsing, validation, transformation, and controlled storage. CSV is useful for interoperable tables but has limited type and metadata support. JSON handles hierarchical data and configuration but still requires an agreed schema. Reliable engineering workflows convert strings explicitly, check units and ranges, distinguish invalid from missing data, and preserve provenance. Efficiency concepts such as O(1), O(n), and O(n log n) support informed choices, but clarity and verification remain essential. Professional practice additionally requires security, reproducibility, traceability, and human review of automated or AI-assisted processing.

Key Terms

Data structure: An organization for storing and accessing data.

String: An immutable sequence of characters.

List: An ordered, mutable collection.

Tuple: An ordered, immutable collection.

Dictionary: A collection of key–value associations.

Array: A typically homogeneous indexed collection designed for efficient numerical storage or operations.

Table: Data arranged as rows and columns.

Record: A group of related fields describing one entity or observation.

Schema: A specification of fields, types, constraints, and relationships.

Parsing: Converting external text or bytes into program data.

Validation: Checking data against structural, type, range, and domain requirements.

CSV: A delimited text format commonly used for tabular data exchange.

JSON: A text format for objects, arrays, and primitive values.

Metadata: Information describing data, such as units, timestamps, and provenance.

Big-O notation: A description of how resource requirements grow with input size.

Immutability: The property that an object cannot be changed after creation.

Serialization: Converting structured data into a storable or transferable representation.

Data provenance: The origin and processing history of data.

Review Questions

Short Answer Questions

1. Why should engineering variable names often include units?
2. State one important difference between a list and a tuple.
3. When is a dictionary preferable to a list of records?
4. Why must numeric CSV fields usually be converted after reading?
5. Name three types of validation applied to measurement data.

Long Answer Questions

1. Compare lists, tuples, dictionaries, and structured records for representing an engineering experiment.
2. Explain the stages of a reliable CSV-processing workflow.
3. Compare CSV and JSON in terms of structure, portability, typing, and typical engineering use.
4. Explain O(1), O(n), and O(n log n), and discuss why complexity is not the only criterion for selecting a structure.
5. Discuss how provenance, units, calibration information, and software version contribute to reproducible engineering analysis.

Analytical / Application Questions

1. Design data structures for a system that records 20 sensors at 1 Hz, permits lookup by sensor ID, and exports daily CSV files. Justify each choice.
2. A file contains temperatures in mixed Celsius and Fahrenheit but has no unit column. Explain how you would detect, quarantine, and resolve the problem without silently combining the values.
3. Develop a validation strategy for a fatigue-test dataset containing specimen ID, cycle count, force, displacement, and timestamp. Include structural, type, range, and relationship checks.

Multiple Choice Questions with Answers

Question 1

Which Python structure is most suitable for direct lookup of a measurement using a unique sensor identifier?

A. Dictionary

B. Tuple

C. String

D. List index only

Correct Answer: A. Dictionary

Question 2

What is the principal limitation of ordinary CSV files for engineering data?

A. They cannot contain text

B. They do not inherently preserve rich data types and units

C. They cannot be opened by spreadsheet software

D. They always require a database server

Correct Answer: B. They do not inherently preserve rich data types and units

Question 3

What type does a value read by csv.DictReader normally have before explicit conversion?

A. Integer

B. Float

C. String

D. Boolean

Correct Answer: C. String

Question 4

For temperatures 22.4 °C, 22.6 °C, and 22.5 °C, what is the arithmetic mean?

A. 22.0 °C

B. 22.4 °C

C. 22.5 °C

D. 22.6 °C

Correct Answer: C. 22.5 °C

Question 5

Which structure best communicates that a fixed group of positions should not be modified?

A. Tuple

B. List

C. Dictionary

D. CSV file

Correct Answer: A. Tuple

Question 6

Which operation is typically O(n) for a Python list?

A. Accessing an element by index

B. Scanning for a value

C. Reading a dictionary key

D. Assigning a variable name

Correct Answer: B. Scanning for a value

Question 7

Why is the with statement recommended when opening files in Python?

A. It automatically converts all values to floats

B. It closes the file when the managed block ends

C. It prevents all malformed data

D. It encrypts the file contents

Correct Answer: B. It closes the file when the managed block ends

Question 8

Which practice most directly supports reproducibility?

A. Deleting the original input after processing

B. Recording units, software version, and processing assumptions

C. Rounding every raw measurement immediately

D. Ignoring rejected rows

Correct Answer: B. Recording units, software version, and processing assumptions

Question 9

What is a suitable representation for a fixed named measurement record in Python?

A. A dataclass

B. A single character

C. An unparsed file path

D. A Boolean value

Correct Answer: A. A dataclass

Question 10

Why should invalid measurement records not simply be discarded without explanation?

A. They may be needed for traceability and quality analysis

B. They always improve the calculated mean

C. They are automatically valid in JSON

D. Discarding them increases file security

Correct Answer: A. They may be needed for traceability and quality analysis