Persistent Data and Application Integration

Introduction

Most useful applications must remember information after a program ends. A contact manager stores contacts, an inventory system records products, and a service application retrieves information from another organization through an API. These applications combine two broad kinds of data structures:

• In-memory structures, such as lists, dictionaries, sets, and objects, which support fast processing during execution.
• Persistent storage, such as text files, JSON documents, and relational databases, which preserves data beyond program termination.

Application integration connects these components to users, files, databases, and external services. Reliable integration requires more than knowing syntax. A program must validate input, handle failures, protect sensitive information, preserve data consistency, and convert information between representations.

This chapter develops practical foundations for integrating Java and Python applications with files, JSON, databases, and web APIs.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain the difference between transient and persistent data.
2. Read, write, append, and safely manage files in Python and Java.
3. Serialize and deserialize structured data using JSON.
4. Select suitable in-memory data structures for application tasks.
5. Use parameterized database queries and basic transactions.
6. Consume an HTTP API and interpret common response statuses.
7. Apply exception handling and input validation appropriately.
8. Design a small application that integrates persistent storage and external services.
9. Recognize security, privacy, reliability, and maintainability concerns.

1. Transient and Persistent Data

A variable, list, or object normally exists only while a process is running. This is transient data. When the process terminates, the data disappears unless it has been written elsewhere. Persistent data remains available after termination, usually in a file or database.

Persistence introduces important design questions:

• What is the data format?
• How is data identified and updated?
• What happens if a write is interrupted?
• Can multiple users change the data simultaneously?
• How are invalid or corrupted records detected?
• Who is authorized to read or modify the data?

A typical data flow is:

User input → validation → in-memory structure → serialization → file/database
                                      ↓
                              application logic
                                      ↓
External API ← request construction ← validated application data

The application should not treat external input as trustworthy. Input may be missing, incorrectly typed, excessively large, malicious, or inconsistent with business rules.

Common storage choices

2. Files and Streams

A file is a named sequence of bytes stored on a device. A program usually accesses it through a stream or file object. The principal modes are:

• Read: obtain existing content.
• Write: create or replace content.
• Append: add content at the end.
• Binary: handle raw bytes rather than text.

Text files require an encoding. UTF-8 is a common choice because it represents a wide range of characters and is supported by both Java and Python.

Python file handling

Python’s with statement closes a file automatically, including when an exception occurs:

from pathlib import Path

path = Path("records.txt")

with path.open("w", encoding="utf-8") as file:
    file.write("Amina,Support\n")

with path.open("r", encoding="utf-8") as file:
    for line in file:
        name, department = line.rstrip("\n").split(",")
        print(name, department)

pathlib.Path is generally preferable to manually constructing path strings because it provides clearer, platform-independent path operations.

Writing with mode "w" replaces existing content. Therefore, a program should use it deliberately. Mode "a" appends, while mode "x" creates a new file and fails if that file already exists.

Java file handling

Java’s try-with-resources statement closes resources automatically:

import java.io.BufferedReader;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;

Path path = Path.of("records.txt");

try (BufferedReader reader = Files.newBufferedReader(path,
        StandardCharsets.UTF_8)) {
    String line;
    while ((line = reader.readLine()) != null) {
        System.out.println(line);
    }
} catch (IOException error) {
    System.err.println("Could not read file: " + error.getMessage());
}

Java’s Files.writeString and Files.readString are convenient for small files. For large files, buffered or streaming approaches avoid loading the entire file into memory.

Safe file practices

1. Use explicit encodings.
2. Validate that a path refers to an allowed location.
3. Avoid using user input directly as a filename.
4. Handle missing files and permission errors.
5. Use temporary files and replacement when atomic updates are important.
6. Consider file size limits.
7. Do not store passwords or secret keys in ordinary text files.

3. JSON and Serialization

Serialization converts an in-memory object into a storable or transferable representation. Deserialization reconstructs an in-memory representation from that form. JSON represents objects, arrays, strings, numbers, Boolean values, and null values.

Example JSON:

{
  "id": 104,
  "name": "Amina Khan",
  "skills": ["Python", "SQL"],
  "active": true
}

JSON has no native date, set, or binary type. Dates should use an agreed textual representation, such as an ISO-style date, and binary information should normally be stored separately or encoded under a defined protocol.

Python JSON

import json
from pathlib import Path

record = {
    "id": 104,
    "name": "Amina Khan",
    "skills": ["Python", "SQL"],
    "active": True
}

path = Path("record.json")
with path.open("w", encoding="utf-8") as file:
    json.dump(record, file, indent=2)

with path.open("r", encoding="utf-8") as file:
    loaded = json.load(file)

if not isinstance(loaded.get("id"), int):
    raise ValueError("id must be an integer")

json.loads and json.dumps operate on strings, whereas json.load and json.dump operate on file-like objects.

Java JSON

The Java standard library does not provide a complete general-purpose JSON object model in the same way as Python’s json module. Applications commonly use a maintained library such as Jackson or Gson. A design should specify the library and its version rather than assuming that arbitrary JSON can be safely mapped to any Java class.

Conceptually, serialization must address:

• Missing fields
• Additional fields
• Null values
• Incorrect types
• Version changes
• Unknown or untrusted content

A robust application validates deserialized data before using it. Parsing successfully does not prove that the data satisfies business rules.

4. Choosing In-Memory Data Structures

Persistence is only part of an application. Data must be organized efficiently while the program processes it.

A dictionary or hash map generally provides average-case constant-time lookup, written as O(1), under a suitable hash distribution. This is an average-case expectation, not a guarantee for every operation or implementation. A list search generally requires O(n) comparisons in the worst case, where n is the number of elements.

For example, if records are frequently retrieved by an integer identifier, a map is more appropriate than repeatedly scanning a list:

records_by_id = {104: {"name": "Amina"}, 105: {"name": "Luis"}}
record = records_by_id.get(104)

The chosen structure should reflect access patterns, memory limits, ordering requirements, and whether duplicate values are meaningful.

5. Relational Databases

A relational database stores data in tables consisting of rows and columns. A primary key uniquely identifies a row. A foreign key represents a relationship to another table. Constraints help prevent invalid data.

For a simple customer system:

CREATE TABLE customers (
    customer_id INTEGER PRIMARY KEY,
    name TEXT NOT NULL,
    email TEXT NOT NULL UNIQUE
);

A database is preferable to a JSON file when an application requires structured queries, relationships, multiple users, constraints, or transactions.

CRUD operations

CRUD means:

• Create: insert a record.
• Read: retrieve records.
• Update: modify records.
• Delete: remove records.

A parameterized query is essential:

import sqlite3

connection = sqlite3.connect("customers.db")
try:
    connection.execute(
        "INSERT INTO customers(customer_id, name, email) VALUES (?, ?, ?)",
        (104, "Amina Khan", "[email protected]")
    )
    connection.commit()
except sqlite3.Error:
    connection.rollback()
    raise
finally:
    connection.close()

The ? placeholders cause the database driver to bind values separately from SQL text. Constructing SQL by string concatenation can permit SQL injection.

In Java, JDBC uses PreparedStatement for the same purpose:

String sql = "SELECT customer_id, name FROM customers WHERE email = ?";
try (PreparedStatement statement = connection.prepareStatement(sql)) {
    statement.setString(1, email);
    try (ResultSet results = statement.executeQuery()) {
        while (results.next()) {
            System.out.println(results.getInt("customer_id"));
        }
    }
}

Transactions

A transaction groups operations into a unit that should satisfy the ACID properties:

• Atomicity: all operations succeed or the transaction is rolled back.
• Consistency: constraints remain satisfied.
• Isolation: concurrent transactions do not produce unacceptable interference.
• Durability: committed results survive a system failure according to the database’s guarantees.

A transaction is not a substitute for validation. The application should validate input, while the database enforces constraints that must remain true regardless of the application that submits the data.

6. HTTP APIs and Application Integration

An API, or application programming interface, defines how one program requests services from another. A REST-style web API commonly uses HTTP methods:

• GET retrieves data.
• POST creates a resource or requests an operation.
• PUT replaces a resource.
• PATCH partially updates a resource.
• DELETE removes a resource.

Common response categories include:

Python example using the standard library:

from urllib.request import urlopen
from urllib.error import HTTPError, URLError
import json

try:
    with urlopen("https://api.example.org/items", timeout=5) as response:
        if response.status != 200:
            raise RuntimeError(f"Unexpected status: {response.status}")
        payload = json.load(response)
except HTTPError as error:
    print("API returned", error.code)
except URLError as error:
    print("Network problem:", error.reason)
except json.JSONDecodeError:
    print("The response was not valid JSON")

An API client should set timeouts, check status codes, validate the response schema, avoid logging secrets, and apply a retry policy only when retrying is safe. Blind retries can duplicate a non-idempotent operation such as a payment or order creation.

7. Exception Handling and Validation

An exception signals an unusual condition or failure. Good exception handling has three parts:

1. Detect a condition that cannot be handled locally.
2. Recover, report, or translate the error at an appropriate boundary.
3. Preserve useful diagnostic information without exposing confidential data.

Python:

try:
    age = int(input("Age: "))
    if age < 0 or age > 130:
        raise ValueError("age is outside the permitted range")
except ValueError as error:
    print("Invalid age:", error)

Java:

try {
    int age = Integer.parseInt(input);
    if (age < 0 || age > 130) {
        throw new IllegalArgumentException("age is outside the permitted range");
    }
} catch (NumberFormatException | IllegalArgumentException error) {
    System.out.println("Invalid age");
}

In Java, NumberFormatException is a subclass of IllegalArgumentException; consequently, listing both in the same multi-catch is not valid because one alternatives is a subtype of another. A correct version would catch NumberFormatException and then separately handle other validation logic, or catch only IllegalArgumentException where appropriate.

Validation should occur at boundaries:

• Type validation: Is the value an integer, string, or Boolean?
• Format validation: Does an email or date have an acceptable form?
• Range validation: Is a quantity positive and within a defined limit?
• Business validation: Is the requested operation permitted by policy?
• Cross-field validation: Are related fields mutually consistent?

Validation is not authorization. A valid user identifier does not prove that the current user may access it.

8. Integrated Application Design

A maintainable application separates responsibilities. A simple layered design contains:

1. Presentation layer: receives commands or displays results.
2. Validation/service layer: applies business rules.
3. Repository/data-access layer: reads and writes persistent data.
4. Integration layer: communicates with external APIs.
5. Domain model: represents meaningful application concepts.

This separation reduces duplication and makes testing easier. For example, the service layer can be tested with a fake repository rather than requiring a live database.

Illustrative case: community training registration system

Background. A training centre needs an application that registers learners for workshops, saves registrations, and retrieves a room capacity from an external scheduling API.

Problem. Staff currently use spreadsheets. Duplicate registrations occur, files are overwritten, and an unavailable scheduling service causes confusing errors.

Analysis. A learner has an identifier, name, and email. A workshop has a code, title, and capacity. A registration links a learner to a workshop. The system must prevent duplicate learner-workshop pairs and must not exceed capacity.

Approach. The application uses:

• A database table for learners, workshops, and registrations.
• A unique database constraint on (learner_id, workshop_code).
• A dictionary in memory for fast lookup of workshop information during one request.
• JSON for API requests and responses.
• Input validation before database operations.
• A transaction for checking capacity and inserting a registration.
• A timeout and clear fallback message when the scheduling API is unavailable.

A simplified transaction is conceptually:

begin transaction
  find workshop and current registration count
  if capacity is full: rollback and report failure
  if learner-workshop pair already exists: rollback and report duplicate
  insert registration
commit

Outcome. The database constraint prevents duplicates even if two staff members submit requests at nearly the same time. The service layer provides understandable messages, while logs record technical details such as request identifiers and failure categories.

Learning points. Persistent storage should enforce important invariants; application validation improves user experience; external services must be treated as unreliable dependencies; and transactions are necessary when several related operations must succeed together.

9. Security, Privacy, and Governance

Integrated applications process data across trust boundaries. Important safeguards include:

• Use parameterized SQL.
• Store passwords as strong salted password hashes, never reversible plaintext.
• Keep API keys in environment variables or a secret-management service.
• Use HTTPS for sensitive network communication.
• Apply least privilege to database and API accounts.
• Validate and constrain uploaded files and paths.
• Avoid placing personal data in logs.
• Define retention and deletion rules.
• Back up persistent data and test restoration.
• Record sufficient audit information for important changes.

Privacy obligations vary by jurisdiction and sector. An application should collect only necessary data, explain its use, restrict access, and retain it for no longer than justified. Security is not only a programming concern; it includes configuration, deployment, staff behavior, monitoring, and incident response.

10. Testing and Reliability

Tests should cover both normal and failure paths:

• Valid and invalid input
• Empty files and malformed JSON
• Missing database records
• Constraint violations
• Network timeouts and non-success statuses
• Duplicate requests
• Partial failures during updates

A useful testing strategy combines unit tests for validation and business logic, integration tests for databases and APIs, and end-to-end tests for major user workflows. Test data should not contain real personal information.

Contemporary applications increasingly use schema validation, typed data models, connection pooling, asynchronous processing, and observability tools. AI-assisted code generation can accelerate development, but generated code must be reviewed for insecure queries, incorrect exception handling, privacy violations, and dependency risks. Responsibility remains with the developer and organization.

Best Practices Checklist

• Choose storage based on data structure, query needs, and concurrency.
• Use context managers in Python and try-with-resources in Java.
• Specify UTF-8 when handling text unless another encoding is required.
• Serialize only well-defined data and validate after parsing.
• Use maps for frequent key-based lookup and sets for uniqueness.
• Use parameterized database statements.
• Commit transactions deliberately and roll back failures.
• Set API timeouts and handle status codes explicitly.
• Separate user-facing messages from technical logs.
• Protect credentials and personal data.
• Test recovery, not only successful execution.

Chapter Summary

Persistent data allows applications to preserve information through files, JSON documents, and databases. Files are simple but require careful encoding, path handling, and resource management. JSON provides portable structured representation, but parsed data must still be validated. In-memory lists, sets, maps, queues, and objects should be selected according to access patterns and uniqueness requirements.

Relational databases provide schemas, constraints, queries, and transactions. Parameterized statements protect against SQL injection, while ACID transactions coordinate related changes. APIs connect applications through HTTP and JSON, but network calls require timeouts, status-code handling, response validation, and safe retry decisions. Exception handling should be specific and purposeful, and validation should address types, formats, ranges, and business rules.

Reliable application integration also depends on layered design, testing, security, privacy, backups, logging, and responsible use of technology. The central principle is to treat every boundary—user input, files, databases, and external services—as a place where assumptions must be checked.

Key Terms

Transient data: Data that exists only during program execution.

Persistent data: Data retained after a program terminates.

Serialization: Converting an in-memory value into a storable or transferable representation.

Deserialization: Reconstructing an in-memory value from a representation.

JSON: A text-based format for structured data exchange.

Stream: An interface for sequentially reading or writing data.

Encoding: A rule for representing characters as bytes.

CRUD: Create, read, update, and delete operations.

Primary key: A column or combination of columns that uniquely identifies a row.

Foreign key: A reference from one table to a key in another table.

Transaction: A group of database operations treated as one logical unit.

ACID: Atomicity, consistency, isolation, and durability properties of transactions.

API: A defined interface through which software components communicate.

HTTP status code: A numeric response classification returned by an HTTP server.

Validation: Checking whether input has acceptable type, form, range, and meaning.

Exception: An indication that an unusual or failure condition has occurred.

SQL injection: An attack caused by treating untrusted input as SQL code.

Repository: A component that abstracts data-access operations.

Idempotent operation: An operation that can be repeated without changing the intended result beyond the first successful application.

Least privilege: Giving a component only the permissions it requires.

Review Questions

Short Answer Questions

1. What is the difference between transient and persistent data?
2. Why should a program specify a text encoding when opening a file?
3. What is the purpose of a primary key?
4. Why are parameterized SQL statements safer than string concatenation?
5. Name two failure conditions that an API client should handle.

Long Answer Questions

1. Compare plain text, JSON, and relational databases for application storage.
2. Explain how serialization and validation work together when processing JSON.
3. Describe the ACID properties and explain why transactions matter.
4. Explain how layered application design improves maintainability and testing.
5. Discuss security and privacy practices for an application that stores personal data and calls an external API.

Analytical / Application Questions

1. Design the data structures and persistence strategy for a small inventory application that supports product lookup, stock updates, and reporting. Justify each choice.
2. An API sometimes returns valid JSON with missing fields and sometimes times out. Propose a robust client-handling strategy, including validation, logging, and user feedback.
3. Two users attempt to register the same learner for the final place in a workshop at the same time. Explain how application validation, database constraints, and transactions should work together to produce a correct result.

Multiple Choice Questions with Answers

Question 1

Which storage option is generally most suitable for related records, constraints, and multi-user transactions?

A. Plain text file

B. Relational database

C. Temporary list

D. Console output

Correct Answer: B. Relational database

Question 2

What does JSON deserialization do?

A. Encrypts a database

B. Deletes invalid files

C. Reconstructs data from a JSON representation

D. Compresses a network connection

Correct Answer: C. Reconstructs data from a JSON representation

Question 3

Which Python structure is most appropriate for fast lookup by a unique identifier?

A. Dictionary

B. List

C. Tuple

D. String

Correct Answer: A. Dictionary

Question 4

Why should SQL parameters be bound through a database API?

A. To increase screen resolution

B. To prevent input from being interpreted as SQL code

C. To remove database constraints

D. To guarantee that a network is available

Correct Answer: B. To prevent input from being interpreted as SQL code

Question 5

Which HTTP status code normally indicates that a new resource was created?

A. 200

B. 201

C. 404

D. 500

Correct Answer: B. 201

Question 6

What is the main purpose of a transaction?

A. To group related database operations into a controlled unit

B. To convert JSON into HTML

C. To rename a programming language

D. To avoid all input validation

Correct Answer: A. To group related database operations into a controlled unit

Question 7

Which practice helps ensure that a file is closed when Python code finishes using it?

A. Using a with statement

B. Using an infinite loop

C. Using string concatenation

D. Disabling exceptions

Correct Answer: A. Using a with statement

Question 8

Which validation checks whether a quantity is within permitted limits?

A. Encoding validation

B. Range validation

C. Path validation only

D. Serialization validation

Correct Answer: B. Range validation

Question 9

Why should an API client use a timeout?

A. To guarantee a successful response

B. To prevent indefinite waiting for an unavailable service

C. To convert HTTP into SQL

D. To make JSON fields mandatory automatically

Correct Answer: B. To prevent indefinite waiting for an unavailable service

Question 10

Which principle gives a database account only the permissions it needs?

A. Eventual consistency

B. Least privilege

C. Serialization

D. Append-only storage

Correct Answer: B. Least privilege