File Systems and Data Persistence

Introduction and Context

A computer system is useful only when it can preserve information beyond the lifetime of a running process. Main memory is fast but volatile: its contents normally disappear when power is removed. File systems provide the operating-system abstraction that makes persistent storage usable. They organize bytes into named objects, control access, locate data efficiently, preserve metadata, and recover from failures.

A file system is more than a collection of files on a disk. It is a set of data structures, algorithms, naming rules, protection mechanisms, and recovery procedures. Its design must reconcile competing goals: high performance, efficient space utilization, reliability, security, scalability, and compatibility. These goals become particularly important in artificial-intelligence and machine-learning systems, where datasets may be extremely large, training checkpoints must survive failures, and many processes may concurrently access shared data.

This chapter examines file and directory abstractions, allocation methods, free-space management, mounting, journaling, consistency and recovery, permissions, virtual file systems, and local and network file systems.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain how operating systems represent files, directories, metadata, and persistent storage.

2. Compare contiguous, linked, indexed, and extent-based allocation.

3. Describe free-space management methods and analyze their trade-offs.

4. Explain mounting, path resolution, file descriptors, and the virtual file system.

5. Distinguish file-system consistency, journaling, and recovery mechanisms.

6. Apply permission, ownership, and access-control concepts to practical systems.

7. Compare local, distributed, and network file systems.

8. Evaluate file-system choices for reliability, performance, security, and AI/ML workloads.

1. The File-System Abstraction

A file is a named, persistent sequence of bytes or a structured object managed by the operating system. The simplest abstraction treats a file as an array of bytes. Programs can read or write at specified offsets, while the operating system hides the physical details of storage devices.

Common file attributes include:

• Name: A human-readable identifier within a directory.
• Identifier: An internal number or object reference, such as a Unix inode number.
• Type: For example, regular file, directory, symbolic link, device, socket, or named pipe.
• Size: The logical length of the file.
• Timestamps: Often including last access, modification, and metadata-change times.
• Ownership and permissions: Information governing access.
• Location metadata: Pointers or extents identifying the blocks containing file data.
• Status flags: Such as read-only, hidden, append-only, or immutable.

The file abstraction separates logical organization from physical placement. A user can refer to `model/checkpoint.bin` without knowing whether its blocks are on a hard disk, solid-state drive, local cache, or remote server.

File Operations

Typical system calls include:

• `create`: Construct a new file and its metadata.
• `open`: Locate a file and create a process-specific file descriptor.
• `read` and `write`: Transfer bytes between a file and memory.
• `seek`: Change the current file offset.
• `close`: Release the descriptor.
• `truncate`: Change file length, often discarding data beyond the new end.
• `stat`: Retrieve metadata.
• `link` and `unlink`: Create or remove directory references.
• `rename`: Atomically change a directory entry on many local file systems.

An open-file description usually contains the current offset, access mode, status flags, and a reference to the underlying file object. A process's file descriptor is a small integer that refers to this kernel-managed object. Multiple descriptors can refer to the same open-file description, so they may share an offset; separately opened descriptors normally have independent offsets.

Directories and Namespaces

A directory maps names to file-system objects. Conceptually, it is a special file containing directory entries, although modern implementations use optimized internal structures rather than exposing raw directory contents as ordinary bytes.

A hierarchical directory tree supports grouping and protection. The root directory is the starting point of an absolute path, such as `/var/log/system.log`. A relative path is interpreted from a process's current working directory, such as `../data/input.csv`.

A hard link is an additional directory entry referring to the same underlying file object. The object remains available until its link count becomes zero and no process still has it open. A symbolic link is a separate file containing a path to another object. Symbolic links can cross file systems and can become dangling when their target is removed.

Path resolution involves parsing components, following directory entries, handling `.` and `..`, checking execute/search permission on directories, and possibly following symbolic links. The kernel must also defend against races and malicious paths. For example, privileged software should avoid assuming that checking a path and later opening it are independent safe actions; an attacker may replace a path component between those operations. Secure APIs can operate relative to an already trusted directory descriptor and impose restrictions on link following.

2. Storage Organization and Allocation

Persistent storage is commonly divided into fixed-size blocks. A file system maps a file's logical blocks to physical or device-level blocks. If the block size is too small, metadata and mapping overhead increase. If it is too large, small files waste space through internal fragmentation and small reads may transfer unnecessary data.

Contiguous Allocation

Contiguous allocation stores a file in consecutive blocks. Metadata can specify a starting block and a length. Sequential access is efficient, and direct access is simple: the physical location of logical block `i` is the starting block plus `i`, subject to the device's addressing scheme.

The main weakness is external fragmentation. Free space may be divided into small gaps, making it difficult to allocate a large file even when total free space is sufficient. Growing a file may require relocation. Contiguous allocation is therefore attractive for read-mostly objects whose sizes are known in advance, but less suitable for general-purpose mutable file systems.

Linked Allocation

In linked allocation, each data block contains a pointer to the next block. A file can grow without requiring contiguous space, and external fragmentation is reduced. However, random access is slow because the chain may need to be traversed. Pointers also consume space and can make corruption of one pointer affect the remainder of a file.

A file-allocation table can move the links from data blocks into a centralized table. This improves some access patterns but introduces table overhead and possible scalability constraints.

Indexed Allocation

Indexed allocation stores pointers to a file's data blocks in one or more index blocks. Direct access is efficient because the system can locate the pointer for a logical block by indexing into the relevant index structure. Large files may require multilevel indexes. Unix inodes traditionally combine direct pointers with single-, double-, or triple-indirect pointers, although modern systems commonly use extent trees.

Index blocks require metadata space. A tiny file may therefore use proportionally more metadata than data. Multilevel indexing also adds lookup steps for very large files, though caching greatly reduces the practical cost.

Extent-Based Allocation

An extent describes a contiguous run of blocks using a starting block and a length. A file can be represented by a small list or tree of extents rather than a pointer for every block. Extents retain much of the sequential performance of contiguous allocation while allowing files to grow in multiple regions.

Extent-based structures are widely used in modern file systems. They are particularly effective for large files, virtual-machine images, databases, and training datasets, which often contain long sequential regions. Fragmentation can still occur when files are repeatedly extended or when free space is heavily used.

Allocation Comparison

No allocation method dominates in every workload. Contiguous allocation provides excellent locality but poor flexibility. Linked allocation grows easily but performs poorly for random access. Indexed allocation supports direct access at the cost of metadata. Extents provide a practical compromise, especially when allocation policies preserve large free runs.

The operating system also uses caching, read-ahead, delayed allocation, and write clustering. Delayed allocation postpones physical placement until more information about the write pattern is available, often improving locality. Its risk is that more recently written data may be lost if a failure occurs before it reaches stable storage.

3. Free-Space Management

A file system must record which blocks are available. This information is used when files grow, directories expand, metadata is created, or deleted space is reused.

Bitmaps

A bitmap contains one bit per allocation unit. One convention marks a free block with 0 and an allocated block with 1; implementations may choose the reverse. Bitmaps make it possible to search for individual free blocks and runs of free blocks. They also support consistency checks because the allocation state can be compared with the blocks referenced by file metadata.

Free Lists and Grouping

A free list links available blocks. It is simple and can be efficient when only one block is needed, but finding a long contiguous run may be expensive. Grouping stores addresses of many free blocks together, reducing traversal overhead. Counting records a starting block and the number of consecutive free blocks, which is useful for extent-based allocation.

Space Reclamation and Fragmentation

When a file is deleted, its blocks become candidates for reuse. The file system may coalesce adjacent free regions or use allocation policies that keep related blocks near one another. Fragmentation is both a performance and capacity issue. On rotating disks, scattered blocks increase seek and rotational delay. On solid-state storage, seek is less important, but fragmentation can still increase metadata operations, reduce read-ahead effectiveness, and complicate garbage collection inside the flash translation layer.

Storage reclamation must also account for snapshots, copy-on-write references, open-but-unlinked files, and delayed deletion. A block cannot be reused while any valid reference still depends on it.

4. File-System Structures and Mounting

A file system typically contains a superblock or equivalent volume metadata describing block size, capacity, feature flags, and locations of important structures. Other structures may include allocation maps, inode tables, directory indexes, journals, checksums, and data blocks.

A partition or logical volume is a region that can hold a file system. A device may contain several partitions, and logical-volume systems can combine or subdivide physical devices. Before a file system can be accessed through a directory hierarchy, it must be mounted.

Mounting associates a file-system root with a directory called a mount point. If a file system is mounted at `/data`, the previous contents of that directory are hidden while the mount is active. The kernel maintains a mount table and consults it during path traversal. Unmounting requires that the file system no longer have active references that would make removal unsafe, although specialized mechanisms may perform lazy or forced operations with additional risks.

Mount options can affect security and behavior. Examples include read-only mounting, disabling execution of binaries, restricting device files, and preventing set-user-ID behavior. Automatic mounting services can improve convenience but increase the importance of authenticating devices and validating file-system metadata.

5. Consistency, Journaling, and Recovery

File-system consistency means that the persistent metadata and data satisfy the structural rules of the file system. For example, a block should not simultaneously be allocated to two unrelated files, a directory entry should refer to a valid object, and free-space accounting should agree with allocation metadata.

Consistency does not necessarily mean that every application-level update is complete. A database transaction or ML checkpoint may require stronger semantic guarantees than the file system alone provides.

Failure Modes

Failures include power loss, kernel crashes, device errors, controller faults, software defects, and malicious tampering. A write may be interrupted after some blocks reach storage but before related metadata does. Volatile caches may contain newer data than the device. Modern storage devices also have their own controllers and persistence guarantees, so an operating system must use appropriate flush or force-unit-access mechanisms when ordering matters.

Journaling

A journaling file system records intended metadata updates, or sometimes data updates as well, in a journal before applying them to their normal locations. A simplified metadata-journaling sequence is:

1. Construct a transaction containing related metadata changes.

2. Write the transaction to the journal.

3. Ensure the journal commit record reaches stable storage.

4. Apply the changes to their home locations.

5. Mark or reclaim the completed journal space.

After a crash, recovery replays committed transactions and discards incomplete ones. This greatly reduces the time required to restore metadata consistency compared with scanning an entire volume.

Journaling modes differ. Metadata journaling protects file-system structure but may leave recently written file data old or partially updated. Ordered journaling ensures that data blocks are written before the metadata that makes them visible. Full data journaling records both data and metadata, providing stronger ordering at a performance cost. Journaling is not a substitute for backups: a journal can preserve structural consistency while failing to recover accidentally deleted or corrupted application content.

Copy-on-Write and Checksums

Copy-on-write file systems write modified blocks to new locations and update references only after the new blocks are ready. This supports snapshots and can provide strong crash-consistency properties. Checksums on metadata and, in some systems, file data help detect silent corruption. Redundant copies or parity can help recover from detected failures.

Recovery Tools

Traditional consistency checkers scan metadata and allocation structures, identify contradictions, and repair them where possible. Such checks may be lengthy for large volumes. Modern systems use journals, trees, checksums, snapshots, redundancy, and background scrubbing to reduce recovery time and detect corruption earlier.

A reliable recovery strategy also includes application-level procedures: replicated data, tested backups, versioned checkpoints, and documented recovery objectives. The recovery point objective concerns how much recent data may be lost; the recovery time objective concerns how quickly service must be restored.

6. Protection, Permissions, and Security

File-system protection controls who may read, modify, execute, rename, or delete objects. In Unix-like systems, a file commonly has an owner, a group, and permission bits for owner, group, and others. For a regular file, read permits content access, write permits modification, and execute permits execution. For a directory, read generally permits listing names, write permits creating or removing entries, and execute permits searching or traversing the directory.

A permission string such as `rwxr-x---` grants read, write, and execute to the owner; read and execute to the group; and no permissions to others. Numeric notation represents each triplet as a three-bit value: read is 4, write is 2, and execute is 1. Thus `750` means owner `7` (`rwx`), group `5` (`r-x`), and others `0`.

Access-control lists (ACLs) extend basic owner-group-other permissions by granting or denying rights to additional users and groups. Mandatory access-control systems impose policy labels and system-wide rules that users cannot freely override. Examples include security frameworks that confine services even when traditional permissions appear permissive.

Other protection mechanisms include encryption at rest, encrypted transport, secure deletion policies, quotas, auditing, and integrity measurement. File names and metadata can reveal sensitive information even when file contents are encrypted. Least privilege requires giving processes only the access necessary for their tasks.

Special cases require care. A process may retain access to an open file after its directory name is removed. Set-user-ID executables can change effective identity and must be protected from unsafe modification. Temporary files should be created with exclusive, unpredictable names to prevent symbolic-link attacks.

7. The Virtual File System

The virtual file system (VFS) is an abstraction layer in the kernel that presents a common interface to different file-system implementations. Applications use operations such as open, read, write, and stat without knowing whether the object is on a local disk, a memory-backed file system, a network share, or a pseudo-file system.

A VFS commonly represents:

• Inodes or vnode-like objects: File identity and metadata.
• Dentries: Components in the directory-name cache.
• File objects: Per-open state, including offsets and flags.
• Superblock-like objects: Information about a mounted file system.

The VFS dispatches operations to implementation-specific methods. A local file system may translate a read into block-cache operations, whereas a network implementation may issue remote protocol requests. Caches improve performance but create coherence and invalidation challenges.

Pseudo-file systems expose kernel or process information through file-like interfaces. Examples include process information, device interfaces, temporary memory-backed storage, and control endpoints. These objects may not correspond to persistent disk blocks at all, illustrating that the file abstraction is broader than storage.

8. Local and Network File Systems

Local file systems directly manage storage attached to the host. Examples in contemporary operating systems include ext4, XFS, APFS, NTFS, and ZFS. Their design choices differ in journaling, copy-on-write behavior, checksumming, snapshots, allocation structures, encryption, and scalability.

A network file system makes remote storage appear as part of a local namespace. Common models include:

• NFS: A stateless or largely stateless protocol family in which clients issue operations to servers, with later versions adding stronger state and security features.
• SMB: Widely used for file and printer sharing, especially in heterogeneous enterprise environments.
• Distributed file systems: Systems designed to scale across many machines, often optimizing for large files, failure tolerance, and high aggregate throughput.
• Object storage gateways: Services that expose objects through APIs rather than traditional hierarchical file semantics.

Network file systems face latency, disconnection, partial failure, authentication, authorization, caching, and consistency problems. A local assumption such as “a file is always available after it is opened” may be false across a network. Caching can improve performance but may produce stale data unless the protocol defines a coherence model.

File semantics also vary. Some systems provide close-to-open consistency, meaning changes are expected to become visible when a writer closes a file and another client opens it. Others provide weaker or application-configurable consistency. Distributed locking can coordinate updates but introduces failure and recovery complexity.

9. File Systems for AI and Machine Learning

AI/ML workloads place unusual demands on persistence. Training datasets may contain millions of small objects or a few very large sharded files. Training often involves sequential reads by many workers, random access to samples, temporary preprocessing data, and frequent checkpoint writes.

For large sequential datasets, extent-friendly local storage, parallel file systems, or object storage with appropriate caching may provide high throughput. For many small files, packaging records into larger shards can reduce metadata operations. Checkpoints should be written using a temporary name, flushed as required, and atomically renamed only after the complete checkpoint is durable. Versioned checkpoints protect against corruption and permit rollback.

Data pipelines should distinguish durable source data from disposable caches. Caches can use faster temporary storage and be regenerated, while source datasets require integrity checks, access control, retention policies, and backup. Dataset manifests containing file names, sizes, hashes, and schema versions improve reproducibility.

File systems also affect privacy and governance. Training data may contain personal or regulated information. Encryption, access logging, retention limits, deletion workflows, and provenance records are essential. Deleting a directory may not remove copies in snapshots, backups, caches, or replicated object stores; governance processes must address the complete data lifecycle.

10. Illustrative Case: Reliable Checkpointing for a Distributed Training Service

Background

Consider an illustrative university research service training a language model on eight worker machines. Each worker writes model-state fragments to shared storage every thirty minutes. A complete checkpoint consists of a manifest and multiple tensor files.

Problem

The service experiences occasional power failures. After restart, some manifests refer to missing or partially written tensor files. In other cases, workers simultaneously update a checkpoint directory, leaving an inconsistent mixture of versions. Restarting training is slow because the system must inspect large directories and manually identify a usable checkpoint.

Analysis

The underlying problem is not simply insufficient disk capacity. The workflow lacks an atomic publication protocol and does not distinguish temporary data from committed data. Network storage also means that visibility and durability may not occur at exactly the same time for all clients.

Approach

The service adopts the following design:

1. Each training attempt writes into a uniquely named temporary directory.

2. Tensor files are written completely and validated with hashes.

3. Workers coordinate through a metadata service to confirm that all fragments exist.

4. The manifest records expected files, sizes, hashes, model version, and training step.

5. The manifest is flushed according to the storage system's durability requirements.

6. The completed directory is renamed or published through an atomic metadata operation.

7. A small `latest` pointer is updated only after publication.

8. Older checkpoints are retained according to a tested retention policy.

9. Recovery scans only published manifests and verifies hashes before loading.

Outcome

A crash may leave an unpublished temporary directory, but it cannot make that directory appear as the latest valid checkpoint. Recovery becomes deterministic, and corrupted fragments are detected before training resumes. The design does not eliminate the need for backups or replication, but it separates atomic visibility from long-term durability and integrity.

Learning Points

The case demonstrates that file-system primitives must be combined with application-level protocols. Atomic rename is useful for publication, but it does not by itself guarantee that every prior write has reached stable storage. Hashes detect corruption but do not restore lost data. A robust system therefore uses clear commit points, durability operations, validation, retention, and recovery testing.

11. Design Trade-offs, Best Practices, and Limitations

File-system design always involves trade-offs. Journaling improves recovery but adds writes. Copy-on-write enables snapshots but may amplify writes and fragment storage. Strong consistency simplifies reasoning but can reduce availability or performance in distributed systems. Encryption protects confidentiality but introduces key-management obligations and sometimes computational overhead.

Recommended practices include:

• Select a file system according to workload, device, scale, and recovery requirements.
• Use backups that are separate from the primary system and regularly test restoration.
• Define and document recovery point and recovery time objectives.
• Apply least privilege and avoid world-writable directories for sensitive data.
• Use atomic publication patterns for generated files and checkpoints.
• Flush data and metadata when durability is a contractual requirement.
• Monitor capacity, inode or object counts, latency, errors, fragmentation, and unusual access.
• Validate backups, snapshots, and replicas rather than assuming they are usable.
• Protect keys separately from encrypted data.
• Use hashes, manifests, and versioning for scientific and ML reproducibility.
• Treat network storage as failure-prone and design for timeouts, retries, and partial failure.

Important limitations remain. No file system can guarantee recovery from every hardware failure without redundancy. Permissions do not prevent a trusted administrator from misusing data. Journaling cannot restore overwritten application content. Snapshots can consume substantial space, and distributed consistency can never be evaluated independently of the application's requirements.

Chapter Summary

File systems provide persistent naming, organization, protection, and recovery for data stored beyond process and system lifetimes. Files are accessed through operations and descriptors, while directories provide hierarchical namespaces using paths, hard links, and symbolic links. Allocation methods include contiguous, linked, indexed, and extent-based designs, each balancing locality, growth, metadata cost, and random access. Bitmaps, free lists, grouping, and counting manage unused space.

Mounting attaches a file system to a namespace. The VFS presents a common interface to local, network, pseudo, and other implementations. Consistency mechanisms include journaling, copy-on-write, checksums, recovery tools, snapshots, and backups; these provide different guarantees and are not interchangeable. Permissions, ACLs, mandatory controls, encryption, auditing, and least privilege protect data. Network and distributed systems introduce latency, caching, consistency, authentication, and partial-failure challenges. AI/ML systems particularly benefit from sharded datasets, atomic checkpoint publication, manifests, hashes, versioning, and tested recovery procedures.

Key Terms

File: A named persistent object containing data.

Directory: A namespace mapping names to file-system objects.

Inode: Metadata structure identifying a file and its attributes.

File descriptor: Process-local reference to an open file object.

Hard link: Directory entry referring to an existing file object.

Symbolic link: File containing a path to another object.

Contiguous allocation: Storage of a file in consecutive blocks.

Extent: Description of a consecutive range of blocks.

Indexed allocation: Allocation using one or more blocks of data pointers.

Bitmap: Bit array representing allocation status.

Mount: Association of a file-system root with a directory in a namespace.

Journaling: Recording updates in a log to support crash recovery.

Copy-on-write: Writing changed data to new locations before updating references.

Virtual file system: Kernel interface abstracting multiple file-system types.

ACL: Access-control list granting permissions to specified principals.

Consistency: Satisfaction of structural and ordering requirements after operations or failures.

Snapshot: Point-in-time representation of a file-system state.

Close-to-open consistency: Model in which updates are expected to appear across close and subsequent open operations.

Recovery point objective: Maximum acceptable amount of data loss measured in time.

Recovery time objective: Target time for restoring service after failure.

Review Questions

Short Answer Questions

1. What information is commonly stored in file metadata?

2. How does a symbolic link differ from a hard link?

3. What problem does a file-system journal address?

4. What is the purpose of mounting?

5. Why are bitmaps useful for free-space management?

Long Answer Questions

1. Compare contiguous, linked, indexed, and extent-based allocation, including their performance and fragmentation trade-offs.

2. Explain how a virtual file system allows applications to use different local and network file systems through a common interface.

3. Describe metadata journaling, ordered journaling, and full data journaling.

4. Explain Unix directory permissions and contrast them with ACLs and mandatory access control.

5. Discuss the major design challenges of network file systems.

Analytical / Application Questions

1. Design a reliable checkpoint-publication protocol for a distributed ML training job. Identify where atomicity, durability, validation, and recovery are required.

2. A storage system has abundant total free space but cannot efficiently allocate a large sequential file. Explain how this can occur and propose remedies.

3. An organization wants to replace tested backups with snapshots. Analyze the security, recovery, capacity, and failure-domain limitations of this decision.

Multiple Choice Questions with Answers

Question 1

Which allocation method represents a file using ranges of consecutive blocks?

A. Linked allocation

B. Extent-based allocation

C. Bitmap allocation

D. Directory allocation

Correct Answer: B. Extent-based allocation

Question 2

What is the primary purpose of a file-system journal?

A. Encrypt every file automatically

B. Increase the logical size of files

C. Support recovery of file-system updates after a crash

D. Replace all user permissions

Correct Answer: C. Support recovery of file-system updates after a crash

Question 3

For a Unix directory, execute permission generally allows a user to:

A. Search or traverse the directory

B. List every file name

C. Modify every file's contents

D. Change the file system's block size

Correct Answer: A. Search or traverse the directory

Question 4

What does mounting do?

A. Compresses all files on a device

B. Associates a file-system root with a directory in a namespace

C. Deletes the previous contents of a partition

D. Converts a hard link into a symbolic link

Correct Answer: B. Associates a file-system root with a directory in a namespace

Question 5

Which structure commonly records whether allocation units are free or occupied?

A. Bitmap

B. Symbolic link

C. File descriptor

D. Superuser shell

Correct Answer: A. Bitmap

Question 6

Which statement about a hard link is correct?

A. It always stores a URL to a remote file

B. It is a separate file containing a target path

C. It is an additional directory entry for the same underlying file object

D. It can refer only to a directory on another file system

Correct Answer: C. It is an additional directory entry for the same underlying file object

Question 7

What is a major advantage of copy-on-write file systems?

A. They never require metadata

B. They support snapshots by preserving older blocks while new blocks are written

C. They eliminate the need for backups

D. They guarantee application-level transaction completion

Correct Answer: B. They support snapshots by preserving older blocks while new blocks are written

Question 8

Which practice best supports reliable publication of an ML checkpoint?

A. Write directly into the directory named latest without validation

B. Publish a fully written and validated temporary directory atomically

C. Delete older checkpoints before writing the new one

D. Disable all storage synchronization operations

Correct Answer: B. Publish a fully written and validated temporary directory atomically

Question 9

What does a virtual file system primarily provide?

A. A common kernel interface for different file-system implementations

B. A physical replacement for all storage devices

C. A method for converting files into processes

D. A guarantee that network storage has zero latency

Correct Answer: A. A common kernel interface for different file-system implementations

Question 10

Which statement best distinguishes backups from journaling?

A. Backups repair only directory permissions

B. Journaling and backups provide exactly the same protection

C. Journaling supports structural crash recovery, while backups can restore lost or deleted data

D. Backups are useful only for temporary files

Correct Answer: C. Journaling supports structural crash recovery, while backups can restore lost or deleted data