File Systems, Paths, and File Metadata

Introduction

Almost every useful Python program interacts with the file system. Programs read configuration files, save reports, process images, create logs, archive data, and exchange information with other applications. Although file operations often appear simple, reliable file-system programming requires careful treatment of names, paths, permissions, operating-system differences, errors, and security.

Python provides three closely related facilities for this work. The pathlib module supplies an object-oriented interface for paths; the os module exposes operating-system services; and shutil provides higher-level operations such as copying, moving, and removing directory trees. Professional Python code commonly uses pathlib for ordinary path manipulation, while retaining os and shutil when their specialized capabilities are useful.

A file-system operation has at least three dimensions:

1. Location—where an object is identified by its path.
2. State—what kind of object it is, such as a regular file, directory, or symbolic link.
3. Access—whether the current process is permitted to inspect or modify it.

Understanding these dimensions is essential before writing programs that manipulate files.

Learning Objectives

By the end of this chapter, you should be able to:

• distinguish files, directories, symbolic links, and file-system paths;
• explain absolute, relative, current, parent, and home-directory paths;
• construct portable paths with pathlib;
• inspect, create, rename, copy, move, and delete file-system objects;
• obtain selected metadata and interpret timestamps and permissions;
• compare the roles of pathlib, os, and shutil;
• design code that behaves safely across operating systems;
• handle common file-system errors and security risks.

1. The File System as a Hierarchical Namespace

A file system organizes persistent data into named objects. A regular file stores bytes, whose interpretation may be text, an image, serialized data, or executable code. A directory stores references to other objects. A symbolic link is a special object containing a reference to another path. Operating systems may also provide devices, sockets, pipes, and other special objects.

A directory hierarchy resembles a tree, although symbolic links can create graph-like relationships and can even form cycles. Each process has a current working directory (CWD). A relative path is interpreted from that directory, not necessarily from the directory containing the Python source file.

For example, suppose the current working directory is:

/home/ada/project

The path data/report.csv identifies:

/home/ada/project/data/report.csv

An absolute path begins at the file-system root. On Unix-like systems, the root is written /; on Windows, an absolute path commonly begins with a drive or UNC location, such as C:\Users\Ada\project or \\server\share\project.

A relative path has meaning only in relation to a base directory. This makes it convenient for project files but potentially ambiguous when a program is launched from an unexpected directory. A robust program should deliberately choose its base directory rather than assume that the CWD is fixed.

Important path concepts include:

The suffix is a naming convention, not a guarantee of file content. A file named photo.jpg may contain arbitrary bytes, and a file without a suffix may still be a valid text file.

2. Portable Path Construction with pathlib

The preferred modern interface for ordinary path manipulation is pathlib.Path. It avoids manually joining strings with / or \\, because the appropriate separator is selected for the host operating system.

from pathlib import Path

project = Path("project")
report = project / "data" / "report.csv"
print(report)

The / operator here is path joining, not arithmetic division. The resulting path is represented using the platform's conventions.

A path object can be inspected without accessing the file system:

path = Path("archive/report.final.csv")

print(path.name)       # report.final.csv
print(path.stem)       # report.final
print(path.suffix)     # .csv
print(path.parent)     # archive
print(path.parts)      # ('archive', 'report.final.csv') on a relative path

Path objects are generally preferable to strings because they express intent and provide methods for common operations. They can be converted to strings with str(path) when an API requires a string. Many modern Python APIs accept path-like objects directly.

2.1 Absolute and Relative Paths

The following operations create or inspect path forms:

from pathlib import Path

relative = Path("data/input.txt")
absolute = relative.absolute()
resolved = relative.resolve()

print(relative.is_absolute())
print(absolute)
print(resolved)

absolute() makes a path absolute, but its treatment of unresolved components should not be confused with full canonicalization. resolve() normally produces an absolute path and resolves . and ..; it may also resolve symbolic links, depending on platform and options. Resolution can fail or behave differently when components do not exist, so code should use it intentionally rather than automatically.

The current working directory can be obtained with:

cwd = Path.cwd()

The user's home directory can be obtained with:

home = Path.home()

These functions are more portable than embedding a username or operating-system-specific path.

A path relative to the current directory is not necessarily relative to the program file. If a script must find a resource stored beside its source file, a common approach is:

BASE_DIR = Path(__file__).resolve().parent
config_path = BASE_DIR / "config" / "settings.json"

This pattern is useful in scripts, but packaging systems may provide more appropriate resource APIs for installed applications. The key principle is to identify the intended base explicitly.

2.2 Normalization and Validation

The components . and .. have conventional meanings, but normalization is not equivalent to security validation. For example, the user-supplied path ../../secret.txt may escape an intended directory after resolution.

A safe directory-confinement check should compare resolved paths:

from pathlib import Path

root = Path("uploads").resolve()
candidate = (root / user_supplied_name).resolve()

try:
    candidate.relative_to(root)
except ValueError:
    raise ValueError("Path escapes the upload directory")

This example addresses lexical path traversal, but production systems must also consider symbolic links, race conditions, permissions, and concurrent changes. Validation should not be treated as a complete security boundary by itself.

3. Inspecting and Manipulating Files

Before modifying an object, code should establish what it is and handle the possibility that it has disappeared or changed. Typical Path predicates include:

path.exists()
path.is_file()
path.is_dir()
path.is_symlink()

These checks are useful for user feedback and control flow, but they introduce a time gap between checking and using a path. Another process may change the file after the check. Therefore, critical code should also handle exceptions raised by the actual operation.

3.1 Creating Directories

from pathlib import Path

output_dir = Path("output")
output_dir.mkdir(parents=True, exist_ok=True)

parents=True creates missing parent directories. exist_ok=True prevents an exception if the directory already exists. Without it, an existing directory normally causes FileExistsError.

3.2 Reading and Writing Text

For small text files, convenience methods are appropriate:

from pathlib import Path

file_path = Path("notes.txt")
file_path.write_text("First line\nSecond line\n", encoding="utf-8")
content = file_path.read_text(encoding="utf-8")

Specifying an encoding is good practice. The default encoding varies by environment, so relying on it can produce different results on different machines. UTF-8 is often a suitable explicit choice, provided that the data source and destination are expected to use it.

For binary data, use bytes methods or open() in binary mode:

image_path = Path("image.bin")
data = image_path.read_bytes()
image_path.write_bytes(data)

For large files, streaming is preferable to loading the entire content into memory:

with Path("large.log").open("r", encoding="utf-8") as source:
    for line in source:
        process(line)

The with statement ensures that the file is closed promptly, including when an exception occurs.

3.3 Renaming, Replacing, and Deleting

old_name = Path("draft.txt")
new_name = Path("final.txt")
old_name.rename(new_name)

replace() expresses the intention to replace an existing destination where supported:

Path("temporary.txt").replace("final.txt")

Deletion can be performed with:

Path("obsolete.txt").unlink()
Path("empty_folder").rmdir()

rmdir() removes only an empty directory. Recursive deletion requires greater care:

import shutil

shutil.rmtree("build_cache")

Recursive deletion is irreversible and should never be applied to an unvalidated or user-controlled path. A program should log or confirm destructive operations where appropriate.

3.4 Searching with Globs

glob() and rglob() support pattern-based discovery:

from pathlib import Path

for csv_file in Path("data").glob("*.csv"):
    print(csv_file)

for python_file in Path("project").rglob("*.py"):
    print(python_file)

Patterns are not regular expressions. generally matches a sequence of characters within a path component, while * is commonly used for recursive matching when supported by the chosen method.

4. The Roles of os and shutil

pathlib is the clearest default for paths, but os remains important for process and operating-system services.

import os

print(os.getcwd())
os.chdir("some_directory")
print(os.environ.get("HOME"))

Changing the CWD affects the whole process and can make libraries interfere with one another. It is usually better to construct explicit paths than to change directories repeatedly.

os also provides lower-level operations such as os.stat(), os.scandir(), and access to file descriptors. os.scandir() can be efficient when processing many directory entries because directory-entry information may be obtained with fewer system calls:

import os

with os.scandir("data") as entries:
    for entry in entries:
        if entry.is_file():
            print(entry.name)

shutil supplies higher-level file operations:

import shutil
from pathlib import Path

shutil.copy2(Path("report.csv"), Path("backup/report.csv"))
shutil.copytree("source_project", "project_backup", dirs_exist_ok=True)
shutil.move("old_location.txt", "archive/old_location.txt")

copy() copies file content and some metadata, while copy2() attempts to preserve more metadata, such as modification time. Preservation is platform-dependent and is not guaranteed to include every attribute, ownership record, or security descriptor.

5. File Metadata

Metadata is information about a file-system object rather than its application content. Common metadata includes size, timestamps, permissions, ownership, object type, and sometimes creation or birth time.

The stat() method returns a platform-dependent structure:

from pathlib import Path

info = Path("report.csv").stat()
print(info.st_size)
print(info.st_mtime)
print(info.st_mode)

Important fields include:

The exact meaning and availability of fields vary. In particular, st_ctime should not be casually labeled “creation time” on every operating system.

5.1 Timestamps and Time Zones

st_mtime is commonly returned as seconds since the system epoch. It is not a formatted date and should be converted carefully:

from datetime import datetime
from pathlib import Path

seconds = Path("report.csv").stat().st_mtime
local_time = datetime.fromtimestamp(seconds)
utc_time = datetime.fromtimestamp(seconds, tz=None)  # naive local-time representation

For distributed systems, timezone-aware UTC handling is generally safer. File-system timestamp precision also varies: some systems store subsecond precision, while others round values. Consequently, programs should avoid assuming that two timestamps differ only when content differs.

5.2 Permissions

On Unix-like systems, permission bits commonly distinguish read (r), write (w), and execute (x) access for the owner, group, and others. A directory's execute permission means that a process may traverse it; read permission controls listing its entries. File permissions therefore do not have exactly the same practical meaning as directory permissions.

Permissions can be inspected with stat or modified cautiously with chmod:

from pathlib import Path

path = Path("private.txt")
print(oct(path.stat().st_mode))
path.chmod(0o600)

The octal literal 0o600 commonly means read and write for the owner and no permissions for group and others on Unix-like systems. This interpretation is platform-dependent, and Windows does not implement Unix permission semantics identically. A successful chmod() call therefore should not be assumed to create identical access behavior everywhere.

Access is determined by more than permission bits. Operating-system policies, ACLs, ownership, elevated privileges, locks, read-only media, and network file systems can all affect the result. The preferred practice is to attempt the required operation and handle its exception rather than relying solely on os.access() as a prediction.

6. Cross-Platform Behavior

Portable programs avoid assumptions about separators, drive letters, case sensitivity, reserved names, maximum path lengths, and permission semantics.

• Use Path joining instead of string concatenation.
• Avoid hard-coded /, \\, or drive letters unless the application specifically targets one platform.
• Do not assume that A.txt and a.txt are distinct. Case sensitivity depends on the file system.
• Do not assume that every path supports the same characters or length.
• Treat Windows reserved names and special device paths carefully.
• Use Path.home() and environment APIs rather than guessing user directories.
• Avoid changing the current working directory as a hidden side effect.

The os.path module remains available for string-oriented compatibility, but pathlib usually communicates intent more clearly. The os module is appropriate when an operation concerns process state, environment variables, file descriptors, or low-level system behavior.

7. Exceptions and Defensive File-System Code

File operations can fail for ordinary reasons: a path may not exist, a destination may already exist, permission may be denied, a component may be a file when a directory is expected, or a network share may be unavailable.

Common exceptions include:

• FileNotFoundError: a required path component does not exist;
• FileExistsError: an operation requires a new object but the destination exists;
• NotADirectoryError: a path component expected to be a directory is not one;
• IsADirectoryError: a file operation was attempted on a directory;
• PermissionError: the process lacks required access;
• OSError: a broader operating-system error.

Catch specific exceptions when the program can respond meaningfully:

from pathlib import Path

try:
    text = Path("settings.ini").read_text(encoding="utf-8")
except FileNotFoundError:
    text = ""
except PermissionError as error:
    raise RuntimeError("Settings file cannot be read") from error

Do not catch every exception and silently continue. Suppressing errors can cause data loss or produce incorrect results. Error messages should identify the operation and path without exposing sensitive data unnecessarily.

For updates to important files, a safer pattern is to write a temporary file and then replace the destination. The exact atomicity guarantees depend on the operating system and storage system, but replacement is generally safer than truncating the original and writing directly to it. Applications requiring strong durability may also need explicit flushing and synchronization policies.

8. Applied Case: A Portable Report Archiver

This is an illustrative case.

Background

A small college analytics application receives CSV reports in an incoming directory. At the end of each day, it must place processed reports in an archive organized by year and month, preserve useful timestamps, and create a log of failures. The program runs on both Linux laboratory computers and Windows staff computers.

Problem

An initial implementation concatenated strings such as "archive/" + year + "/" + filename, assumed that the current working directory was the project directory, and deleted source files after copying. It failed when launched from a desktop shortcut, used the wrong separator on some systems, and risked losing reports if copying failed.

Analysis

The design requires:

1. an explicit project base directory;
2. portable path construction;
3. validation that each input is a regular file;
4. creation of missing archive directories;
5. preservation of selected metadata;
6. failure handling without destructive cleanup;
7. protection against unexpected path traversal or collisions.

Approach

from pathlib import Path
from datetime import datetime
import shutil

BASE = Path(__file__).resolve().parent
INCOMING = BASE / "incoming"
ARCHIVE = BASE / "archive"


def archive_reports():
    now = datetime.now()
    destination_dir = ARCHIVE / f"{now:%Y}" / f"{now:%m}"
    destination_dir.mkdir(parents=True, exist_ok=True)

    results = []
    for source in INCOMING.glob("*.csv"):
        if not source.is_file():
            continue

        destination = destination_dir / source.name
        if destination.exists():
            results.append((source.name, "skipped: destination exists"))
            continue

        try:
            shutil.copy2(source, destination)
            source.unlink()
            results.append((source.name, "archived"))
        except (OSError, shutil.Error) as error:
            results.append((source.name, f"failed: {error}"))

    return results

This code uses Path for construction and discovery, mkdir() for directory creation, and copy2() to preserve commonly available metadata. It deletes the source only after the copy operation reports success. A production version should consider unique naming, checksums, locking, audit logs, partial-copy cleanup, and whether deletion is actually required.

Outcome

The revised design no longer depends on the launch directory or separator conventions. It provides a clear distinction between skipped, successful, and failed operations. It also makes the destructive step visible and conditional.

Learning Points

• A path is data interpreted in a context; define that context explicitly.
• Portability is achieved through APIs, not by manually guessing separators.
• Existence checks improve messages but do not replace exception handling.
• Copying and deleting should be treated as separate operations with separate failure modes.
• Metadata preservation is best-effort and platform-dependent.

9. Security, Governance, and Professional Practice

File-system code can create serious security vulnerabilities. Path traversal occurs when untrusted input allows access outside an intended directory. Symlink attacks occur when a link redirects an apparently safe path to a sensitive target. Temporary-file races can allow another process to substitute a file between validation and use.

Professional practice includes:

• treating filenames and paths from users, uploads, archives, or network messages as untrusted;
• restricting operations to an explicitly resolved base directory;
• avoiding privileged execution when it is unnecessary;
• using least-privilege permissions;
• avoiding predictable temporary filenames;
• recording relevant operations in audit logs;
• limiting file size, number, and recursion depth in uploads or archive extraction;
• preserving privacy by not logging confidential file contents;
• considering retention and deletion requirements imposed by organizational policy or law.

A file extension is not a security control. If a program accepts uploaded documents, it should validate content, size, encoding, and intended use rather than trusting a name such as report.pdf.

10. Best-Practice Checklist

1. Prefer pathlib.Path for path construction and ordinary file operations.
2. Specify text encodings explicitly.
3. Use context managers for open files.
4. Define the intended base directory rather than assuming the CWD.
5. Handle OSError subclasses at the operation that may fail.
6. Use mkdir(parents=True, exist_ok=True) when appropriate.
7. Treat recursive deletion as a high-risk operation.
8. Do not assume Unix permissions, timestamps, or case behavior apply on Windows.
9. Use shutil.copy2() when preserving common metadata is useful, while recognizing its limits.
10. Validate untrusted paths and protect against traversal and symlink-related attacks.
11. Test on the operating systems and file systems that the application supports.
12. Design for races: a file may be changed or removed after it has been inspected.

Chapter Summary

A file system is a hierarchical namespace containing files, directories, links, and other objects. Paths identify objects, but relative paths depend on a base directory, especially the process's current working directory. Absolute paths provide a complete location, while pathlib offers a readable and portable way to construct and inspect both forms.

pathlib supports existence tests, directory creation, reading, writing, renaming, deletion, and pattern-based discovery. os remains valuable for operating-system services, environment information, directory scanning, and lower-level interfaces. shutil supplies higher-level copying, moving, and recursive directory operations.

Metadata includes size, timestamps, type, permissions, and sometimes ownership. Its meaning and availability vary across platforms; for example, st_ctime is not universally a creation timestamp. Permissions, access-control lists, operating-system policies, and locks all influence whether an operation succeeds. Consequently, robust programs perform the intended operation and handle its exceptions.

Professional file-system programming is also a security responsibility. Explicit base directories, careful validation, least privilege, conservative deletion, metadata awareness, and cross-platform testing help produce dependable applications.

Key Terms

Path: A representation used to identify a file-system object.

File system: The operating-system mechanism that stores and organizes persistent objects.

Regular file: An object containing a sequence of bytes.

Directory: A file-system object that contains references to other objects.

Symbolic link: An object that refers to another path.

Absolute path: A path interpreted from a file-system root or equivalent complete location.

Relative path: A path interpreted relative to a base directory.

Current working directory: The directory used to interpret relative paths for a process.

Pathlib: Python's object-oriented path and file-system path library.

os: Python's interface to operating-system services.

shutil: Python's high-level file and directory operation module.

Metadata: Information describing an object rather than its application content.

Permission: An access rule controlling operations such as reading, writing, or traversal.

Timestamp: A recorded time associated with an event such as modification or access.

Glob: A pattern used to match path names.

Path traversal: An attack that escapes an intended directory through crafted path components.

Current working directory (CWD): Another name for the process's current directory.

Race condition: A correctness or security problem caused by changes between related operations.

Least privilege: The practice of granting only the access required for a task.

Review Questions

Short Answer Questions

1. What is the difference between an absolute path and a relative path?
2. Why is pathlib.Path generally preferable to manually concatenated path strings?
3. What is the practical difference between Path.unlink() and Path.rmdir()?
4. Why should st_ctime not universally be described as a creation timestamp?
5. Name two exceptions that may be raised during file-system operations.

Long Answer Questions

1. Explain how the current working directory affects relative paths and describe two strategies for choosing a reliable base directory.
2. Compare the responsibilities and typical uses of pathlib, os, and shutil.
3. Discuss cross-platform differences involving separators, case sensitivity, permissions, and timestamps.
4. Explain how a program should inspect metadata and handle permissions without assuming identical behavior on all systems.
5. Describe the security risks of accepting user-supplied paths and propose controls for reducing those risks.

Analytical / Application Questions

1. Design a directory-processing program that archives all .log files from an input directory into year/month subdirectories. Identify how it should handle existing destinations, failures, and metadata.
2. A script works when launched from a terminal but fails when launched by an IDE or desktop shortcut. Analyze the likely cause and propose a robust path-resolution strategy.
3. A web application receives a filename such as ../../private/config.ini. Explain how you would validate the path, what additional symlink and race-condition risks remain, and how least privilege would reduce the impact of a failure.

Multiple Choice Questions with Answers

Question 1

Which object-oriented Python class is normally preferred for constructing portable file-system paths?

A. pathlib.Path

B. shutil.FilePath

C. os.Directory

D. sys.PathName

Correct Answer: A. pathlib.Path

Question 2

What does a relative path depend on for its interpretation?

A. The file's suffix

B. A base directory, commonly the current working directory

C. The file's modification timestamp

D. The user's preferred text encoding

Correct Answer: B. A base directory, commonly the current working directory

Question 3

Which expression creates a directory and any missing parents without failing if the directory already exists?

A. Path('out').mkdir(parents=True, exist_ok=True)

B. Path('out').rmdir(parents=True)

C. Path('out').unlink(parents=True)

D. Path('out').touch(parents=True)

Correct Answer: A. Path('out').mkdir(parents=True, exist_ok=True)

Question 4

Which method is appropriate for removing a single file or symbolic link?

A. rmdir()

B. unlink()

C. mkdir()

D. glob()

Correct Answer: B. unlink()

Question 5

What does st_size usually represent for a regular file?

A. The file's size in bytes

B. The number of lines in the file

C. The file's permission mask

D. The number of directory entries

Correct Answer: A. The file's size in bytes

Question 6

Why is explicitly specifying encoding='utf-8' often good practice when reading text?

A. It guarantees the file is compressed

B. It avoids relying on a potentially different environment default

C. It grants read permission

D. It converts binary data into an image

Correct Answer: B. It avoids relying on a potentially different environment default

Question 7

Which module provides high-level operations such as copying directory trees?

A. shutil

B. random

C. calendar

D. decimal

Correct Answer: A. shutil

Question 8

What is a major danger of using an unvalidated user-supplied path?

A. It automatically changes the CPU frequency

B. It can enable path traversal outside the intended directory

C. It always produces UTF-8 text

D. It prevents all file metadata from being read

Correct Answer: B. It can enable path traversal outside the intended directory

Question 9

On many Unix-like systems, what does permission mode 0o600 commonly allow?

A. Read and write for the owner, with no group or other permissions

B. Read, write, and execute for everyone

C. Execute permission only for the group

D. Read permission for everyone and write permission for others

Correct Answer: A. Read and write for the owner, with no group or other permissions

Question 10

Why should a program handle exceptions even after calling Path.exists()?

A. The existence test changes the file into a directory

B. Another process may change or remove the object before the actual operation

C. exists() always deletes symbolic links

D. The test disables operating-system permissions

Correct Answer: B. Another process may change or remove the object before the actual operation