Text and Binary File I/O

Introduction

Programs often need to preserve information after they terminate, exchange data with other programs, or process datasets larger than available memory. Files provide persistent storage for these purposes. Python offers a high-level file interface that supports text documents, binary resources, structured data, logs, and temporary files.

File I/O is more than calling open() and read(). Reliable programs must choose an appropriate mode, interpret character encodings correctly, handle platform-specific newline conventions, control buffering when necessary, move deliberately through a file, and guarantee that operating-system resources are released. This chapter develops those skills for professional Python programming.

Learning Objectives

By the end of this chapter, you should be able to:

• open files using suitable modes and encodings;
• read, write, append, and update text and binary files;
• distinguish characters, bytes, and encodings;
• use seek(), tell(), and truncate() safely;
• explain buffering and newline translation;
• guarantee deterministic file closure with context managers;
• identify common file-I/O errors and security risks;
• design a practical file-processing solution.

1. The File Abstraction

A file is a named sequence of persistent data maintained by an operating system or storage service. Python represents an open file through a file object. The object maintains a current position, sometimes called the file pointer, indicating where the next read or write occurs.

A typical interaction has four stages:

1. identify a path;
2. open the file with an appropriate mode;
3. perform operations;
4. close the file.

The path may be relative to the program's current working directory or absolute. pathlib.Path is generally preferable to manual string concatenation because it provides clearer, platform-independent path operations.

from pathlib import Path

path = Path("data") / "sales.txt"
print(path)

A relative path depends on the process's current working directory, not necessarily the directory containing the Python source file. Professional applications should make this assumption explicit.

2. Opening Files and File Modes

The built-in open() function creates a file object:

open(file, mode="r", buffering=-1, encoding=None,
     errors=None, newline=None, closefd=True, opener=None)

The most important arguments are file, mode, encoding, errors, and newline.

Modes may be combined. For example, rb reads bytes, wb writes bytes, and a+ permits reading while all writes occur at the end. The t and b modifiers are mutually exclusive.

The distinction between w and a is especially important. Opening with w immediately discards existing contents by truncating the file to zero length. Opening with a preserves existing contents and positions writes at the end. Use x when overwriting an existing file would be unacceptable.

with open("report.txt", "w", encoding="utf-8") as file:
    file.write("Annual report\n")

The with statement is discussed in detail later, but it is the recommended default.

3. Deterministic Resource Closure

An open file consumes operating-system resources and may hold buffered data that has not yet reached storage. Calling close() releases the resource and flushes buffered output:

file = open("notes.txt", "w", encoding="utf-8")
try:
    file.write("Important information\n")
finally:
    file.close()

The finally block ensures closure even if an exception occurs. The more readable equivalent is a context manager:

with open("notes.txt", "w", encoding="utf-8") as file:
    file.write("Important information\n")

When execution leaves the with block, Python calls the file object's cleanup operation, including when the block exits because of an exception. This is deterministic closure: the programmer defines the point at which the resource is released rather than relying on garbage collection.

A file object can be tested with file.closed. Calling close() more than once is harmless, but operations on a closed file raise ValueError.

4. Text Files: Characters, Bytes, and Encodings

Text files store bytes, but programmers usually interpret those bytes as characters. An encoding specifies the mapping between characters and byte sequences. UTF-8 is the dominant general-purpose encoding because it represents Unicode text and remains compatible with ASCII for common characters.

When a text file is read, Python performs decoding:

bytes on storage → decoded characters in Python

When text is written, Python performs encoding:

characters in Python → encoded bytes on storage

Always specify an encoding when the file format requires predictable behavior:

with open("names.txt", "r", encoding="utf-8") as file:
    names = file.read()

If no encoding is supplied, Python uses a platform-dependent default. This can produce a program that works on one computer and fails on another. UnicodeDecodeError commonly indicates that the chosen encoding cannot interpret the file's bytes. UnicodeEncodeError indicates that a character cannot be represented by the selected output encoding.

The errors parameter controls encoding or decoding failures. Common policies include:

• strict: raise an exception; the safest default for preserving correctness;
• ignore: discard invalid data; useful only when loss is acceptable;
• replace: substitute an error marker; useful for diagnostic display;
• surrogateescape: preserve otherwise-invalid bytes for certain system interfaces.

Silently ignoring encoding errors can corrupt names, identifiers, or financial data. Error handling should therefore be an explicit design decision.

Newline Conventions

Operating systems and file formats may represent line endings differently. Python text mode normally performs newline translation. With newline=None (the default), input recognizes common line endings and returns \n in Python strings; output translates \n according to the platform.

Useful alternatives are:

• newline="": enable universal newline recognition without translating returned line endings;
• newline="\n": specify a particular output line ending;
• newline values such as "\r\n": request a particular translation when writing.

For portable text files, explicitly writing \n with UTF-8 is often suitable. CSV files require additional care; the csv module is designed to manage quoting and newline behavior and is normally used with newline="".

import csv

with open("students.csv", "w", newline="", encoding="utf-8") as file:
    writer = csv.writer(file)
    writer.writerow(["ID", "Name", "Score"])

5. Reading Text Files

read() returns text from the current position. Without an argument it reads to the end; with a size argument it requests up to that many characters, although buffering and text decoding mean that application-level behavior should be understood as a character-oriented interface.

with open("article.txt", encoding="utf-8") as file:
    complete_text = file.read()

readline() reads one line, retaining the line-ending character when present. Iterating over a file is usually the best approach for line-oriented processing because it avoids loading the entire file into memory:

with open("server.log", encoding="utf-8") as file:
    for line_number, line in enumerate(file, start=1):
        if "ERROR" in line:
            print(line_number, line.rstrip("\n"))

readlines() returns a list of all remaining lines. It is convenient for small files but can consume substantial memory for large files.

A robust line-processing pattern preserves meaningful whitespace while removing only the line ending. rstrip() without an argument removes all trailing whitespace, which may accidentally remove significant spaces. Prefer rstrip("\r\n") when only line endings should be removed.

6. Writing and Appending Text

write() writes one string and returns the number of characters accepted by the text stream. It does not automatically add a newline:

with open("output.txt", "w", encoding="utf-8") as file:
    file.write("First line\n")
    file.write("Second line\n")

writelines() writes an iterable of strings, but it also does not insert separators:

lines = ["one\n", "two\n", "three\n"]
with open("output.txt", "w", encoding="utf-8") as file:
    file.writelines(lines)

Appending is appropriate for logs and event histories:

with open("events.log", "a", encoding="utf-8") as file:
    file.write("User signed in\n")

Appending does not, by itself, provide a complete multi-process logging solution. Concurrent writers may interleave data, and a program may need the logging module, file locks, or a database for stronger guarantees.

7. Binary File I/O

Binary mode exposes the file as bytes rather than decoded characters. It is appropriate for images, audio, compressed archives, executable files, serialized formats, and network-oriented protocols.

with open("photo.jpg", "rb") as source:
    header = source.read(16)

A binary read returns a bytes object. Binary writes require bytes-like data:

payload = bytes([0, 1, 2, 255])
with open("data.bin", "wb") as file:
    file.write(payload)

Passing a string to a binary file raises TypeError. Conversely, passing bytes to a text file is invalid because the text layer expects strings.

Binary copying should use chunks rather than loading a large object into memory:

from pathlib import Path

def copy_binary(source: Path, destination: Path, chunk_size: int = 64 * 1024) -> None:
    with source.open("rb") as input_file, destination.open("wb") as output_file:
        while chunk := input_file.read(chunk_size):
            output_file.write(chunk)

The assignment expression stops when read() returns b"", the binary end-of-file value. A chunk size is a performance parameter, not a correctness requirement; it should be chosen according to workload and environment.

8. File Position, seek(), and tell()

A file object maintains a current position. tell() reports that position, while seek() changes it:

with open("sample.txt", "r+b") as file:
    original_position = file.tell()
    file.seek(0)
    first_byte = file.read(1)

The most common form is seek(offset, whence), where whence is:

• os.SEEK_SET (0): position relative to the beginning;
• os.SEEK_CUR (1): position relative to the current position;
• os.SEEK_END (2): position relative to the end.

import os

with open("data.bin", "rb") as file:
    file.seek(-4, os.SEEK_END)
    final_bytes = file.read(4)

In binary streams, offsets are naturally measured in bytes. Text streams are more restricted because variable-width encodings and newline translation make arbitrary byte positions unsafe. A position returned by tell() for a text stream should generally be treated as an opaque value to be passed back to seek(), rather than as a character count.

A common random-access pattern is:

with open("records.bin", "rb") as file:
    file.seek(100)
    record = file.read(20)

This assumes a known binary layout and record size. It is not valid to assume that the 100th byte corresponds to the 100th character in a UTF-8 text file.

9. Truncating Files

truncate(size=None) changes the file's length. If size is omitted, the file is truncated at the current position. If a size is supplied, the file is reduced or extended to that length, subject to the stream and operating-system behavior.

with open("temporary.bin", "r+b") as file:
    file.seek(10)
    file.truncate()

This leaves only the first ten bytes. Truncation is destructive and should be used deliberately. It is not the same as deleting a file, and it does not necessarily overwrite removed data securely on every storage medium.

A safe replacement strategy for important output is to write a complete temporary file, flush and close it, and then replace the destination using an appropriate atomic replacement operation such as Path.replace(). This reduces the chance of leaving a partially written file if the process fails during output.

10. Buffering and Flushing

A file object commonly contains layers: an application-facing text or binary interface, a buffered layer, and an operating-system file descriptor. Buffering groups operations to reduce expensive system calls.

The buffering argument can influence behavior:

• buffering=0 requests unbuffered binary I/O;
• a positive value requests an approximate buffer size;
• buffering=1 requests line buffering where supported, especially for text output;
• -1 selects the default buffering policy.

flush() transfers Python-level buffered data to the underlying stream, but it does not necessarily guarantee that data has reached stable physical storage. For stronger durability requirements, a binary file descriptor may be synchronized with os.fsync(), although this has performance costs and depends on the storage system.

Closing a file flushes its buffers. A program should not use flush() as a substitute for closure. For interactive output, explicit flushing may be useful:

import sys

print("Processing...", end="", flush=True)
sys.stdout.write(" done\n")

11. Exceptions and Defensive File Handling

File operations can fail for reasons outside the program's control. Important exceptions include:

Catch only errors that can be handled meaningfully. Avoid catching broad Exception around an entire function because it may conceal programming defects.

from pathlib import Path

try:
    with Path("config.txt").open(encoding="utf-8") as file:
        configuration = file.read()
except FileNotFoundError:
    configuration = ""
except UnicodeDecodeError as error:
    raise ValueError("Configuration is not valid UTF-8") from error

The from error syntax preserves the original exception as the cause, improving diagnosis.

12. Applied Case: Processing a Large Sensor Log

Background

A manufacturing service receives a UTF-8 text log containing one sensor reading per line. Each line has the form timestamp,device_id,value. The file can exceed several gigabytes, so it cannot safely be loaded into memory at once.

Problem

The service must calculate the number of valid records and the average value while preserving a separate file containing malformed lines for later inspection.

Analysis

The solution should stream line by line, specify UTF-8 explicitly, distinguish malformed input from operational errors, and use a context manager for both files. A record is valid only when it contains exactly three fields and the numeric value can be converted to float.

Solution

from pathlib import Path


def process_log(source: Path, rejected: Path) -> tuple[int, float | None]:
    count = 0
    total = 0.0

    with source.open("r", encoding="utf-8", newline="") as input_file, \
         rejected.open("w", encoding="utf-8", newline="") as rejected_file:
        for line_number, raw_line in enumerate(input_file, start=1):
            line = raw_line.rstrip("\r\n")
            fields = line.split(",")

            if len(fields) != 3:
                rejected_file.write(f"{line_number}: {line}\n")
                continue

            try:
                value = float(fields[2])
            except ValueError:
                rejected_file.write(f"{line_number}: {line}\n")
                continue

            count += 1
            total += value

    average = total / count if count else None
    return count, average

The average is calculated as the sum of valid values divided by the number of valid records. The calculation is applicable only when count > 0; otherwise, the function returns None rather than attempting division by zero.

Outcome

The program uses bounded memory, preserves invalid input, closes both files deterministically, and distinguishes an empty valid dataset from an operational failure. In production, a CSV parser would be preferable if fields may contain quoted commas, and validation would also check timestamps, device identifiers, and acceptable numeric ranges.

Learning Points

The case demonstrates that file-I/O design includes format interpretation, memory management, error policy, resource management, and data-quality decisions. A syntactically functioning program may still be professionally inadequate if it silently loses invalid data or relies on an unspecified encoding.

13. Security, Reliability, and Professional Practice

File paths may contain user-controlled data. Applications should validate permitted locations, avoid unintended path traversal such as ../../secret.txt, and apply least-privilege filesystem permissions. Never assume that a filename is harmless merely because it came from a form or API request.

Avoid constructing shell commands from filenames. Use Python APIs and pass arguments safely when subprocesses are unavoidable. Sensitive files should not be written to publicly accessible directories, and temporary files should be created using secure facilities such as tempfile.

For important files, consider:

• explicit encoding and format documentation;
• atomic replacement rather than direct destructive updates;
• checksums or signatures when integrity matters;
• file locks or a database for concurrent updates;
• bounded reads to prevent memory exhaustion;
• logging that does not expose confidential data;
• access controls and retention policies.

Binary formats may contain executable or malicious content. Reading bytes is not equivalent to safely interpreting them. Deserialization formats such as Python's pickle must never be loaded from untrusted sources because loading can execute arbitrary code.

14. Contemporary Practice and Design Guidance

For simple text and binary files, the standard library is usually sufficient. pathlib improves path manipulation; csv handles tabular text; json handles interoperable structured text; gzip and related modules handle compressed streams; and logging manages application logs.

Asynchronous applications may use asynchronous file libraries, but ordinary local filesystem operations are often mediated by threads or operating-system caching. Async syntax does not automatically make storage faster. For large-scale data, databases, object storage, or columnar formats may provide better indexing, concurrency, and integrity than manually managed files.

A practical decision sequence is:

1. Is the data character text or raw bytes?
2. What encoding and newline rules define the format?
3. Should existing content be preserved, replaced, or rejected if present?
4. Can the data be streamed rather than loaded wholly into memory?
5. What must happen if decoding, permission, or storage errors occur?
6. How will closure, durability, concurrency, and security be guaranteed?

Chapter Summary

Python file I/O provides persistent data access through file objects. The open() mode determines whether a file is read, replaced, appended, exclusively created, or updated, while the b modifier selects binary operation. Text mode decodes bytes into characters and encodes characters when writing; specifying UTF-8 or another deliberate encoding prevents platform-dependent behavior. Newline translation must be considered when creating portable text and CSV files.

read(), readline(), iteration, and readlines() offer different memory and convenience trade-offs. Binary processing should use bytes and, for large resources, bounded chunks. tell() and seek() support controlled navigation, with arbitrary offsets safest in binary streams. truncate() changes file length and is destructive. Buffering improves efficiency, while flush() and closure control when buffered data is transferred. Context managers provide deterministic resource closure and should be the normal pattern.

Professional file handling also requires targeted exception handling, secure path management, explicit data-quality policies, safe replacement strategies, and awareness of concurrency and durability limitations. Good file-I/O code is therefore both technically correct and operationally responsible.

Key Terms

File object: Python object representing an open stream connected to a file.

Path: A relative or absolute name identifying a filesystem location.

Text mode: File mode that decodes and encodes character data.

Binary mode: File mode that reads and writes bytes without text translation.

Encoding: Rule mapping characters to byte sequences.

UTF-8: Variable-length Unicode encoding widely used for text interchange.

Decoding: Converting bytes into characters.

Encoding operation: Converting characters into bytes.

Newline translation: Conversion between platform or format-specific line endings.

Buffering: Temporarily collecting data to reduce I/O operations.

Flush: Transfer of buffered data to an underlying stream.

File pointer: Current position from which the next operation occurs.

seek(): Method that changes the current file position.

tell(): Method that reports the current file position.

truncate(): Method that changes the length of a file.

Context manager: Construct that performs setup and guaranteed cleanup around a block.

Deterministic closure: Predictable release of a resource at a defined program point.

Streaming: Processing data incrementally rather than loading it all into memory.

Atomic replacement: Replacing a destination with a completed file as one filesystem operation where supported.

Path traversal: Unauthorized access caused by manipulating path components.

Review Questions

Short Answer Questions

1. What is the difference between opening a file in w mode and a mode?
2. Why should an encoding normally be specified when opening a text file?
3. What types of values are read from text mode and binary mode respectively?
4. What do seek() and tell() do?
5. Why is a context manager preferred for file operations?

Long Answer Questions

1. Explain the complete process of converting stored bytes into Python text and writing Python text back to storage. Include encoding errors and newline translation.
2. Compare read(), readline(), readlines(), and iteration over a file in terms of use and memory consumption.
3. Explain buffering, flushing, closing, and the limitations of assuming that flushed data is physically durable.
4. Describe how a program should safely process a very large binary file.
5. Discuss the security and reliability issues involved in accepting file paths from users.

Analytical / Application Questions

1. Design a file-processing strategy for a 10 GB UTF-8 log that must produce a summary without loading the whole file into memory. Justify the mode, encoding, iteration strategy, and error policy.
2. A program opens an existing customer database with w mode and then crashes before writing new data. Analyze what happened and propose a safer update procedure.
3. A UTF-8 file contains accented names, but a program reads it successfully on one computer and fails on another. Diagnose the likely cause and propose a portable correction.

Multiple Choice Questions with Answers

Question 1

Which mode opens a text file for writing and immediately truncates existing contents?

A. r

B. a

C. w

D. x

Correct Answer: C. w

Question 2

What does specifying encoding="utf-8" primarily control?

A. The file's permissions

B. Conversion between characters and bytes

C. The file's buffering size

D. The file's current position

Correct Answer: B. Conversion between characters and bytes

Question 3

Which operation is generally most suitable for processing a very large text file line by line?

A. file.read()

B. file.readlines()

C. for line in file

D. file.read(1) repeatedly without iteration

Correct Answer: C. for line in file

Question 4

What type of value does a binary file normally return from read()?

A. str

B. bytes

C. list

D. float

Correct Answer: B. bytes

Question 5

What is the main purpose of a with statement around open()?

A. To encrypt the file

B. To guarantee deterministic cleanup and closure

C. To convert binary data into text

D. To disable buffering

Correct Answer: B. To guarantee deterministic cleanup and closure

Question 6

In binary mode, what does seek(-4, os.SEEK_END) attempt to do?

A. Move four bytes before the end

B. Move four characters after the beginning

C. Delete the last four bytes

D. Move to position four only if the file is text

Correct Answer: A. Move four bytes before the end

Question 7

Which exception commonly indicates that a requested path does not exist?

A. PermissionError

B. FileNotFoundError

C. UnicodeEncodeError

D. IsADirectoryError

Correct Answer: B. FileNotFoundError

Question 8

What does truncate() do?

A. Changes the file length

B. Changes the file encoding

C. Renames the file

D. Converts text to bytes

Correct Answer: A. Changes the file length

Question 9

Why is newline="" commonly used when opening a file for CSV processing?

A. It disables all file reading

B. It lets the CSV module manage newline handling accurately

C. It forces binary mode

D. It automatically validates every field

Correct Answer: B. It lets the CSV module manage newline handling accurately

Question 10

Which mode is appropriate when creating a new file only if it does not already exist?

A. x

B. r

C. a

D. r+

Correct Answer: A. x