Files, Streams, and System Interfaces

Introduction and Context

A C program normally operates on data held in memory, but useful systems must also preserve data, communicate with devices, interact with other processes, and obtain services from the operating system. Files, streams, and system interfaces provide these capabilities. They connect a program’s address space to persistent storage, terminals, networks, processes, and kernel services.

C supports two closely related levels of interaction. The ISO C library provides portable abstractions such as FILE *, fopen, fread, and fprintf. Operating systems additionally provide system-level interfaces such as POSIX file descriptors, open, read, write, lseek, fork, pipe, and mmap. The library level is generally easier and more portable; the system-call level provides finer control and access to operating-system facilities.

A rigorous understanding requires separating three concepts:

1. A file is a named or otherwise identifiable sequence of bytes maintained by a storage or device system.
2. A stream is a flow-oriented interface through which a program reads or writes characters or bytes.
3. A system interface is a defined boundary between an application and the operating system or runtime environment.

The same external object may be accessed through different abstractions. For example, a terminal may be represented by a POSIX file descriptor and wrapped by a C stream such as stdin or stdout.

Learning Objectives

After studying this chapter, a learner should be able to:

• distinguish files, streams, file descriptors, and system calls;
• open, read, write, reposition, flush, and close C streams safely;
• process text and binary data while accounting for portability;
• interpret return values, end-of-file conditions, and error indicators;
• use POSIX-style system interfaces for precise I/O control;
• explain buffering, atomicity, blocking, and partial transfers;
• design reliable and secure file-processing programs;
• evaluate the trade-offs between standard C and operating-system-specific interfaces.

1. The I/O Abstraction Stack

A typical path from an application to storage contains several layers:

Application code
      ↓
C standard I/O library: FILE *, fopen, fread, fprintf
      ↓
Operating-system interface: file descriptor, read, write, open
      ↓
Kernel, device drivers, filesystem and storage device
      ↓
Persistent medium or external device

The layers have different responsibilities. A C stream may buffer data, translate text representations, format values, and maintain a logical position. A system call transfers data between user memory and the kernel. The filesystem maps names and logical file offsets to storage objects. Hardware finally performs the physical transfer.

This layering explains why a successful library call does not always mean that data has reached non-volatile storage. Data may still reside in a user-space buffer, a kernel buffer, or a device cache. Durability requires explicitly considering the guarantees of the operating system and storage hardware.

Important terminology

2. Standard C Streams

The header <stdio.h> defines the standard I/O facilities. Three streams are normally available when a hosted C program starts:

• stdin: standard input;
• stdout: standard output;
• stderr: standard error output.

stdout is often line-buffered when connected to a terminal and fully buffered when redirected to a file. stderr is commonly intended for diagnostic output, although exact buffering behavior should not be assumed beyond the implementation’s documented guarantees.

Opening and closing

The function

FILE *fopen(const char *path, const char *mode);

opens a named file and returns a stream pointer. If the operation fails, it returns NULL and sets an error condition accessible through errno on hosted implementations that provide it.

Common modes include:

The b distinction matters especially on systems where text mode translates line endings or treats certain byte values specially. Portable binary programs should use modes such as "rb" and "wb".

A robust opening pattern is:

FILE *fp = fopen("records.dat", "rb");
if (fp == NULL) {
    perror("records.dat");
    return EXIT_FAILURE;
}

/* use fp */

if (fclose(fp) == EOF) {
    perror("fclose");
    return EXIT_FAILURE;
}

fclose releases the stream and attempts to flush pending output. Its return value must not be ignored when output integrity matters.

Character and line-oriented operations

fgetc reads one character and returns it as an int, not a char. This is necessary because every valid unsigned-character value must be distinguishable from EOF.

int ch;
while ((ch = fgetc(fp)) != EOF) {
    putchar(ch);
}
if (ferror(fp)) {
    perror("read");
}

Testing while (!feof(fp)) before reading is incorrect: the EOF indicator becomes set only after an attempted read passes the end of the input.

fputc writes one character. fgets reads at most one fewer than the supplied buffer size and terminates a successfully read string with \0. It may return a line without its newline if the buffer fills first. fputs writes a string but does not automatically append a newline.

Formatted I/O

fprintf, fscanf, snprintf, and related functions interpret format strings. Format mismatches produce undefined behavior and are a significant security risk. For example, %zu is appropriate for size_t, while %d is appropriate for int.

Input conversion with scanf is often difficult to validate because failed conversion leaves the offending input unread. For robust applications, read a complete line with fgets, then parse it using strtol, strtod, or a dedicated parser while checking range and conversion status.

Block I/O

fread and fwrite transfer arrays of objects:

size_t n = fread(buffer, sizeof buffer[0], capacity, fp);
if (n < capacity) {
    if (ferror(fp)) {
        /* input error */
    } else if (feof(fp)) {
        /* normal end of input */
    }
}

For fread, the returned value is the number of complete objects read. It is not necessarily the number of bytes. The number of bytes transferred is conceptually

where n is the returned object count and s is the object size in bytes, provided the multiplication does not overflow size_t and the object interpretation is valid.

For portable binary exchange, do not assume that writing a structure directly preserves meaning across compilers or machines. Padding, byte order, type sizes, floating-point representation, and alignment may differ. A defined external format should specify field widths, encoding, byte order, and validation rules.

3. Positions, Seeking, and Update Streams

The functions fseek, ftell, and rewind manipulate or report a stream position. fseek uses an offset and an origin such as SEEK_SET, SEEK_CUR, or SEEK_END. ftell returns a long position indicator, subject to the limitations of the implementation and the stream type. For large files, platform-specific alternatives such as POSIX fseeko and ftello may be preferable.

Not every stream is seekable. Regular files commonly are; terminals, pipes, and many sockets are not. Seeking on a non-seekable stream fails.

When a stream is opened for both reading and writing, C imposes sequencing requirements. A read following a write generally requires an intervening fflush or positioning operation, and a write following a read generally requires a positioning operation unless the read encountered end-of-file. Ignoring these rules can produce undefined or implementation-dependent behavior.

4. Buffering and Flushing

Buffering reduces the number of expensive transfers between a program and the operating system. The standard library supports full buffering, line buffering, and unbuffered operation. setvbuf can request a buffering mode immediately after opening a stream and before other operations on it.

fflush(fp) transmits pending output from the C stream to the underlying system. For input streams, the effect is limited and should not be treated as a universal way to discard input. fflush(NULL) attempts to flush all open output streams.

A flush is not automatically equivalent to durable storage. A POSIX program commonly uses fsync on a file descriptor when it needs the kernel to request persistence, but the ultimate guarantee depends on filesystem and hardware semantics. Applications requiring transactional durability often use a protocol involving temporary files, fsync, directory synchronization, and atomic rename.

5. Errors, EOF, and Resource Management

C I/O communicates status through return values and indicators. A program should distinguish:

• successful transfer;
• end of input;
• encoding or conversion failure;
• operating-system error;
• programmer misuse or undefined behavior.

feof(fp) reports whether the stream’s EOF indicator is set. ferror(fp) reports an I/O error indicator. These indicators are independent of the value returned by a particular function and should be checked according to the operation’s contract.

perror prints a message based on errno; strerror converts an error number to a message. Since errno is meaningful only after a function reports failure, it should not be inspected as a general success indicator.

Resource management is part of correctness. Every successful open should have a corresponding close, including paths taken after later failures. In larger C programs, a cleanup convention using a single exit section can make ownership explicit:

FILE *in = NULL;
FILE *out = NULL;
int status = EXIT_FAILURE;

in = fopen("input.txt", "r");
if (!in) goto cleanup;
out = fopen("output.txt", "w");
if (!out) goto cleanup;

/* processing */
status = EXIT_SUCCESS;

cleanup:
if (out && fclose(out) == EOF) status = EXIT_FAILURE;
if (in && fclose(in) == EOF) status = EXIT_FAILURE;
return status;

The goto cleanup pattern is often clearer and safer than duplicated cleanup code, provided control flow remains structured.

6. POSIX File Descriptors and System Calls

POSIX represents an open file-like object with a file descriptor, conventionally an integer. Descriptors 0, 1, and 2 normally refer to standard input, output, and error. open returns a descriptor or -1 on failure:

int fd = open("data.bin", O_RDONLY);
if (fd == -1) {
    perror("open");
}

Flags such as O_WRONLY, O_CREAT, O_TRUNC, O_APPEND, and O_CLOEXEC specify access and creation behavior. When O_CREAT is used, a permission mode is also supplied, and the process umask may remove permission bits.

read and write transfer bytes:

ssize_t n = read(fd, buffer, capacity);
if (n == -1) {
    perror("read");
} else if (n == 0) {
    /* end of file */
}

The return type ssize_t is signed so that -1 can indicate failure. A successful call may transfer fewer bytes than requested. This is normal for pipes, terminals, sockets, signals, nonblocking descriptors, and some devices. A complete-transfer routine must loop, handling interruptions and partial results.

int write_all(int fd, const void *data, size_t length) {
    const unsigned char *p = data;
    size_t done = 0;
    while (done < length) {
        ssize_t n = write(fd, p + done, length - done);
        if (n > 0) {
            done += (size_t)n;
        } else if (n == -1 && errno == EINTR) {
            continue;
        } else {
            return -1;
        }
    }
    return 0;
}

This routine assumes that length fits the interface’s accepted count type and that the memory region is valid. It does not by itself guarantee durable storage.

File descriptors and streams together

POSIX functions fdopen and fileno connect the two abstraction levels. fdopen creates a C stream around an existing descriptor; fileno obtains the descriptor underlying a stream. Mixing operations requires careful coordination because both layers may maintain buffers and positions. After a stream buffers output, a direct write may occur out of order unless the stream is flushed first. Closing ownership must also be decided: fclose closes the underlying descriptor, while merely destroying a wrapper is not a standard operation.

7. Atomicity, Concurrency, and Interprocess Communication

Multiple processes or threads may access the same file. A single system call may have atomicity guarantees that a sequence of library calls does not. O_APPEND, for example, directs each write to the current end of the file, but applications must still consider whether one logical record is written in one operation and whether records can interleave.

A file offset may be shared through duplicated descriptors or inherited descriptors after fork; independent open operations usually have independent open-file descriptions. These distinctions matter for concurrent readers and writers.

Pipes provide one-way byte-stream communication between processes. A pipe has no general seek operation. A write may block when the pipe is full, and a read may block when it is empty. A reader observes end-of-file after all write ends have been closed. Consequently, a parent process must close unused pipe ends, or the reader may wait indefinitely for an EOF that can never occur.

Named pipes, sockets, shared memory, and memory-mapped files extend the system-interface model. Each has different requirements for synchronization, framing, lifetime, failure handling, and security.

8. Text, Binary Data, and External Representation

Text is not merely “bytes that look readable.” It has an encoding, line-ending convention, lexical structure, and possibly normalization rules. UTF-8 is widespread, but C library character functions and locales influence interpretation. A byte-oriented program should distinguish bytes from characters and should not assume that one character occupies one byte.

Binary formats should be specified rather than inferred from in-memory layouts. A robust record format might define:

• a magic identifier and version;
• fixed-width integer fields;
• explicit byte order;
• length fields with bounds checks;
• checksums or authenticated integrity data;
• a policy for unknown fields and future versions.

Never use untrusted file content as a format string. Do not trust length fields before checking that they are nonnegative, representable, and within the available input. Integer overflow in allocation-size calculations can become a memory-safety vulnerability.

9. Security and Governance Considerations

File operations cross a trust boundary. Important threats include path traversal, symbolic-link attacks, unauthorized disclosure, unsafe temporary files, permission errors, and denial of service through extremely large inputs.

Best practices include:

• validate names and enforce an approved directory boundary;
• avoid constructing shell commands from filenames;
• use exclusive creation for sensitive temporary files;
• apply least-privilege permissions;
• check ownership and permissions where threat models require it;
• limit file sizes and parsing depth;
• handle malformed input without crashes;
• avoid time-of-check/time-of-use assumptions;
• use O_CLOEXEC or an equivalent close-on-exec policy for descriptors that must not leak to child processes;
• record security-relevant failures without exposing confidential data.

On POSIX systems, permissions are commonly expressed as read, write, and execute bits for owner, group, and others. The exact security policy also depends on ACLs, labels, mount options, containers, and application identity. Code should therefore rely on explicit authorization design rather than permission bits alone.

10. Applied Case: Crash-Resistant Configuration Update

Background

Consider a service that updates a configuration file containing a version number and operational parameters. A direct rewrite risks leaving a truncated file if the process crashes during writing.

Problem

The service must ensure that readers see either the old complete configuration or the new complete configuration, not a partially written document. It must also validate untrusted content and preserve access permissions where appropriate.

Analysis

A direct sequence of fopen(path, "w"), writing, and closing first truncates the old file. A crash after truncation creates an invalid state. Buffer flushing alone does not solve the atomic replacement problem.

Approach

1. Parse and validate the proposed configuration in memory.
2. Create a temporary file in the same directory, using exclusive creation.
3. Write the complete serialized representation.
4. Check every write and close operation.
5. Request synchronization of the temporary file where durability is required.
6. Atomically rename the temporary file over the destination.
7. Synchronize the containing directory when the platform and durability policy require it.

A rename within one filesystem is commonly atomic with respect to name visibility, but durability guarantees and behavior across filesystems must be verified for the target platform. Readers still need a parser that rejects malformed or unexpected versions.

Outcome

The design separates validation, construction, visibility, and durability. A crash before replacement leaves the old file available; a crash after a successful atomic replacement normally leaves either the old or new name binding, subject to the platform’s documented durability semantics.

Learning points

• atomic visibility and durable persistence are different properties;
• temporary files must be created safely and in the correct directory;
• every I/O result matters;
• file formats require versioning and validation;
• operating-system-specific guarantees must be documented rather than assumed.

11. Contemporary Developments and Design Choices

Modern systems increasingly use asynchronous I/O, event-driven descriptors, memory mapping, cloud-backed filesystems, containers, and sandboxed capability models. These technologies can improve scalability but introduce new failure modes: delayed completion, consistency differences, stale metadata, resource limits, and partial availability.

Memory mapping can make random access convenient, but it does not eliminate page faults, synchronization requirements, or durability questions. Nonblocking I/O avoids indefinite blocking but requires readiness handling and careful treatment of EAGAIN or EWOULDBLOCK. High-performance systems may use direct I/O or specialized asynchronous interfaces, but these often impose alignment and lifetime constraints.

AI systems that ingest documents or logs should treat files as untrusted data sources. File content may contain prompt-injection text, malicious payloads, private information, or misleading instructions. Secure ingestion therefore requires provenance, access control, size limits, content classification, and separation between data and executable instructions.

12. Best-Practice Checklist

1. Check every opening, reading, writing, seeking, flushing, and closing operation.
2. Use the correct mode: text or binary, read or write, append or truncate.
3. Distinguish EOF from an I/O error.
4. Treat short reads and writes as normal possibilities at the system-call level.
5. Do not assume structures are portable file formats.
6. Bound all lengths before allocation or parsing.
7. Keep stream and descriptor operations synchronized when mixed.
8. Make ownership and cleanup paths explicit.
9. Separate visibility, flushing, and durable persistence.
10. Design filenames, permissions, temporary files, and descriptors against realistic threats.
11. Document platform assumptions and portability limitations.
12. Test failure cases, including full disks, interrupted calls, malformed files, permissions, concurrent access, and crashes.

Chapter Summary

Files provide persistent or device-associated byte sequences, while streams provide a portable flow-oriented C abstraction. The standard I/O library uses FILE * objects, buffering, formatting, and character or block operations. Correct programs distinguish successful input, end-of-file, and errors; they check return values and close resources reliably.

POSIX file descriptors and system calls expose lower-level control. They support precise flags, partial transfers, pipes, descriptor inheritance, nonblocking operation, and system-specific mechanisms. Their use requires attention to errno, ssize_t, interruption, atomicity, buffering interactions, and platform guarantees.

Text and binary data require explicit representation decisions. Security requires validating paths and content, controlling permissions, preventing resource exhaustion, and avoiding unsafe temporary-file patterns. Finally, crash-resistant updates demonstrate that atomic replacement, buffering, and durability are distinct concerns. Effective C programmers select the appropriate abstraction level, document assumptions, and design for failure rather than treating I/O as an infallible sequence of calls.

Key Terms

File: A named or identifiable sequence of bytes managed by a storage or device system.

Stream: A C library abstraction representing sequential input or output.

FILE *: Pointer to a C standard-I/O stream object.

File descriptor: POSIX integer identifying an open file-like object.

System call: Controlled request from a user program to the operating system.

Buffer: Memory used to stage input or output.

EOF: Indicator that an input operation reached the end of available data.

errno: Thread-local error indicator used by many system and library interfaces.

Short read: A successful read that transfers fewer bytes than requested.

Short write: A successful write that transfers fewer bytes than requested.

Seek: Movement of a file or stream position.

Text mode: File access in which implementation-specific character translation may occur.

Binary mode: Access intended to preserve byte values without text translation.

Atomicity: Property that an operation appears indivisible to relevant observers.

Durability: Persistence of written data across specified failures or restarts.

Pipe: Kernel-managed byte stream used for interprocess communication.

Buffering: Accumulation of data to reduce transfer overhead.

Symbolic-link attack: Exploitation of pathname indirection to access an unintended object.

Memory mapping: Association of file contents with a process address range.

Serialization: Conversion of structured data into an external representation.

Review Questions

Short Answer Questions

1. What is the difference between a C stream and a POSIX file descriptor?
2. Why is fgetc specified to return int rather than char?
3. How does feof differ from ferror?
4. Why can write transfer fewer bytes than requested?
5. Why is fflush not necessarily equivalent to durable storage?

Long Answer Questions

1. Explain the abstraction stack from a C program to a storage device, including the role of buffering.
2. Describe safe text-file processing using fgets, parsing functions, validation, and error handling.
3. Compare standard C file operations with POSIX descriptor operations in portability, control, and failure handling.
4. Explain why writing a C structure directly to a file is not generally a portable serialization method.
5. Discuss the design of a crash-resistant file replacement protocol and distinguish atomicity from durability.

Analytical / Application Questions

1. Design a routine that copies a potentially large binary file using POSIX read and write. Identify how it handles partial transfers, EINTR, EOF, and resource cleanup.
2. A parent creates a pipe and forks a child, but the child never observes EOF. Analyze the likely descriptor-management error and propose a correction.
3. Evaluate the security risks of an application that accepts a user-supplied pathname, opens it for writing, and stores a user-supplied length followed by data. Propose controls for path security, race prevention, integer safety, permissions, and malformed input.

Multiple Choice Questions with Answers

Question 1

Which condition correctly indicates that a C input stream encountered an I/O error after a read operation?

A. feof(fp) is nonzero

B. ferror(fp) is nonzero

C. errno is always zero

D. The returned character is \0

Correct Answer: B. ferror(fp) is nonzero

Question 2

What does fread return?

A. The number of bytes requested regardless of success

B. The number of complete objects successfully read

C. The file descriptor associated with the stream

D. The number of characters until the next newline

Correct Answer: B. The number of complete objects successfully read

Question 3

Why must a loop generally be used around POSIX write when complete output is required?

A. write always writes exactly one byte

B. write cannot report errors

C. A successful call may transfer fewer bytes than requested

D. write automatically closes the descriptor

Correct Answer: C. A successful call may transfer fewer bytes than requested

Question 4

Which mode opens an existing file for reading and writing without truncating it?

A. "w"

B. "a"

C. "r+"

D. "wb"

Correct Answer: C. "r+"

Question 5

What is the primary purpose of O_APPEND in POSIX file opening?

A. To force binary translation

B. To direct each write to the current end of the file

C. To make the file read-only

D. To delete the file after closing

Correct Answer: B. To direct each write to the current end of the file

Question 6

Why is while (!feof(fp)) an incorrect pattern for reading a file?

A. EOF is set only after an attempted read reaches the end

B. EOF is set before the first read

C. It disables buffering

D. It converts binary data to text

Correct Answer: A. EOF is set only after an attempted read reaches the end

Question 7

Which statement about fflush is most accurate?

A. It guarantees that data survives every hardware failure

B. It flushes pending output from a C stream to its underlying system

C. It closes the stream and invalidates the file

D. It always discards unread input

Correct Answer: B. It flushes pending output from a C stream to its underlying system

Question 8

What is a major portability problem with writing a C structure directly to a binary file?

A. Structures cannot contain integers

B. The compiler always rejects binary output

C. Padding, alignment, type sizes, and byte order may differ

D. Binary files cannot be reopened

Correct Answer: C. Padding, alignment, type sizes, and byte order may differ

Question 9

Which return value from POSIX read conventionally indicates end-of-file?

A. -1

B. 0

C. 1

D. EOF in all contexts

Correct Answer: B. 0

Question 10

Why should a temporary file for a secure update generally be created in the same directory as the destination?

A. It guarantees unlimited storage

B. It permits an atomic rename within the same filesystem in common POSIX designs

C. It prevents all permission checks

D. It makes the file automatically executable

Correct Answer: B. It permits an atomic rename within the same filesystem in common POSIX designs