10. POSIX System Programming and Interprocess Communication

Introduction and Context

Linux applications are built on a layered interface between user programs and the operating-system kernel. At the lowest practical level, this interface is exposed through system calls: controlled requests through which a process opens files, creates processes, allocates memory, communicates with other processes, and responds to external events. POSIX (Portable Operating System Interface) defines a family of standards that make these services consistent across Unix-like systems. Linux implements POSIX interfaces while also providing Linux-specific extensions.

System programming differs from ordinary application programming because the programmer must reason about resources, process identity, scheduling, memory visibility, blocking behavior, failure modes, and concurrency. A program that appears correct in a single-threaded test may fail when two processes access the same file, when a signal arrives between two instructions, or when a pipe becomes full. Consequently, robust Linux programming requires both knowledge of interfaces and an understanding of the kernel abstractions behind them.

This chapter examines POSIX files, processes, threads, signals, pipes, sockets, shared memory, synchronization, asynchronous I/O, and system-call interfaces. The emphasis is on selecting suitable communication mechanisms and designing applications that remain correct under concurrency, interruption, failure, and resource pressure.

Learning Objectives

After studying this chapter, the learner should be able to:

• explain the relationship between library functions, POSIX interfaces, and Linux system calls;
• use file descriptors and blocking or non-blocking I/O safely;
• create, control, and terminate processes and threads;
• distinguish pipes, FIFOs, sockets, message queues, and shared memory;
• design synchronization protocols using mutexes, semaphores, condition variables, and read-write locks;
• explain signal delivery and write signal-safe handlers;
• apply event-driven and asynchronous I/O techniques;
• identify race conditions, deadlocks, resource leaks, and common IPC failures;
• evaluate IPC designs in terms of performance, portability, security, and maintainability.

The POSIX Programming Model

A process is an executing program together with an address space, registers, credentials, open-file descriptors, and other kernel-managed state. Each process has a process identifier (PID). A file descriptor is a small non-negative integer referring to an open file description. Standard input, output, and error conventionally occupy descriptors 0, 1, and 2.

The descriptor is process-local, but the underlying open-file description contains the current file offset and status flags. After `fork()`, parent and child descriptors may refer to the same open-file description; therefore, they share the file offset. This detail explains both useful pipeline behavior and surprising interference when two processes write through duplicated descriptors.

A POSIX function may be implemented by a wrapper in the C library that eventually invokes a system call. For example, `read()` and `write()` transfer data through descriptors, while `open()` establishes access to a file. The system-call boundary validates arguments, checks permissions, changes kernel state, and may put the calling thread to sleep. A failure is generally indicated by a return value such as `-1`, with the reason placed in `errno`. Since `errno` is meaningful only after a failure, programs should inspect it immediately and should not assume that every failure is permanent.

The basic I/O pattern is:

1. acquire a descriptor using `open()`, `socket()`, `pipe()`, or a related interface;

2. perform operations such as `read()` and `write()`;

3. interpret partial transfers, end-of-file, interruption, and errors;

4. close the descriptor on every control path.

A successful `read()` may return fewer bytes than requested. A return value of zero indicates end-of-file for a regular file or an orderly shutdown in some communication contexts. A blocked call may be interrupted by a signal and return `EINTR`; blindly treating this as a permanent error is a common defect.

Processes, Threads, and Program Replacement

Creating process relationships

`fork()` creates a child process by copying the calling process's address space using copy-on-write. Initially, parent and child have equivalent memory contents, but modifications are logically private. Both continue execution after `fork()`, with the parent receiving the child's PID and the child receiving zero. Failure returns `-1` only in the parent context.

A child commonly follows `fork()` with `execve()` or one of its variants. An `exec` call replaces the current process image with a new program; it does not create an additional process. Open descriptors remain open unless marked close-on-exec, while ordinary variables, heap objects, and stack contents disappear. The combination of `fork()`, descriptor setup, and `exec()` is the foundation of shells, service supervisors, and pipeline construction.

A parent collects a terminated child with `wait()` or `waitpid()`. Until collected, a terminated child is a zombie. If a parent terminates first, the child is adopted by a system reaper. A well-designed service records exit status and handles failures in both child creation and child collection.

Threads and shared address spaces

POSIX threads, commonly created with `pthread_create()`, execute within one process and share global memory, heap storage, open descriptors, and signal dispositions. Each thread has its own stack, registers, scheduling state, and thread identifier. Threads avoid the address-space duplication associated with processes and are therefore useful for parallel servers and I/O-bound workloads, but shared memory introduces data races.

A data race occurs when conflicting accesses to shared data are not ordered by synchronization. The result is not merely an incorrect arithmetic value; compiler and processor reordering can make unsynchronized assumptions invalid. A mutex establishes mutual exclusion, while condition variables allow a thread to sleep until a predicate becomes true. The predicate must be tested in a loop because wakeups can be spurious and because another thread may consume the condition first.

`pthread_join()` waits for a joinable thread and retrieves its termination result. Threads that are no longer needed should be joined or explicitly detached. Failure to do so can leak implementation resources.

Choosing processes or threads

Processes provide stronger isolation: a memory error in one process normally does not overwrite another process's address space. Threads offer cheaper communication through shared memory but require disciplined synchronization. A multi-process architecture is often preferable for privilege separation, fault containment, or independently restartable services. A threaded architecture is attractive when shared caches, low-latency communication, or a large number of concurrent activities dominate. Hybrid designs are also common.

Interprocess Communication Mechanisms

IPC should be selected according to communication direction, data volume, topology, persistence, ordering, and failure requirements.

Pipes and FIFOs

An unnamed pipe, created by `pipe()`, supplies a unidirectional byte stream represented by a read and a write descriptor. It is especially appropriate between related processes, such as a parent and child. The writer does not preserve message boundaries. If all write ends are closed, readers observe end-of-file; if no reader remains, a writer may receive `SIGPIPE` or an `EPIPE` error.

A FIFO is a named pipe visible in the filesystem. Unrelated processes can open it, but opening may block until the corresponding endpoint is present, depending on flags. Pipes provide back-pressure: when the kernel buffer is full, a blocking writer waits. This is useful flow control, but it can deadlock if two parties wait for output while neither consumes input.

Message-oriented IPC

POSIX message queues preserve discrete messages and can assign priorities. They are useful when receivers must distinguish commands without defining a byte-stream framing protocol. Their capacity is bounded, so producers still require a policy for full queues. System V message queues and Linux-specific mechanisms provide alternatives, but portability and administration characteristics differ.

Shared memory

Shared memory maps the same physical storage into multiple processes. POSIX shared memory commonly uses `shm_open()`, `ftruncate()`, and `mmap()`. It provides high throughput because data need not be copied through a pipe for every transfer. However, shared memory supplies storage, not coordination. A producer and consumer must establish ownership, visibility, initialization, and synchronization using process-shared mutexes, semaphores, or atomic operations.

A circular buffer illustrates the design. The producer writes at the index `in`, the consumer reads at `out`, and the indices are advanced modulo capacity. Empty and full states must be distinguishable, and updates must be ordered so that a consumer never observes an index before the associated data is visible. Initialization of process-shared synchronization objects must occur exactly once; otherwise, two processes may concurrently initialize the same object.

Unix-domain and network sockets

Sockets support bidirectional communication. Unix-domain sockets communicate between processes on the same host and can provide filesystem-based naming and credential-related security features. Internet sockets support communication across hosts using address families such as IPv4 and IPv6. Stream sockets provide an ordered byte stream; datagram sockets preserve packet boundaries but may lose, duplicate, or reorder packets depending on the protocol.

A TCP server normally creates a socket, binds an address, listens, accepts connections, and then reads and writes each connected stream. `accept()` creates a new connected descriptor while the listening descriptor remains available. Applications must implement framing because one `send()` is not guaranteed to correspond to one `recv()`. Length-prefixing, delimiters, or self-describing formats are common solutions.

Synchronization and the Structure of Correctness

Synchronization is not simply a performance accessory; it defines the ordering relationships that make concurrent observations meaningful. A mutex protects an invariant, not merely a line of code. The critical section should be large enough to preserve the invariant but small enough to avoid unnecessary contention.

Semaphores represent counters and are useful for resource availability, such as the number of filled slots in a buffer. A binary semaphore can resemble a lock, but mutexes usually express ownership more clearly. Read-write locks allow multiple readers or one writer and are useful only when read concurrency compensates for their additional complexity.

Deadlock requires several conditions: mutual exclusion, hold-and-wait, no preemption, and circular wait. Consistent lock ordering is a principal prevention technique. Other approaches include reducing lock scope, using timed acquisition, avoiding calls that may block while holding a lock, and documenting lock hierarchies. A livelock differs from deadlock because threads remain active but repeatedly prevent useful progress.

Condition variables should be paired with a state predicate. For example, a consumer waits while `count == 0`, releases the mutex atomically during the wait, and rechecks the predicate after waking. Signaling without changing the predicate is usually a logic error.

Signals and Asynchronous Events

A signal is a kernel-delivered notification of an event such as child termination, an interrupt, or an invalid memory access. Signals may be generated by hardware, the kernel, another process, or the program itself. A process can install a disposition, ignore some signals, or block delivery temporarily. Standard signals are not queued individually: multiple pending instances may collapse into one. Real-time signals provide queuing and ordering features.

Signal handlers execute asynchronously and therefore may interrupt code at almost any point. Only async-signal-safe functions may be called reliably from a handler. Functions such as `printf()`, `malloc()`, and most stdio operations are unsafe because they may hold internal locks or manipulate inconsistent state. A safe pattern is for the handler to set a `volatile sig_atomic_t` flag or write a byte to a pipe, allowing the main event loop to perform substantial work.

`sigaction()` is preferred to older signal interfaces because it provides explicit flags and masks. `sigprocmask()` controls blocked signals in a thread-aware manner. `sigwaitinfo()` and `sigtimedwait()` offer a synchronous alternative in which a designated thread receives signals as ordinary control flow. This often produces clearer designs than widely shared asynchronous handlers.

Non-Blocking, Multiplexed, and Asynchronous I/O

Blocking I/O is simple: the calling thread sleeps until progress is possible. It becomes problematic when one thread must manage many descriptors. Non-blocking descriptors return `EAGAIN` or `EWOULDBLOCK` when immediate progress is impossible. Readiness multiplexing with `select()`, `poll()`, or Linux `epoll()` allows an event loop to monitor many descriptors.

`select()` has descriptor-set and range limitations. `poll()` uses arrays of descriptor records and is more flexible, though it may scan many entries. `epoll` maintains a kernel event interest structure and is commonly used for high-concurrency Linux servers. Readiness does not guarantee that a complete application message is available; the application must continue handling partial reads and writes.

Asynchronous I/O differs from readiness notification. In readiness designs, the application is told that an operation should not block and then performs it. In completion-based asynchronous I/O, the kernel or subsystem performs the operation and later reports completion. Linux provides interfaces such as POSIX AIO for selected uses and `io_uring` as a modern Linux-specific submission/completion framework. These mechanisms can reduce system-call overhead but introduce more complicated lifetime, cancellation, and buffer-ownership rules.

An Applied Case: A Concurrent Log Ingestion Service

Background

Consider an illustrative service that accepts log records from local applications and remote agents, validates them, and writes them to rotating storage. It must remain responsive during bursts, prevent one client from corrupting another's data, and shut down without losing acknowledged records.

Situation and analysis

A single blocking process that handles one connection at a time fails when a client stalls. A process per connection improves isolation but can become expensive under high connection counts. A thread per client simplifies local state management, yet shared rotation metadata and buffers create synchronization obligations. Network streams also do not preserve record boundaries.

Approach

The service uses a non-blocking listening socket and an event loop based on `epoll`. Each connection has an input buffer. The protocol uses a length prefix, so the service accumulates bytes until a complete record is available. Valid records are placed into a bounded, mutex-protected producer-consumer queue. Worker threads remove records and perform validation and storage. The queue's empty and full states are represented by condition variables or semaphores.

The listening process sets close-on-exec on descriptors, handles `SIGTERM` through a self-pipe or signalfd-style event mechanism, and stops accepting new clients during shutdown. Workers finish records already admitted to the queue, flush storage, close files, and then terminate. Every write handles partial completion and checks disk errors. Client identity and permissions are validated before accepting sensitive records.

Outcome

The design separates event demultiplexing from CPU and storage work. Back-pressure is explicit: when the queue is full, the event loop stops reading or applies a protocol-level rejection policy. This prevents unbounded memory growth. The service also has clear failure boundaries: a malformed client record is rejected, a worker failure is logged and isolated where possible, and shutdown has a defined acknowledgement policy.

Learning points

The case demonstrates that IPC and concurrency mechanisms must be chosen together. Non-blocking sockets solve connection scalability but not message framing. A queue transfers ownership, but synchronization defines safe ownership transfer. Graceful shutdown requires coordination among signals, event loops, workers, descriptors, and durable storage.

Common Errors, Risks, and Professional Considerations

Typical errors include forgetting to close unused pipe ends, causing readers never to observe end-of-file; assuming `write()` transfers all requested bytes; using `strtok()` or other non-reentrant stateful functions across threads without protection; checking a condition variable only once; and calling unsafe library functions inside signal handlers. Another frequent defect is descriptor leakage across `exec`, which can keep files or sockets alive unexpectedly.

Security concerns include file-descriptor inheritance, insecure temporary files, trusting data received over sockets, inadequate permission checks on shared-memory objects, denial of service through unbounded queues, and privilege retention after a service has completed privileged setup. Programs should use least privilege, validate lengths before allocation, avoid predictable temporary names, apply close-on-exec, and define resource limits.

Reliability also depends on observability. Production services should record meaningful error context, distinguish transient errors from permanent failures, expose queue depth and latency metrics, and test under interruption, descriptor exhaustion, slow readers, process termination, and partial writes. Tests should include sanitizers and race-detection tools where appropriate, but these tools complement rather than replace a sound synchronization design.

POSIX portability requires avoiding assumptions about structure sizes, signal numbering, scheduling order, and atomicity beyond what the specification guarantees. Linux-specific facilities can provide substantial performance benefits, but they should be isolated behind interfaces when portability or long-term maintenance matters.

Current Directions

Modern Linux systems increasingly combine event-driven networking, multithreading, zero-copy techniques, and completion-based I/O. `io_uring` supports asynchronous submission and completion for several classes of operations, while newer kernel facilities enable efficient descriptor management and security monitoring. Containers and service managers rely heavily on ordinary POSIX primitives—processes, namespaces, signals, descriptors, and sockets—although they add isolation and lifecycle policies around them.

The continuing trend is toward explicit ownership, bounded resources, structured cancellation, and observable concurrency. High performance is valuable only when it does not compromise correctness, security, or recoverability.

Best Practices

• Check every system-call and library return value.
• Treat short reads, short writes, `EINTR`, and end-of-file explicitly.
• Close unused descriptor copies immediately after `fork()`.
• Prefer `sigaction()` and async-signal-safe handlers.
• Protect invariants, not arbitrary code fragments, with locks.
• Use bounded queues and deliberate back-pressure.
• Define message framing for every byte-stream protocol.
• Set close-on-exec for descriptors that should not cross an `exec` boundary.
• Document ownership, lifetime, lock order, and shutdown behavior.
• Select Linux-specific mechanisms only after evaluating portability and operational cost.

Chapter Summary

POSIX system programming exposes the kernel through disciplined interfaces for files, processes, threads, signals, and communication. File descriptors are central to both ordinary I/O and IPC. Processes provide isolation, while threads provide efficient shared-address-space concurrency at the cost of synchronization complexity. Pipes and FIFOs support streams, message queues preserve message boundaries, shared memory offers high throughput with explicit coordination, and sockets support local or networked communication. Correct concurrent programs require mutexes, condition variables, semaphores, sound memory visibility, and deadlock prevention. Signals must be handled conservatively, and event-driven or asynchronous I/O is useful when many operations must be managed efficiently. Professional designs also account for failure, security, portability, observability, and graceful shutdown.

Key Terms

POSIX: A family of standards specifying portable operating-system interfaces.

System call: A controlled request through which a user process obtains a kernel service.

File descriptor: A process-local integer handle for an open kernel-managed resource.

Process: An executing program with its own virtual address space and kernel state.

Thread: An execution unit sharing an address space with other threads in a process.

`fork()`: An operation that creates a child process using copy-on-write memory semantics.

`exec()`: A family of operations that replaces a process image with a new program.

Zombie: A terminated child whose status has not yet been collected by its parent.

Pipe: A unidirectional kernel-managed byte stream between processes.

FIFO: A named pipe represented in the filesystem namespace.

Socket: A bidirectional communication endpoint for local or network communication.

Shared memory: Memory mapped into multiple processes for direct data exchange.

Mutex: A mutual-exclusion synchronization object protecting a critical section.

Semaphore: A counter-based synchronization object representing permits or resources.

Condition variable: An object enabling a thread to wait for a state predicate to become true.

Signal: An asynchronous notification delivered to a process or thread.

Readiness multiplexing: Monitoring multiple descriptors for operations that can proceed.

Back-pressure: A mechanism that limits producers when consumers or buffers cannot keep up.

Race condition: A defect in which behavior depends on uncontrolled timing or access ordering.

Deadlock: A state in which activities wait indefinitely for resources held by one another.

Review Questions

Short Answer Questions

1. Why can a `read()` call return fewer bytes than requested?

2. What is the difference between `fork()` and `exec()`?

3. Why must a condition variable normally be tested in a loop?

4. What happens to a pipe reader when all write descriptors are closed?

5. Why are `printf()` and `malloc()` unsafe in a signal handler?

Long Answer Questions

1. Explain the lifecycle of a process created with `fork()`, including descriptor inheritance, `exec()`, and `waitpid()`.

2. Compare pipes, message queues, shared memory, and sockets with respect to data model, performance, and appropriate use.

3. Describe how mutexes, semaphores, and condition variables can implement a bounded producer-consumer queue.

4. Explain the design of a non-blocking server using readiness multiplexing and message framing.

5. Discuss the principal causes of deadlock and the techniques used to prevent or diagnose it.

Analytical / Application Questions

1. A parent creates a pipe, forks a child, and forgets to close its copy of the write descriptor. Explain the resulting behavior and propose the correction.

2. Design an IPC architecture for a high-throughput local telemetry collector. Justify your choices among shared memory, Unix-domain sockets, threads, and processes.

3. A multithreaded server occasionally loses records during shutdown. Analyze possible interactions among signals, queue predicates, worker termination, partial writes, and descriptor closure, and propose a reliable shutdown protocol.

22. Multiple Choice Questions with Answers

Question 1

After fork(), which statement is correct about the parent and child processes?

A. They share all ordinary variables permanently.

B. They initially have logically equivalent memory, implemented using copy-on-write.

C. The child immediately replaces the parent process image.

D. The parent cannot communicate with the child through inherited descriptors.

Correct Answer: B. They initially have logically equivalent memory, implemented using copy-on-write.

Question 2

A stream-oriented socket receives only part of an application message. What should the application do?

A. Assume the sender transmitted an incomplete message.

B. Discard the connection immediately.

C. Buffer the bytes and continue reading according to the framing protocol.

D. Call exec() to restart the receiver.

Correct Answer: C. Buffer the bytes and continue reading according to the framing protocol.

Question 3

Why is shared memory normally paired with synchronization?

A. Shared memory automatically preserves message boundaries.

B. Shared memory provides storage but does not order concurrent accesses.

C. Shared memory can only be accessed by the kernel.

D. Synchronization is required only for network communication.

Correct Answer: B. Shared memory provides storage but does not order concurrent accesses.

Question 4

Which action is appropriate inside a conventional asynchronous signal handler?

A. Call printf() to report the event.

B. Allocate a response buffer with malloc().

C. Set a sig_atomic_t flag or write to a notification pipe.

D. Acquire a possibly held mutex.

Correct Answer: C. Set a sig_atomic_t flag or write to a notification pipe.

Question 5

A pipe writer receives EPIPE. What does this most likely indicate?

A. The pipe has no remaining readers.

B. The pipe contains too many readers.

C. The writer has reached end-of-file.

D. The process successfully completed an exec().

Correct Answer: A. The pipe has no remaining readers.

Question 6

Which design best prevents deadlock in code using multiple mutexes?

A. Acquire locks in a consistently documented global order.

B. Hold locks while performing arbitrary blocking I/O.

C. Allow each thread to choose its own lock order.

D. Replace all mutexes with busy waiting.

Correct Answer: A. Acquire locks in a consistently documented global order.

Question 7

What is the principal distinction between readiness notification and completion-based asynchronous I/O?

A. Readiness means the operation has completed; completion means it may begin.

B. Readiness indicates an operation can proceed without blocking; completion reports that it has finished.

C. They are identical terms with no design difference.

D. Completion-based I/O is limited to regular files by definition.

Correct Answer: B. Readiness indicates an operation can proceed without blocking; completion reports that it has finished.

Question 8

Why should a server use bounded queues?

A. To guarantee that producers never block.

B. To prevent unbounded memory growth and make overload behavior explicit.

C. To eliminate the need for message framing.

D. To ensure every client receives identical data.

Correct Answer: B. To prevent unbounded memory growth and make overload behavior explicit.

Question 9

What is the purpose of close-on-exec for a file descriptor?

A. It prevents the descriptor from being used before fork().

B. It causes the descriptor to close automatically across an exec operation.

C. It converts the descriptor into a socket.

D. It makes all writes atomic.

Correct Answer: B. It causes the descriptor to close automatically across an exec operation.

Question 10

Which mechanism is most suitable when unrelated local processes need bidirectional, connection-oriented communication?

A. An unnamed pipe only.

B. A Unix-domain stream socket.

C. A condition variable without shared memory.

D. A signal handler carrying arbitrary-length data.

Correct Answer: B. A Unix-domain stream socket.