1. Operating-System Foundations and Evolution

Introduction and Context

An operating system (OS) is the foundational software that coordinates computing hardware, provides services to application programs, and creates an environment in which users and programs can perform useful work. A modern computer system contains processors, memory, storage devices, networks, timers, and many input/output devices. These resources are powerful but difficult to use directly. The operating system converts their physical complexity into manageable abstractions such as processes, files, virtual memory, sockets, and protection domains.

The OS is therefore more than a collection of utilities. It is both a resource manager and an extended machine. As a resource manager, it allocates processors, memory, storage, and devices among competing programs. As an extended machine, it hides unnecessary hardware details and offers stable interfaces through which software can request services. This dual role explains why operating-system concepts remain relevant across desktop computers, mobile devices, embedded controllers, cloud data centres, and edge platforms.

The historical development of operating systems reflects changes in hardware, application requirements, economics, and patterns of human-computer interaction. Early systems executed one job at a time. Later systems introduced multiprogramming, time sharing, virtual memory, networking, graphical interfaces, virtualization, containers, cloud orchestration, and edge computing. Understanding this evolution provides a basis for analysing modern systems and making informed design decisions.

Learning Objectives

After completing this chapter, the reader should be able to:

1. Explain the purposes and principal responsibilities of an operating system.

2. Distinguish between hardware, kernel, system programs, libraries, and applications.

3. Describe major OS abstractions, including processes, threads, files, virtual memory, and protection domains.

4. Explain user mode, kernel mode, interrupts, exceptions, traps, and system calls.

5. Compare common operating-system structures and their design trade-offs.

6. Describe booting, resource management, and the evolution from batch systems to cloud and edge platforms.

7. Analyse practical OS decisions involving performance, isolation, reliability, security, and manageability.

The Operating System as an Abstraction and Control Layer

A computer system can be viewed as a hierarchy. At the lowest level are electronic circuits and hardware devices. Above them is the kernel, the privileged core of the operating system. System libraries and utilities provide more convenient interfaces, while applications implement user or business functions. Users interact with applications rather than controlling hardware directly.

The OS creates abstractions because direct hardware management is unsafe and inefficient. A physical disk is represented as files and directories; physical memory is represented through address spaces; a processor is presented as an execution environment for processes and threads; and network hardware is exposed through communication endpoints such as sockets. Abstractions reduce programming effort, but they also impose policies and costs. For example, virtual memory simplifies program development and isolation, yet address translation and page faults may affect performance.

An abstraction is useful when it preserves the operations that applications need while hiding irrelevant implementation details. The file abstraction, for example, allows an application to read and write named data without knowing the exact disk sectors involved. However, the abstraction is not magical: permissions, buffering, caching, failures, and persistence semantics still require careful treatment.

Principal Responsibilities

An operating system commonly performs the following responsibilities:

• Process and thread management: creating, scheduling, synchronizing, and terminating units of execution.
• Memory management: allocating memory, translating addresses, providing virtual memory, and enforcing isolation.
• File and storage management: organising persistent data, maintaining metadata, and controlling access.
• Device management: coordinating input/output devices through drivers and buffering mechanisms.
• Protection and security: separating users and processes and enforcing access-control decisions.
• Networking: providing communication interfaces and managing network resources.
• Accounting and monitoring: measuring resource use, recording events, and supporting administration.
• Error handling and recovery: detecting hardware or software faults and limiting their consequences.

These responsibilities interact. Scheduling a process may require memory decisions; a file read may block a process and activate a storage driver; network security may depend on process identity and memory protection. Operating-system design is consequently a systems problem rather than a set of unrelated modules.

Core Abstractions and Terminology

A process is a program in execution together with its address space, registers, open resources, and operating-system state. A program is passive code stored on a device; a process is an active execution entity. A process may contain one or more threads. A thread is a schedulable sequence of instructions that shares an address space and many resources with other threads in the same process. Threads can improve responsiveness and parallelism, but shared memory introduces synchronisation risks.

A program counter identifies the next instruction to execute, while the process control block stores information needed to manage a process, such as its state, registers, scheduling attributes, and resource references. Typical process states include new, ready, running, blocked or waiting, and terminated.

An address space is the set of addresses that a process can reference. Virtual address spaces enable relocation, sharing, and protection. The memory-management unit translates virtual addresses into physical addresses, often using page tables and translation lookaside buffers.

A file is a named abstraction for persistent or device-related data. A directory organises file names and associated metadata. A file descriptor is a process-specific handle used to refer to an open file, device, pipe, or socket. This uniform interface allows programs to use many forms of input and output through similar operations.

A resource is any system facility that can be allocated or consumed, including CPU time, memory pages, storage capacity, device bandwidth, locks, and network connections. Some resources are shareable, whereas others require exclusive access. Concurrency refers to multiple activities making progress during overlapping periods; parallelism refers to simultaneous execution on multiple processing units. Concurrency may exist on a single processor through interleaving, while parallelism requires hardware capable of simultaneous execution.

A protection domain defines the resources and operations available to an execution entity. Authentication establishes an identity; authorization determines what that identity may do. Isolation limits the effect of one process, user, or service on another. These concepts are related but not identical: a system may authenticate correctly while still implementing poor authorization or weak isolation.

Kernel Mode, User Mode, and Controlled Entry

Most general-purpose processors support at least two privilege levels. In user mode, application code has restricted access to instructions, memory regions, and devices. In kernel mode, the operating system can execute privileged instructions, configure hardware, access protected memory, and manage system-wide resources.

This separation is a central protection mechanism. If every application could directly modify page tables, disable interrupts, or issue arbitrary device commands, a defective or malicious program could compromise the entire system. User programs therefore request privileged operations through controlled interfaces.

A system call is a programmed request for an operating-system service. Common categories include process creation, file operations, memory mapping, communication, and security-related operations. A system call normally causes a controlled transition from user mode to kernel mode. The kernel validates arguments, checks permissions, performs or initiates the operation, places a result in a defined location, and returns execution to user mode.

System calls should be distinguished from library calls. A library function may execute entirely in user space, or it may invoke one or more system calls. This distinction matters for performance, portability, and security. A high-level file function, for example, may buffer data in a library before eventually requesting a kernel-level write.

An interrupt is a signal, usually associated with hardware, that causes the processor to suspend its current execution and run a predefined handler. A timer interrupt enables preemptive scheduling. A device interrupt may indicate that an input/output operation has completed. An exception is a processor-detected event associated with the current instruction, such as division by zero or an invalid memory access. A trap is a deliberate exception commonly used to enter the kernel for a system call or debugging operation.

The processor saves sufficient context to resume the interrupted activity. The kernel determines the event, executes the relevant handler, updates system state, and either resumes the interrupted activity or schedules another one. Interrupt handling must be efficient because excessive interrupt processing can consume CPU time and increase latency. Modern systems therefore use techniques such as interrupt moderation, deferred work, polling in selected circumstances, and multiple interrupt queues.

Resource Management and Policy

Resource management involves both mechanisms and policies. A mechanism specifies how an operation can be performed; a policy decides which operation should be chosen. A scheduler may provide mechanisms for placing processes on queues and switching contexts, while a scheduling policy determines whether priority, fairness, response time, throughput, or energy efficiency receives greater emphasis.

CPU management seeks to keep processors productive while meeting application objectives. Important criteria include utilisation, throughput, turnaround time, waiting time, response time, fairness, predictability, and energy consumption. Interactive systems generally value quick response, whereas batch analytics may prioritise throughput. Real-time systems add deadlines and distinguish between soft and hard timing requirements.

Memory management must provide allocation, relocation, sharing, protection, and efficient use of physical memory. Virtual memory allows a process to use an address space larger than available physical memory by moving pages between memory and secondary storage. Excessive paging, known as thrashing, can cause severe performance degradation. Large-scale systems also face non-uniform memory access, cache locality, memory pressure, and the need to coordinate memory among virtual machines or containers.

Storage management includes allocation of persistent space, naming, buffering, caching, consistency, and recovery. A successful write may mean that data reached a process buffer, an OS cache, a device cache, or durable storage; these meanings differ. Applications that require strong durability must use appropriate synchronisation and storage interfaces.

Device management depends on device drivers, which translate general OS requests into device-specific commands. Drivers are powerful and therefore represent an important security and reliability boundary. Input/output may be synchronous, in which the requesting thread waits, or asynchronous, in which the operation proceeds while computation continues. Direct memory access permits a device to transfer data to memory with limited CPU copying, but it requires careful protection and coordination.

System Structures and Design Choices

Operating systems are organised in several ways. A monolithic kernel places many services, such as process management, memory management, filesystems, and drivers, in one privileged address space. This can provide efficient communication and good performance, but a fault in one kernel component may affect the entire system and the kernel may be difficult to modify safely.

A layered system arranges services into levels, with each layer using lower-level services. Layering supports conceptual clarity and verification, although strict boundaries can be difficult to maintain and may introduce performance costs.

A microkernel keeps only essential mechanisms in the privileged kernel, such as low-level address-space management, scheduling, and interprocess communication. Other services run as user-space servers. This can improve fault isolation and modularity, but communication and recovery mechanisms may be more complex. Many practical systems use hybrid designs, combining a substantial kernel with selected modular or user-space services.

A modular kernel allows components to be loaded or unloaded dynamically. It attempts to combine the performance of in-kernel services with some flexibility of decomposition. The choice of structure depends on hardware, performance requirements, reliability goals, development practices, and the trust placed in system components. No single structure is optimal for every environment.

From Batch Processing to Cloud and Edge Computing

Early computers were expensive and commonly operated through batch processing. Users submitted jobs, often using punched cards or other offline media, and an operator collected output later. A resident monitor could automatically load and execute jobs, reducing manual setup time. However, a job that waited for slow input/output left the processor idle.

Multiprogramming addressed this inefficiency by keeping several jobs in memory. When one job waited for input/output, the processor could execute another. This required memory protection, job scheduling, interrupt handling, and more sophisticated resource accounting. Time-sharing extended the idea to interactive users by rapidly switching among processes, creating the impression that each user had a dedicated machine.

The growth of personal computers shifted attention toward usability, graphical interfaces, device support, and application compatibility. Networked and distributed systems then required remote communication, naming, authentication, failure handling, and consistency across machines. Mobile systems added energy constraints, sensors, intermittent connectivity, application sandboxing, and hardware diversity.

Virtualization allows one physical machine to host multiple virtual machines. A hypervisor provides virtual hardware and isolates guest operating systems. Virtualization improves consolidation, portability, testing, and workload management, though it introduces overhead and a larger security boundary. Containers isolate application environments while sharing the host kernel. They usually start faster and use fewer resources than full virtual machines, but their isolation depends substantially on kernel mechanisms and configuration.

Cloud computing builds on virtualization, distributed storage, software-defined networking, automation, and service-based provisioning. Cloud platforms expose resources through APIs and often charge according to consumption. Operating-system concerns include multi-tenancy, noisy neighbours, identity management, image integrity, live migration, observability, and elastic scaling. A cloud service may be operationally successful only when scheduling, storage, security, and failure recovery are considered together.

Edge computing places computation near sensors, users, or data sources rather than exclusively in central data centres. It can reduce latency, conserve bandwidth, and support operation during intermittent connectivity. Edge nodes, however, may have limited power, less physical protection, heterogeneous hardware, and more difficult maintenance. Edge operating environments therefore emphasise remote management, secure updates, lightweight isolation, local autonomy, and eventual synchronisation with central services.

Booting: From Power-On to a Usable System

Booting is the process through which a machine moves from an initial hardware state to execution of a complete operating system. Firmware performs hardware initialisation and selects a boot source. A bootloader locates and loads the kernel, often also loading an initial filesystem or hardware description. The kernel establishes memory management, interrupt handling, process structures, device drivers, and early scheduling. It then starts an initial user-space process or service manager, which launches services and eventually supports user sessions or application workloads.

Booting creates a chain of trust as well as a chain of execution. Secure-boot mechanisms can verify digital signatures on firmware, bootloaders, and kernels before execution. Nevertheless, security depends on key management, update procedures, recovery paths, and protection of later software stages. A technically secure boot process can still be undermined by compromised credentials or unpatched services after startup.

Applied Case: Designing an Operating Environment for Remote Industrial Monitoring

Background

Consider an illustrative company that monitors pumps at water-treatment facilities. Sensors measure vibration, temperature, pressure, and power use. Facilities are connected through unreliable networks, and some sites have limited local computing capacity.

Situation and Problem

The company initially sends all sensor data to a central cloud service. During network outages, data is lost and operators cannot see urgent alarms. Sending every raw sample also increases bandwidth and storage costs. The company needs local responsiveness without losing central analytics and fleet-wide visibility.

Analysis

The system requires several OS-level capabilities. Sensor processes must be isolated because a faulty driver or parser should not disable all monitoring. Time-sensitive alarm processing requires predictable scheduling. Local storage must survive temporary network failures. Secure identity and authorization are needed because remote nodes control industrial equipment. Updates must be possible without requiring technicians to visit every site.

Approach

The company deploys an edge gateway with a hardened operating system. Device drivers run in carefully restricted services, while monitoring applications run in containers with resource limits. A local supervisor restarts failed services and records health metrics. A priority-aware scheduler gives alarm processing precedence over batch compression. The gateway stores recent data in a durable local queue and forwards summaries and selected raw measurements when connectivity returns.

The gateway uses authenticated and encrypted communication with the cloud. Signed software updates are verified before installation, and updates can be rolled back if health checks fail. The cloud performs long-term analytics, fleet-wide comparison, and model training, while the gateway executes immediate threshold rules locally.

Outcome

The design reduces dependence on continuous connectivity and improves alarm response. It also introduces operational responsibilities: monitoring disk usage, rotating credentials, testing updates, and handling clock errors. The case demonstrates that an OS decision is not merely a choice of kernel; it is a coordinated decision about scheduling, isolation, storage, security, recovery, and administration.

Learning Points

1. Local autonomy is valuable when latency and connectivity are uncertain.

2. Isolation limits the effect of defective or compromised components.

3. Reliability requires supervision, durable queues, and recovery procedures.

4. Security must include identity, update integrity, least privilege, and physical assumptions.

5. Cloud and edge systems are complementary rather than mutually exclusive.

Challenges, Risks, and Professional Considerations

Operating systems must balance competing objectives. Strong isolation may reduce performance or complicate communication. Aggressive caching improves speed but can complicate consistency and durability. Preemptive scheduling improves responsiveness but introduces context-switching overhead and concurrency problems. Centralised administration simplifies policy enforcement but can create a single point of failure.

Common engineering mistakes include assuming that a successful system call guarantees durable storage, ignoring resource limits, treating containers as equivalent to complete virtual machines, granting excessive privileges to services, and failing to test recovery after crashes. Race conditions, deadlocks, priority inversion, memory leaks, driver faults, and unbounded logging are recurring operational hazards.

Professional practice requires least privilege, secure defaults, auditable administration, timely patching, and separation of duties. Operating systems also process personal, confidential, and sometimes safety-critical data. Designers should consider privacy, data minimisation, retention, lawful access, accessibility, and the consequences of monitoring users or employees. Reliability claims should be supported by testing and evidence rather than by assumptions about vendor or platform reputation.

Contemporary systems increasingly use hardware-assisted isolation, confidential-computing features, immutable or declarative infrastructure, automated fleet management, and observability pipelines. Machine learning may assist workload prediction, anomaly detection, and energy management, but automated decisions require validation, explainability appropriate to the risk, and safeguards against misleading telemetry. Emerging systems must also address heterogeneous accelerators, disaggregated memory, persistent memory, carbon-aware scheduling, and secure software supply chains.

Best Practices for Operating-System Analysis

When evaluating an operating environment, begin with workload and risk rather than brand names. Identify latency, throughput, availability, isolation, energy, compliance, and maintainability requirements. Map each requirement to an OS mechanism or policy. Define failure modes and determine how the system detects, contains, recovers from, and reports them.

Use least privilege and explicit resource limits. Separate control-plane functions from application workloads where practical. Measure actual response time, CPU utilisation, memory pressure, I/O latency, and failure recovery rather than relying solely on theoretical capacity. Keep kernels, drivers, images, and system services updated through controlled processes. Finally, document assumptions: network availability, clock accuracy, storage durability, physical access, and administrative trust are all part of the operating environment.

Chapter Summary

An operating system provides abstractions over hardware and manages shared resources for applications and users. Its central responsibilities include process execution, scheduling, memory and storage management, device coordination, networking, protection, accounting, and recovery. Processes, threads, address spaces, files, descriptors, and protection domains are core abstractions that simplify programming while imposing performance and security considerations.

User mode and kernel mode establish a privilege boundary. Interrupts, exceptions, traps, and system calls provide controlled interaction between hardware, applications, and the kernel. Monolithic, layered, microkernel, modular, and hybrid structures represent different trade-offs in efficiency, isolation, reliability, and complexity. Operating systems evolved from batch monitors through multiprogramming and time sharing to networked, mobile, virtualized, cloud, and edge environments. Modern design requires integrated attention to performance, security, resilience, automation, and governance.

Key Terms

Operating system: System software that manages hardware resources and provides services to applications.

Kernel: Privileged core of an operating system that controls hardware and essential system mechanisms.

Process: A program in execution together with its execution state and allocated resources.

Thread: A schedulable execution path within a process.

System call: A controlled request by a user program for an operating-system service.

User mode: Restricted processor mode in which ordinary application code executes.

Kernel mode: Privileged processor mode in which the kernel can execute protected operations.

Interrupt: A signal that causes the processor to invoke an event handler.

Exception: A processor-detected event associated with execution of a particular instruction.

Trap: A deliberate exception commonly used for system calls or debugging.

Virtual memory: A memory abstraction that separates process addresses from physical memory locations.

Address space: The set of addresses that an execution entity can reference.

File descriptor: A process-specific handle for an open file, device, pipe, or socket.

Multiprogramming: Keeping multiple jobs in memory so the processor can switch when one waits.

Time sharing: Interactive sharing of processor time among multiple users or processes.

Virtual machine: A software-defined computer environment hosted on physical hardware.

Container: An isolated application environment that generally shares the host kernel.

Hypervisor: Software or firmware that creates and manages virtual machines.

Edge computing: Processing performed near data sources or users rather than only in a central cloud.

Protection domain: A set of resources and operations available to an execution entity.

Review Questions

Short Answer Questions

1. Why is an operating system described both as a resource manager and an extended machine?

2. Distinguish between a program, a process, and a thread.

3. What is the purpose of user mode and kernel mode?

4. Differentiate an interrupt, an exception, and a trap.

5. What role does a device driver play in input/output management?

Long Answer Questions

1. Explain the major abstractions provided by an operating system and discuss the benefits and costs of abstraction.

2. Describe the life cycle of a system call from user-mode invocation to return of a result.

3. Compare monolithic, layered, microkernel, modular, and hybrid operating-system structures.

4. Trace the evolution from batch processing to multiprogramming, time sharing, virtualization, and cloud computing.

5. Explain how CPU, memory, storage, and device management interact in a multiprogrammed system.

Analytical / Application Questions

1. A monitoring application must respond to alarms within a strict time limit while processing large historical datasets. Which scheduling and isolation considerations should guide the OS design? Justify your answer.

2. A containerised service has been compromised but the host remains available. Analyse which OS mechanisms may have limited the damage and which assumptions must be examined.

3. Design an OS-level strategy for an edge gateway that must continue operating during network outages and later synchronise data with a cloud service. Discuss storage, security, scheduling, recovery, and update requirements.

22. Multiple Choice Questions with Answers

Question 1

Which statement best captures the dual role of an operating system?

A. It only translates high-level programming languages into machine code.

B. It manages resources and presents convenient abstractions over hardware.

C. It permanently stores all application data without using hardware drivers.

D. It replaces the need for processors and memory.

Correct Answer: B. It manages resources and presents convenient abstractions over hardware.

Question 2

Why does a system call normally cause a transition to kernel mode?

A. To allow the application to bypass access controls.

B. To permit controlled execution of privileged operating-system services.

C. To convert a process into a thread.

D. To disable all interrupts permanently.

Correct Answer: B. To permit controlled execution of privileged operating-system services.

Question 3

A process is blocked waiting for disk input. Which OS event can allow another process to run?

A. A device completion interrupt followed by scheduling activity.

B. A change from kernel mode to firmware mode only.

C. Deletion of the process control block.

D. Disabling virtual memory.

Correct Answer: A. A device completion interrupt followed by scheduling activity.

Question 4

Which situation illustrates multiprogramming?

A. One job is executed after the computer is manually restarted each time.

B. Several jobs remain in memory, and the CPU runs another when one waits for I/O.

C. A single process uses multiple names for the same file.

D. A user accesses only a graphical interface.

Correct Answer: B. Several jobs remain in memory, and the CPU runs another when one waits for I/O.

Question 5

What is a principal advantage of virtual memory?

A. It eliminates all storage latency.

B. It allows processes to use protected virtual address spaces independent of physical placement.

C. It guarantees that no page fault can occur.

D. It makes kernel mode unnecessary.

Correct Answer: B. It allows processes to use protected virtual address spaces independent of physical placement.

Question 6

Which design choice generally places more services outside the privileged kernel?

A. Microkernel architecture

B. Unmodified monolithic architecture

C. Batch processing

D. Direct memory access

Correct Answer: A. Microkernel architecture

Question 7

Why can a container not automatically be treated as equivalent to a virtual machine?

A. Containers cannot execute applications.

B. Containers generally share the host kernel, so their isolation boundary differs.

C. Virtual machines never use operating systems.

D. Containers require punched cards.

Correct Answer: B. Containers generally share the host kernel, so their isolation boundary differs.

Question 8

An edge gateway continues collecting sensor data during a network outage. Which feature most directly supports this behaviour?

A. A durable local queue combined with later synchronisation.

B. Disabling all local storage.

C. Running every task with unrestricted privilege.

D. Sending every sample immediately to the unavailable cloud.

Correct Answer: A. A durable local queue combined with later synchronisation.

Question 9

Which statement correctly distinguishes concurrency from parallelism?

A. Concurrency requires multiple physical processors, but parallelism does not.

B. Concurrency concerns overlapping progress, while parallelism involves simultaneous execution.

C. They are identical terms in all operating-system contexts.

D. Parallelism can occur only in batch systems.

Correct Answer: B. Concurrency concerns overlapping progress, while parallelism involves simultaneous execution.

Question 10

Which practice best addresses the risk that a successful write has not reached durable storage?

A. Assuming a returned library call always implies physical persistence.

B. Using appropriate durability and synchronisation interfaces and testing their semantics.

C. Disabling buffering for every application without analysis.

D. Granting the application kernel privileges.

Correct Answer: B. Using appropriate durability and synchronisation interfaces and testing their semantics.