3. Operating Systems and Infrastructure Scripting

Introduction and Context

DevOps automation operates at the boundary between software and the computing environment in which software runs. Applications depend on files, directories, users, permissions, processes, services, storage devices, network interfaces, and scheduled activities. These resources are controlled by operating systems. Consequently, an engineer who automates deployments without understanding operating-system behavior is likely to create fragile, insecure, or non-portable systems.

Infrastructure scripting is the practice of using programs and command interpreters to inspect, configure, and maintain computing infrastructure. Python provides a portable, expressive programming environment for this work, while Bash and PowerShell expose operating-system capabilities through shell commands and pipelines. Effective automation does not simply execute commands quickly. It establishes the desired state safely, verifies the result, records meaningful evidence, handles failure, and avoids unintended changes.

This chapter examines operating-system automation from a DevOps perspective. It focuses on Python modules such as `pathlib`, `shutil`, `subprocess`, `os`, `pwd`, `grp`, `psutil`, and `socket`, together with Bash and PowerShell techniques. Examples use Linux and Windows where appropriate, but the central principle is broader: automation should separate portable logic from platform-specific operations.

Learning Objectives

By the end of this chapter, a student should be able to:

1. Explain how operating-system abstractions influence infrastructure automation.

2. Automate files, directories, processes, users, permissions, services, storage, networking, and scheduled tasks.

3. Select appropriately between Python, Bash, PowerShell, and native operating-system APIs.

4. Execute external commands safely and interpret their results.

5. Design idempotent, observable, portable, and reversible infrastructure scripts.

6. Evaluate security, governance, and operational risks in infrastructure automation.

7. Apply these principles to a realistic cross-platform DevOps scenario.

The Operating System as an Automation Surface

An operating system provides abstractions that make physical and virtual resources manageable. A file system presents data as files and directories; a process abstraction represents executing programs; accounts and groups determine identity; permissions regulate access; services provide long-running functions; and network sockets enable communication. Automation interacts with these abstractions through system calls, libraries, command-line utilities, and management interfaces.

Three layers are especially important. The first is the kernel, which controls hardware resources and privileged operations. The second is the user space, containing services, utilities, libraries, and applications. The third is the shell or command interpreter, such as Bash or PowerShell, which translates commands into operations. Python scripts can call libraries directly or invoke shell commands through a subprocess interface.

Direct library or API usage is usually preferable when a stable interface exists. It provides structured data, clearer error handling, and less dependence on command syntax. Shell commands remain valuable for mature administrative tools, concise operational tasks, and platform-specific capabilities. The choice should be deliberate rather than ideological.

A useful model for infrastructure scripts is:

Discover → Decide → Change → Verify → Report

Discovery determines the current state. Decision logic compares it with the desired state. The change phase performs only necessary operations. Verification checks whether the intended result exists, and reporting provides logs, exit status, and metrics. This model is closely related to idempotence: running the same script repeatedly should converge on the same desired state rather than create duplicate or progressively damaging changes.

Files and Directories: The Foundation of Automation

File operations are among the most common infrastructure tasks. They include creating directories, copying configuration files, rotating logs, calculating checksums, managing temporary files, and removing obsolete artifacts. Python's `pathlib` offers an object-oriented and comparatively portable interface:

```python

from pathlib import Path

import shutil

config_dir = Path("/opt/example/conf")

config_dir.mkdir(parents=True, exist_ok=True)

source = Path("app.conf")

target = config_dir / "app.conf"

shutil.copy2(source, target)

print(f"Installed {target}")

```

The use of `Path` avoids manual string concatenation and handles path separators more reliably. However, portability is not automatic. `/opt/example/conf` is a Unix-style location, while Windows may use `C:\ProgramData\Example\conf`. A robust script obtains paths from configuration, environment variables, or platform-specific conventions.

Safe file automation considers atomicity and permissions. Writing directly to a production configuration file can leave a partially written file if the process is interrupted. A safer pattern writes to a temporary file in the same file system, validates its content, and then replaces the destination atomically where the platform supports it. File modes, ownership, and sensitive contents must also be considered. Credentials should not be stored in world-readable files or printed in logs.

Deletion requires particular caution. Scripts should verify the target path, reject unexpected roots, and distinguish files from directories. A relative path supplied by an untrusted user can create a path-traversal vulnerability. Normalizing a path is not sufficient by itself; the script should also confirm that the resolved path remains inside an approved directory.

Bash provides concise file automation:

```bash

set -euo pipefail

install -d -m 0750 /var/lib/example

install -m 0640 app.conf /etc/example/app.conf

```

`set -euo pipefail` improves failure visibility, although it does not eliminate every shell error-handling subtlety. PowerShell provides analogous operations:

```powershell

New-Item -ItemType Directory -Path 'C:\ProgramData\Example' -Force

Copy-Item '.\app.conf' 'C:\ProgramData\Example\app.conf' -Force

```

The `-Force` option should not be treated as a safety mechanism; it may overwrite existing data. Scripts should first establish whether replacement is intended.

Processes, Commands, and Services

A process is an executing instance of a program. Infrastructure automation may need to launch a process, collect its output, terminate it, identify resource consumption, or verify that a service is running. Python's `subprocess` module is the principal interface for external commands.

```python

import subprocess

result = subprocess.run(

["systemctl", "is-active", "--quiet", "nginx"],

check=False,

capture_output=True,

text=True,

)

if result.returncode != 0:

raise RuntimeError("nginx is not active")

```

Passing an argument list rather than a single shell command reduces interpretation ambiguity. In contrast, `shell=True` invokes a shell and can create command-injection risk when any part of the command is influenced by external input. It should be reserved for situations that genuinely require shell features, with strict validation and controlled inputs.

A production script should distinguish at least three outcomes: successful completion, an expected operational condition, and an exceptional failure. It should set timeouts for commands that may hang, capture output without exposing secrets, and preserve the command's exit status. A command that returns text successfully may still have produced an incorrect operational result, so verification must examine meaning rather than only process completion.

For process inspection, libraries such as `psutil` expose CPU, memory, process, and network information in structured form. A script can use a process identifier, executable path, or service manager rather than unreliable text matching. Process identifiers can be reused after termination; therefore, a verification routine should also inspect process identity or start time when precision matters.

Services are long-running processes managed by an operating-system service framework. Linux systems commonly use `systemd`, while Windows uses the Service Control Manager. A deployment normally follows a sequence such as install unit or service definition, reload the manager, enable startup behavior, start or restart the service, and verify health. Restarting a service unconditionally is disruptive and may conceal configuration errors. A stronger workflow validates configuration first and restarts only when the effective configuration has changed.

Examples of service commands include:

```bash

sudo systemctl daemon-reload

sudo systemctl enable --now example.service

sudo systemctl is-active --quiet example.service

```

```powershell

Set-Service -Name 'ExampleService' -StartupType Automatic

Start-Service -Name 'ExampleService'

Get-Service -Name 'ExampleService'

```

Service state is not the same as application health. A process may be running while its HTTP endpoint, database connection, or queue consumer is unusable. Infrastructure automation should therefore combine service-manager state with application-level health checks.

Identity, Permissions, and Privilege

Operating systems enforce access through identities and authorization rules. A user account identifies a principal, a group aggregates principals, and permissions determine which operations are allowed. Unix-like systems commonly represent access through owner, group, and other permissions such as read, write, and execute. Windows uses access-control entries in an access-control list (ACL).

The principle of least privilege requires a script or service to receive only the permissions necessary for its task. Running every automation step as root or administrator is convenient but expands the consequences of a defect or compromise. Privileged actions should be minimized, isolated, audited, and, where possible, delegated through narrowly defined mechanisms.

User and group management is inherently platform-specific. Linux commands include `useradd`, `usermod`, `groupadd`, and `id`; Windows provides PowerShell cmdlets such as `New-LocalUser`, `Add-LocalGroupMember`, and `Get-LocalUser`. Python can inspect Unix account databases through `pwd` and `grp`, but account creation often requires platform-specific tools or identity-management APIs.

A safe account-management script is declarative. It checks whether the account exists, compares required properties, changes only differences, and avoids replacing an existing home directory or credentials unintentionally. Passwords should never appear in command-line arguments because process listings and audit records may expose them. Secrets should be obtained from an approved secret manager or protected input channel.

Permissions should be verified after deployment. For example, a configuration file containing a database password may need ownership by the service account and mode `0640` or stricter. Permission models differ across operating systems, so portability requires an explicit capability test and a platform-specific implementation rather than silently assuming Unix semantics.

Storage and Resource Capacity

Storage automation includes mounting volumes, creating directories for application data, monitoring capacity, setting quotas, and managing temporary or archival files. The most frequent operational error is confusing available disk capacity with application health. A file system may have free space while inodes, quotas, or an underlying volume limit has been exhausted.

Python can inspect disk usage with `shutil.disk_usage`, while `psutil.disk_partitions` and related functions provide broader system information. A capacity policy should define warning and critical thresholds, measurement locations, retention periods, and the response to each threshold. Deleting files automatically is risky unless ownership, age, file type, and retention obligations have been established.

Storage changes can be destructive. Formatting a device, deleting a volume, or changing a mount point requires explicit confirmation, inventory validation, and preferably a backup or recovery plan. Automation should refuse to act when a device identity does not match the expected inventory. In regulated environments, retention and legal-hold requirements may prohibit deletion even when capacity is low.

Networking and Name Resolution

Network automation covers interface configuration, route inspection, DNS resolution, port testing, firewall rules, and service endpoint validation. Python's `socket` module can resolve names and create connections; `ipaddress` can validate addresses and networks. These libraries are useful for portable checks, but they do not replace platform-specific network configuration systems.

A connectivity test should identify what it proves. Successful DNS resolution proves that a name was resolved, not that the application is healthy. Successful TCP connection proves that a port accepted a connection, not that authentication or a request succeeded. A layered diagnostic approach tests DNS, routing, transport, protocol, and application behavior separately.

```python

import socket

host, port = "service.internal", 443

with socket.create_connection((host, port), timeout=3):

print("TCP connection established")

```

Network scripts must handle timeouts, transient failures, IPv4/IPv6 differences, proxies, and credential protection. Firewall and route changes are high-impact operations and should be reviewed, logged, and tested with a rollback plan. Exposing a management interface broadly to solve a connectivity problem is usually an unacceptable shortcut.

Scheduled Tasks and Recurring Operations

Scheduled automation includes log cleanup, certificate renewal, backups, health checks, and report generation. Unix systems commonly use `cron` or systemd timers; Windows uses Task Scheduler. Python itself can implement a loop, but operating-system schedulers are generally more reliable for restart behavior, centralized administration, and execution history.

A scheduled task should define its interval, time zone, concurrency policy, timeout, environment, output destination, and failure notification. Two copies of a job running concurrently may corrupt files or duplicate business actions. Lock files, operating-system mutexes, or a scheduler's native concurrency controls can prevent overlap.

Time is a frequent source of defects. Daylight-saving changes, clock drift, leap behavior, and ambiguous local times can alter execution. Where possible, use UTC for infrastructure timestamps and make the intended time zone explicit. Scheduled scripts must use absolute paths and a deliberate environment because cron and Task Scheduler do not necessarily provide the interactive user's `PATH` or working directory.

Shell Choice and Python Integration

Bash and PowerShell are not merely different syntaxes; they embody different operating models. Bash commonly treats command output as text, whereas PowerShell pipelines objects with properties and types. Bash is strong for compact Unix composition, while PowerShell is deeply integrated with Windows administration and .NET. Python is preferable for substantial logic, data transformation, testing, cross-platform abstraction, and reusable modules.

A practical decision framework asks:

• Is the operation exposed by a stable Python library or API?
• Is the task tightly coupled to a platform-native administration tool?
• Are structured objects required, or is line-oriented text sufficient?
• How will errors, timeouts, credentials, and exit codes be handled?
• Can the operation be tested without changing production state?

A common anti-pattern is a Python script that constructs a large shell command containing interpolated user or inventory data. Better designs pass argument arrays, validate inputs, use environment variables only for non-sensitive configuration, and isolate platform-specific adapters behind a common interface.

Portability, Idempotence, and Observability

Portability has several dimensions: operating-system compatibility, processor architecture, shell availability, privilege model, path conventions, encoding, and installed utilities. A script that runs on one Linux distribution may fail on another because a command option, service manager, or file location differs. Portability therefore requires explicit support boundaries, capability detection, and tests on each supported platform.

Idempotence means that applying an operation more than once has the same intended effect as applying it once. `mkdir(..., exist_ok=True)` is commonly idempotent; blindly appending a line to a configuration file is not. Idempotent scripts compare current and desired state, use markers or structured configuration, and avoid unnecessary restarts.

Observability makes automation accountable. Logs should record what was attempted, the target resource, result, duration, and correlation identifier, while excluding secrets. Exit codes should communicate success or failure to CI/CD systems. Metrics may record task duration, failure rate, drift count, and resource thresholds. Dry-run mode is valuable, but it must accurately model changes rather than merely print optimistic messages.

Testing should include unit tests for decision logic, integration tests in disposable virtual machines or containers, permission tests, failure injection, and platform-specific tests. Infrastructure code deserves the same review discipline as application code because its blast radius may be larger.

Illustrative Case Study: Cross-Platform Agent Deployment

Background

A fictional organization, Northstar Retail, operates a monitoring agent on Linux servers and Windows application hosts. Its existing deployment process consists of manually copied binaries, inconsistent service settings, and undocumented firewall rules. Incidents frequently arise when an agent is running but unable to reach the monitoring endpoint.

Situation and Problem

The platform team is asked to create a Python-based deployment utility. The utility must install the agent, create its service account, place a configuration file, configure startup, permit outbound communication, and verify application health. It must support repeated execution without unnecessary restarts and must not expose the API token.

Analysis

The team identifies four risks. First, Linux and Windows have different account, service, permission, and firewall models. Second, a successful process launch does not prove endpoint connectivity. Third, configuration changes must be atomic and protected. Fourth, a partially completed deployment must be diagnosable and recoverable.

Approach and Solution

The utility is divided into a portable orchestration layer and platform adapters. The orchestration layer performs discovery, validates inputs, computes the desired state, and emits structured logs. The Linux adapter uses `pathlib`, `subprocess` with argument lists, a dedicated system user, a systemd unit, and restrictive file modes. The Windows adapter uses PowerShell cmdlets through controlled subprocess calls and applies an ACL to the configuration directory.

The configuration is rendered to a temporary file, parsed locally, and replaced only after validation. A hash of the effective configuration is stored as deployment metadata. The service is restarted only if the binary or configuration hash changes. The utility checks service-manager state, resolves the monitoring hostname, opens a TCP connection with a timeout, and performs an authenticated health request without logging the token.

Each stage returns a typed result. Failure at account creation stops the process; failure at the final health check leaves the service state recorded and produces a remediation message. A dry-run reports the intended account, files, service changes, and firewall operation. Integration tests run in disposable Linux and Windows environments.

Outcome

The organization obtains repeatable deployments and clearer incident evidence. Re-running the utility becomes a no-op when the desired state already exists. Configuration errors are detected before service restart, and health verification distinguishes a running service from a reachable monitoring endpoint.

Learning Points

The case demonstrates that cross-platform automation is not achieved by hiding every difference. It is achieved by defining common desired-state behavior while implementing platform-specific actions explicitly. It also shows that idempotence, least privilege, atomic configuration updates, health verification, and observability are mutually reinforcing design requirements.

Risks, Governance, and Professional Practice

Infrastructure scripts can cause outages, data loss, unauthorized access, and non-compliance. Risks include command injection, excessive privilege, insecure temporary files, secret leakage, race conditions, accidental broad deletion, unbounded retries, and insufficient rollback. Scripts should undergo code review, static analysis, dependency scanning, and controlled release management.

Change-management policies may require approval, maintenance windows, segregation of duties, and evidence of testing. Audit logs should identify the operator or automation identity, target system, action, timestamp, and outcome. Automation must respect privacy, licensing, retention, and access-control obligations. A script should not circumvent organizational controls merely because it can technically perform an operation.

Emerging practice is moving toward policy-as-code, declarative configuration, immutable images, ephemeral test environments, and agentless management APIs. AI-assisted scripting can generate commands or propose remediation, but generated actions require validation, least privilege, human approval for high-impact changes, and protection against hallucinated or unsafe commands. The more autonomous the system, the more important bounded permissions, simulation, and explainable logs become.

Best Practices for Infrastructure Scripting

1. Define the desired state before writing the change logic.

2. Prefer structured Python APIs or native management interfaces over fragile text parsing.

3. Pass subprocess arguments as lists and avoid untrusted shell interpolation.

4. Use least privilege and separate privileged actions from ordinary logic.

5. Make operations idempotent, atomic where possible, and safe to retry.

6. Validate paths, devices, hosts, and configuration before modifying them.

7. Set timeouts and handle partial failure explicitly.

8. Verify service health at both operating-system and application levels.

9. Keep secrets out of source code, arguments, logs, and temporary files.

10. Test on every supported platform and maintain rollback procedures.

11. Log useful evidence without disclosing sensitive information.

12. Treat infrastructure code as production software subject to review and change control.

Chapter Summary

Operating systems expose the resources that DevOps automation must manage: files, processes, identities, permissions, services, storage, networks, and scheduled tasks. Python supplies portable abstractions and structured control, while Bash and PowerShell provide powerful platform-native operations. Safe scripting follows a discover–decide–change–verify–report cycle and aims for idempotence, least privilege, observability, portability, and recoverability. External commands should be invoked carefully, destructive operations should be validated and governed, and service verification should include application health rather than process state alone. The Northstar case illustrates how a portable orchestration layer and explicit platform adapters can produce reliable cross-platform infrastructure automation.

Key Terms

Operating system: Software that manages hardware resources and provides services to applications.

Infrastructure scripting: Programmatic administration of computing resources and operating-system services.

Shell: A command interpreter used to execute programs and compose administrative operations.

Process: An executing instance of a program with its own operating-system-managed state.

Service: A long-running process managed by an operating-system service framework.

Idempotence: The property that repeated application of an operation produces the same intended state.

Desired state: The condition that automation is intended to establish and maintain.

Least privilege: Granting only the permissions required for a task.

Path traversal: Unauthorized access caused by manipulating file paths beyond an approved directory.

Atomic replacement: Updating a file so observers see either the old complete version or the new complete version.

ACL: An access-control list containing rules that grant or deny permissions to identities.

Privilege escalation: Obtaining permissions beyond those initially authorized.

Subprocess: A child process launched and managed by another process or script.

Exit status: A numeric result returned by a command to indicate its outcome.

Health check: A test that determines whether a service is functioning as required.

Dry run: An execution mode that reports intended changes without applying them.

Capability detection: Identifying whether a platform supports a required command, API, or feature.

Observability: The ability to understand system behavior through logs, metrics, and traces.

Configuration drift: Divergence between actual infrastructure state and its defined desired state.

Rollback: Reversing a change to restore a previously known working state.

Review Questions

Short Answer Questions

1. Why is direct use of a structured Python API often preferable to parsing shell output?

2. Define idempotence in the context of infrastructure scripting.

3. What security risk is associated with `shell=True` in Python?

4. Why is service-manager state alone insufficient to establish application health?

5. State two reasons why scheduled tasks should use explicit paths and environments.

Long Answer Questions

1. Explain the discover–decide–change–verify–report model and show how it supports safe automation.

2. Compare Bash, PowerShell, and Python as tools for operating-system automation.

3. Discuss how least privilege, secret management, and audit logging reduce infrastructure risk.

4. Describe a robust process for deploying and validating a configuration file.

5. Explain the principal portability challenges faced by cross-platform infrastructure scripts.

Analytical / Application Questions

1. A script appends a configuration line on every execution and restarts a service unconditionally. Analyze the defects and redesign the workflow.

2. A server reports that a monitoring agent is running, but the monitoring platform shows it as offline. Design a layered diagnostic sequence using the concepts in this chapter.

3. Design a safe automation plan for deleting files when disk usage exceeds a critical threshold. Identify validation, retention, authorization, and rollback considerations.

22. Multiple Choice Questions with Answers

Question 1

Which sequence best represents a safe infrastructure-automation workflow?

A. Change, restart, discover, report, verify

B. Discover, decide, change, verify, report

C. Report, change, retry, discover, delete

D. Verify, delete, discover, change, approve

Correct Answer: B. Discover, decide, change, verify, report

Question 2

Why is passing a list of arguments to subprocess.run generally safer than constructing one interpolated shell command?

A. It automatically grants administrative privileges

B. It prevents the command from returning an exit status

C. It reduces unintended shell interpretation and injection risk

D. It guarantees that the command succeeds

Correct Answer: C. It reduces unintended shell interpretation and injection risk

Question 3

Which practice best demonstrates idempotent service deployment?

A. Restart the service on every script execution

B. Append the same configuration line every time

C. Compare effective configuration and restart only when it changes

D. Delete and recreate the service on every execution

Correct Answer: C. Compare effective configuration and restart only when it changes

Question 4

A process is active, but its HTTP endpoint cannot be reached. What is the most accurate conclusion?

A. The operating system is necessarily malfunctioning

B. The service is running but application health is not established

C. The endpoint must be healthy because the process exists

D. The process identifier must be invalid

Correct Answer: B. The service is running but application health is not established

Question 5

Which action most directly follows the principle of least privilege?

A. Running all deployment steps as root or administrator

B. Giving a monitoring agent only the permissions required to read its data

C. Storing credentials in a world-readable configuration file

D. Allowing every service account to modify system files

Correct Answer: B. Giving a monitoring agent only the permissions required to read its data

Question 6

What is the principal purpose of writing a configuration file to a temporary file before replacement?

A. To guarantee that the file is encrypted

B. To make the update more atomic and reduce partial-file risk

C. To avoid checking file permissions

D. To remove the need for configuration validation

Correct Answer: B. To make the update more atomic and reduce partial-file risk

Question 7

Which test provides the strongest evidence that an application endpoint is usable?

A. Checking that a directory exists

B. Checking that a process identifier exists

C. Checking that DNS resolves only

D. Performing an application-level health request with appropriate authentication

Correct Answer: D. Performing an application-level health request with appropriate authentication

Question 8

Why should a destructive storage script validate the target device against inventory?

A. To increase the speed of formatting

B. To prevent an operation on the wrong device

C. To ensure that all devices become writable

D. To bypass change approval

Correct Answer: B. To prevent an operation on the wrong device

Question 9

Which design best supports portability between Linux and Windows?

A. Assuming identical paths, permissions, and service commands

B. Using one large shell command for both platforms

C. Separating common orchestration logic from platform-specific adapters

D. Ignoring platform differences until production execution

Correct Answer: C. Separating common orchestration logic from platform-specific adapters

Question 10

Which information should normally be excluded from infrastructure logs?

A. Operation duration

B. Target resource identifier

C. Command outcome

D. An API token used by the service

Correct Answer: D. An API token used by the service