Automation, DevOps, and Infrastructure as Code

Introduction and Context

Cloud computing has changed the speed and scale at which organisations can create digital services. However, cloud platforms do not automatically produce reliable systems. A poorly designed manual process can reproduce mistakes rapidly, create inconsistent environments, and make security weaknesses difficult to detect. Automation and DevOps address this problem by treating software delivery and infrastructure operations as repeatable, collaborative, and measurable engineering activities.

DevOps is a set of practices, cultural principles, and technical capabilities that brings development and operations closer together. Its purpose is not simply to deploy software faster. Effective DevOps seeks to improve the flow of value from idea to working service while maintaining reliability, security, compliance, and operational visibility.

Infrastructure as Code (IaC) is a central cloud practice within this approach. IaC represents infrastructure—such as networks, virtual machines, databases, identity policies, and containers—in machine-readable configuration files. These files can be reviewed, tested, versioned, and applied automatically. Together, DevOps, continuous integration and continuous delivery, configuration management, IaC, policy as code, and automated provisioning form a modern operating model for cloud services.

Learning Objectives

By the end of this chapter, you should be able to:

• explain the principles and business value of DevOps;
• distinguish continuous integration, continuous delivery, and continuous deployment;
• describe the stages and controls in a CI/CD pipeline;
• compare configuration management with infrastructure provisioning;
• explain how version control supports collaboration and auditability;
• identify the main approaches, workflow, and risks of IaC;
• describe the purpose of policy as code and automated compliance;
• design a basic automated provisioning process for a cloud application; and
• evaluate the security, governance, reliability, and organisational implications of automation.

1. DevOps as an Operating Model

Traditional delivery often separates development, operations, security, and quality assurance into functional silos. Developers may be measured by the features they produce, while operations teams may be measured by stability. These objectives can conflict: a development team wants frequent change, whereas an operations team wants to reduce operational risk. DevOps creates shared responsibility for the complete service lifecycle.

The principal DevOps ideas are:

1. Collaboration: Teams communicate across organisational boundaries and share responsibility for outcomes.
2. Automation: Repetitive activities are performed by reliable, repeatable tools rather than manual intervention.
3. Short feedback cycles: Automated tests, monitoring, and user feedback reveal problems early.
4. Small, reversible changes: Smaller changes are easier to test, deploy, troubleshoot, and roll back.
5. Continuous improvement: Teams use operational evidence and retrospectives to improve processes.
6. Shared ownership: Developers participate in service operation, while operations specialists contribute to design and delivery.

DevOps is therefore both technical and organisational. Installing a pipeline tool without changing ownership, testing, or communication usually creates only superficial automation.

DevOps and the Software Delivery Lifecycle

A simplified lifecycle includes planning, coding, building, testing, releasing, deploying, operating, and monitoring. In a traditional sequential model, feedback may arrive only after a large release. In a DevOps model, these activities form a continuous loop. Monitoring data informs planning; tests inform coding; deployment results inform design.

A useful distinction is between speed and flow. Speed concerns how quickly an individual task is completed. Flow concerns how efficiently valuable work moves through the entire system. A team that produces code quickly but waits weeks for testing or approval has poor flow. DevOps attempts to reduce queues, hand-offs, rework, and avoidable manual effort.

2. Continuous Integration and Continuous Delivery

Continuous Integration

Continuous integration (CI) is the practice of frequently merging code changes into a shared repository and automatically validating those changes. A CI process commonly performs:

• source-code retrieval;
• dependency installation;
• static analysis and formatting checks;
• unit tests;
• compilation or packaging;
• vulnerability and secret scanning; and
• creation of a versioned build artifact.

CI reduces integration risk because conflicts and defects are discovered near the time they are introduced. It requires practices such as small commits, automated tests, a stable main branch, and clear build instructions.

Continuous Delivery and Continuous Deployment

Continuous delivery means that validated software is kept in a deployable state. A release may still require an explicit human approval, particularly where regulatory, business, or operational controls demand it.

Continuous deployment goes further: every change that passes the defined automated controls is deployed to production without a separate manual release decision.

These concepts are related but not identical. A pipeline can support continuous delivery without continuous deployment. For example, an organisation may automatically build and test every change, but require an authorised manager to approve production release.

Pipeline Stages and Controls

A practical pipeline may contain the following stages:

1. Commit: A developer submits a change to version control.
2. Validate: The system checks syntax, formatting, dependencies, and policy requirements.
3. Build: Source code is converted into a deployable artifact.
4. Test: Unit, integration, security, performance, and acceptance tests are run as appropriate.
5. Package: The artifact is stored in an immutable registry.
6. Provision or update: Infrastructure and application configuration are applied to a target environment.
7. Release: Approval or an automated rule permits promotion.
8. Observe: Logs, metrics, traces, alerts, and health checks are evaluated.
9. Recover: The team rolls back, rolls forward, or performs remediation when necessary.

A pipeline should fail safely. If a security scan detects a critical vulnerability, the process should prevent promotion or route the result to an authorised exception process. A pipeline that always succeeds because failures are ignored is not automated assurance.

Deployment Strategies

A rolling deployment updates instances gradually. It limits the immediate impact of a change but can temporarily run different software versions.

A blue-green deployment maintains two production-like environments. Traffic is directed to the new environment after validation, and traffic can be redirected to the old environment if a problem occurs. This can require additional capacity and careful management of data compatibility.

A canary deployment sends a small proportion of users or requests to the new version. Metrics are compared with the existing version before wider release. Canary deployment is useful when observability is strong and the organisation can define safe thresholds.

A feature flag separates code deployment from feature activation. A feature can be deployed but disabled, or enabled for a selected group. Flags require lifecycle management; unused flags create complexity and may become security risks.

3. Version Control and Collaboration

Version control is the foundation of traceable automation. It records changes to application source code, IaC files, pipeline definitions, policies, documentation, and sometimes operational runbooks.

Important version-control concepts include:

• Repository: A managed collection of files and their history.
• Commit: A recorded change with an author, timestamp, and message.
• Branch: An independent line of development.
• Merge or pull request: A controlled method for integrating reviewed changes.
• Tag: A named reference to a particular version, often used for releases.
• Revert: A new change that reverses an earlier change.

A professional workflow normally protects the main branch, requires peer review, runs automated checks, and records approvals. Commit messages should explain intent rather than merely repeat a file name. Sensitive values such as passwords, private keys, and access tokens must not be committed. If a secret is exposed, deleting it from the latest file is insufficient: it must be revoked and removed from repository history where appropriate.

Branching models should match the team’s delivery needs. Long-lived branches may isolate unfinished work but create difficult merge conflicts. Short-lived branches and frequent integration reduce divergence. The correct choice depends on release frequency, testing maturity, and governance requirements.

4. Configuration Management and Provisioning

Configuration management and provisioning are related but distinct.

Provisioning creates or changes resources: networks, subnets, load balancers, compute instances, databases, queues, and access roles.

Configuration management installs and configures software on those resources. It may create users, install packages, configure services, place application files, and manage service settings.

For example, a provisioning tool may create a virtual machine and its network security rule. A configuration-management tool may install a web server and configure its listening port.

Modern cloud designs often prefer immutable infrastructure. Instead of changing a running server repeatedly, the organisation builds a new image or container, tests it, and replaces the old instance. This reduces configuration drift and makes rollback more predictable. Mutable configuration remains useful in some environments, but it requires strong control over changes and state.

5. Infrastructure as Code

Definition and Characteristics

IaC expresses infrastructure in declarative or procedural files. A declarative description states the desired result—for example, “a private network with three subnets”—while the tool determines the actions required to reach that result. A procedural script specifies a sequence of commands. Declarative approaches often improve repeatability, although they do not eliminate the need to understand dependencies and provider behaviour.

Important IaC characteristics are:

• Repeatability: The same definition can create equivalent environments.
• Consistency: Shared modules and standards reduce accidental variation.
• Reviewability: Infrastructure changes can use pull requests and peer review.
• Auditability: History shows who changed what and when.
• Reusability: Modules can represent approved patterns.
• Automation: Changes can be tested and applied by pipelines.
• Drift awareness: The actual environment can be compared with the declared configuration.

IaC does not mean that every environment will be identical. Parameters may legitimately differ between development and production. The important distinction is between intentional variation and undocumented drift.

Declarative Workflow

A typical IaC workflow is:

1. Define resources and parameters in version-controlled files.
2. Validate syntax and run static policy checks.
3. Generate a proposed change plan.
4. Review the plan and approve it according to organisational controls.
5. Apply the approved change using a controlled identity.
6. Verify resource health and application behaviour.
7. Record outputs, logs, and the resulting version.

Many IaC systems maintain a state representation that maps declared resources to real provider resources. State enables the tool to calculate differences efficiently. State is operationally sensitive: it may contain resource identifiers, configuration values, or secrets. It should be stored in an access-controlled remote backend, encrypted where supported, and protected against concurrent updates. State locking or an equivalent mechanism prevents two operators from applying conflicting changes simultaneously.

Idempotence and Convergence

An operation is idempotent when applying it repeatedly produces the same intended result as applying it once. In infrastructure automation, idempotence is highly desirable. A configuration task such as “ensure the web service is installed and running” should not reinstall or corrupt the service each time it runs.

IaC tools generally seek convergence: they compare the desired state with the observed state and perform actions that reduce the difference. Convergence is not magic. External changes, provider limitations, non-deterministic scripts, and resources with random names can still produce unexpected results. Operators must understand what the tool manages and what it does not manage.

Modules and Environments

A module is a reusable, parameterised collection of infrastructure definitions. Modules can standardise network layouts, logging, identity roles, or application platforms. Good modules expose necessary inputs while hiding unnecessary implementation detail. They should be versioned, documented, tested, and governed like software libraries.

Environment separation reduces accidental impact. Development, test, staging, and production should normally have distinct accounts, projects, subscriptions, or equivalent boundaries. Promotion should use the same tested artifact and approved infrastructure definitions, with controlled parameter differences. Copying and manually editing files for each environment increases risk.

6. Policy as Code and Security Automation

Policy as code represents organisational, security, or regulatory rules in machine-evaluable form. Examples include:

• storage must use encryption;
• public network access is prohibited for a database;
• resources must include an owner and cost-centre tag;
• deployments must use approved regions;
• containers must not run with excessive privileges; and
• production changes must originate from an approved pipeline.

Policies can run before provisioning, during deployment, or continuously against existing resources. Preventive controls stop non-compliant changes. Detective controls identify violations after they occur. Corrective controls may automatically remediate a violation, but automated remediation must be carefully designed to avoid disrupting legitimate services.

Policy as code supports consistent governance, but a policy engine cannot understand every business context. Exceptions should be explicit, time-limited, documented, and approved. A rule that is too broad may block valuable work; a rule that is too narrow may provide false assurance.

Security must be integrated throughout the pipeline. This practice is often called DevSecOps. Controls include dependency scanning, static application security testing, dynamic testing, container-image scanning, infrastructure scanning, secret detection, least-privilege identities, and audit logging. Security checks should be prioritised according to risk rather than added indiscriminately until the pipeline becomes unusably slow.

7. Automated Provisioning in Practice

Automated provisioning should begin with a desired architecture and a clear ownership model. A basic cloud application may require:

• a network boundary and private subnets;
• an application service or compute platform;
• a managed database;
• identity and access roles;
• encryption keys or certificates;
• monitoring, logging, and alerting; and
• backup and recovery settings.

A suitable process is:

1. Design: Identify dependencies, trust boundaries, data classifications, and availability needs.
2. Encode: Represent resources and configuration in IaC modules.
3. Validate: Check syntax, module versions, security policies, naming, tags, and cost controls.
4. Plan: Generate a human-readable change plan.
5. Review: Obtain peer and, where required, security or change-management approval.
6. Apply: Use a controlled pipeline identity to provision resources.
7. Test: Verify connectivity, access boundaries, health endpoints, backups, and alerts.
8. Operate: Monitor the service and reconcile drift.
9. Retire: Remove resources and revoke access when the service is decommissioned.

The pipeline identity should have only the permissions necessary for its tasks. Separate identities may be used for planning and applying changes, with stronger controls on production application. Credentials should be short-lived where possible and retrieved from a secrets-management service rather than embedded in files.

8. Measuring DevOps and Automation

Measurement should encourage healthy system behaviour rather than reward unsafe speed. Common delivery measures include:

• Deployment frequency: How often a service is deployed.
• Lead time for changes: The time from a change being committed or authorised to its deployment.
• Change failure rate: The proportion of deployments that cause incidents, rollback, or urgent remediation.
• Time to restore service: The time required to recover after a failure.

These measures are useful only when definitions are consistent. A high deployment frequency is not beneficial if it is accompanied by frequent outages. Teams should also measure test reliability, pipeline duration, failed-change causes, vulnerability remediation time, and infrastructure drift.

Operational metrics should include service availability, latency, error rates, saturation, and business outcomes. A pipeline can be technically successful while the application is unusable. Effective observability connects deployment events to service behaviour.

9. Illustrative Case: Automating a Student Assessment Platform

Background

A fictional college operates an online assessment platform. Its application runs in the cloud and uses a web tier, an API, and a managed relational database. Initially, an administrator creates environments manually. Development and production differ in network rules, runtime versions, and monitoring configuration.

Problem

Releases occur once every several weeks because testing and production changes require coordination among separate teams. A previous release failed because a production environment lacked a required database parameter. Recovery took several hours. The college also cannot reliably demonstrate who changed a firewall rule or whether storage is encrypted.

Analysis

The main problems are configuration drift, manual approval without a consistent evidence trail, insufficient automated testing, and excessive privilege. The college decides that automation must improve both delivery and control, not merely increase release speed.

Approach

The college adopts the following design:

• Application and infrastructure definitions are stored in separate version-controlled repositories.
• Pull requests require peer review and automated checks.
• IaC modules create the network, application service, database, logging, and alerting configuration.
• Environment-specific values are supplied through controlled parameters rather than copied files.
• A pipeline performs syntax validation, unit tests, dependency scanning, IaC security checks, and a provisioning plan.
• Production application requires an authorised approval after the plan is reviewed.
• The database is private, encrypted, backed up, and accessible only through the application role.
• Policy checks require ownership tags, approved regions, encryption, and prohibition of public database access.
• A canary release sends a small percentage of traffic to a new application version.
• Monitoring compares error rate and latency with defined thresholds before wider promotion.
• Secrets are stored in a managed secrets service and are never placed in source control.

Outcome

The platform gains repeatable environments and a documented change history. The team detects the missing database parameter during automated validation rather than during release. A failed canary is stopped before full deployment, and traffic is returned to the previous version. The college still retains human approval for production, satisfying its governance requirements.

Learning Points

First, IaC is valuable because it makes infrastructure changes visible and testable, not because files are inherently superior to consoles. Second, automation must include security and recovery controls. Third, continuous delivery can be appropriate even when continuous deployment is not. Finally, reliable results depend on process design, observability, and team ownership as much as on tooling.

10. Challenges, Limitations, and Governance

Complexity and Tool Dependence

Cloud automation can involve source control, pipeline services, IaC tools, configuration tools, registries, scanners, secrets managers, policy engines, and monitoring platforms. Excessive tool diversity increases training and integration costs. Organisations should standardise common patterns while allowing justified exceptions.

State and Drift

Manual console changes may create drift between declared and actual infrastructure. Drift detection should be scheduled and investigated. Automatically overwriting every difference can destroy an intentional emergency change, so remediation should follow a defined policy.

Secrets and Sensitive Data

IaC state, pipeline logs, build artifacts, and crash reports may expose sensitive information. Secret values should be masked, access-controlled, rotated, and excluded from logs. Encryption in transit and at rest is necessary but does not replace access governance.

Failure at Scale

Automation can magnify an error across many environments. Safe rollout mechanisms, testing, approval boundaries, quotas, and the ability to halt or roll back are essential. Production permissions should be separated from development permissions.

Compliance and Accountability

Automated decisions must remain explainable and auditable. Organisations should preserve pipeline logs, approvals, code versions, test results, deployment records, and policy outcomes for an appropriate retention period. Regulatory requirements vary by jurisdiction and sector; automation should support, not replace, professional legal and compliance advice.

Human and Ethical Considerations

Automation changes roles rather than eliminating responsibility. Teams need training in cloud security, testing, observability, and incident response. An organisation should avoid measuring individuals by raw deployment counts, because this can encourage unsafe changes. Access decisions and automated remediation should be reviewed for unintended effects, especially where services affect students, employees, patients, or other vulnerable groups.

11. Contemporary Developments

Platform engineering builds internal self-service capabilities for development teams. A platform may provide approved templates, deployment workflows, observability, and secure defaults while hiding unnecessary infrastructure complexity. This approach can improve consistency when platform teams treat developers as users and measure usability as well as technical compliance.

GitOps applies version control and reconciliation principles to operational environments. A declared configuration in a repository becomes the authoritative source, and an agent detects and applies approved changes. GitOps can improve traceability, but repository security and reconciliation failure modes must be managed.

Artificial intelligence is increasingly used to suggest pipeline fixes, summarise incidents, generate configuration drafts, and identify anomalous behaviour. Such tools can assist engineers but may produce insecure or incorrect code, expose confidential context, or reinforce poor patterns. AI-generated infrastructure should undergo the same review, testing, least-privilege analysis, and approval as human-generated infrastructure. Responsibility remains with the organisation and authorised professionals.

Best-Practice Checklist

A certificate-level practitioner should be able to apply the following principles:

• Store application, infrastructure, policy, and pipeline definitions in version control.
• Review changes before applying them to shared or production environments.
• Use automated tests and security checks as early as practical.
• Build immutable, versioned artifacts and promote the same artifact between environments.
• Keep secrets out of source code, images, logs, and unprotected state files.
• Use least-privilege, short-lived pipeline identities.
• Generate and review IaC plans before applying changes.
• Protect state with access control, encryption, backup, and locking.
• Detect and investigate infrastructure drift.
• Use progressive deployment and observable rollback mechanisms.
• Encode repeatable security and governance requirements as policies.
• Measure reliability and change outcomes, not only delivery speed.
• Maintain human accountability for automated decisions.

Chapter Summary

DevOps combines collaboration, automation, feedback, and shared responsibility to improve software and service delivery. Continuous integration validates frequent changes; continuous delivery keeps software ready for release; and continuous deployment releases validated changes automatically. CI/CD pipelines should include testing, security, approvals where required, deployment controls, observability, and recovery mechanisms.

Version control provides the history and review process for application code, infrastructure, policies, and pipeline definitions. Provisioning creates cloud resources, while configuration management prepares those resources to run software. Infrastructure as Code expresses desired infrastructure in reusable, reviewable, and repeatable definitions. Its effectiveness depends on controlled state, idempotent operations, environment separation, testing, and drift management.

Policy as code makes security, compliance, and architectural rules machine-evaluable. DevSecOps integrates security throughout delivery rather than treating it as a final inspection. Automated provisioning must use least privilege, protected secrets, controlled approvals, logging, monitoring, and rollback. Finally, successful automation is organisational as well as technical: teams require clear ownership, appropriate measures, training, and accountability.

Key Terms

DevOps: An operating model combining development and operations through collaboration, automation, feedback, and shared responsibility.

Continuous integration: Frequent integration of code changes with automated validation.

Continuous delivery: Maintaining software in a deployable state, commonly with an optional approval before production.

Continuous deployment: Automatically releasing every change that passes required controls.

CI/CD pipeline: An automated workflow that builds, tests, approves, deploys, and observes software changes.

Version control: A system for recording, reviewing, and managing changes to files and their history.

Artifact: A versioned output, such as a package, container image, or executable, produced for deployment.

Configuration management: Managing software settings, packages, services, and system configuration.

Provisioning: Creating and modifying infrastructure resources.

Infrastructure as Code: Representing infrastructure in machine-readable, version-controlled definitions.

Declarative configuration: A description of the desired result rather than a complete sequence of actions.

Idempotence: The property that repeating an operation produces the same intended result as performing it once.

Configuration drift: Unintended divergence between declared configuration and actual infrastructure.

State: Information used by an IaC tool to track managed resources and their relationships.

Module: A reusable and parameterised collection of infrastructure definitions.

Policy as code: Encoding governance, security, or compliance rules so that tools can evaluate them.

DevSecOps: Integrating security practices and controls throughout the DevOps lifecycle.

Canary deployment: Releasing a change to a small proportion of traffic before wider rollout.

Feature flag: A control that enables or disables application functionality independently of deployment.

Observability: The ability to understand system behaviour through logs, metrics, traces, and related signals.

Review Questions

Short Answer Questions

1. What is the main difference between continuous delivery and continuous deployment?
2. Why is version control important for infrastructure definitions?
3. Distinguish provisioning from configuration management.
4. What is configuration drift, and how can it be detected?
5. Give two examples of rules that may be implemented as policy as code.

Long Answer Questions

1. Explain how DevOps changes the relationship between development, operations, security, and quality assurance.
2. Describe the stages of a secure CI/CD pipeline and explain the purpose of each stage.
3. Explain the benefits and risks of Infrastructure as Code, including state management and idempotence.
4. Compare rolling, blue-green, and canary deployment strategies.
5. Discuss why automation requires governance, observability, and human accountability.

Analytical / Application Questions

1. A company provisions production servers manually, stores passwords in deployment scripts, and has no record of firewall changes. Propose a phased plan using version control, IaC, secrets management, policy as code, and CI/CD controls.
2. A canary release shows a small increase in latency but no increase in errors. What evidence and decision rules should the team use before expanding or stopping the release?
3. An automated compliance tool repeatedly removes a firewall rule that an operations team added during an incident. Analyse the conflict and recommend a safer design for emergency changes, exceptions, and later reconciliation.

Multiple Choice Questions with Answers

Question 1

Which statement best describes continuous integration?

A. Deploying every change directly to production

B. Frequently merging code and automatically validating the changes

C. Manually configuring production servers after each release

D. Creating a separate cloud account for every developer

Correct Answer: B. Frequently merging code and automatically validating the changes

Question 2

What is the primary purpose of Infrastructure as Code?

A. To replace all application testing

B. To represent and manage infrastructure through version-controlled definitions

C. To prevent the use of cloud APIs

D. To store passwords in deployment files

Correct Answer: B. To represent and manage infrastructure through version-controlled definitions

Question 3

Which practice most directly helps prevent two operators from applying conflicting IaC changes?

A. State locking

B. Increasing application memory

C. Adding more source-code comments

D. Disabling audit logs

Correct Answer: A. State locking

Question 4

Which deployment strategy initially sends only a small proportion of traffic to a new version?

A. Rolling deployment

B. Blue-green deployment

C. Canary deployment

D. Manual deployment

Correct Answer: C. Canary deployment

Question 5

Which item should normally not be committed to a source-code repository?

A. A pipeline definition

B. A version tag

C. A peer-reviewed infrastructure module

D. A production access token

Correct Answer: D. A production access token

Question 6

What does policy as code enable?

A. Machine-evaluable security and governance rules

B. Unlimited administrator access

C. Removal of all human accountability

D. Automatic elimination of software defects

Correct Answer: A. Machine-evaluable security and governance rules

Question 7

A pipeline creates a tested container image and promotes that same image from testing to production. What practice does this support?

A. Using immutable, versioned artifacts

B. Introducing configuration drift

C. Storing secrets in images

D. Avoiding automated testing

Correct Answer: A. Using immutable, versioned artifacts

Question 8

Which principle limits the damage if a pipeline identity is compromised?

A. Least privilege

B. Long-lived shared credentials

C. Disabling monitoring

D. Using one administrator account everywhere

Correct Answer: A. Least privilege

Question 9

What is the main purpose of a feature flag?

A. To encrypt infrastructure state

B. To separate feature activation from code deployment

C. To replace version control

D. To create a database backup

Correct Answer: B. To separate feature activation from code deployment

Question 10

Which measure indicates the proportion of deployments that cause incidents, rollback, or urgent remediation?

A. Deployment frequency

B. Lead time for changes

C. Change failure rate

D. Storage capacity

Correct Answer: C. Change failure rate