Infrastructure as Code and Configuration Automation

Introduction

Cloud environments are created from many interacting resources: networks, subnets, identity policies, virtual machines, containers, databases, monitoring services, and security controls. When these resources are created manually through a console, the resulting environment may work, but it is difficult to reproduce, audit, test, or repair. Infrastructure as Code (IaC) addresses this problem by representing infrastructure in machine-readable files that can be reviewed, versioned, tested, and applied repeatedly.

Configuration automation extends this idea to the operating systems and software running on infrastructure. IaC commonly creates resources, while configuration-management tools install packages, create users, place files, configure services, and enforce desired operating conditions. Together, these practices support DevOps goals: rapid delivery, consistency, traceability, and reliable recovery.

This chapter explains declarative infrastructure, configuration management, modules, state, drift detection, secrets, and policy validation. Examples use Terraform-style concepts and Ansible-style configuration tasks, but the principles also apply to AWS CloudFormation, Azure Bicep, Pulumi, OpenTofu, Chef, Puppet, and similar technologies.

Learning Objectives

By the end of this chapter, a learner should be able to:

1. Explain the difference between imperative scripts and declarative infrastructure.
2. Design a repeatable cloud environment using resources, variables, outputs, and modules.
3. Distinguish infrastructure provisioning from configuration management.
4. Explain state files, locking, backends, and state security.
5. Identify and respond to configuration and infrastructure drift.
6. Manage sensitive values without exposing secrets in source control or logs.
7. Apply testing and policy validation before infrastructure reaches production.
8. Evaluate practical, security, operational, and governance trade-offs.

1. From Manual Provisioning to Desired State

An imperative procedure specifies the sequence of actions to perform. For example, a script may create a network, then create a server, then install a web server. Imperative automation can be useful for one-time tasks, but it must explicitly handle ordering, failures, retries, and the possibility that an object already exists.

A declarative description specifies the desired end state. The automation engine determines which operations are necessary to move the current environment toward that state. A declaration might state that a virtual network must have a particular address range and that three application instances must exist. The tool compares the declaration with its knowledge of the environment and proposes a plan.

The central principle is idempotence. An operation is idempotent when applying it repeatedly produces the same intended result as applying it once, assuming the desired inputs have not changed. Creating a user only if that user does not exist is idempotent; blindly appending the same line to a configuration file on every run is not.

Declarative automation does not eliminate logic. Providers, dependency graphs, refresh operations, lifecycle rules, and error handling still influence the result. Nor does it guarantee that every external change will be corrected automatically. Teams must define what is authoritative and how conflicts are handled.

Desired state and convergence

A useful conceptual model is:

Here, D is the desired configuration, P is the proposed plan, A is the set of actions approved for execution, and E is the resulting environment. The arrow does not imply that convergence is always successful. Permissions, quotas, unavailable services, invalid parameters, and unmanaged dependencies can prevent the environment from reaching the desired state.

A mature workflow therefore separates planning from applying. A plan is reviewed, tested, and approved before changes are made. In a production pipeline, the plan should be generated from a known commit and applied by a controlled identity.

2. Infrastructure as Code Architecture

An IaC repository commonly contains four layers:

A resource is an object managed by the tool, such as a subnet or database. An input variable allows a configuration to be adapted without copying it. An output exposes a value for another module or pipeline. A data source reads information that already exists, such as an approved image identifier or an existing network.

Dependencies may be explicit or inferred. If a server refers to a subnet identifier produced by a network resource, the reference creates a dependency edge. The engine can then create the network before the server and destroy the server before the network. Explicit dependency declarations should be used sparingly because excessive manual ordering makes configurations harder to understand.

Example: a small environment

The following simplified Terraform-style example illustrates concepts rather than a complete provider configuration:

variable "environment" {
  type    = string
  default = "dev"
}

variable "instance_count" {
  type    = number
  default = 2
}

resource "cloud_network" "app" {
  name = "app-${var.environment}"
  cidr = "10.20.0.0/16"
}

resource "cloud_subnet" "private" {
  name       = "private-${var.environment}"
  network_id = cloud_network.app.id
  cidr       = "10.20.1.0/24"
}

resource "cloud_instance" "app" {
  count     = var.instance_count
  subnet_id = cloud_subnet.private.id
  image     = "approved-linux-image"
}

output "network_id" {
  value = cloud_network.app.id
}

The network identifier is not hard-coded into the subnet; it is obtained from the network resource. This creates a relationship in the dependency graph. The instance count is an input, so the same design can support a development environment with two instances or a test environment with four.

The example is incomplete for production. It would normally require identity roles, encryption, security groups, logging, backups, tags, availability-zone strategy, and policy checks. A concise example should not be mistaken for a secure production architecture.

3. Configuration Management

Provisioning creates or changes infrastructure resources. Configuration management prepares the software environment on those resources. Typical tasks include:

• installing a specified package version;
• creating service accounts with controlled permissions;
• rendering configuration files from templates;
• enabling and restarting services only when necessary;
• applying operating-system security settings;
• deploying application artifacts; and
• verifying that services are healthy.

Ansible represents these tasks in playbooks. A simplified example is:

- name: Configure application hosts
  hosts: app
  become: true
  tasks:
    - name: Install web server
      ansible.builtin.package:
        name: nginx
        state: present

    - name: Install application configuration
      ansible.builtin.template:
        src: app.conf.j2
        dest: /etc/app/app.conf
        owner: root
        group: root
        mode: '0640'
      notify: Restart application

    - name: Ensure application is enabled and running
      ansible.builtin.service:
        name: app
        enabled: true
        state: started

  handlers:
    - name: Restart application
      ansible.builtin.service:
        name: app
        state: restarted

The template task notifies a handler only when the file changes. This avoids unnecessary service restarts. Modules are preferred to arbitrary shell commands because modules generally provide clearer parameters, safer idempotence, and structured results.

Provisioning and configuration management can be combined, but their responsibilities should remain clear. A cloud resource definition should not become a large script containing operating-system commands. Conversely, a configuration playbook should not be responsible for creating an entire network unless that is an intentional architectural choice.

Immutable and mutable approaches

In a mutable approach, an existing server is updated in place. This can be efficient but may accumulate undocumented changes. In an immutable approach, a new image or instance is built and tested, then replaces the old one. Immutable infrastructure improves repeatability and rollback, although it requires suitable deployment patterns, data handling, and operational maturity.

A practical organisation often uses IaC to create networks and compute resources, image pipelines to build standard operating-system images, and configuration automation for limited runtime configuration. The correct boundary depends on scale, application architecture, and recovery requirements.

4. Modules and Reuse

A module is a reusable unit of infrastructure configuration. A well-designed module has a clear interface: typed inputs, documented assumptions, predictable outputs, and a controlled set of resources. For example, a network module may accept a name, address range, and subnet definitions, and return network and subnet identifiers.

Good modules reduce duplication without hiding important decisions. A module should not expose every provider argument merely for flexibility. Excessive parameters create a difficult interface and make safe defaults unclear.

Recommended module practices include:

• use a naming convention and consistent tags;
• validate input ranges and allowed values;
• provide secure defaults, such as encryption enabled;
• document required permissions and outputs;
• pin provider and module versions where stability matters;
• maintain compatibility deliberately; and
• test modules independently and in representative environments.

A module is not automatically a security boundary. Users of a module still need permissions to create resources, and a poorly designed module can distribute insecure practices at scale. Code review should examine both the module implementation and the values supplied by each environment.

5. State, Backends, and Collaboration

Declarative tools need a record of resources they manage. State maps configuration objects to real-world object identifiers and records attributes needed to calculate changes. It is not merely a cache. It can contain resource metadata and, depending on the resource, sensitive values.

A local state file is unsuitable for a team because concurrent users can overwrite one another, credentials may be exposed, and there is no reliable locking or central history. A remote backend should normally provide:

• encryption at rest and in transit;
• access control based on least privilege;
• versioning and recovery from accidental deletion;
• state locking or another concurrency mechanism;
• audit logging; and
• controlled separation between environments.

Locking prevents two operations from modifying the same state simultaneously. It does not make unsafe changes safe. A team still needs review, pipeline controls, and recovery procedures.

State should be treated as sensitive operational data. Access to state can reveal identifiers, network details, and sometimes secret values. State access should be restricted to the automation identity and authorised operators. State backups must receive the same protection as the primary state.

State lifecycle

A typical workflow is:

1. initialise providers and the backend;
2. validate syntax and types;
3. refresh or read relevant remote information;
4. generate a plan;
5. review and approve the plan;
6. acquire a lock;
7. apply the approved actions;
8. release the lock; and
9. record logs, outputs, and results.

Manual state editing is a high-risk recovery activity. If an object has been created outside the tool, an import operation may associate it with a declared resource. If a resource must no longer be managed, a deliberate removal-from-state operation may be appropriate, but it should be distinguished from destroying the real resource.

6. Drift Detection and Reconciliation

Drift is a difference between the declared configuration and the real environment. It may result from console changes, emergency repairs, provider defaults, failed deployments, or changes made by another automation system.

Drift detection commonly compares three things:

1. the declared configuration;
2. the recorded state; and
3. the actual remote resource.

A plan that proposes changes is evidence of a difference, but it does not automatically identify the cause. The team should ask whether the change is legitimate, whether the declaration should be updated, and whether the external change indicates a process failure.

A useful response pattern is:

• detect and classify the difference;
• determine ownership and business impact;
• preserve evidence where security or compliance is involved;
• either update the declaration or reconcile the environment;
• apply the approved correction; and
• prevent recurrence through permissions, policy, or process.

Some attributes should be ignored only when there is a documented reason. Broadly ignoring changes can conceal security drift. Conversely, enforcing every provider-managed attribute can create perpetual changes if the provider intentionally manages that value.

Drift monitoring is especially important for firewall rules, identity policies, encryption settings, public exposure, backup configuration, and audit logging. Detection frequency should reflect risk and operational capability; a daily check may be adequate for some development resources but insufficient for a sensitive production boundary.

7. Secrets and Sensitive Configuration

Secrets include passwords, API tokens, private keys, certificates, database credentials, and encryption material. They should not be embedded in source files, committed to version control, placed in command-line arguments, or printed in CI logs.

A secure pattern is to store secrets in a managed secrets service, grant the workload or deployment identity narrowly scoped access, and retrieve the value at the point of use. Secret rotation should be designed from the beginning. Rotation may require overlapping credentials, application reloads, and a rollback plan.

Important controls include:

• secret discovery scanning before commits and in CI;
• masking sensitive values in logs;
• encryption in transit and at rest;
• short-lived credentials where practical;
• separation of duties between code authors and production approvers;
• audit records for access; and
• immediate revocation when exposure is suspected.

Marking a variable as sensitive generally affects display; it does not necessarily encrypt the value or remove it from state. This distinction is essential. A secret passed into a resource may still be recorded by the provider or IaC state. The design must therefore consider the complete data path, not only the source file.

8. Testing and Policy as Code

Automation should be tested at several levels. Static validation checks syntax, types, formatting, and provider schemas. Unit-like tests evaluate modules with controlled inputs. Integration tests create resources in an isolated account or subscription and verify actual behaviour. Acceptance tests may check connectivity, encryption, backup restoration, or application health.

Policy as code expresses organisational rules in machine-evaluable form. Examples include:

• storage must not be publicly readable;
• production resources must have an owner and cost-centre tag;
• databases must use encryption;
• compute resources must use approved images;
• security groups must not allow unrestricted administrative access; and
• regions must belong to an approved list.

Policy can be preventive, detective, or corrective. A preventive policy blocks a plan before application. A detective policy reports violations. Corrective automation changes the resource, but automatic correction should be used cautiously because it can disrupt applications or destroy evidence.

A reliable pipeline may contain the following gates:

Figure: A controlled infrastructure pipeline with validation, policy, approval, application, and feedback.

The pipeline should produce an auditable record linking the commit, plan, approver, execution identity, result, and post-deployment checks. A green pipeline is not proof that an architecture is appropriate; tests and policies must reflect real risks.

9. Applied Case: Illustrative Online Retail Platform

Background

An illustrative online retailer operates a web application in a public cloud. Its development team originally created resources manually. Production environments differed from test environments, and emergency console changes were common.

Problem

The retailer experienced three issues: a missing firewall rule delayed a release, a database backup setting differed between environments, and a former employee’s access remained in an unmanaged account. The organisation needed repeatable environments without exposing database credentials.

Analysis

The team classified resources into networking, application compute, database, observability, and identity. It identified which values were environment-specific and which controls had to be universal. It also decided that production changes required approval, while development changes could be applied automatically after policy checks.

Approach and solution

A network module created private application subnets and restricted database subnets. A database module required encryption, backup retention, and deletion protection as inputs with secure defaults. A separate identity configuration managed roles and group membership. Configuration automation installed the application agent and monitoring package on approved images.

The pipeline performed formatting, syntax validation, secret scanning, static security analysis, module tests, and policy checks. Policies rejected public database access, missing ownership tags, unapproved regions, and unrestricted administrative ports. State was stored in an encrypted remote backend with versioning and locking.

Database credentials were placed in a managed secret store. The application received permission to retrieve only its own secret. The pipeline never printed the secret and did not place it in source control. Rotation was tested in a staging environment before production use.

A scheduled drift plan detected console changes. Operators were permitted to make emergency changes, but each emergency change required a ticket and a follow-up update to the declaration or an explicit approved exception.

Outcome

The environments became reproducible, and the database controls were consistent. Reviewers could inspect proposed changes before production application. Drift reports exposed unmanaged identity changes, and the organisation removed the former employee’s access. The team also discovered that some provider-generated attributes changed frequently; those attributes were documented and handled rather than broadly ignored.

Learning points

The case demonstrates that IaC is not simply a replacement for clicking in a console. It is a governance and engineering system. Reusable modules, secure state, identity controls, policy tests, and operational feedback must work together. Automation increased consistency, but it did not remove the need for architectural judgement or incident procedures.

10. Challenges, Limitations, and Governance

IaC introduces its own risks. A defective module can create insecure resources across many environments. A broad execution role can make a compromised pipeline highly damaging. State corruption can interrupt recovery. Provider changes may alter defaults or deprecate resource arguments. Large plans can become difficult to review, and dependencies on externally created resources can produce ambiguity.

Teams should use small, composable changes and avoid applying an unreviewed plan generated long before execution. Provider and module versions should be controlled, but updates should be tested rather than postponed indefinitely. Production access should use short-lived credentials and workload identity where available.

Governance should address ownership, approval thresholds, exception expiry, evidence retention, separation of duties, and emergency changes. Regulatory obligations may require access logging, data-location controls, encryption, retention, and demonstrable change management. Automation records can support an audit, but they do not replace a risk assessment.

Ethically, practitioners should consider the consequences of automation failures. A misconfigured identity policy can expose personal data; an automated deletion can interrupt essential services. Safe defaults, staged rollout, least privilege, backups, and human review for high-impact actions are therefore professional responsibilities.

11. Contemporary Developments and AI

Platform engineering teams increasingly provide internal modules, templates, and self-service workflows. These platforms can improve developer productivity by offering approved building blocks, but they must avoid hiding cost, security, or operational consequences.

Policy engines are becoming more integrated with pull requests and deployment pipelines. Continuous drift monitoring and cloud security posture tools can correlate configuration findings with ownership and business impact. Open standards and compatible tooling are also important because organisations may need to migrate providers or avoid excessive dependence on one vendor.

AI assistants can help explain plans, generate starter configurations, identify repeated patterns, and suggest tests. However, generated infrastructure must be treated as untrusted code until reviewed. AI may invent unsupported provider arguments, omit security controls, reproduce secrets from context, or misunderstand organisational policy. Human approval, automated validation, and restricted execution identities remain necessary.

Best-Practice Checklist

• Store IaC and playbooks in version control.
• Separate reusable modules from environment-specific values.
• Use typed variables, validation, secure defaults, and documented outputs.
• Keep state remote, encrypted, versioned, access-controlled, and locked.
• Separate plan, approval, and apply responsibilities for sensitive environments.
• Treat state and logs as possible sources of sensitive information.
• Prefer idempotent modules over unrestricted shell commands.
• Scan for secrets and insecure resources before deployment.
• Detect drift and establish a documented response process.
• Pin and regularly test provider, module, image, and action versions.
• Use least-privilege, short-lived deployment credentials.
• Test recovery, rollback, backup restoration, and secret rotation.
• Record exceptions and emergency changes with an expiry or follow-up action.

Chapter Summary

Infrastructure as Code represents desired infrastructure in reviewable, repeatable files. Declarative tools calculate changes from desired configuration, state, and provider information, while configuration-management tools enforce software and operating-system conditions. Idempotence supports safe repetition, and modules provide controlled reuse.

State is essential for mapping declarations to real resources, but it can contain sensitive information and must be protected with encryption, access control, versioning, and locking. Drift occurs when the actual environment differs from the declared or recorded configuration; effective teams detect, investigate, reconcile, and prevent recurring drift.

Secrets require protection throughout their lifecycle, including source control, pipelines, state, logs, runtime access, rotation, and revocation. Testing and policy as code provide preventive and detective controls for security, compliance, reliability, and cost. Successful automation combines technology with governance, professional judgement, and continuous operational feedback.

Key Terms

Infrastructure as Code: Managing infrastructure through machine-readable, versioned definitions.

Declarative configuration: A description of the required end state rather than a fixed action sequence.

Imperative automation: Automation that specifies the steps or commands to execute.

Idempotence: The property that repeated application produces the same intended result as one application.

Desired state: The condition an automation system is intended to establish.

Resource: A managed infrastructure object such as a subnet, instance, or database.

Module: A reusable, parameterised unit of infrastructure configuration.

Provider: An integration that allows an IaC tool to communicate with a platform or service.

State: Data linking declared objects with real-world resources and recorded attributes.

Backend: The storage and coordination system used for IaC state.

State locking: A mechanism that prevents conflicting concurrent state operations.

Drift: A difference between declared or recorded configuration and the actual environment.

Configuration management: Automated installation and enforcement of operating-system and software settings.

Handler: A configuration-management action triggered when a related task changes something.

Secret: Sensitive data such as a password, token, private key, or certificate.

Policy as code: Machine-evaluable rules that enforce organisational or regulatory requirements.

Plan: A proposed set of changes produced before infrastructure is applied.

Immutable infrastructure: Replacing altered resources with newly built resources rather than updating them in place.

Least privilege: Granting only the permissions required for a task.

Reconciliation: Correcting an environment so that it conforms to an approved desired state.

Review Questions

Short Answer Questions

1. What is the main difference between declarative and imperative automation?
2. Why is idempotence important in configuration management?
3. What information does an IaC state file provide?
4. Give two controls that should protect a remote state backend.
5. What is infrastructure drift, and how can it be addressed?

Long Answer Questions

1. Explain how resources, variables, outputs, dependencies, and modules work together in a reusable IaC design.
2. Compare mutable and immutable infrastructure, including advantages and limitations of each approach.
3. Describe a secure end-to-end workflow for managing secrets in an infrastructure pipeline.
4. Explain how testing and policy as code can prevent insecure cloud deployments.
5. Discuss the technical and governance responsibilities involved in operating a production IaC system.

Analytical / Application Questions

1. A team discovers that a production database was changed through the cloud console to disable encryption. Analyse the possible causes and propose a safe drift-response process.
2. Design a module interface for a private application network. Identify at least four inputs, three outputs, secure defaults, and validation rules.
3. An organisation wants to allow developers to apply development infrastructure automatically but requires approval for production. Design a pipeline control model that preserves speed while maintaining security and auditability.

Multiple Choice Questions with Answers

Question 1

Which statement best describes declarative infrastructure automation?

A. It specifies the exact command sequence for every operation.

B. It describes the desired end state and lets the tool determine required actions.

C. It can only be used for operating-system configuration.

D. It eliminates the need for testing and review.

Correct Answer: B. It describes the desired end state and lets the tool determine required actions.

Question 2

What is the main purpose of an IaC state file?

A. To store application source code.

B. To replace all provider authentication mechanisms.

C. To map declared resources to real infrastructure and record relevant attributes.

D. To encrypt every secret used by an application automatically.

Correct Answer: C. To map declared resources to real infrastructure and record relevant attributes.

Question 3

Which practice best supports safe collaboration on shared infrastructure?

A. Keeping an unencrypted state file on each developer’s laptop.

B. Using a remote encrypted backend with access control and locking.

C. Allowing several users to apply changes simultaneously.

D. Editing state manually for every change.

Correct Answer: B. Using a remote encrypted backend with access control and locking.

Question 4

A configuration task installs a package only when it is absent. Which property does this illustrate?

A. Idempotence

B. Drift

C. Inheritance

D. Deprovisioning

Correct Answer: A. Idempotence

Question 5

Which action is the safest general approach for a database password?

A. Commit it to a private source repository.

B. Place it in a command-line argument visible to CI logs.

C. Store it in a managed secrets service and grant narrowly scoped runtime access.

D. Embed it in a reusable public module.

Correct Answer: C. Store it in a managed secrets service and grant narrowly scoped runtime access.

Question 6

What is infrastructure drift?

A. A planned change that has passed policy checks.

B. A difference between the declared or recorded configuration and the actual environment.

C. A provider plugin that cannot be downloaded.

D. A module containing too few variables.

Correct Answer: B. A difference between the declared or recorded configuration and the actual environment.

Question 7

Which item is an example of policy as code?

A. A developer verbally requesting a larger instance.

B. A rule rejecting storage resources that are publicly readable.

C. A manually written incident report.

D. A dashboard showing CPU utilisation only.

Correct Answer: B. A rule rejecting storage resources that are publicly readable.

Question 8

Why are handlers useful in the Ansible-style example?

A. They ensure a service restarts only when a related configuration changes.

B. They store Terraform state remotely.

C. They create cloud-provider accounts automatically.

D. They bypass idempotence requirements.

Correct Answer: A. They ensure a service restarts only when a related configuration changes.

Question 9

Which statement about marking a variable sensitive is most accurate?

A. It guarantees that the value is never stored anywhere.

B. It usually reduces display exposure but does not by itself encrypt all data paths.

C. It makes the value public to all pipeline users.

D. It removes the need for secret rotation.

Correct Answer: B. It usually reduces display exposure but does not by itself encrypt all data paths.

Question 10

A team wants production changes to be auditable. Which workflow is most appropriate?

A. Apply directly from an unmanaged laptop without a plan.

B. Allow console changes without recording them.

C. Generate a plan from a versioned commit, review it, approve it, and apply using a controlled identity.

D. Disable state locking to increase speed.

Correct Answer: C. Generate a plan from a versioned commit, review it, approve it, and apply using a controlled identity.