Infrastructure as Code and Configuration Automation

Introduction

Cloud infrastructure is increasingly created, modified, and retired through software rather than through manual work in a console. This practice is known as Infrastructure as Code (IaC). IaC expresses infrastructure resources—such as networks, virtual machines, databases, IAM policies, and load balancers—in machine-readable configuration files. These files can be reviewed, versioned, tested, and executed repeatedly.

For DevOps teams, IaC addresses a central operational problem: an environment that works today may be difficult to reproduce tomorrow. Manual configuration creates undocumented differences, inconsistent security settings, and dependence on individual administrators. IaC replaces undocumented activity with an explicit, reviewable desired state.

This chapter introduces Terraform and provider-native templates, with emphasis on AWS and Google Cloud Platform (GCP). It explains state, resources, providers, modules, variables, outputs, environments, version control, testing, policy as code, and drift detection. The objective is not merely to create infrastructure, but to create it safely, consistently, and audibly.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain the purpose and operating model of IaC.
2. Distinguish Terraform from provider-native tools such as AWS CloudFormation and Google Cloud Deployment Manager or Config Connector.
3. Create basic Terraform configurations using providers, resources, variables, outputs, and modules.
4. Explain state management, locking, remote backends, and sensitive information.
5. Organise development, staging, and production environments safely.
6. Apply version control, validation, testing, policy as code, and drift detection.
7. Evaluate the operational, security, governance, and ethical implications of automated infrastructure.

1. The Foundations of Infrastructure as Code

1.1 Desired state and convergence

IaC generally describes a desired state: the resources and properties that should exist. An IaC engine compares this desired state with the observed or recorded state and performs actions needed to converge the environment.

A simplified model is:

Here, Plan is the proposed set of changes, DesiredState is the configuration, CurrentState is the known or observed infrastructure, and ProviderCapabilities are the operations supported by the cloud platform. The model is conceptual rather than a universal mathematical algorithm. Actual tools may use stored state, refresh operations, provider APIs, and dependency graphs.

The desired-state approach differs from an imperative script. An imperative script may say “create a network, then create a server.” A declarative configuration says “a network and a server with these properties must exist.” The engine determines an appropriate operation sequence.

1.2 Benefits and trade-offs

Important benefits include:

• Repeatability: the same configuration can create comparable environments.
• Consistency: security groups, labels, regions, and naming conventions can be standardised.
• Reviewability: changes can be examined through pull requests.
• Automation: infrastructure can be integrated into continuous delivery pipelines.
• Auditability: version history records who proposed and approved changes.
• Recovery: environments can be rebuilt when configurations and required data are available.

IaC also introduces risks. A configuration error can affect many resources quickly. State files may contain sensitive values. Provider APIs may change. Poorly designed modules can hide important behaviour. IaC improves control only when supported by access management, testing, approvals, backups, and monitoring.

1.3 Idempotence and immutability

An operation is idempotent when applying it repeatedly produces the same intended result after the first successful application. Declarative tools aim for idempotent behaviour, although external systems, timestamps, random values, and non-deterministic APIs can complicate this goal.

Immutable infrastructure treats deployed instances as replaceable rather than manually modified. When a change is required, a new image or resource is created and the old one is retired. This reduces configuration drift inside servers, but it does not remove the need for persistent-data management, backups, migrations, and operational controls.

2. Terraform and Provider-Native Templates

2.1 Terraform’s operating model

Terraform uses HashiCorp Configuration Language (HCL) to describe infrastructure. Its principal concepts are:

• Provider: a plugin that translates Terraform operations into API calls, such as AWS or Google.
• Resource: an object managed by Terraform, such as an AWS VPC or a GCP network.
• Data source: information retrieved from an external system, such as an existing image.
• State: a record that maps configuration objects to real infrastructure objects.
• Module: a reusable collection of Terraform configuration.
• Plan: a preview of proposed changes.
• Apply: an operation that executes an approved plan.

A minimal example is:

terraform {
  required_version = ">= 1.6.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = var.aws_region
}

resource "aws_s3_bucket" "logs" {
  bucket = var.bucket_name

  tags = {
    Purpose = "application-logs"
    Managed = "terraform"
  }
}

Provider versions should be constrained and committed through the dependency lock file. Uncontrolled provider upgrades can change defaults or resource behaviour.

2.2 Provider-native approaches

AWS CloudFormation expresses AWS resources in YAML or JSON and is operated as an AWS service. AWS Cloud Development Kit (CDK) generates CloudFormation templates from languages such as TypeScript or Python. GCP has provider-native approaches including Google Cloud Deployment Manager (historically used for template-driven deployments) and Config Connector, which represents Google Cloud resources as Kubernetes objects.

Terraform supports multiple providers through a common workflow. Provider-native tools may offer earlier access to platform-specific features and close integration with the cloud provider. Terraform may offer a consistent multi-cloud workflow and a broad ecosystem. “Multi-cloud” does not automatically mean portable architecture: networking, identity, managed services, and operational semantics remain provider-specific.

The correct choice depends on organisational skills, compliance requirements, platform scope, lifecycle model, and integration needs.

3. Configuration Structure: Variables, Outputs, and Dependencies

3.1 Variables and locals

Variables separate reusable logic from environment-specific values.

variable "aws_region" {
  type        = string
  description = "AWS deployment region"
  default     = "eu-west-1"
}

variable "environment" {
  type        = string
  description = "Deployment environment"

  validation {
    condition     = contains(["dev", "staging", "prod"], var.environment)
    error_message = "Environment must be dev, staging, or prod."
  }
}

locals {
  name_prefix = "orders-${var.environment}"
}

A variable has a type, optional default, description, and possibly validation. Sensitive variables should be marked sensitive = true, but this does not encrypt the value in all locations. Secrets should normally come from a secrets manager or protected pipeline variable, not from a repository.

3.2 Outputs

Outputs expose useful values to users, modules, or later pipeline stages.

output "bucket_arn" {
  description = "ARN of the application log bucket"
  value       = aws_s3_bucket.logs.arn
}

Outputs can unintentionally disclose endpoints, identifiers, or credentials. Security review must therefore include outputs as well as resources.

3.3 Dependency graphs

Terraform builds a dependency graph from references. If a resource refers to another resource’s attribute, Terraform normally creates the dependency automatically. An explicit depends_on should be used only when a real dependency is not visible through an attribute reference. Excessive explicit dependencies reduce parallelism and make configurations harder to understand.

4. State Management

4.1 Purpose of state

Terraform state maps a configuration address, such as aws_s3_bucket.logs, to a real object identifier. It may also contain attributes needed to calculate future changes. State is not simply a cache and must be treated as an operationally important asset.

A team should use a remote backend with:

• encryption at rest;
• restricted access through least privilege;
• versioning and recovery where supported;
• state locking or an equivalent concurrency mechanism;
• audit logging;
• separation between environments.

For example, an AWS backend may use an S3 bucket with encryption, versioning, and a locking design appropriate to the Terraform version in use. GCP teams may use a versioned, access-controlled Cloud Storage bucket.

4.2 Locking and concurrency

Two administrators applying different plans to the same state can produce conflicting operations. Locking ensures that only one state-changing operation proceeds at a time. Locking does not replace code review or eliminate conflicts; it prevents simultaneous state writers from corrupting coordination.

Never edit state manually unless a documented recovery procedure requires it. Commands such as terraform state mv, import, and rm can be useful but should be reviewed carefully because they change Terraform’s understanding of ownership.

4.3 Sensitive data

Marking a value sensitive usually suppresses it in terminal output; it does not make the value absent from state. State access must therefore be governed like access to a sensitive database. Avoid passing passwords directly into resource arguments when a managed secret reference is available.

5. Modules and Environment Design

A module is a reusable interface around infrastructure. A good module has a clear purpose, documented inputs, predictable outputs, safe defaults, and minimal hidden behaviour.

Example module call:

module "network" {
  source  = "./modules/network"
  name    = "orders-${var.environment}"
  region  = var.aws_region
  cidr    = var.vpc_cidr
  tags    = local.common_tags
}

Modules should be versioned when consumed from a registry or another repository. Avoid creating a “universal” module with dozens of flags. A small number of focused modules is generally easier to test and upgrade.

5.1 Environment patterns

Common patterns include:

1. Separate directories and state: each environment has an independent root module and backend. This gives strong isolation and explicit promotion.
2. Workspaces: one configuration uses different state instances. Workspaces can be convenient, but they may be insufficient where production requires substantially different permissions or architecture.
3. Separate repositories: useful where teams, compliance boundaries, or release lifecycles differ.

A production environment should not be distinguished only by a variable such as environment = "prod". It should also have separate credentials, state, approval rules, account or project boundaries where appropriate, and stronger policy controls.

6. A Safe IaC Workflow

A professional workflow commonly follows these stages:

1. Design: identify ownership, dependencies, data classification, cost, recovery, and security requirements.
2. Author: write configuration and documentation.
3. Format and validate: run terraform fmt and terraform validate.
4. Static analysis: scan for insecure patterns, missing tags, public exposure, and policy violations.
5. Plan: generate a plan using the intended variable set and credentials.
6. Review: inspect the plan through peer review.
7. Apply: execute in a controlled pipeline rather than from an unmanaged laptop.
8. Verify: test service health, security controls, monitoring, and outputs.
9. Observe: detect drift and operational changes.

Plans should be treated as time-sensitive: infrastructure or provider data can change between planning and application. For sensitive production changes, save and apply the reviewed plan using the same relevant context, while recognising that external conditions may still require verification.

Version control practices

Repositories should include provider constraints, lock files, README documentation, examples, ownership information, and a change policy. Commits should be small enough to review and should describe intent. Secrets, private keys, state files, and generated plans should not be committed unless a carefully reviewed process explicitly protects them.

7. Testing and Policy as Code

IaC testing operates at several levels:

Policy as code converts governance requirements into executable rules. A policy might require encryption, prohibit public storage, require cost-centre tags, or restrict approved regions. Policies should provide actionable messages and a controlled exception process. A rule that blocks every unusual case without a risk-based exception process encourages workarounds.

Example policy logic in plain language:

IF a storage resource is publicly readable
THEN deny the plan
UNLESS an approved exception identifier is present and unexpired

Policy enforcement should occur before deployment and, where possible, after deployment through cloud configuration and security monitoring.

8. Drift Detection and Reconciliation

Drift is a difference between the declared configuration, the recorded state, and the actual infrastructure. It can result from console edits, emergency changes, provider defaults, failed operations, or changes made by another automation system.

A periodic refresh or plan can reveal drift, but a clean plan is not proof of complete compliance. Some external settings may not be represented by the resource schema, and monitoring may have incomplete permissions. Drift processes should therefore combine:

• scheduled plan or refresh operations;
• cloud configuration monitoring;
• audit-log analysis;
• ownership metadata;
• alerts for unauthorised changes;
• documented reconciliation procedures.

When drift is found, the team must decide whether the configuration or the infrastructure reflects the intended design. The correct response may be to apply the code, update the code, import an unmanaged resource, or remove the resource. Blindly overwriting emergency changes can worsen an incident.

9. Applied Case: A Multi-Environment Orders Service

The following is an illustrative case.

Background

A small retailer operates an orders service on AWS and plans to deploy a reporting component on GCP. Development environments are created manually, while production is managed by two administrators. Incidents have revealed inconsistent tags, open security rules, and unclear ownership of resources.

Problem

The company needs repeatable environments, controlled production changes, secure state, and a way to identify console changes. It also needs separate development, staging, and production accounts or projects.

Analysis

The team identifies the following requirements:

• network and identity boundaries must differ by environment;
• production changes require approval;
• storage must be encrypted and private by default;
• resources require application, owner, environment, and cost-centre tags;
• state must not be stored in developer laptops;
• modules must support AWS and GCP-specific implementations without pretending that the services are identical.

Approach

The team creates separate root configurations for each environment and uses remote, locked state. A versioned network module supplies subnets and routing, while cloud-specific modules handle AWS and GCP details. Provider versions are constrained and lock files are committed.

A pull request pipeline runs formatting, validation, static security checks, and policy tests. Policies deny public object storage and resources without mandatory tags. A plan is attached to the pull request. Production application requires approval from an authorised reviewer and uses a deployment identity with narrowly scoped permissions.

A scheduled drift job generates plans and sends alerts when unexpected changes occur. Emergency console changes are recorded, reviewed, and either incorporated into code or removed through a controlled reconciliation.

Outcome

Development environments become reproducible, production changes become auditable, and the team can identify unauthorised changes more quickly. The solution does not eliminate all risk: provider outages, incorrect modules, excessive permissions, and application-level failures remain possible. These risks are managed through layered controls rather than through IaC alone.

Learning points

The case demonstrates that successful IaC is an operating model, not merely a collection of files. Environment isolation, state protection, review, policy, testing, and monitoring are as important as syntax.

10. Governance, Ethics, and Contemporary Practice

IaC can encode organisational power. A flawed rule may deny legitimate access, expose personal data, or impose costs on teams that cannot influence the policy. Teams should document ownership, review high-impact policies, retain audit records, and provide an appeal or exception process.

Least privilege applies to both humans and automation. A pipeline that can delete every production resource is a significant concentration of risk. Use separate identities, short-lived credentials where feasible, approval gates, and protected branches.

Cost governance should be part of design. Resource labels, budgets, quotas, and automated expiration of temporary environments reduce waste. However, automatic deletion must account for retention requirements, backups, legal holds, and business continuity.

Artificial intelligence can assist with generating HCL, explaining plans, and suggesting policy rules. Generated configurations must not be trusted automatically. Review is required for security, licensing, provider accuracy, data leakage, and unintended destructive actions. Do not submit secrets or confidential architecture to an unapproved AI service.

Emerging practices include GitOps reconciliation, infrastructure testing in ephemeral environments, software supply-chain signing, and policy engines integrated into deployment pipelines. These practices strengthen automation but increase the need for clear ownership and observability.

Challenges and Limitations

1. Provider variation: AWS and GCP expose different abstractions, so complete portability is unrealistic.
2. State complexity: state can contain sensitive values and become difficult to migrate.
3. Destructive changes: renaming or changing resource arguments may cause replacement.
4. External dependencies: DNS, certificates, quotas, and APIs may fail independently of code quality.
5. Long-lived resources: databases and networking require careful migration and backup planning.
6. Organisational resistance: teams may continue making undocumented console changes.
7. False confidence: a successful plan does not prove that an application is secure or operationally correct.

Best-Practice Checklist

• Store IaC in version control with peer review.
• Pin provider and module versions.
• Use remote, encrypted, locked, and access-controlled state.
• Separate environments and credentials.
• Prefer least privilege and short-lived deployment identities.
• Validate, lint, scan, test, and review plans before applying.
• Enforce essential controls with policy as code.
• Detect and investigate drift.
• Document modules, ownership, recovery, and exceptions.
• Protect persistent data and test destructive changes.
• Treat generated code and AI suggestions as untrusted until reviewed.

Chapter Summary

Infrastructure as Code expresses cloud infrastructure as versioned, reviewable configuration. Terraform provides a provider-based, declarative workflow built around resources, data sources, modules, variables, outputs, plans, applies, and state. Provider-native tools offer close integration with AWS or GCP and may be preferable for platform-specific requirements.

Reliable IaC depends on more than configuration syntax. Remote state, locking, environment isolation, dependency management, version control, automated testing, policy as code, controlled approvals, and drift detection form a complete operating model. Modules should be focused and documented; sensitive values require secure handling; production automation requires least privilege and governance.

IaC improves repeatability and auditability but does not remove risk. Provider differences, destructive changes, state exposure, cost, policy errors, and unauthorised modifications must be managed through layered technical and organisational controls.

Key Terms

Infrastructure as Code: Managing infrastructure through machine-readable, version-controlled definitions.

Declarative configuration: A description of the desired result rather than a step-by-step procedure.

Idempotence: The property that repeated execution produces the same intended result.

Terraform: A declarative IaC tool supporting multiple infrastructure providers.

Provider: A plugin translating configuration into platform API operations.

Resource: An infrastructure object managed by an IaC tool.

Data source: Information queried from an external system for use in configuration.

State: A record connecting configuration objects with real infrastructure.

Backend: The location and mechanism used to store state.

State locking: A mechanism preventing concurrent state-changing operations.

Module: A reusable collection of infrastructure configuration.

Variable: A parameter that supplies configurable input.

Output: A value exposed by a configuration or module.

Drift: A difference between declared, recorded, and actual infrastructure.

Policy as code: Executable rules that enforce governance requirements.

Immutable infrastructure: Infrastructure replaced rather than manually modified.

Least privilege: Granting only the permissions required for a task.

Remote backend: A shared, controlled service used to store state outside a local machine.

Provider-native template: A template or controller designed primarily for one cloud platform.

Reconciliation: Bringing actual infrastructure back into alignment with intended configuration.

Review Questions

Short Answer Questions

1. What is the principal difference between declarative and imperative infrastructure automation?
2. Why must Terraform state be protected as sensitive operational data?
3. What are the purposes of variables and outputs in a Terraform module?
4. Define infrastructure drift and identify two causes.
5. What is policy as code?

Long Answer Questions

1. Explain Terraform’s operating model, including providers, resources, plans, applies, and state.
2. Compare Terraform with AWS CloudFormation and GCP provider-native approaches.
3. Describe a secure design for development, staging, and production environments.
4. Explain how version control, testing, peer review, and approval gates reduce IaC risk.
5. Discuss the benefits and limitations of immutable infrastructure in cloud operations.

Analytical / Application Questions

1. A production bucket was made public through a console change. Describe the investigation and reconciliation process you would use without blindly overwriting possible emergency work.
2. Design a module and pipeline for a service deployed to AWS and GCP. Identify which concerns can be standardised and which remain provider-specific.
3. An organisation wants one highly privileged pipeline identity for all environments. Evaluate this proposal and recommend a safer alternative using isolation and least privilege.

Multiple Choice Questions with Answers

Question 1

Which Terraform component translates configuration into cloud-provider API operations?

A. Provider

B. Backend

C. Variable

D. Output

Correct Answer: A. Provider

Question 2

What is the primary purpose of Terraform state?

A. To store application source code

B. To map configuration objects to real infrastructure objects

C. To replace cloud audit logs

D. To encrypt all provider credentials automatically

Correct Answer: B. To map configuration objects to real infrastructure objects

Question 3

Which practice best reduces concurrent state-write conflicts?

A. Using local state on every laptop

B. Disabling plan reviews

C. Using a remote backend with locking

D. Removing provider version constraints

Correct Answer: C. Using a remote backend with locking

Question 4

Which statement about a sensitive Terraform variable is correct?

A. It can never appear in state

B. It is automatically deleted after apply

C. It is always encrypted by every backend

D. Its display may be suppressed, but state still requires protection

Correct Answer: D. Its display may be suppressed, but state still requires protection

Question 5

What is the most appropriate use of a Terraform module?

A. To provide a focused, reusable infrastructure component

B. To avoid documenting infrastructure

C. To combine every organisational resource into one file

D. To bypass provider authentication

Correct Answer: A. To provide a focused, reusable infrastructure component

Question 6

What does drift describe?

A. A provider plugin version constraint

B. A difference between intended or recorded and actual infrastructure

C. A syntax error in a variable declaration

D. A successful state lock

Correct Answer: B. A difference between intended or recorded and actual infrastructure

Question 7

Which control is an example of policy as code?

A. A developer verbally requesting a firewall exception

B. A rule denying public storage during plan evaluation

C. A manually edited production resource

D. A cloud provider outage

Correct Answer: B. A rule denying public storage during plan evaluation

Question 8

Why should provider versions be constrained?

A. To prevent all infrastructure changes permanently

B. To ensure configurations cannot use variables

C. To reduce unexpected behaviour from uncontrolled provider upgrades

D. To eliminate the need for state

Correct Answer: C. To reduce unexpected behaviour from uncontrolled provider upgrades

Question 9

Which environment practice provides the strongest production isolation?

A. Using only a different workspace name

B. Using separate credentials, state, and suitable account or project boundaries

C. Allowing all developers production administrator access

D. Applying development plans directly to production

Correct Answer: B. Using separate credentials, state, and suitable account or project boundaries

Question 10

What is a key limitation of using AI-generated Terraform configuration?

A. It cannot produce any text files

B. It removes the need for security review

C. It always requires a physical server

D. It may contain inaccurate, insecure, or destructive recommendations

Correct Answer: D. It may contain inaccurate, insecure, or destructive recommendations