9. Cloud, Containers and Hybrid Infrastructure

Introduction and Context

Infrastructure as Code (IaC) becomes especially significant when infrastructure is distributed across public clouds, private data centres, container platforms, and managed services. A modern application may use a virtual network in one cloud, a managed database in another, Kubernetes for application execution, an on-premises identity service, and Ansible for operating-system configuration. The technical challenge is not merely creating resources. It is coordinating dependencies, security boundaries, operational ownership, portability, and change management across unlike environments.

Terraform and Ansible address different but complementary layers of this problem. Terraform is primarily a declarative provisioning and infrastructure-lifecycle tool. It represents desired infrastructure in configuration files, maintains state, calculates a dependency graph, and invokes provider APIs. Ansible is primarily an automation and configuration-management tool. It connects to hosts or APIs and performs tasks that bring systems to a desired operational condition. In practice, Terraform may create a virtual network, cluster, instances, storage, and identity bindings, while Ansible configures operating systems, installs agents, deploys applications, and performs post-provisioning tasks.

Cloud and hybrid IaC therefore requires architectural judgement. A configuration that is technically portable may fail to exploit valuable provider capabilities. Conversely, deep dependence on one provider can improve performance and simplicity while increasing switching costs. The central concern of this chapter is how to make these choices deliberately.

Learning Objectives

After studying this chapter, the student should be able to:

1. Explain how Terraform and Ansible are applied to public-cloud, private-cloud, container, and hybrid infrastructure.

2. Design IaC for networking, identity, storage, managed services, and Kubernetes.

3. Distinguish portable abstractions from provider-specific capabilities.

4. Evaluate immutable images, configuration management, and the boundary between provisioning and operations.

5. Analyse multi-cloud and hybrid-cloud trade-offs, including security, cost, resilience, governance, and operational complexity.

6. Recommend controls and best practices for reliable, secure, and auditable cloud infrastructure.

A Layered View of Cloud Infrastructure

Cloud infrastructure is best understood as a set of dependent layers rather than a collection of isolated resources. A typical sequence is:

• Organisational layer: accounts, subscriptions, projects, billing boundaries, policies, and quotas.
• Identity layer: users, service accounts, roles, groups, workload identities, and privileged-access controls.
• Network layer: virtual networks, subnets, routes, gateways, firewalls, load balancers, DNS, and connectivity to external networks.
• Compute layer: virtual machines, autoscaling groups, serverless functions, and container clusters.
• Data layer: block, object, and file storage; databases; queues; caches; and backup services.
• Platform layer: Kubernetes, service meshes, ingress controllers, observability systems, and deployment platforms.
• Workload layer: applications, configuration, secrets, and runtime policies.

The order is not always strictly linear. Identity policies may be required before resources can be created, while DNS and certificates may depend on load balancers. IaC tools model these relationships through explicit references, implicit dependencies, modules, inventories, and execution order.

A sound design assigns ownership to the appropriate layer. Terraform is generally well suited to durable resources whose lifecycle should be reviewed through a plan. Ansible is well suited to host configuration, software installation, procedural orchestration, and tasks that cannot conveniently be represented as cloud resources. Kubernetes manifests or Helm charts may manage resources inside a cluster, but they should not automatically be used to manage the entire cloud foundation.

Terraform Across Public and Private Clouds

Terraform uses providers as adapters to APIs. A provider exposes resources and data sources, while the Terraform engine maintains a dependency graph and state. The same language can describe resources from different providers, but syntactic similarity does not imply semantic equivalence. A virtual network, security group, managed database, or identity role may have substantially different behaviours in different clouds.

A practical Terraform architecture commonly separates:

• reusable modules for networks, clusters, databases, and monitoring;
• environment-specific root modules for development, testing, and production;
• variable and output interfaces between modules;
• remote state with locking and encryption;
• provider configuration and version constraints;
• policy checks and automated plan review.

A module should expose meaningful inputs such as network address ranges, availability requirements, or database size, rather than exposing every provider argument. Excessive abstraction produces a lowest-common-denominator module that hides important differences. A useful module abstracts repeated design decisions while still allowing deliberate escape hatches for provider-specific features.

Private-cloud platforms may expose APIs through technologies such as VMware, OpenStack, or platform-specific tools. Terraform can provision these environments when a suitable provider exists, although provider maturity, API consistency, and resource coverage must be assessed. Ansible can then configure virtual machines, network appliances, and platform services. In a private cloud, IaC must also account for physical capacity, IP address management, hardware failure domains, licensing, and operational teams—concerns that public-cloud abstractions may conceal.

Networking as an IaC Design Problem

Networking is often the foundation on which other resources depend. A production design normally specifies address spaces, subnet segmentation, routing, ingress and egress controls, private service access, DNS, load balancing, and connectivity to corporate networks.

Terraform is appropriate for declaring networks and their durable relationships. For example, a module may create a virtual network, private application subnets, restricted data subnets, route tables, security groups, and private endpoints. The configuration should make trust boundaries visible. A database subnet should not become reachable merely because an application subnet and a management subnet share a broad rule.

Hybrid connectivity introduces additional dependencies: virtual private networks or dedicated links, route exchange, firewall policies, overlapping address spaces, and DNS forwarding. IaC should validate these assumptions before deployment. A useful practice is to maintain an explicit network design document or machine-readable contract that records CIDR ranges, owners, permitted flows, and failure domains.

Ansible can configure network appliances or operating-system firewalls, but it is risky to use ad hoc scripts as the sole record of network intent. Configuration drift becomes difficult to detect when changes are made directly through consoles or device-specific commands. Where possible, both cloud network resources and appliance configuration should be represented in version control and tested in a safe environment.

Identity, Secrets, and Governance

Identity is infrastructure. A cloud deployment is incomplete if it creates compute resources without defining who or what may administer, read, modify, or delete them. IaC should establish least-privilege roles, service identities, group assignments, resource policies, and audit settings.

Long-lived access keys embedded in Terraform variables or Ansible inventories are a serious design weakness. Prefer short-lived credentials, federated identity, workload identity, and secret-manager integration. A CI/CD pipeline should authenticate using a controlled identity with narrowly scoped permissions. Secrets should not be stored in source code, Terraform state in plaintext, command output, or unprotected artifacts. State itself may contain sensitive attributes and must therefore be encrypted, access-controlled, backed up, and retained according to policy.

Governance can be implemented through policy as code. Rules may prohibit public storage buckets, require encryption, restrict geographic locations, mandate tags, or prevent excessively broad firewall rules. These policies should run before apply, not merely after an incident. Tagging and naming standards also support cost allocation, ownership, incident response, and automated cleanup.

Storage and Managed Services

Cloud storage has different durability, availability, performance, and consistency properties. IaC must distinguish block storage attached to a compute instance, shared file storage, object storage, and managed databases. The configuration should specify encryption, backup, retention, deletion protection, replication, and recovery objectives rather than treating storage as an anonymous volume.

Managed services reduce operational work but introduce provider dependence. A managed database may offer automated patching, backups, scaling, and monitoring, yet its extensions, network model, export tools, or replication features may not be portable. The decision should compare total operational cost and risk, not merely the monthly service price.

Destructive operations deserve particular attention. Terraform may correctly plan the replacement of a database when an immutable attribute changes, but correctness does not make the operation acceptable. Production resources should use deletion protection, lifecycle rules, backup validation, and approval gates. Data migration and rollback procedures should be tested independently of the IaC code.

Immutable Images and Configuration Management

An immutable infrastructure approach creates a machine image containing the operating system, packages, baseline configuration, and agents. Instances are replaced rather than manually modified. Image pipelines can use tools such as Packer, after which Terraform provisions instances from approved image versions. This improves repeatability, reduces configuration drift, and makes rollback more predictable.

Immutability does not mean that no runtime configuration exists. Environment-specific values, certificates, feature flags, and secrets are usually injected at deployment or startup. Ansible remains valuable for image construction, post-provisioning, controlled operational tasks, and systems that cannot be replaced easily. However, using Ansible to make frequent, undocumented changes to long-lived production servers weakens the benefits of immutable design.

A useful boundary is:

• Terraform: infrastructure resources and their lifecycle.
• Image pipeline: operating-system and common software baseline.
• Ansible: host configuration, operational automation, and integration tasks.
• Kubernetes or deployment tooling: application workloads and release behaviour.

These boundaries are conventions rather than laws. The correct division depends on the organisation’s skills, platform, compliance requirements, and recovery model.

Kubernetes and Container Infrastructure

Containers package applications with their dependencies, but they still require infrastructure: registries, networks, nodes, identity, storage classes, ingress, observability, and security controls. Kubernetes adds an orchestration layer with its own declarative API and reconciliation loops.

Terraform is often used to create the cloud foundation and managed Kubernetes control plane, node pools, identity bindings, registries, and supporting services. Kubernetes providers, Helm, GitOps controllers, or Ansible may then install cluster-level components and applications. Separating these layers reduces the risk that a failed application deployment will modify foundational networking or destroy a cluster.

Container infrastructure must address image provenance, vulnerability scanning, admission policies, resource requests and limits, pod security, network policies, secret handling, and persistent-volume behaviour. A container image is not a security boundary by itself. The host kernel, runtime, registry, admission process, and cloud identity model remain important.

Kubernetes stateful workloads require special care. A persistent volume claim may be declarative, but the underlying storage class, availability zone, backup method, and recovery process are platform-specific. IaC should document which resources are disposable and which contain irreplaceable data.

Hybrid and Multi-Cloud Architecture

Hybrid infrastructure combines on-premises and cloud resources; multi-cloud uses more than one public cloud. These approaches may be justified by regulatory location requirements, acquisition history, specialised services, resilience objectives, customer demand, or negotiation strategy. They are not automatically more resilient. A common identity failure, shared DNS dependency, poorly tested failover process, or central CI/CD outage can affect every environment.

Portability exists at several levels:

1. Syntax portability: similar Terraform or Kubernetes configuration can be reused.

2. API portability: equivalent resource operations are available across platforms.

3. Operational portability: teams can monitor, secure, recover, and troubleshoot each platform consistently.

4. Workload portability: applications and data can move with acceptable effort, performance, and cost.

The first level is easy to achieve and the last is difficult. A Kubernetes workload may deploy on multiple clouds, but storage performance, load-balancer behaviour, identity integration, egress pricing, and managed database dependencies may prevent practical migration.

A decision matrix should evaluate capability, cost, latency, compliance, skills, resilience, exit effort, and operational complexity. Standardising every capability across clouds can reduce portability to a minimal common set and increase engineering effort. Using each cloud’s strongest services may produce a better system but requires specialised skills and creates switching costs. The appropriate strategy is usually selective portability: keep critical interfaces and data-export paths portable while using provider-specific capabilities where their value is substantial.

An Illustrative Case: Northstar Retail’s Hybrid Platform

Background

Northstar Retail operates physical stores, an on-premises warehouse system, and a cloud-based customer platform. The company plans to modernise its order service. Store and warehouse systems must remain connected to the private data centre, while customer-facing services require elastic cloud capacity.

Situation and Problem

The initial proposal uses two public clouds to avoid dependence on one provider. However, the engineering team has limited experience with both platforms. Network ranges overlap with an existing data centre, database replication is untested, and administrators manually create cloud identities and firewall rules. The organisation needs a safer delivery model without promising unrealistic active-active portability.

Analysis

The main risks are not Terraform syntax or container packaging. They are identity inconsistency, data movement, network addressing, recovery time, and insufficient operational expertise. A two-cloud design would double the number of firewalls, monitoring integrations, IAM models, and incident procedures. The business requires recovery within hours, not necessarily simultaneous processing in both clouds.

Approach

Northstar adopts a primary-cloud and warm-recovery architecture. Terraform modules create the primary network, private connectivity, cluster, registry, managed database, logging, and identity bindings. A smaller recovery environment is provisioned in the second cloud using separate modules rather than pretending that all resources are identical. Ansible builds approved images and configures warehouse gateways and monitoring agents.

Application containers follow Kubernetes standards, but the platform team documents provider-specific ingress, storage, and identity mappings. Database backups are exported in a documented format and restored regularly in the recovery environment. Policy checks reject public database endpoints, unencrypted storage, missing ownership tags, and unrestricted administrative ports. Terraform state is stored remotely with locking and restricted access.

Outcome

The company gains repeatable environments and a tested recovery process. It does not achieve effortless workload portability, but it achieves a more realistic objective: the ability to rebuild essential services in a second environment within the approved recovery target. Operational runbooks identify which steps are automated and which require human approval.

Learning Points

The case demonstrates that multi-cloud value depends on tested recovery and organisational capability. Reusing code is helpful, but portable operations, data recovery, identity management, and observability matter more. A deliberately asymmetric architecture can be more economical and reliable than two fully identical environments.

Challenges, Risks, and Common Mistakes

Cloud IaC fails in recognisable ways. Teams may place all environments in one large state file, allowing unrelated changes to block one another and increasing blast radius. They may create overly generic modules that hide important provider behaviour, or use copy-and-paste configurations that drift across environments. Other common mistakes include storing secrets in repositories, allowing manual console changes without reconciliation, failing to pin provider and module versions, and treating a successful plan as proof that an application will operate correctly.

State corruption, provider bugs, API throttling, eventual consistency, quota exhaustion, and partial failure also require operational planning. A pipeline should retain plans and logs, use concurrency controls, and define recovery procedures for failed applies. Drift detection should distinguish authorised emergency changes from unapproved configuration. Importing existing infrastructure into state may be necessary during migration, but imported resources should subsequently be normalised and governed.

The principal limitation of IaC is that it cannot eliminate architectural ambiguity or organisational weakness. Code can reproduce a poor design very efficiently. It also cannot guarantee that external systems, human approvals, data semantics, or operational procedures behave as intended.

Professional, Ethical, and Governance Considerations

Infrastructure decisions affect confidentiality, availability, accessibility, and public trust. Engineers must minimise unnecessary collection and exposure of data, apply retention rules, and consider jurisdictional requirements. Automated deletion or replacement must not be designed without safeguards for data owners. Access should be auditable, and emergency privileges should be time-limited and reviewed.

Professional responsibility also includes honest communication about resilience. Describing a system as “multi-cloud” does not prove independence or failover capability. Recovery claims should be supported by exercises and measured recovery time and data-loss objectives. Cost forecasts should include network egress, managed-service consumption, support plans, licensing, and staff capability. Sustainability considerations include resource right-sizing, autoscaling policies, image lifecycle management, and avoiding unnecessary duplicate environments.

Emerging Directions and Best Practices

Platform engineering increasingly provides approved templates, self-service interfaces, policy guardrails, and standard observability while allowing application teams to work at an appropriate abstraction level. GitOps extends declarative management into continuous reconciliation, particularly for Kubernetes. Infrastructure testing is also becoming more systematic through static analysis, policy testing, ephemeral environments, integration tests, and disaster-recovery exercises.

Artificial intelligence may assist with configuration explanation, anomaly detection, cost analysis, and remediation suggestions. It should not receive unrestricted production credentials or apply infrastructure changes without review. Generated recommendations require validation because cloud context, security requirements, and organisational policy are not always visible to a model.

Effective practice includes:

• design layers and ownership boundaries before writing modules;
• keep state remote, encrypted, locked, backed up, and access-controlled;
• pin versions and review plans through pull requests;
• use short-lived identities and secret managers;
• test networks, policies, images, backups, and recovery procedures;
• prefer immutable replacement for suitable workloads;
• document provider-specific assumptions and exit paths;
• separate foundation, cluster, and application lifecycles;
• monitor drift, cost, security posture, and operational SLOs;
• measure portability by tested recovery and migration effort, not code reuse alone.

Chapter Summary

Cloud, container, and hybrid infrastructure requires IaC to coordinate several dependent layers: identity, networking, compute, storage, managed services, Kubernetes, and applications. Terraform provides declarative provisioning and lifecycle management across providers, while Ansible supports configuration, image construction, operational automation, and integration. Immutable images reduce drift, but runtime configuration and provider-specific behaviour remain important.

Portability has technical and operational dimensions. Multi-cloud and hybrid designs can support regulatory, resilience, or capability objectives, but they increase complexity and do not automatically provide failover. Sound architecture uses modularity, policy as code, secure state management, tested recovery, selective abstraction, and explicit governance. The most valuable outcome is not identical infrastructure everywhere; it is infrastructure that can be understood, changed, secured, and recovered with confidence.

Key Terms

Infrastructure as Code: Managing infrastructure through machine-readable, version-controlled definitions.

Provider: A Terraform plugin that communicates with a platform or service API.

Terraform state: The persisted mapping between configuration and resources managed by Terraform.

Module: A reusable, parameterised unit of Terraform configuration.

Immutable infrastructure: Infrastructure replaced from approved artefacts rather than modified indefinitely in place.

Configuration drift: Difference between recorded desired configuration and actual infrastructure state.

Managed service: A provider-operated service that reduces customer responsibility for underlying operations.

Virtual network: A logically isolated network defined within a cloud or virtualisation platform.

Workload identity: An identity assigned to an application or service rather than a human user.

Policy as code: Machine-enforced rules governing infrastructure and deployment decisions.

Container: A process package containing an application and its required user-space dependencies.

Kubernetes: A platform for orchestrating containerised workloads through declarative APIs.

Persistent volume: Storage exposed to a workload with lifecycle characteristics beyond a container instance.

Hybrid cloud: An architecture combining private infrastructure with public-cloud services.

Multi-cloud: Use of services from more than one public-cloud provider.

Portability: The ability to move or reproduce workloads across environments with acceptable effort and impact.

State locking: A mechanism preventing conflicting concurrent changes to infrastructure state.

Least privilege: Granting only the permissions required for a task or identity.

Recovery point objective: The maximum acceptable amount of data loss measured in time.

Recovery time objective: The target time for restoring a service after disruption.

Review Questions

Short Answer Questions

1. How do Terraform and Ansible differ in their primary responsibilities?

2. Why should Terraform state be protected as sensitive information?

3. What is meant by selective portability in multi-cloud architecture?

4. State two benefits of immutable infrastructure.

5. Why are Kubernetes persistent workloads more complex than stateless workloads?

Long Answer Questions

1. Explain how IaC can be structured across identity, networking, compute, storage, Kubernetes, and application layers.

2. Discuss the benefits and risks of using managed services in a hybrid-cloud architecture.

3. Compare immutable images with post-provisioning configuration management using Ansible.

4. Analyse the organisational and technical consequences of adopting a multi-cloud strategy.

5. Describe a governance model for secure Terraform and Ansible delivery in production.

Analytical / Application Questions

1. An organisation wants identical Terraform modules for two clouds, but their databases and identity systems behave differently. Propose an abstraction strategy and justify where provider-specific code should remain.

2. A Terraform plan proposes replacing a production database after a configuration change. Explain the checks, approvals, safeguards, and recovery tests required before proceeding.

3. Design a hybrid recovery approach for a containerised application whose primary database and network connectivity are located in a private data centre.

22. Multiple Choice Questions with Answers

Question 1

Which responsibility most naturally belongs to Terraform in a layered cloud IaC design?

A. Installing a package interactively on a running server

B. Creating a virtual network, subnets, and managed database

C. Performing a one-time application data correction

D. Responding manually to an application alert

Correct Answer: B. Creating a virtual network, subnets, and managed database

Question 2

Why should Terraform state be encrypted and access-controlled?

A. It contains only public provider documentation

B. It may contain sensitive attributes and defines managed infrastructure relationships

C. Encryption is required only for Kubernetes manifests

D. State is used solely to calculate application logs

Correct Answer: B. It may contain sensitive attributes and defines managed infrastructure relationships

Question 3

What is the principal danger of an overly generic multi-cloud module?

A. It prevents version control from being used

B. It always increases network bandwidth

C. It may hide important provider-specific behaviour and reduce useful capability

D. It makes Terraform unable to create any resource

Correct Answer: C. It may hide important provider-specific behaviour and reduce useful capability

Question 4

Which approach best represents immutable infrastructure?

A. Manually patching production servers every week

B. Replacing instances from a tested image when the baseline changes

C. Allowing administrators to modify servers without recording changes

D. Disabling image versioning to simplify deployments

Correct Answer: B. Replacing instances from a tested image when the baseline changes

Question 5

A team wants to use two clouds for resilience. Which evidence most strongly demonstrates that the design provides recovery capability?

A. The same application name exists in both clouds

B. Both clouds have identical colour-coded dashboards

C. A tested recovery exercise meets defined recovery objectives

D. The team has copied Terraform files between repositories

Correct Answer: C. A tested recovery exercise meets defined recovery objectives

Question 6

Which practice best protects a CI/CD pipeline’s cloud access?

A. Embedding a permanent administrator key in the repository

B. Using a short-lived, narrowly scoped workload identity

C. Sharing one personal credential among all engineers

D. Printing credentials in pipeline logs for auditing

Correct Answer: B. Using a short-lived, narrowly scoped workload identity

Question 7

Why should Kubernetes foundation resources and application workloads often have separate lifecycles?

A. Kubernetes cannot run applications on cloud infrastructure

B. It reduces the chance that an application change will alter or destroy foundational infrastructure

C. Terraform cannot manage more than one resource type

D. Applications never require configuration

Correct Answer: B. It reduces the chance that an application change will alter or destroy foundational infrastructure

Question 8

Which statement best distinguishes syntax portability from operational portability?

A. Syntax portability concerns reusable configuration; operational portability concerns the ability to run and recover consistently

B. They are identical because equivalent code guarantees equivalent operations

C. Operational portability applies only to physical servers

D. Syntax portability requires using no provider APIs

Correct Answer: A. Syntax portability concerns reusable configuration; operational portability concerns the ability to run and recover consistently

Question 9

A managed database is convenient but uses provider-specific extensions. What trade-off does this illustrate?

A. Lower operational effort may be accompanied by increased provider dependence

B. Managed services always eliminate the need for backups

C. Provider-specific features guarantee effortless migration

D. Database services cannot be governed through IaC

Correct Answer: A. Lower operational effort may be accompanied by increased provider dependence

Question 10

Which control most directly prevents a public database endpoint from being deployed?

A. A policy-as-code rule evaluated before apply

B. A larger container image

C. A manual naming convention without enforcement

D. A second application deployment

Correct Answer: A. A policy-as-code rule evaluated before apply