10. Testing, Quality and Reliability EngineeringIntroduction and ContextInfrastructure as Code (IaC) changes infrastructure from an manually executed operational activity into a versioned, reviewable, and repeatable engineering process. Terraform configurations describe desired resources and relationships, while Ansible playbooks describe configuration and orchestration procedures. This shift creates major benefits, but it also creates a new obligation: infrastructure code must be tested with the same discipline applied to application code. An infrastructure change can fail in several ways. It may be syntactically invalid, violate organizational policy, produce an incorrect dependency graph, configure a service insecurely, or work in a test account but fail under production constraints. A successful deployment is therefore not sufficient evidence of quality. Quality must be evaluated across multiple dimensions: correctness, security, maintainability, repeatability, recoverability, performance, and operational suitability. Testing in IaC is not one activity or one tool. It is a layered engineering practice that begins with formatting and validation, progresses through static analysis and policy testing, and extends to unit, contract, integration, and end-to-end testing. Ephemeral environments provide realistic test contexts, while observability, failure injection, and recovery testing assess whether infrastructure remains dependable when conditions are abnormal. For postgraduate DevOps professionals, the central question is not simply “Does the code run?” It is “What evidence demonstrates that this infrastructure is safe to change, behaves as intended, satisfies organizational constraints, and can be recovered when assumptions fail?” Learning ObjectivesBy the end of this chapter, students should be able to: 1. Explain the role of testing and quality engineering in Terraform and Ansible delivery pipelines. 2. Distinguish formatting, validation, linting, static analysis, policy, unit, contract, integration, and end-to-end tests. 3. Design a layered test strategy using Terraform test frameworks and Ansible testing tools. 4. Use ephemeral environments to test infrastructure realistically without creating uncontrolled cost or risk. 5. Interpret test evidence, plan changes, drift signals, and recovery results in engineering decisions. 6. Identify common testing failures, governance concerns, and limitations of infrastructure test automation. 7. Develop quality gates and reliability practices appropriate for professional DevOps environments. Infrastructure Quality as an Engineering PropertyQuality in IaC is multidimensional. Functional correctness asks whether resources and configurations satisfy their declared purpose. Structural quality concerns modularity, naming, dependency clarity, idempotence, and maintainability. Security quality concerns identity, network exposure, secrets, encryption, and compliance. Reliability quality concerns availability, failure tolerance, recovery, and predictable behavior during change. These dimensions often interact. A configuration may be functionally correct but operationally unsafe because it permits public access. A highly available design may be expensive or unnecessarily complex. A playbook may complete successfully while repeatedly changing the same file, demonstrating a lack of idempotence. Quality engineering therefore requires explicit acceptance criteria rather than an assumption that successful execution equals correctness. A useful quality model evaluates infrastructure through four questions: - Is the change valid? The code is well formed and can be interpreted by the relevant tools.
- Is the change intended? The plan or execution matches the requested design and does not contain unexplained side effects.
- Is the change permitted? Security, policy, compliance, cost, and architectural rules are satisfied.
- Is the change dependable? The resulting system operates, tolerates expected failures, and can be restored or rolled back.
Testing should produce evidence for each question. No single test type can answer all of them. The Testing Pyramid for InfrastructureThe testing pyramid is a useful model for balancing speed, cost, and realism. Lower-level tests are fast and numerous; higher-level tests are slower, more expensive, and closer to actual user or operator experience. At the base are formatting, parsing, validation, linting, and static analysis. These tests identify defects before provider calls or remote execution. Above them are unit and module tests, which assess local logic and expected resource construction. Contract tests verify assumptions between modules, modules and providers, or automation and operating systems. Integration tests apply code to real or realistic services and inspect the resulting state. End-to-end tests assess complete workflows, such as provisioning a network, deploying an application, accessing it through its public interface, and destroying the environment. The pyramid is not a rigid sequence. A security policy check may be fast but conceptually operate at a higher level than syntax validation. Similarly, a small smoke test after deployment may be more valuable than a large collection of shallow unit tests. The principle is to obtain the earliest reliable feedback possible and reserve expensive tests for behavior that cannot be established earlier. The First Quality Gates: Formatting, Validation and LintingFormatting establishes a shared representation of code. Terraform’s `terraform fmt` applies canonical formatting, reducing irrelevant differences in reviews and making automated comparison more reliable. Ansible does not have a single equivalent formatting command for every file type, but YAML formatters, editor integrations, and style rules can enforce indentation, quoting, line length, and document consistency. Validation checks whether a configuration is structurally acceptable. `terraform validate` verifies internal consistency and configuration syntax within an initialized working directory. It does not prove that a design is secure or that a cloud provider will accept every operation. Ansible syntax checks, YAML parsers, and collection validation identify malformed playbooks, undefined structures, and certain module errors. Linting applies style and heuristic rules. Tools such as TFLint can detect suspicious Terraform constructs, provider-specific issues, and practices that are technically valid but risky. Ansible Lint identifies deprecated modules, command use where an idempotent module is preferable, weak task names, missing handlers, and other maintainability concerns. Linting is valuable because it captures institutional knowledge that is difficult to express through syntax alone. These gates should run locally and in continuous integration. They should be deterministic, quick, and easy to reproduce. A team should distinguish mandatory correctness failures from advisory style warnings; otherwise developers may learn to ignore all output. Static Analysis and Policy as CodeStatic analysis examines code or planned changes without necessarily creating infrastructure. Terraform plan output can be analyzed by tools such as Checkov, tfsec, or custom policy engines. Open Policy Agent and its Rego language can express rules such as “storage resources must use encryption,” “production subnets must not have direct internet routes,” or “deployments require approved regions.” Sentinel is another policy-as-code approach used in some Terraform-oriented environments. Policy checks should be specific, explainable, and linked to an owner. A policy that simply rejects a plan without identifying the violated principle creates frustration and encourages workarounds. Exceptions should be explicit, time-limited, documented, and auditable. Policy-as-code is not a replacement for professional judgment; it is a mechanism for making repeatable organizational decisions visible and enforceable. Static analysis has limitations. It may not understand runtime behavior, organization-specific context, or the effect of data returned by external systems. A rule that blocks every public load balancer may be inappropriate for a public website. Policies therefore require testing themselves, including positive cases, negative cases, and exception cases. Unit, Contract and Module TestingA Terraform module can be treated as a unit when its inputs, outputs, resources, and observable behavior are defined clearly. Unit-level testing may inspect generated plans, validate variable constraints, test conditional resources, and confirm output values. Terraform’s native testing framework, invoked through `terraform test`, supports test files that exercise module configurations and can use assertions against planned or applied results. Depending on the test design, tests may use mocked providers or real provider operations; the latter provide stronger environmental evidence but incur cost and side effects. Unit tests should focus on decisions made inside the module. For example, a network module may test whether enabling a private endpoint creates the expected endpoint resources and whether an invalid CIDR is rejected. A test should not merely repeat the implementation in a different syntax, because such a test can reproduce the same defect. Contract tests verify interfaces. A module contract may state that a subnet module exports subnet identifiers, that a database module requires a private subnet set, or that an Ansible role expects a variable with a particular type and default. Contract testing is especially important when separate teams own modules and consumers. It detects breaking changes before integration environments are affected. Ansible role testing commonly uses Molecule. Molecule can create a test instance or container, converge a role, verify the resulting state, and destroy the test instance. Verification may use Ansible assertions, Testinfra, pytest, or platform-specific checks. A strong role test checks both positive behavior and idempotence: after convergence, running the role again should produce no unexpected changes. Integration and End-to-End TestingIntegration testing applies infrastructure to actual provider APIs or realistic operating systems. It can verify that security groups, identity bindings, routing, managed services, and configuration agents work together. Tools such as Terratest, written in Go, are frequently used to deploy Terraform configurations, inspect cloud resources, execute commands, and destroy the environment. Kitchen-Terraform and other harnesses can provide similar workflows. Integration tests should define lifecycle boundaries. Setup must be repeatable, test data must be isolated, and teardown must run even when assertions fail. State files, credentials, temporary DNS records, and cloud resources require explicit ownership and cleanup. Tags, labels, and cost budgets help identify abandoned resources. End-to-end tests evaluate a complete user or operational journey. An example might provision a load balancer, application instances, a database, monitoring, and identity permissions; then verify that a request reaches the application, data persists, alerts are produced, and the environment can be destroyed. End-to-end tests are expensive and can be unstable because they depend on many systems. They should therefore be limited to critical workflows and supported by diagnostic logging and correlation identifiers. A deployment smoke test is a narrow end-to-end test intended to provide rapid confidence after release. It should test essential availability and basic functionality, not every possible behavior. Smoke testing complements, rather than replaces, deeper integration tests. Terraform Testing in a Delivery PipelineA mature Terraform pipeline commonly follows this sequence: 1. Check formatting and repository conventions. 2. Initialize with controlled provider and module versions. 3. Validate configuration. 4. Run linting and static analysis. 5. Generate a plan using the intended workspace, variables, and backend controls. 6. Evaluate policy against the plan or its normalized representation. 7. Run module and contract tests. 8. Apply integration tests in an isolated account or project when appropriate. 9. Review the plan through human approval for sensitive environments. 10. Apply, execute smoke tests, and record evidence. 11. Destroy ephemeral resources and publish test artifacts. The plan is a particularly important test artifact. It communicates proposed additions, changes, and destructions, but it is not a complete guarantee of runtime success. Unknown values, provider behavior, permissions, eventual consistency, and external dependencies may only become visible during application. A quality process treats plan review as necessary but not sufficient. Terraform state also affects testing. Shared state can cause tests to interfere with one another, while isolated state improves repeatability. Remote state access must be protected because it may contain sensitive values or resource identifiers. Test pipelines should use least-privilege credentials and avoid placing secrets in plans, logs, or assertion output. Ansible Testing and IdempotenceAnsible’s declarative modules are designed to converge systems toward a desired state. Idempotence means that applying the same playbook to an already compliant host produces no unnecessary changes. It is a central reliability property because non-idempotent automation creates noise, extends deployment time, and may cause service disruption. Ansible testing should examine syntax, lint quality, role behavior, idempotence, and failure handling. Molecule provides a lifecycle for dependency installation, instance creation, convergence, verification, and destruction. Testinfra or pytest can inspect packages, services, ports, files, permissions, users, and processes from an external perspective. Ansible’s check mode can provide useful previews, although not all modules support it perfectly. A role should be tested on the operating systems and versions it claims to support. Differences in package managers, service names, filesystem paths, and security defaults can invalidate assumptions. Where containers do not reproduce systemd, networking, or kernel behavior adequately, virtual machines or dedicated test hosts are more appropriate. Failure-path tests are often neglected. A reliable role should behave sensibly when a package repository is unavailable, a required variable is missing, a service fails to start, or a configuration template is invalid. Handlers, blocks, rescue sections, retries, and explicit assertions should be tested rather than assumed to work. Ephemeral Environments and Test DataAn ephemeral environment is created for a test or short-lived validation period and then destroyed. It may be a temporary cloud account, project, namespace, virtual network, virtual machine set, or container environment. Ephemeral infrastructure improves isolation and reduces contamination from previous experiments. It also makes destructive testing more acceptable. Effective ephemeral environments require automation for the entire lifecycle: creation, configuration, testing, collection of evidence, and destruction. Naming conventions and expiration tags support governance. Budget limits, quotas, and automatic cleanup jobs reduce financial risk. Credentials should be scoped to the test environment and revoked or rotated after use. Ephemeral environments are not automatically representative. A small test account may have different quotas, network rules, identities, service limits, or data volumes from production. Teams should document which properties are faithfully reproduced and which are approximated. Production-like testing should avoid copying sensitive production data unless it has been anonymized and approved. Test data must be deterministic where possible. Random identifiers can prevent collisions but should be recorded for diagnostics. Data generators should create realistic relationships without exposing personal or confidential information. Destruction tests should verify that temporary data and dependent resources are removed, not merely that the primary resource disappears. Reliability Engineering: Failure, Recovery and DriftReliability testing asks how infrastructure behaves under adverse conditions. Relevant scenarios include a failed instance, unavailable dependency, expired credential, partial deployment, interrupted pipeline, corrupted configuration, and loss of a region or availability zone. The goal is not to create disorder for its own sake, but to validate documented assumptions about resilience and recovery. Recovery objectives provide measurable targets. The recovery time objective specifies how quickly service should be restored; the recovery point objective specifies the acceptable amount of data loss. IaC supports recovery by making environments reproducible, but reproducibility depends on external state, backups, secrets, provider availability, and correct version pinning. A script that can recreate servers but cannot restore data is not a complete recovery strategy. Drift occurs when actual infrastructure differs from the declared configuration. It can arise from emergency console changes, provider defaults, manual maintenance, or failed automation. Scheduled plan checks, configuration management scans, and cloud inventory comparisons can identify drift. Teams must decide whether to reconcile the environment, update the code, or formally accept the deviation. Automatic correction may be dangerous if the code is stale or if the manual change was an emergency safeguard. Reliability evidence should include successful recovery exercises, not only design documents. A restore test, failover test, or rebuild from an empty account can reveal missing permissions, undocumented dependencies, and assumptions about ordering. Applied Case Study: Testing a Multi-Environment PlatformBackgroundThis illustrative case concerns Northstar Retail, an organization operating an online commerce platform. Its infrastructure was managed through Terraform modules for networking, Kubernetes, databases, identity, and observability. Ansible roles configured administrative hosts and selected supporting services. Development deployments were frequent, but production changes were associated with occasional outages and unexplained manual corrections. Situation and ProblemThe organization’s existing pipeline performed `terraform plan` and required an approval before apply. It did not enforce formatting, security policies, module contracts, or Ansible idempotence. A networking change once removed an expected route, while an Ansible role repeatedly restarted a service on every run. The incidents were not caused by one defective tool; they resulted from insufficient evidence at several levels. AnalysisThe platform team mapped failure modes to test layers. Syntax and style problems belonged in fast pre-commit checks. Security and regional restrictions belonged in policy tests. Module interface assumptions required contract tests. Provider interactions required integration tests. Customer-facing availability required end-to-end and smoke tests. Recovery claims required deliberate rebuild and restore exercises. The team also identified governance risks. Shared test accounts made cleanup unreliable, credentials were broader than necessary, and plans sometimes exposed sensitive values in pipeline logs. Therefore, testing itself required security and lifecycle controls. Approach and SolutionThe team introduced a staged pipeline. Terraform formatting, validation, TFLint, and static security analysis ran on every change. A policy engine rejected unencrypted databases, unrestricted administrative ingress, and unapproved regions. Module tests asserted network outputs, database encryption settings, and variable validation. Contract tests ran whenever a shared module version changed. Molecule scenarios tested Ansible roles on supported operating-system images. Verification used Testinfra to inspect services and permissions, and a second convergence confirmed idempotence. Integration tests used Terratest to create an isolated cloud environment, deploy the platform, verify private connectivity and database access, and destroy all resources. A small end-to-end test sent authenticated requests through the load balancer and checked application health. A scheduled recovery exercise rebuilt a non-production environment from version-controlled code and restored a recent database backup. Test accounts used expiring credentials, resource tags, quotas, and automated cleanup. OutcomeThe new process detected invalid routes, missing encryption settings, unsupported operating-system assumptions, and non-idempotent tasks before production deployment. Pipeline duration increased for integration stages, so the team separated rapid pull-request checks from scheduled or release-level tests. The organization accepted that not every test belonged on every commit, but required traceable evidence before high-risk changes. Learning PointsThe case demonstrates that reliability is produced by a portfolio of controls rather than a single framework. Fast tests reduce feedback time; realistic tests reveal provider and system behavior; policy tests protect organizational boundaries; recovery tests validate operational claims. It also shows that testing strategy is a management decision involving cost, risk, speed, and evidence quality. Challenges, Limitations and Common MistakesA common mistake is testing only whether a command exits successfully. Successful execution may conceal incorrect values, insecure exposure, missing dependencies, or a service that is technically running but unusable. Assertions should inspect meaningful outcomes. Another mistake is overusing mocks. Mocks provide speed and isolation, but they can hide provider behavior, permission requirements, or eventual consistency. They should be balanced with a smaller number of real integration tests. Flaky tests are especially harmful in infrastructure pipelines. They may result from eventual consistency, shared environments, timing assumptions, rate limits, or incomplete cleanup. Retries should be bounded and accompanied by diagnosis; indiscriminate retries can conceal real defects. Teams also frequently test only the happy path. Invalid inputs, denied permissions, partial failure, rollback, interruption, and destruction deserve deliberate attention. Conversely, exhaustive end-to-end testing is often economically impractical. Risk-based prioritization is more effective than attempting to test every possible provider combination. Version drift is another challenge. Provider releases, operating-system images, Ansible collections, Terraform versions, and cloud APIs change over time. Pinning versions improves reproducibility, while controlled upgrade tests prevent indefinite stagnation. Test dependencies should be updated deliberately rather than silently. Governance, Ethics and Professional ResponsibilityInfrastructure testing operates with powerful privileges. A test may create public endpoints, access data stores, alter identity permissions, or destroy resources. Least privilege, environment separation, approval controls, secret management, and audit logging are therefore professional obligations. Security tests must avoid exposing real personal data. If production-like data is needed, it should be minimized, anonymized, or synthetically generated. Destructive tests require explicit authorization and clearly bounded targets. Teams should define ownership for test environments and incident response procedures for failed automation. Quality gates should be transparent. A policy should not become an unexplained barrier that disproportionately affects certain teams or encourages bypasses. Exceptions must be governed, and test results should be retained sufficiently to support audit, incident analysis, and learning. The purpose of governance is not merely to prevent change; it is to make responsible change possible. Emerging Practices and Best PracticesModern pipelines increasingly treat plans, test reports, policy results, logs, and resource inventories as machine-readable evidence. This supports trend analysis, quality dashboards, and automated release decisions. Supply-chain security is also becoming more important: teams verify provider and collection sources, pin dependencies, scan modules, sign artifacts where appropriate, and review changes to reusable components. Policy engines are being integrated earlier into developer workflows, while ephemeral preview environments allow reviewers to inspect functional changes before merge. Artificial intelligence may assist with test generation, anomaly detection, or plan explanation, but generated tests require expert review. AI systems can reproduce incorrect assumptions, mishandle secrets, or produce assertions that merely confirm implementation details. Effective practice can be summarized as follows: - Test early, but include realistic provider-level tests for important behavior.
- Treat plans as evidence requiring review, not as proof of correctness.
- Test interfaces, idempotence, failure paths, and destruction as well as successful creation.
- Isolate environments, credentials, state, and test data.
- Make policies explainable and exceptions auditable.
- Use risk to determine which tests run on each change.
- Record artifacts, ownership, versions, and cleanup outcomes.
- Test recovery in practice and revise documentation from the results.
Chapter SummaryTesting IaC is a layered quality and reliability discipline. Formatting and validation establish basic correctness; linting and static analysis identify maintainability and security risks; policy as code enforces organizational boundaries. Unit and module tests examine local decisions, contract tests protect interfaces, and integration tests verify behavior against real providers and operating systems. End-to-end and smoke tests establish confidence in complete workflows. Terraform’s native testing capabilities, plan analysis, Terratest, and related tools support infrastructure verification. Ansible Lint, Molecule, Testinfra, pytest, and idempotence checks support role and playbook quality. Ephemeral environments improve isolation and realism when their credentials, costs, data, and destruction are governed properly. Reliability engineering extends testing to drift, failure, recovery, backups, and rebuilds. The strongest strategy is risk-based and evidence-oriented. It balances fast feedback with realistic testing, avoids excessive reliance on mocks, controls flakiness, protects sensitive information, and treats recovery as a capability that must be demonstrated rather than assumed. Key TermsInfrastructure as Code (IaC): Management of infrastructure through version-controlled, machine-readable definitions. Quality gate: An automated or human-controlled condition that must be satisfied before progression in a delivery process. Idempotence: The property that repeated execution produces no unnecessary changes after the desired state is reached. Static analysis: Examination of code or configuration without executing the complete infrastructure workflow. Policy as code: Encoding organizational, security, compliance, or architectural rules in executable form. Terraform plan: A representation of proposed infrastructure changes before application. Module contract: The documented and testable interface of a reusable infrastructure component. Contract test: A test that verifies assumptions between separately developed components. Integration test: A test that evaluates interactions among components or with real external services. End-to-end test: A test of a complete workflow from an entry point to its expected operational outcome. Ephemeral environment: A temporary environment created for testing and removed afterward. Drift: A difference between declared infrastructure configuration and actual deployed state. Smoke test: A focused test of essential functionality after deployment or change. Recovery time objective (RTO): The target time within which service should be restored. Recovery point objective (RPO): The maximum acceptable amount of data loss measured in time. Flaky test: A test whose result is inconsistent despite no relevant code change. Molecule: A framework commonly used to develop and test Ansible roles. Terratest: A Go-based testing framework frequently used to deploy and verify infrastructure. Test artifact: A retained output, report, plan, log, or result that provides evidence of testing. Review QuestionsShort Answer Questions1. Why is a successful Terraform apply insufficient evidence of infrastructure quality? 2. What is the difference between validation and linting in IaC? 3. Define idempotence and explain its importance in Ansible. 4. What purpose does a contract test serve in a Terraform module ecosystem? 5. Why should ephemeral environments use automatic cleanup and expiration controls? Long Answer Questions1. Explain the infrastructure testing pyramid and discuss the trade-off between speed, realism, and cost at different levels. 2. Describe a complete Terraform quality pipeline from formatting through post-deployment smoke testing. 3. Discuss how Molecule, Testinfra, and Ansible Lint can be combined to test an Ansible role. 4. Explain how policy as code contributes to security and governance, including the management of exceptions. 5. Analyze why recovery testing is a necessary part of reliability engineering for IaC-managed systems. Analytical / Application Questions1. A Terraform plan creates an encrypted database but also permits unrestricted administrative access from the internet. Identify which test layers should detect the problem and explain the appropriate quality gates. 2. An Ansible role reports changed status on every execution and occasionally restarts a production service. Design a testing and remediation approach. 3. Design an ephemeral integration environment for a multi-tier application. Specify isolation, credentials, test evidence, failure cleanup, and the limitations of its representativeness. 22. Multiple Choice Questions with AnswersQuestion 1Which test layer is most appropriate for detecting that a Terraform module has changed the name and type of an output consumed by another team? A. Formatting test B. Contract test C. End-to-end smoke test D. Recovery test Correct Answer: B. Contract test Question 2A Terraform configuration is syntactically valid but creates an unencrypted storage resource. Which control is most directly designed to reject it before deployment? A. Policy as code B. YAML formatting C. Ansible convergence D. DNS smoke testing Correct Answer: A. Policy as code Question 3What does a second successful convergence of an Ansible role primarily demonstrate? A. Provider version compatibility B. Idempotence C. Disaster recovery D. Terraform state locking Correct Answer: B. Idempotence Question 4Which statement best describes an ephemeral test environment? A. A permanent production replica used for all releases B. A temporary isolated environment created and destroyed through automation C. A local text editor configuration D. A provider policy that prevents resource deletion Correct Answer: B. A temporary isolated environment created and destroyed through automation Question 5Why should integration tests supplement Terraform mocks? A. Mocks always create real cloud resources B. Integration tests reveal provider, permission, and runtime behavior that mocks may hide C. Integration tests eliminate the need for policy checks D. Mocks cannot test variable validation Correct Answer: B. Integration tests reveal provider, permission, and runtime behavior that mocks may hide Question 6A pipeline retries a failing infrastructure test indefinitely until it passes. What is the principal quality risk? A. The test becomes too deterministic B. Retries may conceal genuine defects and increase resource cost C. The infrastructure becomes automatically idempotent D. The plan becomes easier to review Correct Answer: B. Retries may conceal genuine defects and increase resource cost Question 7Which evidence most directly validates a documented recovery claim? A. A successful formatting check B. A code review approval C. A practical rebuild and data-restore exercise D. A lint warning count Correct Answer: C. A practical rebuild and data-restore exercise Question 8What is the primary purpose of analyzing a Terraform plan in a delivery pipeline? A. To guarantee that provider APIs cannot fail B. To inspect intended additions, changes, and destructions before application C. To replace all runtime testing D. To remove the need for state management Correct Answer: B. To inspect intended additions, changes, and destructions before application Question 9Which practice best protects sensitive information during infrastructure testing? A. Copying production data into every test account B. Printing all variables in pipeline logs C. Using least-privilege credentials and sanitized or synthetic test data D. Disabling audit logging Correct Answer: C. Using least-privilege credentials and sanitized or synthetic test data Question 10An organization wants rapid pull-request feedback but also needs realistic cloud validation. Which strategy is most appropriate? A. Run only end-to-end tests on every commit B. Use fast validation and policy checks for every change, with risk-based integration tests at suitable stages C. Remove all automated tests and rely on approval D. Use only mocked tests for production changes Correct Answer: B. Use fast validation and policy checks for every change, with risk-based integration tests at suitable stages
|