DevOps, Infrastructure as Code and Delivery Automation

Introduction

Serverless and microservices architectures change the operational responsibilities of a software team. A cloud provider manages servers, operating-system patching and much of the runtime infrastructure, but the application team remains responsible for source code, configuration, permissions, event definitions, API contracts, observability and deployment safety. The result is not “no operations”; it is a shift from server administration to automated systems engineering.

DevOps is the organisational and technical practice of integrating development, security and operations so that software can be delivered rapidly, reliably and with appropriate controls. Infrastructure as Code (IaC) expresses infrastructure in machine-readable definitions that can be versioned, reviewed, tested and reproduced. Continuous integration and continuous delivery or deployment (CI/CD) automate the path from a code change to a validated release.

For serverless microservices, automation is especially important because a system may contain many independently deployable functions, queues, topics, API routes, databases, policies and environment-specific settings. Manual configuration creates configuration drift, inconsistent environments and difficult-to-reproduce failures.

Learning Objectives

By the end of this chapter, a learner should be able to:

• explain the relationship between DevOps, CI/CD, IaC and policy as code;
• design a pipeline for serverless microservices;
• distinguish automated testing levels, including contract testing;
• use local emulation appropriately and understand its limits;
• compare deployment strategies such as rolling, blue-green, canary and feature flags;
• design rollback and environment-promotion mechanisms;
• identify security, governance and operational risks in delivery automation; and
• evaluate a deployment process using meaningful reliability and delivery metrics.

1. The DevOps Delivery Model

A delivery system is a controlled flow of changes through stages. A typical flow is:

Commit → Build → Unit tests → Static analysis → Package
       → Infrastructure validation → Integration tests
       → Contract tests → Deploy to development
       → Acceptance tests → Promote → Observe → Release

Continuous integration means that changes are merged frequently into a shared repository and automatically built and tested. Continuous delivery means that every change which passes the required controls is kept in a releasable state. Continuous deployment goes further: an approved change is automatically released to production without a separate manual deployment action.

These practices are related but not identical. A team may practise continuous integration without automatically deploying to production. Similarly, a pipeline can deploy frequently while still lacking meaningful tests or effective rollback. Automation is valuable only when it provides repeatability, evidence and safe decision-making.

DevOps principles for serverless microservices

1. Small, independently deployable units: A service should have a clear responsibility and a bounded interface.
2. Version everything: Source code, IaC, policy, dependency definitions, API specifications and pipeline configuration should be version controlled.
3. Automate repeatable decisions: Builds, tests, validation and deployment should not depend on an individual’s workstation.
4. Make changes observable: Each deployment should be associated with a commit, artifact, environment and outcome.
5. Prefer reversible changes: Releases should support rapid rollback or forward correction.
6. Shift controls left: Security, dependency, quality and compliance checks should run before production deployment.
7. Design for failure: A pipeline should handle failed tests, partial deployment, unavailable services and invalid configuration predictably.

A useful distinction is between a pipeline and a release process. The pipeline is the automated implementation. The release process includes human approvals, risk classification, change records, incident procedures and business communication.

2. Infrastructure as Code

Infrastructure as Code represents resources such as functions, API gateways, queues, databases, roles, networks, alarms and dashboards as declarative or programmatic definitions. A declarative definition states the desired end state; an IaC engine determines the operations necessary to reach that state. An imperative script instead specifies a sequence of commands.

Desired state and reconciliation

Suppose a service requires one function, one queue and a permission allowing the function to consume messages. The IaC definition describes these resources and their relationship. During deployment, the provisioning engine compares the declared configuration with the current state and creates, updates or removes resources as necessary.

The key benefits are:

• Reproducibility: equivalent definitions can create equivalent environments;
• Auditability: changes are reviewed as code and linked to commits;
• Consistency: development, test and production can use a common structure;
• Speed: a complete environment can be created without manual steps; and
• Recovery: infrastructure can be reconstructed after accidental deletion or regional failure, subject to data-recovery arrangements.

IaC does not automatically guarantee identical environments. Differences may arise from provider-managed defaults, region-specific services, quotas, secrets, data and out-of-band changes. IaC must therefore be combined with drift detection, version pinning and environment validation.

Modules, parameters and state

Reusable modules encapsulate common patterns such as an HTTP function with logging, alarms and least-privilege permissions. Modules should expose a small, documented interface and avoid hiding important security or cost consequences.

Environment-specific values should be supplied through controlled parameters or secret-management services, not embedded in source code. Examples include database endpoints, feature settings and external credentials. Secrets should be encrypted in storage and transit, access-controlled and excluded from logs.

Many IaC tools maintain state describing resource identities and relationships. State is operationally sensitive: it should be stored in an encrypted, access-controlled backend with locking or concurrency control. A state file should not be edited casually. If two pipeline executions modify the same state simultaneously, the result may be inconsistent or destructive.

Idempotence and drift

An operation is idempotent if applying it repeatedly produces the same intended result after the first successful application. IaC tools aim for idempotent convergence, although provider APIs and custom scripts may violate this property. A custom provisioning action should be designed so that retries do not create duplicate resources or corrupt data.

Drift is the difference between declared configuration and actual deployed configuration. It may be caused by console edits, emergency changes, provider defaults or failed deployments. Teams should either prohibit out-of-band changes or document an emergency process in which the change is rapidly incorporated into IaC and reviewed.

3. Policy as Code and Secure Delivery

Policy as Code expresses governance rules in a form that can be automatically evaluated. Examples include:

• a storage resource must not be publicly readable;
• functions must use an approved runtime;
• encryption must be enabled for queues and databases;
• production deployments require an approved source branch;
• IAM permissions must not use unrestricted administrative actions; and
• resources must include owner, cost-centre and data-classification tags.

Policies can be evaluated against IaC before deployment, against cloud resources after deployment, or against pipeline events. Preventive controls block unsafe changes; detective controls report violations and support remediation. A mature system uses both.

Security controls should follow the principle of least privilege. A build job should receive only the permissions needed to build and deploy its target. Production credentials should not be available to pull-request jobs from untrusted forks. Short-lived federated credentials are generally preferable to long-lived static keys.

A secure pipeline should include dependency scanning, secret detection, software composition analysis, static application security testing and, where appropriate, dynamic testing. These tools produce findings rather than certainty. Teams must define severity thresholds, exception expiry dates and ownership for remediation.

4. Designing a CI/CD Pipeline

A pipeline should be divided into stages with explicit inputs, outputs and failure conditions. A practical serverless pipeline may contain the following stages.

4.1 Source and change validation

A pull request triggers formatting checks, linting, type checking, secret detection and unit tests. Branch protection requires successful checks and review before merging. Commit messages or pull-request metadata may identify the service, change type and risk level.

4.2 Build and dependency resolution

The build stage installs dependencies using a lock file, compiles code where necessary and creates an immutable artifact. An artifact may be a deployment bundle, container image, function package or infrastructure template. The artifact should be identified by a content digest or commit identifier rather than a mutable label such as “latest”.

4.3 Infrastructure validation

The pipeline performs syntax checks, template validation, policy checks and a preview of proposed changes. Destructive operations, such as replacement of a production database, should require an explicit approval or be prohibited by policy.

4.4 Automated tests

Unit, component, integration, contract and end-to-end tests should be selected according to risk. Fast tests run early; expensive or environment-dependent tests run later. Test results and coverage reports should be retained as evidence, but coverage percentage alone is not a quality guarantee.

4.5 Deployment and verification

The pipeline deploys the immutable artifact to an isolated environment, runs smoke and acceptance tests, and checks service health. Verification should include logs, metrics, traces, queue depth, error rates and functional probes. A deployment is not successful merely because the provider accepted the template.

4.6 Promotion

Promotion should move the same tested artifact between environments whenever possible. Rebuilding separately for each environment can produce a different dependency graph and undermines confidence. Configuration should vary through controlled parameters, while executable content remains identical.

Example pipeline policy

A production promotion may require:

• all mandatory tests passing;
• no unapproved critical security findings;
• an IaC change preview with no prohibited destructive actions;
• successful deployment to a production-like environment;
• an approved change record for high-risk changes; and
• a defined rollback or forward-fix plan.

5. Testing Serverless Microservices

Testing must consider both code and cloud interactions. A function can be correct in isolation but fail because of an incorrect event schema, permission, timeout, retry policy or downstream contract.

Contract testing

A contract is an explicit agreement between a service and its consumers. For an HTTP interface, it may specify method, path, headers, status codes and JSON structure. For an event interface, it may specify event type, version, required fields, data types and delivery semantics.

Consumer-driven contract testing allows a consumer to state the interactions it requires. The provider verifies that its implementation satisfies those expectations before release. This detects breaking changes earlier than a complete end-to-end test suite.

A contract test should distinguish required and optional fields. Adding an optional response field is often backward compatible, while removing or changing the type of a required field is usually breaking. Compatibility also depends on behaviour: status-code meanings, idempotency, ordering, retry handling and error formats may be part of the contract.

Contracts should be versioned and published in a registry or repository. A provider should not claim compatibility merely because a schema validates; semantic behaviour must also be tested.

Property-based and resilience testing

Property-based testing generates many inputs to verify general properties, such as “a valid request never produces a negative quantity” or “repeating an idempotent command does not create a second order.” Resilience testing deliberately introduces timeouts, duplicate events, delayed messages and dependency failures. These tests are particularly relevant because serverless platforms commonly retry asynchronous invocations.

6. Local Emulation and Test Environments

Local emulation allows developers to run a close approximation of functions, API routes, event payloads and selected cloud services on a workstation. It reduces feedback time and supports offline development. Container-based emulators and provider-specific local runtimes are common approaches.

Local emulation is useful for:

• syntax and handler validation;
• rapid unit and component testing;
• experimenting with event payloads;
• testing basic routing and serialization; and
• reproducing deterministic defects.

It cannot perfectly reproduce provider-managed behaviour such as IAM evaluation, regional service limits, network latency, managed retries, eventual consistency, throttling and proprietary event delivery. Therefore, local success is evidence of limited correctness, not proof of cloud correctness.

A sound test strategy uses a pyramid: many local unit tests, a substantial set of component and contract tests, fewer cloud integration tests, and a small number of end-to-end journeys. Test environments should be isolated, automatically provisioned and destroyed or reset safely. Test data must not contain real personal or confidential information unless specifically authorised and protected.

7. Deployment Strategies

Deployment strategy determines how a new version receives traffic and how the team detects failure.

Rolling deployment

Instances or versions are replaced gradually. It can be efficient, but old and new versions coexist temporarily. In serverless systems, the equivalent may be gradual replacement of aliases or routes. Compatibility with shared data and events is essential.

Blue-green deployment

Two production-like environments exist: blue serves current traffic and green contains the candidate. Traffic switches after validation. Blue-green deployment enables rapid reversal by switching back, but it may temporarily double infrastructure cost and does not automatically undo database changes.

Canary deployment

A small percentage of traffic is sent to the candidate before wider release. The team compares error rate, latency, business outcomes and resource consumption against the baseline. Canary analysis must use adequate traffic and a meaningful observation window; a small sample can conceal rare failures.

A simple error-rate measure is:

The formula assumes that requests are counted consistently and that failures are clearly defined. It is not sufficient by itself: latency percentiles, severity, traffic volume and business impact should also be considered.

Feature flags

Feature flags separate code deployment from feature activation. They support gradual exposure, user targeting and rapid deactivation. Flags create operational complexity and should have owners, expiry dates, secure evaluation rules and tests for both states. A flag is not a substitute for safe deployment; inactive code may still affect packaging, permissions or startup behaviour.

Strategy comparison

8. Rollback, Recovery and Database Change

Rollback means returning service behaviour to a previously known version. It is straightforward for immutable function code but more complex when a release changes schemas, events or persistent data.

A robust rollback plan answers four questions:

1. How is the previous artifact identified?
2. How is traffic redirected or the previous version activated?
3. How are incompatible data changes handled?
4. How are side effects, messages and external calls reconciled?

Database migrations should normally be expand and contract. First, add the new schema elements while retaining compatibility with the old version. Next, deploy code that can use the expanded schema. After all old consumers are removed, contract the schema by deleting obsolete elements. This avoids requiring an instantaneous, system-wide switch.

Some changes should not be automatically rolled back. For example, an order submitted to an external payment provider cannot necessarily be undone by redeploying code. Operational runbooks should include idempotency keys, compensating actions, reconciliation jobs and human escalation.

Rollback triggers should be explicit. Examples include a sustained increase in server errors, a statistically significant latency regression, failed health checks, elevated duplicate processing or a critical security alert. Automatic rollback should be bounded and observable so that repeated oscillation does not occur.

9. Environment Promotion and Configuration Management

A typical promotion path is:

Feature branch → Pull request → Development → Test/Integration
                 → Staging/Production-like → Production

Environments should differ intentionally, not accidentally. Development may use emulators and synthetic data; staging should resemble production in architecture, permissions and scale characteristics; production contains real workloads and stricter controls.

The same artifact should be promoted through environments. The pipeline records artifact digest, source commit, IaC version, configuration version, test results and approvals. This creates provenance: the organisation can answer what is running, who approved it and which evidence supported deployment.

Configuration should be separated into:

• non-sensitive configuration, such as timeouts and feature settings;
• secrets, such as credentials and signing keys; and
• environment identity, such as account, region and resource names.

Configuration validation should occur before deployment. Missing values, invalid ranges and incompatible combinations should fail early rather than causing runtime errors.

10. Observability and Delivery Metrics

Automation must be evaluated by outcomes, not only pipeline duration. Important delivery metrics include:

• Deployment frequency: how often successful releases reach production;
• Lead time for changes: elapsed time from a committed change to production;
• Change failure rate: proportion of deployments causing rollback, incident or urgent remediation;
• Mean time to restore: average time to restore service after an incident; and
• Pipeline success and queue time: reliability and waiting time of the delivery system.

These measures should be interpreted together. Increasing deployment frequency while change failure rate rises is not an improvement. Teams should avoid using metrics to punish individuals; the purpose is to identify system constraints and improve flow.

For serverless systems, operational observability should include structured logs, correlation identifiers, distributed traces, invocation duration, throttling, cold-start indicators where relevant, queue age, retry counts, dead-letter volume and cost anomalies. Personally identifiable information should not be written to logs unnecessarily.

11. Applied Case: Illustrative Online Claims Service

Background

An insurance organisation operates a serverless claims platform. An API receives claim submissions, a validation function stores metadata, an event is published for document processing and a notification service informs customers. Three teams independently modify services, but deployments are manual and shared test environments are frequently inconsistent.

Problem

A release that added a required field to the claim event caused the document processor to fail. The API team had tested its service successfully, but no consumer contract test existed. Operators discovered the problem only after production retries filled a dead-letter queue. The rollback restored the old function code but did not remove already-published incompatible events.

Analysis

The failure involved several weaknesses:

• an undocumented event contract;
• incompatible schema evolution;
• no automated integration or contract gate;
• manual environment configuration;
• insufficient monitoring of dead-letter queues; and
• a rollback plan that considered code but not messages already in flight.

Approach

The organisation introduced a versioned event schema. The old consumer accepted version 1 while a new consumer was developed for version 2. An expand-and-contract migration allowed both versions during transition. Consumer-driven contract tests ran on every pull request, and a contract registry recorded compatible provider versions.

The teams created IaC modules for queues, permissions, alarms and dead-letter handling. Each pull request generated an infrastructure preview and policy scan. A pipeline built one immutable artifact, deployed it to development and staging, executed cloud integration tests, and then performed a 10 percent canary release.

Canary verification examined API error rate, document-processing success, queue age and customer-notification latency. A failed threshold automatically stopped promotion and restored the previous alias. A reconciliation job safely reprocessed valid failed messages using idempotency keys.

Outcome

The organisation reduced manual deployment steps, detected incompatible event changes before production and gained an auditable promotion history. The principal improvement was not a particular cloud product; it was the combination of explicit contracts, reproducible infrastructure, observable deployment and recovery designed around real side effects.

Learning points

1. Interface compatibility includes event behaviour, not only JSON syntax.
2. Code rollback cannot reverse every external side effect.
3. IaC and policy checks are most effective when integrated into normal review.
4. Canary deployment requires domain-specific health indicators.
5. Dead-letter queues require monitoring, ownership and a recovery procedure.

12. Challenges, Limitations and Governance

Automation can amplify mistakes. A faulty IaC module may reproduce an insecure configuration across every environment. An overly broad deployment role can turn a compromised pipeline into an organisation-wide incident. Excessive end-to-end testing can slow delivery and encourage teams to bypass the pipeline.

Important governance controls include separation of duties for high-risk changes, protected production environments, approval evidence, retention of deployment records, access reviews and disaster-recovery exercises. Regulatory obligations may require data residency, auditability, retention limits, encryption and demonstrable change control. The exact requirements depend on jurisdiction and sector.

AI-assisted coding and test generation can increase productivity, but generated code and policies require human review. AI tools may reproduce insecure patterns, expose confidential source material or create tests that assert implementation details rather than meaningful behaviour. Organisations should define approved tools, data-handling rules, provenance requirements and review responsibilities.

Best-Practice Checklist

• Store application, IaC, policy and pipeline definitions in version control.
• Build immutable, traceable artifacts and promote the same artifact.
• Use least-privilege, short-lived deployment credentials.
• Validate IaC and policy before applying changes.
• Keep unit tests fast and add contract tests for service boundaries.
• Test cloud-specific permissions, retries and event behaviour in real or representative environments.
• Use canary or blue-green release for high-risk services.
• Design rollback around data, messages and external side effects.
• Monitor both technical and business indicators.
• Detect drift and prohibit undocumented console changes.
• Give every feature flag and exception an owner and expiry date.
• Practise restoration and rollback rather than relying on written plans alone.

Chapter Summary

DevOps for serverless microservices combines collaborative practice, automated controls and operational feedback. CI/CD pipelines provide repeatable movement from source change to validated release. Infrastructure as Code makes cloud resources reviewable, reproducible and recoverable, while policy as code enforces security, compliance and architectural rules.

Effective testing uses multiple levels. Unit and component tests provide rapid feedback; integration tests validate cloud interactions; contract tests protect service boundaries; and end-to-end tests confirm critical journeys. Local emulation accelerates development but cannot reproduce every provider-managed behaviour.

Deployment strategies determine exposure and recovery. Rolling deployment is efficient, blue-green deployment supports rapid traffic reversal, canary deployment provides evidence from limited exposure and feature flags separate activation from deployment. Rollback must account for persistent data, asynchronous messages and irreversible external side effects. Safe environment promotion uses immutable artifacts, controlled configuration, observable evidence and explicit approvals.

Finally, delivery automation must be secure and measurable. Least privilege, secret management, policy checks, provenance, drift detection and meaningful reliability metrics are essential. The goal is not automation for its own sake, but a delivery system that enables frequent, safe and explainable change.

Key Terms

DevOps: A practice integrating development, security and operations to improve software delivery and reliability.

Continuous integration: Frequent integration of changes with automated build and test activities.

Continuous delivery: Keeping validated software in a releasable state.

Continuous deployment: Automatically releasing validated changes to production.

Infrastructure as Code: Machine-readable management of infrastructure configuration and lifecycle.

Declarative configuration: A description of the desired end state rather than a command sequence.

Idempotence: The property that repeating an operation produces the same intended result.

Infrastructure drift: Difference between declared infrastructure and actual deployed infrastructure.

Policy as Code: Machine-evaluated rules governing systems, resources or delivery actions.

Immutable artifact: A versioned build output that is not modified after creation.

Contract testing: Testing whether a provider and consumer satisfy an agreed interface contract.

Local emulation: Running a local approximation of cloud services or runtimes.

Canary deployment: Releasing a new version to a limited proportion of traffic first.

Blue-green deployment: Maintaining old and new environments and switching traffic between them.

Feature flag: A controlled runtime switch that enables or disables behaviour.

Rollback: Returning service execution to a previous known version or configuration.

Expand and contract: A migration pattern that preserves compatibility while schemas evolve.

Deployment provenance: Evidence linking a running release to its source, artifact, configuration and approvals.

Change failure rate: The proportion of deployments that cause failure, rollback or urgent remediation.

Dead-letter queue: A destination for messages that cannot be processed successfully after defined attempts.

Review Questions

Short Answer Questions

1. Distinguish continuous delivery from continuous deployment.
2. What is infrastructure drift, and how can it be detected?
3. Why should deployment artifacts be immutable?
4. State two limitations of local cloud emulation.
5. What is the purpose of a consumer-driven contract test?

Long Answer Questions

1. Explain how Infrastructure as Code, policy as code and CI/CD work together in a serverless microservices organisation.
2. Compare rolling, blue-green and canary deployment strategies, including their principal risks.
3. Describe a testing strategy for a serverless service that publishes versioned events.
4. Explain why database and message changes complicate rollback, and describe the expand-and-contract approach.
5. Design a secure promotion workflow from pull request to production, including approvals, evidence and observability.

Analytical / Application Questions

1. A pipeline passes unit tests, but production shows rising queue age and duplicate processing after release. Analyse possible causes and propose tests, metrics and rollback actions.
2. An IaC preview proposes replacing a production database because a resource identifier changed. What controls should stop or review the change, and what safer alternatives might be considered?
3. A team wants to increase deployment frequency by removing all manual approvals. Evaluate this proposal using risk classification, automated policy, canary deployment, auditability and recovery requirements.

Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes continuous delivery from continuous deployment?

A. Continuous delivery requires no automated tests.

B. Continuous delivery keeps software releasable, while continuous deployment automatically releases validated changes.

C. Continuous deployment applies only to infrastructure.

D. Continuous delivery means deploying only once per year.

Correct Answer: B. Continuous delivery keeps software releasable, while continuous deployment automatically releases validated changes.

Question 2

What is the primary purpose of Infrastructure as Code?

A. To eliminate the need for application testing.

B. To describe and manage infrastructure in versioned, repeatable definitions.

C. To store production passwords in source code.

D. To replace all monitoring systems.

Correct Answer: B. To describe and manage infrastructure in versioned, repeatable definitions.

Question 3

Which practice most directly reduces the risk of simultaneous IaC state modification?

A. Using mutable artifacts

B. Disabling source control

C. State locking or concurrency control

D. Increasing log retention only

Correct Answer: C. State locking or concurrency control

Question 4

Which test is specifically designed to detect incompatibility between a service provider and its consumer?

A. Unit test

B. Contract test

C. Load-balancing test

D. Formatting test

Correct Answer: B. Contract test

Question 5

A canary deployment sends a new version to:

A. All users permanently

B. No users under any circumstances

C. A limited proportion of traffic before wider release

D. Only the infrastructure team’s source repository

Correct Answer: C. A limited proportion of traffic before wider release

Question 6

Why is local emulation insufficient as the only test environment?

A. It cannot run any application code.

B. It cannot reproduce every provider-specific behaviour such as IAM, throttling and managed retries.

C. It always requires production data.

D. It prevents unit testing.

Correct Answer: B. It cannot reproduce every provider-specific behaviour such as IAM, throttling and managed retries.

Question 7

Which migration pattern supports compatibility while a database schema changes?

A. Delete and rebuild immediately

B. Expand and contract

C. Disable all consumers permanently

D. Deploy the new schema after deleting the old one

Correct Answer: B. Expand and contract

Question 8

If 25 requests fail out of 1,000 total requests, what is the error rate using the chapter’s definition?

A. 0.25%

B. 2.5%

C. 4%

D. 25%

Correct Answer: B. 2.5%

Question 9

Which control is an example of policy as code?

A. A developer’s informal verbal preference

B. A rule that blocks publicly readable storage resources

C. A customer support script

D. A manually edited desktop wallpaper

Correct Answer: B. A rule that blocks publicly readable storage resources

Question 10

Why should the same immutable artifact ideally be promoted between environments?

A. To ensure the tested executable content is not changed between environments.

B. To guarantee that all environments use identical secrets.

C. To avoid recording deployment provenance.

D. To prevent configuration validation.

Correct Answer: A. To ensure the tested executable content is not changed between environments.