CI/CD and Release Engineering

Introduction and context

Cloud deployment is not complete when an application has been written or packaged. A reliable service must be built reproducibly, tested automatically, checked for security weaknesses, stored as a traceable artifact, and released to infrastructure in a controlled way. Continuous integration and continuous delivery/deployment (CI/CD) provide the automation and governance for this process. Release engineering extends CI/CD by treating software releases as engineered production systems rather than as occasional manual events.

In AWS and Google Cloud, a pipeline may use services such as AWS CodePipeline, CodeBuild, CodeDeploy, ECR and S3, or Google Cloud Build, Artifact Registry, Cloud Deploy, Cloud Run and GKE. These managed services are useful, but the underlying principles are provider-independent: versioned source code, immutable artifacts, automated verification, controlled promotion, observable deployment and rapid recovery.

A poorly designed pipeline can accelerate the delivery of defects, expose secrets, or make a failed deployment difficult to reverse. A well-designed pipeline reduces repetitive work while preserving appropriate human judgement at high-risk points.

Learning objectives

By the end of this chapter, a learner should be able to:

1. Explain CI, continuous delivery, continuous deployment and release engineering.
2. Design a Git-based workflow and a pipeline containing build, test, security, artifact and deployment stages.
3. Manage immutable, traceable artifacts in AWS or Google Cloud.
4. Apply approvals, environment promotion and policy controls.
5. Compare rolling, blue-green, canary and feature-flag release strategies.
6. Plan rollback, roll-forward and database migration procedures.
7. Evaluate a pipeline using delivery, reliability, security and operational metrics.

1. The CI/CD model

Continuous integration

Continuous integration (CI) means that developers integrate changes into a shared version-control repository frequently, often through short-lived branches and pull requests. Each change triggers automated activities such as compilation, unit testing, linting and dependency checks. The purpose is not merely to run a build server; it is to detect integration problems while they are still inexpensive to correct.

A CI system should produce the same result from the same source revision whenever practical. Builds should therefore declare dependencies, use pinned or lock-file versions, and avoid depending on a developer's workstation. Build logs should identify the commit, tool versions, inputs and outputs.

Continuous delivery and continuous deployment

Continuous delivery means that every change that passes the pipeline is kept in a releasable state. Deployment to production may require an explicit approval or a scheduled business decision.

Continuous deployment goes further: every change satisfying automated policy is deployed to production without a routine manual approval. Continuous deployment is appropriate only when testing, observability, rollback and organizational controls are sufficiently mature.

These terms are related but not identical:

A pipeline as a chain of controls

A production pipeline should be viewed as a sequence of evidence-producing controls:

1. Source control establishes what changed and who reviewed it.
2. Build converts source into a versioned package or container image.
3. Automated tests provide evidence of functional correctness.
4. Security checks identify vulnerable code, dependencies, images and configurations.
5. Artifact management stores an immutable release candidate.
6. Deployment changes a controlled environment.
7. Verification checks health, functionality, performance and business signals.
8. Promotion or recovery either advances the release or stops and restores service.

A later stage should not silently compensate for a missing earlier control. For example, production monitoring is essential, but it is not a substitute for testing or secure dependency management.

2. Git-based workflows

Git provides the history and identity foundation for modern release engineering. A useful workflow defines how changes move from development to production and which rules protect important branches.

Trunk-based development

In trunk-based development, developers integrate small changes into a main branch frequently. Work that is incomplete can be hidden behind a feature flag. Short-lived branches reduce merge divergence and encourage fast feedback. This approach fits high-frequency delivery, but it requires good automated tests and disciplined review.

Feature-branch and pull-request workflows

A developer creates a branch, makes a focused change, and opens a pull request. Automated checks run before merge, and one or more reviewers assess the change. Branch protection can require successful checks, signed commits, approval from a designated team, and no unresolved review comments.

Long-lived release branches can support regulated or scheduled products, but they create merge overhead and may allow the production branch to diverge from active development. A team should choose a branching model deliberately rather than copy one mechanically.

Commit and tag practices

A release should be traceable to:

• a repository and exact commit identifier;
• the build definition and toolchain version;
• dependency lock files;
• test and security results;
• the resulting artifact digest;
• the deployment event and approving identity.

A Git tag can name a release, but a tag alone is not a sufficient integrity mechanism if it can be moved. Protected tags, signed commits or tags, artifact digests and an auditable deployment system provide stronger evidence.

Pull requests as controls, not ceremonies

A review should examine design, security, failure handling, tests, observability and operational impact. Automated checks should enforce objective rules, while reviewers address questions that tools cannot reliably answer. Self-approval, bypassing required checks and merging untested emergency changes should be exceptional, documented and audited.

3. Designing the pipeline

A practical pipeline separates fast feedback from slower, expensive verification. A typical sequence is:

Commit → Validate → Build → Unit test → Scan → Package
       → Integration test → Publish artifact → Deploy staging
       → Verify → Approval/policy → Deploy production → Observe

Source and validation stage

The pipeline obtains source from a repository using a trusted connection. It validates the commit, checks required files and confirms that the build definition itself is allowed. Secrets must not be embedded in source, Dockerfiles or pipeline logs. Secret values should come from a managed secret service, such as AWS Secrets Manager or AWS Systems Manager Parameter Store, or Google Secret Manager, with least-privilege access.

Build stage

The build should be deterministic enough that source revision r produces an identifiable artifact a. A container build should use a small, maintained base image, a multi-stage Dockerfile where appropriate, and a non-root runtime user. Build caches can improve speed, but cached material must not bypass security or cause stale dependencies to be used unknowingly.

The build environment itself is part of the supply chain. Pinning action versions, using approved builders, restricting network access and recording tool versions reduce risk. In AWS, CodeBuild can execute a declared build specification; in Google Cloud, Cloud Build can execute a declarative build configuration.

Test stages

Tests should be layered:

A pipeline should avoid relying only on end-to-end tests. Such tests are valuable but slower, more brittle and less diagnostic than unit and integration tests. Test data must be isolated and credentials must be temporary and restricted.

Security scanning

Security checks should occur throughout the lifecycle:

• Secret scanning searches source and history for credentials and tokens.
• Software composition analysis examines third-party dependencies and known vulnerabilities.
• Static application security testing (SAST) analyzes source or bytecode for insecure patterns.
• Container scanning checks image layers and operating-system packages.
• Infrastructure-as-code scanning checks Terraform, CloudFormation, Kubernetes manifests or similar files.
• Dynamic application security testing (DAST) tests a running application from the outside.
• Software bill of materials (SBOM) records components included in an artifact.

A vulnerability result should have a documented policy: block, allow temporarily with an expiry, or accept based on risk. A scanner does not prove that a system is secure. False positives, unreachable code, newly disclosed vulnerabilities and configuration-dependent weaknesses require human review.

Artifact management

An artifact is the deployable output of a build: for example, a container image, a package, a serverless bundle or a static website archive. Artifact repositories include Amazon ECR and S3, and Google Artifact Registry and Cloud Storage.

Important practices include:

• publish once and promote the same artifact across environments;
• use immutable version identifiers and preferably content digests;
• retain provenance linking artifact to source, builder and checks;
• apply retention and legal-hold policies deliberately;
• replicate critical artifacts where recovery requirements justify it;
• scan before release and rescan when vulnerability intelligence changes;
• restrict who can publish, delete or promote artifacts.

Rebuilding separately for staging and production is risky because the two outputs may differ. Build once, then configure the deployment environment externally.

4. Environments, configuration and approvals

A useful environment progression is development, integration, staging and production. Environments should be similar enough to reveal deployment problems, but they need not be identical in scale. Infrastructure should be defined as code and reviewed like application code.

Configuration contains environment-specific values such as endpoints, capacity and feature settings. Secrets are a special category of configuration and must be encrypted, access-controlled and rotated. Configuration should not require rebuilding the artifact.

Approvals should be risk-based. A low-risk documentation change may pass automatically; a database schema change or public authentication change may require peer, security or operations approval. Approval records should identify the release, evidence reviewed, approver and time. An approval gate that simply asks someone to click “approve” without presenting useful evidence is process theatre.

Policy-as-code can enforce rules such as:

• production deployment must use a signed artifact;
• a critical vulnerability must have an approved exception;
• infrastructure changes must pass a policy check;
• the deployer must not have unrestricted administrative permissions;
• a change must have an associated issue or review.

5. Release strategies

Rolling deployment

A rolling deployment replaces instances or pods in batches. It avoids maintaining a complete duplicate environment and is efficient for many services. However, old and new versions coexist temporarily, so backward-compatible APIs and database changes are important. If the new version is defective, rollback may require replacing already-updated instances.

Blue-green deployment

Blue-green deployment maintains two production-capable environments. Blue serves users while green receives the new version. After validation, traffic switches to green. The old environment is retained for rapid recovery, subject to data and cost constraints.

Blue-green is especially useful when a whole-environment switch is feasible. It does not automatically solve database compatibility: both versions may access the same database during transition.

Canary deployment

A canary sends a small, controlled proportion of traffic to the new version, then increases exposure when defined success criteria are met. Criteria may include error rate, latency, saturation, conversion rate and support signals. The canary group should be selected meaningfully—for example by percentage, region or tenant—without creating unfair or unsafe exposure.

Canarying requires reliable traffic routing and representative traffic. A small canary cannot reveal a problem that occurs only for a rare customer type unless that type is included in the test group.

Feature flags

A feature flag separates deployment of code from activation of behavior. Teams can release dormant code, enable it for internal users, or disable a problematic capability without rebuilding. Flags should have an owner, purpose, creation date, expiry date and tested default. Permanent flags increase complexity and create paths that are easy to forget.

Feature flags are not a substitute for access control. A flag that exposes confidential information must be combined with authorization. Flag evaluation should be observable, and emergency changes must be audited.

6. Deployment safety and recovery

A deployment is safe only when the team can detect failure and recover within an acceptable time. Before release, define health indicators, abort thresholds, owner responsibilities and the exact recovery action.

Rollback and roll-forward

Rollback returns service to a previous application version or configuration. It is attractive when the previous version is known to work, but it may be unsafe after an irreversible schema change or data transformation. Roll-forward deploys a corrective version and is often preferable when data has already changed or when the previous version has a known vulnerability.

Database migrations should generally be expand and contract:

1. Expand: add a backward-compatible schema element.
2. Deploy code that can use both old and new forms.
3. Migrate or backfill data safely.
4. Switch reads and writes to the new form.
5. Contract: remove the old element only after old application versions are gone.

This approach supports rolling and canary releases more safely than a single destructive migration.

Observability and automated gates

Logs, metrics, traces and deployment events should be correlated with a release identifier. A post-deployment check can compare the new version with a baseline. Useful signals include request failure percentage, latency percentiles, queue age, resource saturation, crash frequency and business transaction success.

A release gate should define a measurement window and a decision rule. For example, a service may pause canary expansion if its error rate exceeds the agreed threshold for a sustained interval. Thresholds must account for normal variation; otherwise the pipeline will either stop healthy releases or ignore real failures.

Reliability metrics

The DORA framework commonly discusses deployment frequency, lead time for changes, change failure rate and time to restore service. These metrics should be interpreted together. Increasing deployment frequency while change failure rate and recovery time worsen is not an improvement. Metrics should encourage learning rather than punish teams for reporting incidents.

For a defined observation period, a simple change failure rate can be expressed as:

Here, a “deployment causing a production failure” must be defined consistently—for example, a deployment requiring rollback, hotfix or customer-impacting remediation. The metric is meaningful only when the numerator and denominator cover the same period and deployment population. It should not be used to compare teams with different service risk without context.

7. AWS and Google Cloud implementation patterns

A representative AWS pipeline may use GitHub or CodeCommit as the source, CodePipeline for orchestration, CodeBuild for compilation and tests, ECR for container storage, and CodeDeploy, ECS, EKS or CloudFormation for release. Traffic can be controlled through load balancers, target groups, service deployment mechanisms or weighted routing such as Route 53, depending on the architecture.

A representative Google Cloud pipeline may use a Git repository, Cloud Build for build and test execution, Artifact Registry for images or packages, and Cloud Deploy for promotion to GKE or other supported targets. Cloud Run revisions support traffic splitting and quick revision changes; GKE can support rolling or progressive delivery through suitable deployment tooling.

In both providers, the following controls remain essential:

• use service accounts or IAM roles with least privilege;
• separate build identity from deployment identity;
• store secrets in managed secret services rather than pipeline variables where possible;
• enable audit logging;
• restrict artifact deletion;
• use private networking or controlled egress where required;
• encrypt artifacts and logs according to organizational policy;
• define infrastructure and pipeline configuration as code;
• test the recovery path, not only the forward deployment.

The cloud service does not determine the maturity of the delivery process. A managed pipeline can still be insecure, untested or impossible to recover.

8. Illustrative case: a retail checkout service

Background

A fictional online retailer operates a checkout API on a managed container platform. The team releases approximately twice monthly. Deployments are performed manually from engineers’ laptops, and a failed release once caused duplicate order attempts.

Problem

The organization needs more frequent releases, but it must protect payment and order integrity. The existing process lacks a reproducible build, artifact history, automated integration tests and a reliable recovery procedure.

Analysis

The most serious risks are not simply slow delivery. They are uncertainty about what was deployed, insufficient testing of idempotency and payment interactions, and the absence of a controlled traffic strategy. A full automatic production deployment would increase risk rather than reduce it.

Approach and solution

The team adopts trunk-based development with protected pull requests. Each merge triggers a pipeline that:

1. runs linting, unit tests and secret scanning;
2. builds a container using a locked dependency set;
3. generates an SBOM and performs dependency, image and IaC scans;
4. runs integration tests against an isolated database and payment sandbox;
5. publishes one immutable image to Artifact Registry or ECR;
6. deploys that exact image to staging;
7. runs checkout smoke tests and contract tests;
8. requires an operations approval for production during the initial rollout.

Production uses a canary strategy. Five percent of eligible traffic is routed to the new revision for a defined observation period. The gate checks payment success, duplicate-order rate, error rate and latency against baseline. If the gate fails, traffic returns to the previous revision. Database changes use expand-and-contract migration, and a feature flag controls a new discount calculation separately from the deployment.

The pipeline records the commit, image digest, scan results, approver, routing change and final outcome. Secrets are retrieved at runtime using a narrowly scoped service identity.

Outcome

The retailer can release smaller changes more frequently and can identify the exact artifact associated with a customer-impacting event. The canary limits exposure, while the feature flag permits the discount behavior to be disabled independently. The process does not eliminate defects, but it makes defects less likely, easier to detect and less costly to recover from.

Learning points

1. Automation must be paired with evidence and policy.
2. The same immutable artifact should move through environments.
3. Business metrics, such as duplicate-order rate, can be more important than infrastructure metrics.
4. Deployment and feature activation are separate decisions.
5. Database compatibility determines whether rollback is genuinely possible.

9. Governance, ethics and professional practice

Release engineering affects users, employees and organizational accountability. Teams should protect personal data in test environments, avoid copying production data without controls, and ensure that logs do not expose credentials or sensitive payment information. Automated decisions must be auditable, especially when a pipeline blocks a release or changes customer-facing behavior.

Segregation of duties may require that the person who writes a change cannot be the sole approver for production. Emergency access should be time-limited, logged and reviewed afterward. Regulatory requirements may impose retention, traceability, change approval or data-residency obligations; these should be translated into pipeline controls with advice from security, legal and compliance specialists.

AI-assisted coding and test generation can improve productivity, but generated code may contain vulnerabilities, license concerns, hallucinated APIs or untested assumptions. Human review, approved tools, confidential-data restrictions and normal security gates remain necessary. AI should assist professional judgement, not replace accountability for the released system.

10. Best-practice checklist

A diploma-level practitioner should be able to verify that a pipeline:

• has one authoritative source repository and protected production branches;
• runs automatically on change and reports actionable results;
• builds from declared, reproducible inputs;
• tests behavior at multiple levels;
• scans source, dependencies, images and infrastructure;
• creates an SBOM or equivalent component record;
• publishes immutable artifacts and promotes the same artifact;
• separates configuration and secrets from the artifact;
• uses least-privilege identities and audit logs;
• supports approvals appropriate to risk;
• defines health indicators and abort criteria;
• supports a tested rollback or roll-forward procedure;
• uses backward-compatible database migration patterns;
• removes obsolete feature flags;
• measures delivery speed together with reliability and security.

Chapter Summary

CI/CD is a system for integrating changes, verifying them and making them releasable; release engineering adds the disciplined design of release processes, artifacts, environments, risk controls and recovery. A strong pipeline begins with Git-based traceability, builds reproducible outputs, applies layered testing and security scanning, and stores immutable artifacts in a controlled repository. The same artifact should be promoted rather than rebuilt for each environment.

Approvals and policy-as-code provide governance when risk justifies them. Rolling, blue-green, canary and feature-flag strategies address different operational needs and constraints. Safe release engineering also depends on observability, meaningful quality gates, backward-compatible database migrations and tested recovery. AWS and Google Cloud provide useful managed services, but neither removes the need for sound engineering judgement, least privilege, auditability and ethical handling of data.

Key Terms

CI: Continuous integration; frequent merging of changes with automated verification.

Continuous delivery: Keeping every accepted change in a releasable state, often with an approval before production.

Continuous deployment: Automatically releasing every change that satisfies defined controls.

Release engineering: The discipline of designing, automating and governing software releases.

Pipeline: An automated sequence of activities that moves a change from source to deployment.

Artifact: A deployable output such as a package, image or application bundle.

Artifact digest: A content-derived identifier used to refer to an exact artifact.

Immutable artifact: An artifact that cannot be silently altered after publication.

SBOM: Software bill of materials; a record of components included in software.

SAST: Static application security testing performed on source or compiled code.

DAST: Dynamic application security testing performed against a running application.

Feature flag: A controlled runtime switch that changes behavior independently of deployment.

Blue-green deployment: Operation of two production-capable environments with traffic switched between them.

Canary deployment: Gradual release to a limited and controlled traffic group.

Rollback: Returning to a prior version or configuration.

Roll-forward: Deploying a corrective version instead of reverting.

Expand-and-contract migration: A staged, backward-compatible approach to database schema change.

Policy-as-code: Machine-enforced rules that express organizational or security policy.

Change failure rate: The percentage of deployments that cause a defined production failure.

Review Questions

Short Answer Questions

1. Distinguish continuous delivery from continuous deployment.
2. Why should a team promote the same artifact across environments?
3. State two purposes of a pull request in a Git workflow.
4. What is the main purpose of a feature flag?
5. Why can a database migration make rollback unsafe?

Long Answer Questions

1. Explain the stages of a secure CI/CD pipeline from commit through production verification.
2. Compare rolling, blue-green and canary deployment strategies, including their operational constraints.
3. Discuss how artifact management supports traceability, security and recovery.
4. Explain how testing, security scanning and approvals should be combined rather than treated as isolated activities.
5. Describe an expand-and-contract database migration and explain why it supports progressive delivery.

Analytical / Application Questions

1. A team has frequent deployments but rising customer-impacting failures. Analyze which pipeline controls and metrics should be reviewed before increasing deployment frequency further.
2. Design a release plan for a payment service that introduces a new checkout calculation. Select a deployment strategy, define quality gates, and explain rollback, feature-flag and data-migration decisions.
3. An image passed staging tests, but a production canary shows increased latency for one customer region. Decide what evidence should be collected and whether to continue, pause, roll back or roll forward. Justify the decision using release-engineering principles.

Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes continuous delivery from continuous deployment?

A. Continuous delivery keeps changes releasable, while continuous deployment automatically releases qualifying changes.

B. Continuous delivery applies only to infrastructure, while continuous deployment applies only to applications.

C. Continuous delivery excludes automated testing, while continuous deployment requires only manual testing.

D. Continuous delivery uses no artifacts, while continuous deployment always rebuilds artifacts.

Correct Answer: A. Continuous delivery keeps changes releasable, while continuous deployment automatically releases qualifying changes.

Question 2

What is the strongest reason for promoting the same immutable artifact from staging to production?

A. It guarantees that production has more dependencies than staging.

B. It prevents the deployed output from changing between environments.

C. It removes the need for production monitoring.

D. It allows developers to bypass source control.

Correct Answer: B. It prevents the deployed output from changing between environments.

Question 3

Which security practice is most appropriate for a pipeline secret?

A. Store it in the application repository in an encrypted text file.

B. Place it in a container image so every stage can read it.

C. Retrieve it from a managed secret service using a least-privilege identity.

D. Print it in the build log for troubleshooting.

Correct Answer: C. Retrieve it from a managed secret service using a least-privilege identity.

Question 4

Which deployment strategy sends a small, controlled proportion of traffic to a new version before wider release?

A. Rolling deployment

B. Canary deployment

C. Big-bang deployment

D. Source branching

Correct Answer: B. Canary deployment

Question 5

What is a principal risk of a permanent feature flag?

A. It makes source code impossible to compile.

B. It prevents all forms of monitoring.

C. It creates flag debt and increases behavioral complexity.

D. It automatically exposes secrets to users.

Correct Answer: C. It creates flag debt and increases behavioral complexity.

Question 6

Which approach best supports rollback during a database-backed progressive release?

A. Perform a destructive schema change before deploying application code.

B. Use an expand-and-contract migration with backward compatibility.

C. Delete old columns as soon as the canary starts.

D. Allow only the new application version to access the database immediately.

Correct Answer: B. Use an expand-and-contract migration with backward compatibility.

Question 7

A service had 2 deployments that caused production failures out of 40 total deployments in a measurement period. What is its change failure rate using the chapter definition?

A. 2%

B. 5%

C. 20%

D. 38%

Correct Answer: B. 5%

Question 8

Which control most directly records the components included in a software artifact?

A. SBOM generation

B. Traffic splitting

C. Branch deletion

D. Load balancing

Correct Answer: A. SBOM generation

Question 9

Why are layered tests preferable to relying only on end-to-end tests?

A. They eliminate the need for source control.

B. They provide earlier, faster and often more diagnostic feedback.

C. They guarantee that no production defect can occur.

D. They prevent the use of integration environments.

Correct Answer: B. They provide earlier, faster and often more diagnostic feedback.

Question 10

Which statement about rollback is most accurate?

A. Rollback is always safe after any database change.

B. Rollback means deploying a corrective version rather than reverting.

C. Rollback may be unsafe after an irreversible data transformation.

D. Rollback removes the need for observability.

Correct Answer: C. Rollback may be unsafe after an irreversible data transformation.