7. Continuous Integration, Delivery and Deployment

Introduction and Context

Cloud platforms have changed the economics and architecture of software delivery. Teams can provision infrastructure through code, build services automatically, and release applications to globally distributed environments. Yet cloud capability alone does not create delivery performance. Without disciplined integration, testing, security controls, and deployment practices, cloud environments can increase the speed at which defects and vulnerabilities are propagated.

Continuous integration and continuous delivery/deployment (CI/CD) provide the operating model for converting source-code changes into reliable software releases. CI integrates changes frequently and validates them through automated checks. Continuous delivery keeps software in a releasable state, while continuous deployment automatically promotes approved changes into production. These practices are related but not identical. They combine technical automation with organizational decisions about risk, accountability, compliance, and customer impact.

A mature pipeline is therefore more than a sequence of build commands. It is a controlled socio-technical system connecting developers, source control, test environments, artifact repositories, cloud infrastructure, security tools, operational monitoring, and governance. This chapter examines how to design such pipelines across AWS, Google Cloud, and Microsoft Azure, with particular attention to branching, testing, environment control, release strategies, and secure automation.

Learning Objectives

By the end of this chapter, students should be able to:

1. Distinguish continuous integration, continuous delivery, and continuous deployment.

2. Design a CI/CD workflow using source control, branching, testing, artifacts, and controlled environments.

3. Compare trunk-based development, feature branching, and release-branch approaches.

4. Apply security, governance, and supply-chain controls to pipelines.

5. Select suitable rolling, blue-green, canary, and feature-flag deployment strategies.

6. Evaluate pipeline performance using delivery and reliability metrics.

7. Analyse trade-offs in an enterprise cloud deployment scenario.

From Code Change to Operational Release

A CI/CD pipeline usually begins with a change committed to a version-control repository. The pipeline retrieves the source, validates its structure, compiles or packages it, executes tests and security checks, creates an immutable artifact, and promotes that artifact through controlled environments. At later stages, infrastructure definitions may provision or modify cloud resources, while deployment automation changes the running service. Monitoring then supplies evidence about the effect of the release.

The central principle is separation between building and deploying. A build should produce a versioned artifact once; the same artifact should be tested and promoted across environments. Rebuilding separately for testing and production creates uncertainty because the binaries, dependencies, or compilation settings may differ. Artifact immutability therefore supports reproducibility and auditability.

Continuous integration is a development practice in which team members integrate small changes into a shared mainline frequently. Each integration triggers automated validation. CI reduces long-lived divergence and detects conflicts close to the time they are introduced. It does not mean that every test must be completed in seconds, but the initial feedback cycle should be sufficiently rapid to influence developer behaviour.

Continuous delivery extends CI by automating packaging, environment deployment, and release validation so that every accepted change can be released on demand. A human or accountable business process may still authorize production release. Continuous deployment goes further: changes that pass the defined controls are released automatically. The distinction is principally one of release authorization, not merely tool choice.

A useful pipeline model is:

Commit → verify → build → test → secure → package → publish → deploy → validate → observe → learn.

The model is iterative. Monitoring findings can create new backlog items, test cases, or architectural changes. Deployment is therefore part of a feedback system rather than the final technical step.

Source Control and Branching as Risk Controls

Git-based source control supplies version history, collaboration, review, and rollback information. A repository should contain application code, test code, configuration templates, pipeline definitions, and—where practical—infrastructure as code. Secrets must not be committed. Instead, pipelines should retrieve short-lived credentials or references to managed secret stores such as AWS Secrets Manager, AWS Systems Manager Parameter Store, Google Secret Manager, or Azure Key Vault.

Branching strategy influences integration frequency and release complexity. In trunk-based development, developers integrate small changes into a shared trunk, often using short-lived branches or direct, protected pull requests. Automated tests and feature flags allow incomplete functionality to coexist without exposing it to users. This approach minimizes merge divergence and generally supports frequent deployment.

A feature-branch model gives each change an isolated branch and merges it after review and testing. It is useful when changes require independent review, but branches that live for weeks accumulate integration risk. Teams should therefore keep branches short-lived and merge frequently.

Gitflow-style models use separate development, release, and hotfix branches. They can be appropriate for products with formal release trains or multiple supported versions, but they often introduce synchronization overhead and delayed feedback. No branching model is universally superior. The choice should reflect release frequency, regulatory requirements, team topology, compatibility constraints, and the cost of parallel maintenance.

Protected branches, mandatory reviews, signed commits where appropriate, status checks, and ownership rules provide governance without replacing technical testing. A review is not evidence that software works; it is one control among several.

Pipeline Architecture and Quality Gates

A pipeline is often organized into stages, although modern systems may execute independent checks in parallel. A practical design includes the following controls.

Source and change validation. The pipeline checks commit identity, branch policy, syntax, formatting, dependency declarations, and change metadata. Pull requests can require peer review and successful status checks before merging.

Build and unit testing. Compilation or packaging confirms that the application can be produced. Unit tests examine small components in isolation and should provide fast feedback. Failed tests should stop promotion unless an explicitly documented exception exists.

Integration and contract testing. Integration tests examine interactions with databases, queues, APIs, and cloud services. Contract tests verify that service providers and consumers agree on interfaces. These tests identify failures that unit tests cannot expose, but they require controlled dependencies and careful data management.

System, acceptance, and performance testing. System tests exercise a complete service. Acceptance tests assess business behaviour, while load, stress, and resilience tests evaluate operational characteristics. These checks should be proportionate to risk; running an expensive performance suite on every documentation change is inefficient, while never testing performance before a major release is dangerous.

Security testing. Static application security testing examines source or compiled code; software composition analysis identifies vulnerable dependencies; secret scanning searches for exposed credentials; dynamic testing examines a running application. Infrastructure-as-code scanning detects insecure cloud configurations. Security checks should begin early and continue at deployment and runtime rather than being isolated in a final stage.

Release gates. A gate may require test success, vulnerability thresholds, approvals, change-ticket linkage, policy compliance, or a healthy deployment metric. Gates should be explicit and version-controlled. Excessive manual gates create queues and encourage bypasses; insufficient gates create uncontrolled risk.

Pipeline failures require a clear ownership model. A broken mainline should be treated as urgent because every later change is affected. Teams should distinguish product defects, environmental failures, flaky tests, and pipeline defects. Flaky tests are particularly harmful because they weaken trust in automation.

Artifacts, Dependencies, and Reproducibility

An artifact is a deployable output such as a container image, package, binary, Helm chart, or serverless bundle. Repositories such as Amazon Elastic Container Registry and CodeArtifact, Google Artifact Registry, and Azure Container Registry and Artifacts can store these outputs. The repository should support access control, retention, replication where required, vulnerability scanning, and provenance information.

Artifacts should be identified by immutable versions, preferably including a source revision or digest. A mutable label such as `latest` is convenient for experimentation but unsafe as a production reference because its meaning changes. Dependency versions should be pinned or constrained through a controlled policy, and lockfiles should be reviewed as part of source changes.

Software supply-chain security requires confidence about where an artifact came from and what it contains. Useful controls include signed artifacts, build provenance, isolated build workers, minimal base images, dependency allowlists, and separation of build and deployment permissions. A pipeline that can alter its own approval logic or deploy with unrestricted cloud credentials represents a significant concentration of risk.

Environment Design and Promotion

Environments commonly include development, integration or test, staging, and production. Their purpose is not simply to create copies of infrastructure. Each environment should have an explicit role, data policy, access model, and promotion rule. Development environments favour speed and experimentation; staging should approximate production sufficiently to reveal deployment and configuration problems; production requires strict access, observability, resilience, and recovery controls.

Configuration should be externalized from application artifacts. Environment-specific values—such as endpoints, scaling limits, and feature settings—belong in controlled configuration systems, not in hard-coded source. Secrets should be injected at runtime or deployment time and should be rotated. Production data should not be copied into lower environments without authorization, minimization, masking, and a legitimate purpose.

Infrastructure as code makes environment creation repeatable. Terraform, AWS CloudFormation, AWS CDK, Google Cloud Deployment Manager or Terraform-based Google infrastructure, and Azure Bicep or ARM templates can describe resources declaratively. Plans should be reviewed before application, and state must be protected against unauthorized modification. Policy-as-code tools can block prohibited regions, public storage, excessive permissions, or missing encryption.

Promotion should normally move one tested artifact forward rather than rebuilding it. Deployment identities should have only the permissions needed for their stage. A production deployment may require stronger authentication, approval, separation of duties, or a time-bound authorization than a development deployment.

Deployment Strategies for Cloud Applications

Deployment strategy determines how new software is introduced and how failure affects users.

Rolling Deployment

A rolling deployment replaces instances or pods gradually while maintaining service capacity. It is efficient because it does not require a complete duplicate environment. Health checks, readiness probes, minimum capacity, and rollback thresholds are essential. Rolling deployment is well suited to backward-compatible changes, but it can expose users to mixed versions. Database schema changes must therefore support both old and new application versions during the transition.

AWS services such as Amazon ECS and EKS, Google Kubernetes Engine, and Azure Kubernetes Service support rolling patterns through orchestration controls. The platform mechanism differs, but the design principles are shared.

Blue-Green Deployment

Blue-green deployment maintains two production-like environments. Blue serves users while green receives the new version. Traffic is switched after validation, generally through a load balancer, service routing layer, or platform deployment controller. Rollback can be rapid by switching traffic back to blue. The principal disadvantages are the cost of duplicated capacity and the difficulty of handling stateful resources or incompatible database changes.

Canary Deployment

A canary releases the new version to a small, selected proportion of traffic or users before wider promotion. Automated analysis can compare error rate, latency, saturation, conversion, or domain-specific indicators between versions. Canary analysis must account for traffic composition: a canary serving only internal users is not equivalent to one serving a representative customer population. A defined abort policy is necessary; otherwise a canary becomes an indefinite experiment.

Feature Flags

Feature flags separate code deployment from feature activation. A feature can be deployed but disabled, enabled for employees, exposed to a percentage of users, or activated by geography or account type. Flags reduce release coupling and support experimentation, but they introduce conditional paths and operational debt. Every flag should have an owner, purpose, expiry or review date, and secure default. Flags must not be used to conceal authorization failures or bypass essential security controls.

These strategies can be combined. For example, a new version may be rolled onto a small canary pool, while a feature flag keeps unfinished functionality disabled. The appropriate selection depends on blast radius, rollback speed, infrastructure cost, data compatibility, observability maturity, and regulatory obligations.

Observability, Rollback, and Measurement

A deployment is successful only if the service continues to meet its operational and business objectives. Logs, metrics, traces, health checks, deployment events, and audit records should be correlated with a release identifier. Important indicators include availability, latency, error rate, saturation, queue depth, and business transaction success.

Rollback should be designed before deployment. It may involve reverting an application artifact, switching traffic to a previous environment, disabling a feature flag, or applying a forward fix. Database rollback is often difficult and sometimes unsafe; backward-compatible, expand-and-contract migration patterns are preferable. For example, a team may add a nullable column, deploy code that writes both old and new fields, migrate data, and remove the old field only after all versions no longer depend on it.

Delivery performance can be assessed using deployment frequency, lead time for changes, change failure rate, and time to restore service. These measures should guide improvement rather than become targets that encourage unsafe releases. High deployment frequency with frequent incidents is not mature delivery; speed and stability must be interpreted together.

Security, Governance, and Professional Responsibility

CI/CD pipelines are privileged automation systems. They can access source code, artifact repositories, cloud accounts, production data, and deployment controls. Threats include credential theft, malicious dependency substitution, compromised build agents, unauthorized pipeline changes, poisoned artifacts, and excessive permissions.

A secure design applies least privilege, workload identity or short-lived credentials, network controls, encrypted communication, centralized audit logging, approval separation for high-risk changes, and protected pipeline definitions. Build agents should be ephemeral or regularly rebuilt, and untrusted code should not execute with production credentials. Secrets should be masked in logs, but masking is not a substitute for preventing secret exposure.

Governance must be proportionate. Regulated organizations may need evidence of who approved a change, which artifact was deployed, what tests ran, which infrastructure version was used, and whether required segregation of duties was maintained. Automated evidence collection is more reliable than manually assembled records. Compliance controls should be embedded in the delivery path rather than treated as paperwork after deployment.

Ethically, engineers must consider the consequences of automated release decisions. A feature may be technically deployable but harmful to vulnerable users, discriminatory in effect, or unsafe in a critical service. Business pressure must not justify suppressing known security findings, manipulating quality metrics, or bypassing incident disclosure obligations.

Illustrative Case Study: A Multi-Region Payment Service

Background. A fictional financial-technology company, MeridianPay, operates a payment API on a managed Kubernetes platform. It serves customers in several regions and must maintain strong availability and traceability. Before improvement, teams deployed manually from developer workstations, rebuilt packages for each environment, and used a shared production credential.

Problem. Releases were infrequent because deployment was considered risky. A configuration mismatch caused one release to fail in production, and the team could not immediately determine which source version was running. Database changes also made rollback unreliable.

Analysis. The organization identified four related weaknesses: lack of artifact immutability, insufficient environment parity, excessive deployment privilege, and inadequate release telemetry. The problem was not solved merely by purchasing a pipeline tool. The delivery process needed architectural and governance changes.

Approach. MeridianPay adopted protected trunk-based development with short-lived pull requests. Each merge initiated unit, contract, dependency, secret, and infrastructure-policy checks. A signed container image was built once and stored in a managed artifact registry with a source revision and software bill of materials. Development and staging used the same image digest as production.

The pipeline obtained cloud permissions through workload identity and separate roles for build, staging deployment, and production deployment. Infrastructure changes were reviewed as plans. Production releases used a canary deployment beginning with five percent of representative traffic. Automated analysis evaluated payment success, latency, error rate, and saturation for a defined observation period. A feature flag controlled a new payment-routing algorithm independently of the application deployment.

Database changes followed an expand-and-contract pattern. The service was instrumented with trace and release identifiers, and an automated rollback disabled the feature flag or returned traffic to the previous version when thresholds were exceeded. Production deployment remained subject to an approval requirement for high-risk changes, while low-risk changes could proceed through continuous delivery controls.

Outcome. The fictional case demonstrates how faster delivery resulted from reducing uncertainty, not from removing safeguards. The company gained reproducible artifacts, clearer audit evidence, smaller release blast radius, and faster recovery. It also accepted continuing costs: maintaining observability, reviewing feature flags, and testing compatibility across regions.

Learning points. First, deployment frequency depends on confidence in the entire system. Second, security identity design is part of pipeline architecture. Third, canary and feature-flag controls are effective only when supported by meaningful telemetry. Finally, database and state-management decisions often determine whether rollback is genuinely possible.

Common Failure Patterns and Limitations

Several mistakes recur in CI/CD programmes. Teams may automate a fragile manual process without simplifying it, create long-lived branches that make integration painful, or measure only pipeline duration while ignoring escaped defects. A green pipeline can still produce a poor release if tests are shallow, test data is unrealistic, or observability is absent.

Another failure is treating staging as proof of production safety. Environments differ in scale, identity, network policy, data volume, and external dependencies. Progressive delivery reduces this uncertainty but does not eliminate it. Similarly, blue-green deployment is not automatically safe if both environments share an incompatible database.

Feature flags can accumulate into untested combinations and become a hidden configuration system. Security scanners can generate so many low-value findings that teams ignore important ones. Manual approvals may provide accountability, but they can become ceremonial if approvers lack evidence or authority. Finally, platform standardization can improve reliability while restricting local experimentation; governance should define safe boundaries rather than impose unnecessary uniformity.

Current Developments and Best Practices

Cloud delivery is increasingly influenced by platform engineering, internal developer platforms, policy-as-code, software bills of materials, artifact provenance, and automated progressive delivery. GitOps approaches use a version-controlled desired state and reconcile it with running infrastructure, improving traceability but requiring careful control of repository and controller permissions. AI-assisted coding and test generation may increase change volume, making review, provenance, security analysis, and human accountability more important rather than less important.

Effective practice includes the following principles:

• Build once and promote an immutable, identifiable artifact.
• Keep integration frequent and branches short-lived.
• Make the fastest valuable tests run first, while retaining deeper risk-based testing.
• Treat security and infrastructure policy as pipeline requirements.
• Use least privilege and short-lived identities for automation.
• Design rollback, database compatibility, and observability before deployment.
• Select rolling, blue-green, canary, or feature-flag methods according to risk and context.
• Give every exception, flag, and manual gate an owner and review date.
• Measure both delivery performance and operational outcomes.
• Make the pipeline itself versioned, monitored, and tested.

Chapter Summary

CI/CD is a controlled path from source change to operational learning. Continuous integration validates frequent integration; continuous delivery keeps changes releasable; continuous deployment automates production release when controls are satisfied. Reliable pipelines use appropriate branching, automated and risk-based testing, immutable artifacts, managed environments, infrastructure as code, least-privilege identities, and explicit quality gates. Rolling, blue-green, canary, and feature-flag strategies offer different balances of cost, speed, blast radius, and rollback capability. Security, governance, observability, and database compatibility must be designed into the pipeline. Mature organizations measure both delivery speed and service stability, using automation to improve confidence rather than to remove responsibility.

Key Terms

Continuous integration: Frequent integration of code into a shared mainline with automated validation.

Continuous delivery: A practice of keeping validated software ready for release on demand.

Continuous deployment: Automatic release of qualifying changes into production.

Pipeline: An automated workflow that validates, packages, promotes, and deploys software.

Artifact: A versioned deployable output such as a container image or binary.

Artifact immutability: The principle that a published artifact is not changed after creation.

Trunk-based development: Frequent integration into a shared mainline using short-lived branches.

Feature branch: A temporary branch used to develop and review a particular change.

Quality gate: A defined condition that must be satisfied before pipeline progression.

Contract testing: Testing that verifies compatibility between service consumers and providers.

Software composition analysis: Examination of third-party dependencies for vulnerabilities and policy issues.

Infrastructure as code: Declarative or programmatic management of infrastructure through versioned definitions.

Rolling deployment: Gradual replacement of running instances with a new version.

Blue-green deployment: Traffic switching between separate old and new production environments.

Canary deployment: Limited release of a new version before broader promotion.

Feature flag: A runtime control that governs whether functionality is active.

Workload identity: A mechanism allowing workloads to obtain controlled cloud permissions without long-lived secrets.

Progressive delivery: Incremental exposure of a release supported by automated observation and control.

Change failure rate: The proportion of deployments that cause incidents, rollback, or remediation.

Software bill of materials: An inventory of components contained in a software artifact.

Review Questions

Short Answer Questions

1. Distinguish continuous delivery from continuous deployment.

2. Why should a pipeline build an artifact once and promote it across environments?

3. State two risks associated with long-lived feature branches.

4. What is the purpose of a quality gate?

5. Why are feature flags both useful and potentially risky?

Long Answer Questions

1. Explain how source control, automated testing, artifact repositories, and environment controls form an integrated CI/CD system.

2. Compare rolling, blue-green, and canary deployment strategies in terms of cost, rollback, blast radius, and operational requirements.

3. Discuss how least privilege, workload identity, artifact signing, and audit logging secure a cloud pipeline.

4. Explain why database compatibility is central to safe deployment and rollback.

5. Design a CI/CD workflow for a cloud-hosted web service, justifying its branching, testing, promotion, and release controls.

Analytical / Application Questions

1. A team deploys frequently but experiences frequent production incidents. Using the delivery and reliability concepts in this chapter, diagnose the likely weaknesses and propose corrective measures.

2. An enterprise must release a major API change while old clients remain active. Select a deployment and data-migration strategy, and explain how you would limit user impact.

3. A pipeline uses a shared production credential, mutable container tags, and manual testing before release. Analyse the security and reliability risks and redesign the pipeline.

22. Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes continuous delivery from continuous deployment?

A. Continuous delivery integrates code, while continuous deployment only runs unit tests.

B. Continuous delivery keeps software releasable with controlled authorization, while continuous deployment automatically releases qualifying changes.

C. Continuous delivery applies only to infrastructure, while continuous deployment applies only to applications.

D. Continuous delivery requires blue-green deployment, while continuous deployment requires canary deployment.

Correct Answer: B. Continuous delivery keeps software releasable with controlled authorization, while continuous deployment automatically releases qualifying changes.

Question 2

Why is rebuilding an application separately for staging and production considered poor practice?

A. It prevents unit testing from running.

B. It makes source control unnecessary.

C. It can produce different binaries or dependencies, weakening reproducibility.

D. It always increases cloud storage costs beyond acceptable limits.

Correct Answer: C. It can produce different binaries or dependencies, weakening reproducibility.

Question 3

A team wants to minimize branch divergence while integrating small changes frequently. Which approach is most appropriate?

A. Long-lived release branches

B. Trunk-based development with short-lived branches

C. Manual deployment from local workstations

D. A separate permanent branch for every developer

Correct Answer: B. Trunk-based development with short-lived branches

Question 4

Which deployment strategy initially exposes a new version to a limited and representative portion of traffic?

A. Canary deployment

B. Full replacement deployment

C. Static provisioning

D. Batch compilation

Correct Answer: A. Canary deployment

Question 5

What is the primary security advantage of workload identity in a CI/CD pipeline?

A. It eliminates the need for authorization.

B. It provides every build agent with administrator access.

C. It replaces long-lived credentials with controlled workload-based access.

D. It prevents all application vulnerabilities.

Correct Answer: C. It replaces long-lived credentials with controlled workload-based access.

Question 6

A database schema must support both old and new application versions during a rolling deployment. Which approach is most suitable?

A. Immediate deletion of all old columns

B. Expand-and-contract migration

C. Rebuilding the database after every instance update

D. Disabling health checks during deployment

Correct Answer: B. Expand-and-contract migration

Question 7

Which statement about blue-green deployment is correct?

A. It always requires only one environment.

B. It gradually modifies the same instance without traffic control.

C. It can enable rapid rollback by switching traffic to the previous environment.

D. It eliminates the need for database compatibility planning.

Correct Answer: C. It can enable rapid rollback by switching traffic to the previous environment.

Question 8

What is a significant operational risk of unmanaged feature flags?

A. They prevent any form of rollback.

B. They create conditional paths and configuration debt.

C. They make source code impossible to version.

D. They require every release to use a new artifact repository.

Correct Answer: B. They create conditional paths and configuration debt.

Question 9

Which control most directly helps establish what components are contained in a deployable artifact?

A. A software bill of materials

B. A longer branch name

C. A manual server restart

D. A network load balancer alone

Correct Answer: A. A software bill of materials

Question 10

A canary release shows acceptable latency but a sharp decline in successful payment transactions. What should the pipeline or release controller do?

A. Ignore the metric because latency is healthy.

B. Increase traffic immediately to obtain more data.

C. Apply the defined abort or rollback policy and investigate the business metric.

D. Delete the monitoring data and repeat the deployment.

Correct Answer: C. Apply the defined abort or rollback policy and investigate the business metric.