Deployment Strategies and CI/CD

Introduction

Cloud applications are rarely delivered as a single installation. They are changed repeatedly through source-code commits, dependency updates, configuration changes, database migrations, and infrastructure modifications. A deployment strategy defines how a new version moves from a build environment into production, how risk is controlled, how users are exposed to change, and how service is restored when a release is defective.

Continuous integration and continuous delivery provide the engineering process around deployment. Continuous integration (CI) frequently integrates small code changes and verifies them through automated checks. Continuous delivery keeps software in a releasable state, while continuous deployment automatically releases approved changes to production. These practices are related but not identical: delivery may require a human approval, whereas deployment is fully automated after defined controls pass.

For cloud infrastructure professionals, deployment is not only an application concern. It includes container images, operating-system packages, infrastructure-as-code, secrets, policies, observability configurations, and database schemas. A reliable delivery pipeline therefore treats all deployable components as versioned, testable artifacts.

Learning Objectives

By the end of this chapter, you should be able to:

• explain CI/CD pipelines, artifacts, environments, gates, and approvals;
• distinguish rolling, recreate, blue-green, canary, feature-flag, immutable, and progressive deployments;
• select a deployment strategy according to availability, risk, cost, and reversibility;
• design automated testing, artifact promotion, rollback, and approval controls;
• describe GitOps and its relationship to declarative infrastructure;
• evaluate security, governance, observability, and operational limitations in delivery systems.

Foundations of a Delivery Pipeline

A delivery pipeline is a controlled sequence through which a change is transformed into a deployable release. A typical pipeline contains the following stages:

1. Source and change control: A developer commits code, configuration, or infrastructure definitions to a version-control repository. Pull requests provide review, traceability, and policy enforcement.
2. Build: The system compiles code, resolves dependencies, creates a container image or package, and records its version.
3. Verification: Unit, integration, security, policy, performance, and infrastructure tests are executed.
4. Packaging and publication: A versioned artifact is stored in a trusted registry or repository.
5. Deployment to a lower environment: The artifact is installed in development, test, or staging using an automated procedure.
6. Promotion: Evidence is evaluated and the same artifact is promoted to the next environment rather than rebuilt.
7. Production release: A selected deployment strategy controls exposure to users.
8. Observation and decision: Metrics, logs, traces, alerts, and automated analysis determine whether to continue, pause, or roll back.

A pipeline should be reproducible: the same source, dependency versions, build instructions, and configuration inputs should produce an equivalent artifact. Reproducibility reduces the risk that testing one binary and deploying another creates false confidence.

Artifacts and Promotion

An artifact is a versioned output that can be deployed. Examples include a container image identified by a digest, a package, a serverless bundle, a Helm chart, or an infrastructure plan. Tags such as latest are convenient but mutable; immutable identifiers such as cryptographic digests provide stronger traceability.

Artifact repositories should provide access control, retention rules, vulnerability scanning, provenance metadata, and replication where required. Promotion should normally copy or authorize an existing artifact across environments. Rebuilding separately for production can introduce compiler, dependency, or environment differences.

A release consists of more than an application binary. It may include configuration, database migration scripts, infrastructure definitions, and a release manifest. Secrets should not be embedded in artifacts. They should be retrieved at deployment or runtime from an approved secret-management service, with access audited and rotated.

Pipeline Controls

Important controls include:

• branch protection and mandatory peer review;
• dependency and container-image scanning;
• software composition and license checks;
• secret detection before code leaves the repository;
• infrastructure policy checks, such as prohibiting public storage by default;
• signed commits, images, or attestations where supply-chain assurance is required;
• separation of duties for high-risk production changes;
• approval gates based on evidence rather than informal communication;
• audit records linking a change, artifact, approver, deployment, and outcome.

A manual approval is useful for risk acceptance, regulatory control, or business coordination, but it should not compensate for missing automated tests. Excessive approvals can create queues and encourage unsafe bypasses.

Automated Testing and Release Readiness

Testing should be layered. Unit tests provide fast feedback on isolated logic. Component and integration tests verify interactions with databases, queues, identity systems, and external services. Contract tests check that independently deployed services agree on interfaces. End-to-end tests validate critical user journeys but are usually slower and more fragile. Security tests include static analysis, dependency scanning, dynamic testing, and configuration checks.

A useful pipeline test sequence places fast, inexpensive checks early and expensive environment-dependent checks later. Tests must be maintained like production code: unreliable tests create false failures, while inadequate tests create false confidence.

Release readiness should include operational evidence, not only functional correctness. Teams should confirm that dashboards, alerts, runbooks, backups, capacity limits, and rollback procedures exist. A deployment is not ready merely because its code passes unit tests.

Major Deployment Strategies

Recreate Deployment

In a recreate deployment, the existing version is stopped and the new version is started. It is simple and can be appropriate for development or workloads that tolerate downtime. Its principal disadvantage is an explicit service interruption and a potentially risky cutover. It is unsuitable for highly available services unless redundancy exists elsewhere.

Rolling Deployment

A rolling deployment replaces instances gradually. A scheduler removes or updates a subset of old instances and starts new ones, repeatedly, until the fleet is updated. During the transition, old and new versions coexist.

Rolling deployment reduces capacity shock and normally avoids complete downtime. It requires backward compatibility: clients may reach either version, and both versions may access the same database. Health checks must verify readiness, not merely process existence. Poorly chosen batch sizes can cause capacity loss, while an incompatible schema can make rollback difficult.

A common operational policy is to maintain a minimum number of healthy instances and restrict the maximum number of unavailable instances. The exact settings depend on workload capacity, startup time, and failure tolerance; there is no universally correct percentage.

Blue-Green Deployment

Blue-green deployment maintains two production-capable environments. The active environment, blue, serves users; green contains the candidate version. After testing green, a router, load balancer, service mesh, or DNS mechanism shifts traffic to it. Rollback can be rapid by directing traffic back to blue, provided blue remains available and data changes are compatible.

The strategy provides a clear separation between versions and a strong rollback path, but it may require nearly double compute capacity. DNS switching can be slow because of caching, so a load-balancer or service-routing switch is often more predictable. Database migrations remain a central challenge: reverting application traffic does not automatically revert data.

Canary Deployment

A canary deployment exposes a small proportion of traffic to the new version before wider release. The canary may be selected by percentage, region, customer group, or request characteristic. Engineers compare error rate, latency, saturation, business outcomes, and security signals between canary and baseline populations.

Canaries require representative traffic and statistically useful observations. A small percentage does not guarantee safety if the canary receives unusual requests or too little traffic. Automated promotion should include explicit thresholds and a pause period. For example, a team might halt promotion when the candidate's error rate exceeds the baseline by a defined tolerance, but the threshold must reflect service objectives and normal measurement noise.

Feature-Flag Deployment

Feature flags separate code deployment from feature activation. Code can be deployed while a feature remains disabled. It can then be enabled for internal users, a cohort, or all users without installing another binary.

Flags support experimentation, gradual exposure, and rapid functional disablement. They also introduce operational debt. Every flag needs an owner, purpose, default state, expiry date, and testing plan. A flag that remains indefinitely creates multiple untested code paths. Flags should not be used to hide security controls or to expose incomplete behavior accidentally.

Immutable Deployment

An immutable deployment replaces rather than modifies running infrastructure. A new virtual-machine image, container image, or serverless version is created and deployed; existing instances are not patched in place. Immutability improves consistency, auditability, and rollback because a known image can be redeployed.

Immutable infrastructure does not mean that data is immutable. Databases, queues, and object stores remain stateful and require independent backup, migration, and recovery strategies. Image construction must be secured, and image updates must be performed regularly to address vulnerabilities.

Progressive Deployment

Progressive deployment is the broader practice of increasing exposure in controlled stages while monitoring evidence. Canary, blue-green transitions, staged regional release, and feature-flag activation can all be progressive. A progression may be 1%, 10%, 25%, 50%, and 100%, but stages should be selected according to risk, traffic volume, blast radius, and the time needed to detect failures.

Progressive delivery combines deployment automation with observability and policy. It is not simply a slower rollout. It requires predefined promotion, pause, and rollback criteria.

Choosing a Strategy

Selection should consider availability objectives, data compatibility, startup time, traffic patterns, cost, regulatory obligations, rollback feasibility, and team maturity. A service may combine strategies: immutable images deployed by a rolling mechanism, with canary traffic controlled by a service mesh and feature flags controlling business behavior.

Rollback, Recovery, and Database Change

Rollback means restoring an acceptable previous application state. It may involve routing traffic to a previous environment, redeploying a previous artifact, disabling a feature flag, or applying a corrective release. Rollback is not always safer than forward repair. If a schema migration is destructive or a security vulnerability has been introduced, returning to the old version may be impossible or undesirable.

Database changes should follow an expand-and-contract pattern where practical:

1. Expand: Add new, backward-compatible schema elements.
2. Migrate: Deploy application code that can use old and new structures, then move data gradually.
3. Contract: Remove obsolete elements only after all old application versions are gone.

This pattern supports rolling and blue-green coexistence. Teams should test migration duration, locking behavior, backup recovery, and application compatibility. A rollback plan must state what happens to partially migrated data.

Recovery objectives also influence strategy. Recovery time objective (RTO) is the maximum acceptable time to restore service after disruption. Recovery point objective (RPO) is the maximum acceptable amount of data loss measured in time. Blue-green routing may reduce application recovery time, but it does not by itself provide a suitable RPO for a database.

Observability and Automated Decisions

A progressive pipeline needs signals at three levels:

• Technical: availability, error rate, latency, CPU, memory, saturation, queue depth, and restart counts.
• Dependency: database errors, downstream latency, API rejection, message-processing failures, and timeout rates.
• Business: successful transactions, conversion, payment completion, cancellation, or other domain outcomes.

Metrics should be segmented by version, region, endpoint, and customer cohort where legally and technically appropriate. Logs should be structured and correlation identifiers should connect a request across services. Traces help locate latency and dependency failures.

The four common service-level indicators are availability, latency, error rate, and throughput. If a service-level objective is defined, deployment policy can consume its error budget. However, automated decisions must account for insufficient traffic, delayed telemetry, and noisy data. A pipeline should fail safely: when monitoring is unavailable, it should pause rather than silently promote.

GitOps

GitOps uses a version-controlled repository as the authoritative declaration of desired infrastructure or application state. An agent in the target environment compares the desired state with actual state and reconciles the difference. In a push model, the pipeline directly invokes deployment commands. In a pull-based GitOps model, an agent retrieves approved configuration and applies it from inside the environment.

GitOps offers auditability, repeatability, drift detection, and a clear review workflow. A commit can represent a desired image version, replica count, policy, or service configuration. The reconciliation agent should use least privilege and should not grant repository writers unrestricted access to the underlying cloud account.

GitOps is not automatically secure or fully continuous. Poorly reviewed changes can still be dangerous, secrets may be mishandled, and reconciliation can repeatedly reapply an incorrect configuration. Secret references, encrypted secret stores, policy checks, and emergency procedures are essential. Operators also need a controlled method for urgent changes and a process for reconciling emergency changes back into Git.

Applied Case: Illustrative Online Retail Release

Background. An online retailer operates a containerized checkout service across three cloud zones. Checkout is financially important, and the business cannot tolerate a planned outage. The service uses a relational database and a payment provider.

Problem. A new checkout interface and tax calculation library must be released. A defect could cause failed payments or incorrect totals. The database schema also requires two additional nullable columns.

Analysis. A recreate deployment is rejected because it creates downtime. A simple rolling deployment is possible, but the library change could affect only some customer regions and the database must support old and new application versions. Blue-green would provide a clear switch but would require additional capacity. The team selects immutable container images, an expand-and-contract migration, and a progressive canary release.

Approach. A pull request triggers unit, contract, dependency, container, and infrastructure-policy tests. The build produces an image addressed by digest and signs its provenance. The migration first adds nullable columns. The image is deployed to a staging environment, where synthetic checkout and payment tests run. After approval, 2% of production traffic is routed to the candidate in one zone. Dashboards compare candidate and baseline payment success, error rate, p95 latency, and tax-calculation discrepancies. If signals remain within policy for the observation period, traffic increases to 10%, 25%, 50%, and 100%.

A feature flag keeps the new interface disabled initially. If the interface is defective, the flag can be disabled without changing the image. If the service-level indicators breach policy, the routing controller stops promotion and returns traffic to the prior version. The old image remains available until the release is confirmed. After all instances use the new schema and code, a later change removes obsolete fields only after backup and recovery validation.

Outcome. The retailer obtains a small initial blast radius, a rapid application rollback path, and a separately controlled feature switch. The design does not eliminate risk: payment-provider behavior, data migration, and monitoring quality remain dependencies requiring operational testing.

Learning points. The case demonstrates that deployment strategy is a system design decision. Immutable artifacts improve traceability, progressive exposure reduces blast radius, feature flags separate technical release from business activation, and database compatibility determines whether rollback is genuinely possible.

Security, Governance, and Professional Practice

Delivery pipelines are privileged systems. A compromised pipeline can modify production code or infrastructure. Protect pipeline identities with short-lived credentials, workload identity, least privilege, environment separation, protected branches, and audited approvals. Do not place long-lived cloud keys in source repositories or build logs.

Supply-chain security includes trusted base images, dependency pinning, vulnerability response, artifact signing, provenance, isolated builds, and verification at deployment. A vulnerability scanner is a decision aid, not proof that software is safe; teams must assess exploitability, exposure, compensating controls, and remediation urgency.

Governance should define who may approve production releases, what evidence is required, how emergency changes are recorded, and how long records are retained. Privacy obligations require care when using production data in test environments. Sensitive customer data should be masked or replaced with synthetic data. Feature experiments must respect consent, fairness, accessibility, and applicable data-protection law.

Challenges and Limitations

No strategy guarantees a successful release. Common limitations include:

• hidden coupling between services;
• incompatible database or message schemas;
• insufficient traffic for meaningful canary evidence;
• monitoring that detects infrastructure failure but not business failure;
• rollback blocked by irreversible data changes;
• excessive feature flags and configuration combinations;
• duplicated blue-green environments increasing cost;
• unreliable tests that encourage bypasses;
• cloud-provider or registry outages;
• human overconfidence in automated gates.

The remedy is not simply more automation. Teams need architecture designed for compatibility, realistic failure testing, clear ownership, operational rehearsals, and a culture in which small reversible changes are preferred.

Contemporary Developments

Modern delivery platforms increasingly combine policy-as-code, software supply-chain attestations, ephemeral test environments, automated dependency updates, service-mesh traffic control, and observability-driven promotion. Artificial intelligence can assist with test generation, log summarisation, anomaly detection, and change-risk analysis. These systems should support—not replace—human accountability. AI-generated code and configuration require the same review, testing, licensing, security, and provenance controls as manually produced changes. Automated recommendations should be explainable enough for operators to challenge them, especially in regulated or safety-sensitive environments.

Best-Practice Checklist

1. Build once and promote the same immutable artifact.
2. Keep deployment configuration versioned and reviewable.
3. Automate tests, policy checks, and evidence collection.
4. Use health checks that represent real readiness.
5. Design application and database changes for coexistence.
6. Define promotion, pause, rollback, and escalation criteria before release.
7. Monitor technical, dependency, and business indicators.
8. Keep rollback artifacts and runbooks available.
9. Give pipeline identities only the permissions they require.
10. Remove expired feature flags and review deployment performance after each release.

Chapter Summary

CI/CD transforms version-controlled changes into verified, traceable, and deployable artifacts. A robust pipeline builds, tests, stores, promotes, observes, and governs those artifacts. Recreate deployments are simple but interrupt service; rolling deployments replace instances gradually; blue-green deployments separate active and candidate environments; canaries limit initial exposure; feature flags decouple activation from deployment; immutable infrastructure improves consistency; and progressive delivery increases exposure according to evidence. Effective rollback depends on artifact retention, routing control, observability, and compatible data changes. GitOps applies declarative desired state through version control and reconciliation. Security, supply-chain integrity, privacy, approvals, and professional accountability must be designed into the pipeline rather than added after deployment.

Key Terms

Continuous integration: Frequent integration of changes with automated verification.

Continuous delivery: Keeping software in a releasable state through automated processes.

Continuous deployment: Automatically releasing validated changes to production.

Artifact: A versioned deployable output such as an image or package.

Artifact promotion: Moving an already-built artifact between environments.

Rolling deployment: Replacing instances gradually while versions may coexist.

Recreate deployment: Stopping the old version before starting the new one.

Blue-green deployment: Operating separate active and candidate environments.

Canary deployment: Releasing a new version to a limited user or traffic group.

Feature flag: A control that activates or deactivates behavior independently of deployment.

Immutable infrastructure: Infrastructure replaced from versioned definitions rather than modified in place.

Progressive delivery: Controlled expansion of release exposure using evidence.

Rollback: Returning service to a previous acceptable state.

Expand-and-contract: A compatibility pattern for safely changing schemas in stages.

GitOps: Declarative, version-controlled operations reconciled by an automated agent.

Drift: Difference between declared desired state and actual deployed state.

Health check: An automated test of instance or service readiness and viability.

Provenance: Evidence describing how an artifact was produced and from which sources.

RTO: Maximum acceptable time to restore service.

RPO: Maximum acceptable period of data loss measured in time.

Review Questions

Short Answer Questions

1. Distinguish continuous delivery from continuous deployment.
2. Why should production normally use the same artifact tested in staging?
3. State one benefit and one limitation of a rolling deployment.
4. What is the purpose of a feature flag?
5. Why are backward-compatible database changes important during rolling releases?

Long Answer Questions

1. Explain the stages of a professional CI/CD pipeline, including testing, artifact management, promotion, approval, and observation.
2. Compare recreate, rolling, blue-green, and canary deployments in terms of availability, cost, risk, and rollback.
3. Explain how immutable infrastructure and GitOps improve traceability and operational consistency.
4. Describe an expand-and-contract approach to database migration and explain why it supports rollback.
5. Discuss the security and governance controls required to protect a cloud delivery pipeline.

Analytical / Application Questions

1. A financial service cannot tolerate planned downtime, has limited spare capacity, and receives highly variable traffic. Select a deployment strategy or combination of strategies and justify the choice.
2. During a canary release, technical error rates remain normal but payment completion falls sharply for the canary cohort. What should the pipeline do, and what investigation should follow?
3. A team wants to deploy an application that requires a destructive database migration. Analyse why a simple rollback may fail and propose a safer release design.

Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes continuous delivery from continuous deployment?

A. Continuous delivery requires that every change be deployed automatically to production.

B. Continuous delivery keeps changes releasable, while continuous deployment automatically releases approved changes.

C. Continuous delivery applies only to infrastructure, while continuous deployment applies only to applications.

D. Continuous delivery eliminates the need for automated testing.

Correct Answer: B. Continuous delivery keeps changes releasable, while continuous deployment automatically releases approved changes.

Question 2

What is the principal characteristic of a rolling deployment?

A. The old environment is always destroyed before the new environment is created.

B. Traffic is permanently divided equally between two environments.

C. Instances are replaced gradually, so old and new versions may coexist.

D. A feature is activated without deploying code.

Correct Answer: C. Instances are replaced gradually, so old and new versions may coexist.

Question 3

Which strategy provides a separate candidate environment and permits a relatively direct traffic switch?

A. Blue-green deployment

B. Recreate deployment

C. Feature-flag deployment

D. Dependency scanning

Correct Answer: A. Blue-green deployment

Question 4

Why is an immutable artifact preferable to a mutable tag such as latest for production promotion?

A. It guarantees that the application contains no defects.

B. It identifies a specific build and improves traceability and reproducibility.

C. It removes the need for access control in the registry.

D. It automatically performs database backups.

Correct Answer: B. It identifies a specific build and improves traceability and reproducibility.

Question 5

What is the main purpose of a canary deployment?

A. To expose a new version to a limited population before wider release.

B. To prevent any two application versions from coexisting.

C. To replace automated tests with manual approval.

D. To modify production servers directly.

Correct Answer: A. To expose a new version to a limited population before wider release.

Question 6

Which database migration sequence best supports coexistence of old and new application versions?

A. Delete old columns, deploy new code, then restore data if necessary.

B. Deploy new code first, then add the columns it requires.

C. Add compatible structures, migrate usage or data, then remove obsolete structures later.

D. Stop all services until the entire database is redesigned.

Correct Answer: C. Add compatible structures, migrate usage or data, then remove obsolete structures later.

Question 7

In GitOps, what is normally treated as the authoritative desired state?

A. An undocumented operator command history

B. A version-controlled declarative repository

C. A developer's local workstation

D. A mutable container running in production

Correct Answer: B. A version-controlled declarative repository

Question 8

Which item is a business-level deployment signal?

A. CPU utilisation

B. Container restart count

C. Payment completion rate

D. Memory allocation

Correct Answer: C. Payment completion rate

Question 9

What should a progressive pipeline generally do when its monitoring system becomes unavailable?

A. Promote immediately because no failures are visible.

B. Pause or fail safely until adequate evidence is available.

C. Delete the previous artifact.

D. Disable all security controls.

Correct Answer: B. Pause or fail safely until adequate evidence is available.

Question 10

Which practice most directly protects a delivery pipeline from excessive cloud-account access?

A. Using one permanent administrator key for every stage

B. Storing credentials in source code for convenience

C. Applying least privilege and short-lived workload credentials

D. Disabling audit logs to reduce pipeline latency

Correct Answer: C. Applying least privilege and short-lived workload credentials