Continuous Delivery and Deployment Strategies

Introduction

Modern software users expect frequent improvements, rapid security fixes, and dependable service. DevOps addresses this expectation by connecting development, testing, operations, security, and business decision-making through automation and shared responsibility. A central mechanism is the delivery pipeline: an automated path that transforms a version-controlled change into a tested, approved, observable release.

Continuous delivery and continuous deployment are related but distinct practices. Both seek to reduce manual effort and shorten feedback cycles. Continuous delivery ensures that every change that passes the required checks is ready for release, while continuous deployment automatically releases qualifying changes to production. The distinction is operational and governance-related rather than merely technical.

This chapter examines pipeline construction, environment promotion, approvals, feature flags, rollback, and major deployment strategies. It also considers reliability, security, compliance, cost, and organisational implications.

Learning Objectives

By the end of this chapter, you should be able to:

1. Distinguish continuous integration, continuous delivery, continuous deployment, and release management.
2. Construct a delivery pipeline with quality gates, environment promotion, approvals, and evidence.
3. Explain deployment strategies including rolling, blue-green, canary, and feature-flag-based release.
4. Select an appropriate strategy according to risk, architecture, traffic, and organisational constraints.
5. Design rollback and recovery procedures and evaluate their limitations.
6. Apply these principles to a realistic cloud-based service.

1. From Code Change to Production Service

A software change passes through several logically different activities:

• Build: source code is compiled or packaged, dependencies are resolved, and an immutable artifact is produced.
• Verification: automated tests, static analysis, security scans, and policy checks evaluate the artifact.
• Release preparation: configuration, infrastructure definitions, deployment manifests, and documentation are assembled.
• Deployment: the artifact is installed in a target environment.
• Release activation: traffic or users are enabled for the new version.
• Operation and observation: health, performance, security, and business outcomes are monitored.

These activities should be distinguished. A deployment can install software without exposing it to all users; conversely, a feature can be activated through configuration without deploying a new binary. Separating deployment from release gives teams more control over risk.

An artifact should be immutable: the same identified artifact is tested and promoted, rather than rebuilt differently for each environment. Environment-specific values such as database endpoints, credentials, and feature settings should be supplied through controlled configuration and secret-management systems.

Important terminology

• Pipeline: an automated sequence of activities, conditions, and controls that moves a change toward operation.
• Artifact: a versioned deployable output, such as a container image, package, or serverless bundle.
• Environment: an operational context such as development, test, staging, or production.
• Promotion: moving an already verified artifact to a subsequent environment.
• Quality gate: a condition that must be satisfied before the pipeline proceeds.
• Release: making a capability available to users, whether by deployment, configuration, or both.
• Rollback: returning service to a previously known working state.
• Roll-forward: correcting a problem by deploying a new version rather than reverting.

2. Continuous Delivery and Continuous Deployment

Continuous integration (CI) means that developers frequently merge changes into a shared repository and validate them automatically. CI primarily addresses integration risk.

Continuous delivery (CD) means that software is maintained in a releasable condition. Every change that passes the pipeline can be deployed to production through a reliable, usually automated, process. A human may still approve the production release.

Continuous deployment goes further: qualifying changes are automatically deployed to production without a routine human approval step. Automated tests, policies, observability, and recovery mechanisms therefore carry greater responsibility.

Continuous deployment is not automatically superior. A regulated medical, financial, or safety-critical system may require separation of duties, documented approval, or a scheduled release window. Continuous delivery can provide strong automation while preserving a deliberate approval decision.

A mature pipeline treats production access as a controlled capability. It records who or what initiated a release, which artifact was used, which checks passed, what configuration was applied, and what happened after release.

3. Constructing a Delivery Pipeline

A practical pipeline commonly contains the following stages:

Figure: A delivery pipeline combines automated verification, controlled promotion, release, and operational feedback.

3.1 Source and build controls

A pipeline should begin from a traceable commit or pull request. Branch protection, peer review, signed commits where appropriate, and required automated checks reduce the probability of unreviewed changes. The build should pin dependency versions, generate a unique artifact identifier, and record its provenance. Dependency caching may improve speed, but caches must not allow stale or untrusted content to bypass verification.

3.2 Automated verification

Verification should be layered. Fast unit tests provide immediate feedback; integration and contract tests examine interactions; end-to-end tests validate important user journeys. Static analysis, dependency scanning, container scanning, infrastructure-as-code checks, and secret detection address additional risks.

No test suite proves that software is defect-free. A quality gate is a risk decision, not a guarantee. Teams should monitor flaky tests, test duration, escaped defects, and coverage of critical behaviour rather than treating code coverage as a complete quality measure.

3.3 Infrastructure and environment validation

Infrastructure should be described declaratively where practical. A pipeline can validate infrastructure plans, policy rules, network boundaries, identity permissions, and resource quotas before applying changes. Infrastructure changes deserve the same review discipline as application code.

Test environments should resemble production in relevant ways, especially runtime versions, data access patterns, authentication, observability, and networking. Perfect duplication may be economically impractical, so teams should identify and document meaningful differences.

3.4 Quality gates and approvals

A gate may require successful tests, no critical vulnerabilities, a valid change record, acceptable performance, or an approved separation-of-duties decision. Gates should be explicit and reproducible. Manual approvals should not become informal testing or an unexplained queue; they should address a specific risk or governance requirement.

A useful approval record identifies the artifact, evidence reviewed, approver, timestamp, scope, and expiration if relevant. Emergency changes should use an expedited but auditable path rather than bypassing all controls.

4. Environment Promotion

A common promotion path is development → test → staging → production. The exact names are less important than the purpose of each environment.

• Development supports rapid feedback and experimentation.
• Test or integration validates service interactions and automated acceptance checks.
• Staging or pre-production provides a production-like rehearsal, including deployment, migration, monitoring, and recovery procedures.
• Production serves real users and requires the strongest access, observability, and change controls.

Promotion should normally move the same artifact, not a source revision rebuilt at every step. Configuration is environment-specific but should be versioned, reviewed, and separated from secrets. Secrets must be injected through a secure secret manager and never committed to source control or printed in logs.

Data management is a frequent obstacle. Production data may contain personal or confidential information and should not be copied casually into non-production environments. Masking, tokenisation, synthetic data, retention controls, and least-privilege access help reduce exposure.

Database changes require special care because application rollback does not automatically reverse a schema migration. A backward-compatible expansion-and-contraction approach is often safer:

1. Add a compatible schema element.
2. Deploy code that can work with both old and new forms.
3. Migrate or backfill data gradually.
4. Switch reads and writes.
5. Remove obsolete elements only after the old version is no longer needed.

5. Release Controls and Feature Flags

A feature flag is a controlled runtime decision that enables or disables behaviour. Flags can support dark launches, user segmentation, experimentation, emergency deactivation, and separation of deployment from release.

Flags should have an owner, purpose, default state, evaluation rules, audit history, and planned removal date. Long-lived flags increase complexity and can create untested combinations. Sensitive flags must be protected by access control and change logging. A flag is not a substitute for secure authorization: users must still be checked against proper identity and permission rules.

Feature flags are especially useful when a team wants to deploy code before exposing it. For example, a new recommendation engine may be installed for all instances but enabled only for internal users, then for 1% of customers, and finally for all customers after evidence is reviewed.

Release decisions should use both technical and business indicators. Technical indicators include error rate, latency, saturation, and availability. Business indicators may include successful transactions, conversion, cancellation, or support contacts. A change that passes unit tests but reduces successful payments is not a successful release.

6. Deployment Strategies

6.1 Recreate deployment

A recreate deployment stops the old version and starts the new version. It is simple and inexpensive but creates downtime and gives little opportunity for comparison. It may be acceptable for a low-use internal service or a system whose architecture cannot support coexistence.

6.2 Rolling deployment

A rolling deployment replaces instances gradually. During the transition, old and new versions coexist. The deployment controller should wait for readiness and health checks before replacing more instances. Parameters include batch size, pause duration, maximum unavailable capacity, and rollback conditions.

Advantages include lower additional infrastructure cost and generally low downtime. Risks include version incompatibility, mixed-version behaviour, and partial failure. APIs and messages should therefore remain backward compatible during the transition.

6.3 Blue-green deployment

Blue-green deployment maintains two production-capable environments. One environment, such as blue, serves users while the other, green, receives the new version. After validation, traffic is switched to green. The former environment remains available for rapid traffic reversal, subject to data compatibility and infrastructure cost.

Blue-green deployment offers a clear cutover and a straightforward application rollback. It may be difficult when databases, long-running sessions, caches, or external side effects cannot support two versions. Traffic switching must be tested; simply changing a load-balancer rule does not undo data already written by the new release.

6.4 Canary deployment

A canary deployment sends a small proportion of traffic to the new version, observes results, and expands exposure in stages. A canary may be selected by random traffic, geography, tenant, device type, or internal-user status. The selection must avoid unfairly or dangerously concentrating risk on a vulnerable group.

Canary analysis compares the new version with a baseline using predefined indicators and an observation window. A small sample can miss rare failures, while a long window delays delivery. Automated rollback should be based on meaningful thresholds and sufficient data, not on a single noisy measurement.

6.5 Feature-flag release

Feature-flag release can expose a capability independently of infrastructure deployment. It supports gradual activation and rapid disablement, but it does not eliminate defects in shared code paths, data migrations, or resource consumption. Flags should be combined with monitoring and a tested operational response.

7. Rollback, Roll-Forward, and Recovery

Rollback is a planned operational capability, not merely a button. A recovery plan should define the trigger, decision authority, traffic action, data implications, communication path, and verification steps.

Rollback is usually easiest when the change is stateless and the previous artifact remains available. It becomes difficult when the release has changed schemas, queues, external contracts, billing effects, or irreversible data. For such systems, backward-compatible migrations, idempotent operations, compensating actions, and roll-forward fixes may be safer.

A basic recovery sequence is:

1. Detect the deviation through alerts or user reports.
2. Confirm whether the new release is causally related.
3. Stop further promotion and preserve evidence.
4. Reduce exposure, disable the feature, shift traffic, or deploy a fix.
5. Verify service health and business outcomes.
6. Communicate status and document the incident.
7. Conduct a blameless review and improve controls.

Health checks should test meaningful readiness, not merely whether a process is running. Observability should include logs, metrics, traces, deployment markers, and correlation identifiers. Alerts need owners and actionable thresholds.

Useful delivery measures include deployment frequency, lead time for changes, change failure rate, and time to restore service. These measures should be interpreted together. Increasing deployment frequency while reliability declines is not improvement.

8. Choosing a Strategy

Strategy selection should consider:

• Criticality: harm from failure, downtime, or incorrect transactions.
• Reversibility: whether traffic, code, and data can be safely returned.
• Architecture: statelessness, session handling, service contracts, and database coupling.
• Traffic control: ability to route users by percentage, tenant, region, or header.
• Observability: quality and speed of feedback.
• Capacity and cost: ability to run duplicate environments or spare instances.
• Compliance: approval, evidence, residency, audit, and separation-of-duties requirements.
• Team capability: operational experience and on-call readiness.

A low-risk stateless service may use rolling deployment. A high-impact customer-facing service with strong routing and monitoring may use canary or blue-green deployment. A service with incompatible schema changes may require staged migrations before any traffic strategy is safe.

9. Illustrative Case: CloudPay Checkout Service

Background

CloudPay is an illustrative online payment platform running containerised services in a public cloud. Its checkout service processes customer payments and communicates with inventory, fraud detection, and a bank gateway. Releases previously occurred monthly through a manual procedure.

Problem

A rushed release caused elevated payment failures. Operators discovered that the deployment package used in production was not identical to the package tested in staging. Database changes also required manual intervention. The organisation wanted more frequent releases without increasing financial or regulatory risk.

Analysis

The main risks were artifact inconsistency, insufficient contract testing, weak observability, and a rollback plan that ignored database state. A simple recreate deployment was unacceptable because payment availability was a business requirement. Automatic production deployment was also initially inappropriate because evidence and approval controls had to be strengthened.

Approach and solution

CloudPay implemented the following design:

1. Pull requests triggered unit, integration, API-contract, dependency, secret, and infrastructure-policy checks.
2. A single signed container artifact was produced and stored in a registry.
3. The same artifact was promoted to test and staging; only configuration and secrets differed.
4. A staging rehearsal tested deployment, health checks, dashboards, alerts, and database migration.
5. Database changes followed an expand-and-contract pattern.
6. Production used a canary deployment beginning with internal transactions and a small controlled customer segment.
7. Error rate, payment completion, gateway latency, and fraud-service response were monitored against the baseline.
8. A feature flag controlled the new checkout calculation, with an owner and expiry date.
9. The release required an authorised approval because of payment and audit requirements.
10. If thresholds were exceeded, traffic was shifted back, the feature was disabled, and the incident process began.

Outcome

The process reduced the size of each change and improved traceability. The organisation retained a human approval for production while automating nearly all preparation and evidence collection. It did not assume that rollback alone could repair a data problem; compatibility and migration safety were treated as design requirements.

Learning points

The case demonstrates that deployment strategy cannot compensate for poor architecture or absent observability. Continuous delivery is achievable even where continuous deployment is restricted. The strongest design combines immutable artifacts, staged evidence, controlled exposure, reversible activation, and rehearsed recovery.

10. Governance, Security, and Responsible Automation

Pipelines are part of the organisation’s control environment. Pipeline identities should follow least privilege, use short-lived credentials where possible, and separate duties for code review, pipeline administration, and production approval. Build systems and artifact registries are high-value targets and should be protected with access control, logging, isolation, and vulnerability management.

Audit records should be retained according to legal and organisational requirements. They should support reconstruction of what changed, who authorised it, which artifact ran, and what configuration was used. Privacy principles apply to logs, test data, customer segmentation, and monitoring identifiers.

Automation must fail safely. A broken approval service should not silently approve a high-risk change. Automated rollback should not repeatedly oscillate between versions or destroy diagnostic evidence. Policy-as-code can improve consistency, but policies require review because an incorrect policy can block safe work or permit unsafe work at scale.

Artificial intelligence can assist with test generation, anomaly detection, log summarisation, and change-risk analysis. These tools should be treated as advisory unless their outputs are independently validated. Generated tests may reproduce existing assumptions, anomaly models may produce false alarms, and confidential source or operational data must not be sent to an unauthorised service. Human accountability remains essential for consequential release decisions.

11. Best Practices and Common Failure Modes

Best practices

• Build once and promote the identified artifact.
• Keep pipeline definitions and infrastructure under version control.
• Make quality gates explicit, fast where possible, and risk-based.
• Use production-like testing for critical paths.
• Design APIs and schemas for compatibility during transition.
• Define rollback triggers before deployment begins.
• Monitor technical and business outcomes.
• Give feature flags owners, expiry dates, and audit trails.
• Practise recovery through controlled exercises.
• Keep production access least-privileged and auditable.

Common failure modes

• Treating a manual checklist as a substitute for automated evidence.
• Rebuilding artifacts separately for staging and production.
• Using code coverage as the sole definition of quality.
• Performing canary releases without sufficient traffic or baseline comparison.
• Calling a process blue-green when the old environment cannot actually receive traffic.
• Assuming application rollback reverses database or external side effects.
• Accumulating feature flags until combinations become untestable.
• Suppressing alerts rather than improving noisy detection.
• Allowing emergency procedures to bypass all accountability.

Chapter Summary

Continuous delivery keeps every verified change ready for production, whereas continuous deployment automatically releases qualifying changes. A dependable pipeline builds an immutable artifact, verifies it through layered tests and security checks, promotes it through controlled environments, records evidence, and observes the result.

Environment promotion should use the same artifact with controlled configuration and secure secrets. Approvals are most valuable when they address explicit governance or risk requirements. Feature flags separate deployment from release but require ownership and removal discipline. Rolling, blue-green, canary, and recreate strategies offer different balances of cost, exposure, reversibility, and complexity. Rollback must account for databases, messages, external effects, and evidence preservation; sometimes a roll-forward fix is safer.

The appropriate strategy depends on criticality, architecture, observability, traffic control, cost, compliance, and team capability. Professional DevOps practice combines automation with security, auditability, responsible decision-making, and rehearsed recovery.

Key Terms

Continuous integration: Frequent integration of changes with automated verification.

Continuous delivery: Maintaining software in a state that can be released reliably at any time.

Continuous deployment: Automatically releasing qualifying changes to production.

Pipeline: An automated sequence that builds, verifies, promotes, and releases software.

Artifact: A versioned deployable software output.

Immutable artifact: An artifact that is not modified after creation.

Environment promotion: Moving a verified artifact to a later environment.

Quality gate: A condition that must be satisfied before pipeline progression.

Approval: An authorised decision allowing a controlled action to proceed.

Feature flag: Runtime configuration that controls whether behaviour is active.

Rollback: Returning service to an earlier known state.

Roll-forward: Correcting a fault by deploying a subsequent version.

Rolling deployment: Gradually replacing instances with a new version.

Blue-green deployment: Switching traffic between two production-capable environments.

Canary deployment: Gradually exposing a new version to a limited audience or traffic share.

Readiness check: A test indicating that an instance can receive traffic.

Observability: The ability to understand system state through metrics, logs, and traces.

Expand-and-contract migration: A compatible staged approach to changing data schemas.

Change failure rate: The proportion of deployments that require remediation or cause failure.

Policy as code: Machine-readable rules that automatically enforce governance conditions.

Review Questions

5 Short Answer Questions

1. Define continuous delivery and explain how it differs from continuous deployment.
2. Why should a pipeline promote the same artifact through environments?
3. State two advantages and two risks of a rolling deployment.
4. What is the purpose of a feature flag?
5. Why can a database migration make rollback difficult?

5 Long Answer Questions

1. Describe the stages of a secure, observable delivery pipeline from source commit to production.
2. Compare blue-green and canary deployment with respect to traffic control, cost, risk, and rollback.
3. Explain how environment promotion, approvals, and audit evidence support governance.
4. Discuss why automated tests cannot by themselves guarantee a safe production release.
5. Explain the expand-and-contract approach and its value in a continuously delivered system.

3 Analytical / Application Questions

1. A customer-facing service has high availability requirements, supports percentage-based traffic routing, and has strong monitoring. Recommend a deployment strategy and justify your choice. Identify two conditions that could invalidate your recommendation.
2. A release passes all automated tests, but payment completion falls by 8% during a canary. Design the immediate response, evidence collection, and follow-up investigation.
3. An organisation wants continuous deployment for a regulated financial service. Propose a pipeline control model that preserves automation while satisfying approval, separation-of-duties, audit, and recovery requirements.

Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes continuous delivery from continuous deployment?

A. Continuous delivery excludes automated testing, while continuous deployment requires it.

B. Continuous delivery keeps software releasable, while continuous deployment automatically releases qualifying changes.

C. Continuous delivery applies only to infrastructure, while continuous deployment applies only to applications.

D. Continuous delivery requires production downtime, while continuous deployment prevents all downtime.

Correct Answer: B. Continuous delivery keeps software releasable, while continuous deployment automatically releases qualifying changes.

Question 2

Why is promoting the same artifact through environments a recommended practice?

A. It ensures the tested output is the output released to production.

B. It allows each environment to compile different source code.

C. It removes the need for environment-specific configuration.

D. It guarantees that the artifact contains no defects.

Correct Answer: A. It ensures the tested output is the output released to production.

Question 3

Which deployment strategy gradually replaces instances while old and new versions coexist?

A. Recreate

B. Blue-green

C. Rolling

D. Manual release

Correct Answer: C. Rolling

Question 4

What is the principal purpose of a canary deployment?

A. To expose a new version to limited traffic and evaluate evidence before wider release.

B. To eliminate the need for monitoring.

C. To guarantee that database changes are reversible.

D. To deploy without producing an artifact.

Correct Answer: A. To expose a new version to limited traffic and evaluate evidence before wider release.

Question 5

Which statement about feature flags is correct?

A. They replace authentication and authorization controls.

B. They can separate deployment from release but require ownership and lifecycle management.

C. They make all database migrations automatically safe.

D. They remove the need for automated tests.

Correct Answer: B. They can separate deployment from release but require ownership and lifecycle management.

Question 6

Which situation most strongly supports blue-green deployment?

A. A system cannot run two versions at once and has no traffic router.

B. A service can switch traffic between two compatible, production-capable environments.

C. A service requires a complete shutdown during every release.

D. A team has no monitoring or health checks.

Correct Answer: B. A service can switch traffic between two compatible, production-capable environments.

Question 7

Why may application rollback fail to restore a system fully?

A. Previous artifacts are always deleted immediately.

B. Rollback cannot affect application binaries.

C. Database changes or external side effects may not be reversible by changing the application version.

D. Traffic routing is never involved in deployment.

Correct Answer: C. Database changes or external side effects may not be reversible by changing the application version.

Question 8

Which practice best protects secrets in a delivery pipeline?

A. Committing secrets to a private source repository.

B. Printing secrets in deployment logs for traceability.

C. Storing and injecting secrets through a controlled secret-management system.

D. Embedding one shared password in every artifact.

Correct Answer: C. Storing and injecting secrets through a controlled secret-management system.

Question 9

In a staged database migration, what is the purpose of the expansion phase?

A. To remove all fields used by the previous version immediately.

B. To add compatible structures so old and new application versions can coexist.

C. To disable monitoring during deployment.

D. To prevent any application deployment until the database is empty.

Correct Answer: B. To add compatible structures so old and new application versions can coexist.

Question 10

Which is an appropriate reason to retain a human production approval?

A. To compensate for having no automated tests at all.

B. To satisfy a defined governance or regulatory control requiring an authorised decision.

C. To make the pipeline slower in every circumstance.

D. To prevent the use of immutable artifacts.

Correct Answer: B. To satisfy a defined governance or regulatory control requiring an authorised decision.