Continuous Delivery, Deployment and Release Management

Introduction and Context

Continuous integration (CI) verifies that changes can be integrated into a shared codebase. Continuous delivery (CD) extends this practice by keeping software in a releasable state through automated build, test, security, packaging and deployment processes. Continuous deployment goes further: every change that satisfies the required controls is released automatically to production. Release management governs how a particular version is introduced, observed, communicated, supported and, when necessary, reversed.

These practices are central to DevOps because they connect development activity with operational outcomes. A pipeline is not merely a sequence of scripts. It is a controlled flow of evidence, artifacts, approvals and deployment decisions. Effective CD reduces manual error and lead time while preserving safety through testing, policy enforcement, progressive exposure and rapid recovery.

This chapter explains how to design multi-stage CD pipelines, promote immutable artifacts, use approvals and feature flags, and select suitable rolling, blue-green and canary deployment strategies.

Learning Objectives

By the end of this chapter, you should be able to:

• distinguish continuous delivery, continuous deployment and release management;
• design a multi-stage CD pipeline with quality gates and approvals;
• explain artifact immutability, provenance and promotion;
• select and apply feature flags safely;
• compare rolling, blue-green and canary deployments;
• plan rollback and roll-forward procedures;
• define useful release metrics and governance controls; and
• evaluate a deployment approach in a realistic organizational scenario.

1. Continuous Delivery and Continuous Deployment

1.1 Three related concepts

Continuous integration means that developers frequently merge changes and receive rapid automated feedback. Continuous delivery means that validated changes are automatically prepared for release and can be deployed to production at any time. A human decision may still be required before production release. Continuous deployment removes that routine manual release decision: validated changes proceed automatically, subject to automated policies.

The distinction is important. An organization may have an automated pipeline but still practice continuous delivery rather than continuous deployment because a product owner or operations approver authorizes production release. Neither approach is universally superior. Regulated systems, high-risk changes and organizations with immature observability may appropriately retain explicit approval gates.

1.2 Deployment versus release

A deployment places software in an environment. A release makes functionality available to users. These events can be separated. For example, a new service version may be deployed with a feature disabled, and the feature may later be released through a feature flag. This separation allows technical deployment and business exposure to be controlled independently.

Release management therefore includes:

• release planning and scope definition;
• dependency and risk assessment;
• artifact identification;
• environment promotion;
• communication and change records;
• feature exposure decisions;
• monitoring and acceptance; and
• rollback, roll-forward or retirement.

2. The Multi-Stage CD Pipeline

A robust pipeline progressively increases confidence. Early stages are fast and inexpensive; later stages are more realistic and consequential. A typical design is:

1. Source and trigger – a merge, tag, scheduled event or approved manual action starts the pipeline.
2. Build – source is compiled or packaged, dependencies are resolved and an immutable version is produced.
3. Unit and static checks – unit tests, linting, code analysis and policy checks run.
4. Integration and contract tests – components and service interfaces are tested together.
5. Artifact publication – the artifact and metadata are stored in a trusted repository.
6. Development deployment – the exact artifact is deployed to a development environment.
7. System and security tests – end-to-end, performance, vulnerability and configuration tests execute.
8. Staging deployment – production-like infrastructure and data-handling conditions are used.
9. Approval or policy gate – automated thresholds and, where required, human approval are evaluated.
10. Production deployment – a selected deployment strategy exposes the artifact progressively or immediately.
11. Verification and closure – health checks, business metrics and audit records confirm the result.

A pipeline should fail safely. If a required test or policy check fails, promotion stops and the failure is visible to the responsible team. A pipeline that allows teams to bypass failed controls without documented authority is not a reliable control system.

2.1 Quality gates

A quality gate is a condition that must be satisfied before a pipeline proceeds. Examples include:

• all mandatory tests pass;
• no critical security vulnerability is present;
• code coverage meets an agreed threshold, where coverage is meaningful;
• an artifact is signed and traceable to source;
• infrastructure policy checks pass;
• database migration compatibility is verified;
• service-level indicators remain within limits; and
• required approvals are recorded.

Thresholds must be meaningful rather than ceremonial. A high coverage percentage does not prove that tests are effective, and a green pipeline does not prove that a system is safe if important risks were never tested.

2.2 Approvals and separation of duties

Approvals should be explicit, attributable and connected to evidence. A production approval record should identify the artifact version, test results, change scope, risk assessment, approver, time and any conditions. Approval should not mean that a person manually repeats automated testing. It should mean that an authorized person accepts the residual risk and confirms operational readiness.

Separation of duties may require that the person who writes a change cannot unilaterally approve its production release. This is common in regulated or safety-critical environments. In lower-risk environments, automated policy gates and peer review may provide sufficient control. The appropriate design depends on risk, regulation and organizational capability.

3. Artifacts, Promotion and Supply-Chain Trust

An artifact is a deployable output such as a container image, package, binary, chart or serverless bundle. A strong CD system builds an artifact once and promotes that same artifact through environments. Rebuilding separately for development, staging and production can introduce differences and undermine test evidence.

3.1 Immutability

An immutable artifact cannot be silently changed after publication. It should be identified by a unique version, digest or content address rather than only by a mutable label such as latest. Configuration that varies by environment should be injected at deployment time, while the application artifact remains unchanged.

A useful provenance record links:

• source commit or tag;
• build pipeline and builder identity;
• dependency versions;
• test and scan results;
• artifact digest;
• signing information;
• deployment environments; and
• approvals and release timestamps.

This evidence supports incident investigation and software supply-chain security.

3.2 Promotion model

Promotion is the controlled movement of an artifact reference from one environment or repository stage to another. Promotion should not mean copying an unverified binary manually. It should invoke policy checks and preserve the artifact's identity.

A simplified model is:

Source commit
     ↓
Build once → Test and scan → Publish immutable artifact
                                      ↓
                         Development → Staging → Production
                                      ↑              ↑
                              automated gates   approval/policy

Figure: A build-once, promote-the-same-artifact CD flow.

Promotion also applies to infrastructure definitions, database migration packages and configuration bundles. Compatibility must be considered across all of them, not only the application binary.

3.3 Configuration and secrets

Environment-specific configuration should be externally managed, versioned where appropriate and validated before deployment. Secrets should be stored in a dedicated secrets-management system, accessed through least-privilege identities and never embedded in source code or ordinary logs. A pipeline should prevent accidental publication of credentials and should rotate compromised secrets promptly.

4. Release Strategies

A release strategy determines how much infrastructure and user traffic is exposed to a new version, how quickly exposure increases and how recovery occurs.

4.1 Recreate deployment

A recreate deployment stops the old version and starts the new version. It is simple but produces downtime and is unsuitable when continuous availability is required. It can be acceptable for internal tools or planned maintenance windows.

4.2 Rolling deployment

A rolling deployment replaces instances incrementally. During the transition, old and new versions usually coexist. The system must therefore tolerate mixed-version operation.

A rolling deployment requires:

• readiness checks that represent actual service readiness;
• capacity planning so available capacity remains adequate;
• connection draining for old instances;
• backward-compatible APIs and data changes;
• a defined maximum unavailable capacity; and
• monitoring throughout each replacement batch.

Its advantages are efficient infrastructure use and gradual change. Its risks include prolonged mixed-version states and more complicated rollback when database schemas or message formats are incompatible.

4.3 Blue-green deployment

Blue-green deployment maintains two production-capable environments. The active environment, say blue, serves users while the inactive green environment receives the new version. After validation, traffic is switched to green. If a problem appears and the old environment remains healthy, traffic can be switched back to blue.

Advantages include a clear cutover and rapid application rollback. Disadvantages include the cost of maintaining duplicate capacity and the difficulty of reversing irreversible data changes. A traffic switch may be implemented using a load balancer, service mesh, DNS, gateway or platform routing mechanism. DNS-based switching can be affected by caching and therefore may not provide an immediate universal cutover.

4.4 Canary deployment

A canary deployment exposes the new version to a small subset of traffic, hosts or users before wider rollout. Exposure may progress from 1% to 5%, 25%, 50% and finally 100%, but percentages should reflect capacity, risk and statistical usefulness rather than a fixed recipe.

Canary analysis compares the candidate with the stable version using technical and business indicators, such as:

• error rate;
• latency percentiles;
• saturation and resource consumption;
• conversion or transaction success;
• support contacts; and
• user-specific failure patterns.

A canary is not safe merely because it is small. The selected traffic must be representative, and monitoring must be capable of detecting harm. A canary involving only internal employees may miss problems affecting external customers or particular regions.

4.5 Strategy comparison

5. Feature Flags and Progressive Delivery

A feature flag is a runtime-controlled decision that determines whether functionality is enabled. Flags can target all users, a percentage, a region, a customer group or an experiment cohort. They make it possible to deploy code without immediately releasing its behavior.

5.1 Types of flags

• Release flags hide incomplete functionality until it is ready.
• Operational flags disable an expensive or unstable capability.
• Permission flags control access by role or entitlement.
• Experiment flags support controlled comparison of alternatives.
• Kill switches provide a rapid way to disable harmful behavior.

Flags add control but also add system complexity. They should have an owner, purpose, default behavior, expiry date and audit trail. Temporary release flags should be removed after rollout; otherwise the codebase accumulates decision paths that are difficult to test.

5.2 Safe flag design

The default behavior must be defined for failure of the flag service. A critical service should not become unavailable because a remote flag provider cannot be reached. Applications may cache flag values, use secure local defaults or fail closed when security requires it.

Flags must not be used to bypass authorization. A user interface flag can hide a button, but server-side authorization must still enforce permission. Sensitive data should not be delivered to an unauthorized client merely because a display flag is off.

6. Rollback, Roll-Forward and Recovery

Rollback returns a system to an earlier known-good version. Roll-forward fixes the defect by deploying a new version. Rollback is often fast for stateless application code, but it may be unsafe after irreversible database changes, data transformations or external side effects.

A recovery plan should specify:

• the rollback trigger and decision authority;
• the last known-good artifact;
• traffic or instance reversal steps;
• database and message compatibility requirements;
• treatment of in-flight requests and queued messages;
• customer and stakeholder communication;
• evidence preservation; and
• post-incident review.

6.1 Expand-and-contract database changes

A compatible database migration commonly follows three phases:

1. Expand – add new columns, tables or indexes without removing the old structure.
2. Migrate and transition – deploy code that can read or write both representations, backfill data and switch behavior gradually.
3. Contract – remove obsolete structures only after all old application versions are gone and recovery risk is acceptable.

This approach supports rolling and blue-green deployment better than a single destructive migration. Backups are necessary but do not replace tested restoration procedures.

6.2 Recovery objectives

Operational planning often distinguishes:

• Recovery time objective (RTO): maximum acceptable time to restore service.
• Recovery point objective (RPO): maximum acceptable amount of data loss measured in time.

These are business requirements, not properties automatically guaranteed by a pipeline. A rapid application rollback may satisfy RTO while failing to restore data consistency; both must be assessed.

7. Observability and Release Metrics

A CD pipeline should measure both delivery performance and service outcomes. Useful delivery metrics include deployment frequency, lead time for changes, change failure rate and time to restore service. These metrics should encourage reliable improvement, not reward unsafe speed.

Service-level indicators may include availability, latency, error rate and successful transaction rate. During progressive delivery, compare the candidate with the baseline over an appropriate period and traffic volume. Define thresholds before exposure begins to reduce biased decision-making.

A practical release dashboard should show:

• artifact and commit identity;
• deployment stage and exposure percentage;
• health-check status;
• error and latency trends;
• saturation indicators;
• business success measures;
• active feature flags; and
• current rollback action.

Logs, metrics and traces should be correlated with release identifiers. Monitoring must respect privacy: avoid collecting unnecessary personal information, restrict access and define retention periods.

8. Illustrative Case: Online Retail Checkout Service

Background

The fictional company Northstar Retail operates an online checkout service. Releases occur weekly, but production incidents are frequent because the team manually rebuilds packages for each environment. The service has a payment API dependency and a database containing orders.

Problem

A new discount-calculation feature is ready. The company needs to release it without disrupting payments. A direct replacement is considered too risky, while a permanent duplicate production environment is expensive.

Analysis

The team identifies several risks:

• a rebuild could produce a different artifact from the one tested;
• the discount code may increase checkout latency;
• old and new application versions may coexist during rollout;
• payment requests must not be duplicated during retries; and
• an immediate database-column removal would prevent rollback.

Approach and solution

The team designs the following pipeline:

1. A merge request triggers unit, integration, dependency and static-security checks.
2. The pipeline builds one container image and records its digest and source commit.
3. The image is published to the registry and promoted unchanged to development and staging.
4. Contract tests verify compatibility with the payment provider.
5. A staging test uses production-like traffic patterns and checks checkout latency.
6. The database change follows expand-and-contract design: a nullable discount field is added without removing existing fields.
7. A product owner and operations representative approve production exposure after reviewing evidence.
8. The image is deployed by a rolling strategy with readiness checks and connection draining.
9. The discount feature remains disabled by a release flag.
10. The flag is enabled for internal users, then 5% of representative customers, while dashboards compare error rate, latency and successful payment completion with the stable cohort.
11. If thresholds are exceeded, the flag is disabled first. If the application itself is unhealthy, traffic is returned to the previous version and the incident process begins.
12. After a stable period, exposure increases and the old flag-controlled code is scheduled for removal.

Outcome

The organization separates deployment from release, avoids rebuilding artifacts, limits customer exposure and preserves a recovery path. It also discovers that a low error rate alone is insufficient: payment completion and latency are more relevant business indicators.

Learning points

The case demonstrates that safe delivery depends on the interaction of artifact integrity, compatibility, progressive exposure, observability and decision authority. No single deployment strategy eliminates risk. Risk is managed through layered controls.

9. Governance, Ethics and Professional Practice

Release systems can affect availability, financial transactions, safety and personal data. Governance should therefore be proportional to impact. High-impact releases may require documented risk assessment, independent approval, audit records and tested recovery procedures.

Ethically, teams should not use customers as uninformed test subjects for material risk. Canary and experimentation programs require clear limits, monitoring and, where applicable, consent or legally appropriate notice. Feature targeting must not create discriminatory outcomes or expose sensitive groups to disproportionate failure.

Access to pipelines and production systems should follow least privilege, multi-factor authentication and strong identity management. Logs must be protected against tampering, and approvals must not be backdated. Supply-chain controls should include dependency review, signed artifacts where supported, vulnerability response and careful use of third-party actions or plugins.

Artificial intelligence can assist with test generation, anomaly detection, log summarization and release-risk analysis. It should not be treated as an unquestionable approver. Generated tests may miss important scenarios, and automated recommendations can inherit biased or incomplete data. Human accountability, reproducible evidence and explicit policy remain necessary.

10. Best Practices and Common Failure Modes

Best practices

• Build once and promote the immutable artifact.
• Keep pipeline definitions and infrastructure configuration under version control.
• Make stages repeatable, observable and independently diagnosable.
• Use automated checks for routine evidence and human approval for residual-risk decisions.
• Design APIs, schemas and messages for compatibility during transition.
• Define rollback triggers before deployment begins.
• Monitor business as well as technical outcomes.
• Give every feature flag an owner and removal date.
• Test restoration and rollback, not merely deployment.
• Use least privilege and retain auditable evidence.

Common failure modes

Using mutable tags: The tag may point to a different image later, invalidating test evidence. Use immutable digests or unique versions.

Treating approval as a substitute for testing: A person cannot reliably inspect every technical property manually. Automate repeatable checks and use approval for risk acceptance.

Rolling out without observability: Progressive exposure only helps when failure can be detected quickly.

Ignoring mixed-version operation: Rolling deployments can expose incompatibilities in APIs, schemas and messages.

Keeping flags indefinitely: Permanent flags create hidden complexity and untested combinations.

Assuming rollback is always possible: Data mutations, external effects and incompatible migrations may make reversal unsafe.

Optimizing only for deployment frequency: Fast releases that increase incidents are not an improvement. Reliability and customer outcomes must be included.

Chapter Summary

Continuous delivery keeps validated software ready for production, while continuous deployment automatically releases changes that satisfy defined controls. Release management coordinates artifacts, approvals, exposure, monitoring and recovery.

A multi-stage pipeline should build an immutable artifact once, test it progressively, promote it with evidence and enforce quality, security and governance gates. Deployment and release can be separated through feature flags and progressive delivery. Rolling deployments use incremental replacement, blue-green deployments switch between complete environments, and canary deployments increase representative exposure gradually.

Rollback and roll-forward require compatibility planning, especially for databases and external side effects. Effective release engineering combines automation with observability, explicit decision rights, supply-chain integrity, privacy protection and tested recovery. The most reliable strategy is not the one with the fewest manual steps alone; it is the one that provides rapid, evidence-based change while limiting customer harm.

Key Terms

Continuous delivery: Practice of keeping validated software deployable and ready for release.

Continuous deployment: Practice of automatically releasing validated changes to production.

Release management: Planning, authorization, exposure, monitoring and recovery associated with a software release.

Artifact: A deployable output such as a container image, package or binary.

Artifact promotion: Controlled movement of an artifact between environments or repository stages.

Immutable artifact: An artifact that cannot be silently altered after publication.

Quality gate: A condition that must be satisfied before pipeline progression.

Approval gate: An explicit authorization required before a controlled action.

Feature flag: A runtime control that determines whether functionality is enabled.

Progressive delivery: Gradual exposure of a change using techniques such as flags or canaries.

Rolling deployment: Incremental replacement of running instances.

Blue-green deployment: Deployment using two production-capable environments and a traffic switch.

Canary deployment: Limited initial exposure followed by evidence-based expansion.

Rollback: Return to a previous known-good version or state.

Roll-forward: Correction through deployment of a subsequent version.

Readiness check: Test indicating that an instance can safely receive traffic.

Artifact provenance: Evidence connecting an artifact to its source, build and validation history.

RTO: Maximum acceptable time to restore service.

RPO: Maximum acceptable time-based amount of data loss.

Expand-and-contract migration: Compatibility-oriented process of adding new data structures, transitioning usage and later removing obsolete structures.

Review Questions

Short Answer Questions

1. Distinguish continuous delivery from continuous deployment.
2. Why should a CD pipeline build an artifact once and promote it unchanged?
3. What is the purpose of a quality gate?
4. State one advantage and one limitation of blue-green deployment.
5. Why should feature flags have owners and expiry dates?

Long Answer Questions

1. Explain the stages of a multi-stage CD pipeline and the evidence expected at each major transition.
2. Compare rolling, blue-green and canary deployments in terms of infrastructure, exposure, observability and rollback.
3. Explain how approvals and separation of duties support responsible production release.
4. Describe how expand-and-contract database migration supports deployment and rollback.
5. Discuss the relationship between artifact provenance, software supply-chain security and auditability.

Analytical / Application Questions

1. A team can deploy rapidly but has no reliable health checks or business metrics. Recommend changes before adopting continuous deployment and justify them.
2. During a canary release, technical error rates remain normal but payment completion falls for one customer region. What decisions and investigations should occur before expanding exposure?
3. Design a release approach for a service that requires database changes, must remain available, and has limited budget for duplicate infrastructure. Explain your choice of deployment strategy, feature-flag use and recovery plan.

Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes continuous delivery from continuous deployment?

A. Continuous delivery requires no automated testing, while continuous deployment does.

B. Continuous delivery keeps software ready for release, while continuous deployment automatically releases validated changes.

C. Continuous delivery applies only to infrastructure, while continuous deployment applies only to applications.

D. Continuous delivery always requires two production environments, while continuous deployment requires one.

Correct Answer: B. Continuous delivery keeps software ready for release, while continuous deployment automatically releases validated changes.

Question 2

What is the primary reason for building an artifact once and promoting the same artifact through environments?

A. It guarantees that all tests will pass.

B. It prevents the need for environment-specific configuration.

C. It reduces differences between the tested artifact and the production artifact.

D. It eliminates the need for production monitoring.

Correct Answer: C. It reduces differences between the tested artifact and the production artifact.

Question 3

Which deployment strategy uses two production-capable environments and switches traffic between them?

A. Rolling

B. Recreate

C. Canary

D. Blue-green

Correct Answer: D. Blue-green

Question 4

What is a major requirement for a safe rolling deployment?

A. All old instances must be permanently removed before any new instance starts.

B. The application must tolerate temporary coexistence of old and new versions.

C. No readiness checks are needed because replacement is gradual.

D. Database changes must always be destructive.

Correct Answer: B. The application must tolerate temporary coexistence of old and new versions.

Question 5

A feature flag primarily allows an organization to do what?

A. Replace authorization controls with user-interface logic

B. Separate deployment of code from exposure of functionality

C. Avoid testing the disabled code path

D. Guarantee that a failed release cannot affect users

Correct Answer: B. Separate deployment of code from exposure of functionality

Question 6

Which action is usually the safest first response when a newly enabled feature causes harmful behavior but the underlying application remains healthy?

A. Disable the feature flag

B. Delete the production database

C. Rebuild every environment with a different artifact

D. Disable all monitoring

Correct Answer: A. Disable the feature flag

Question 7

Which migration sequence best supports compatibility during a rolling deployment?

A. Delete old fields, deploy new code, then restore backups

B. Expand, transition and migrate, then contract after old versions are gone

C. Stop all services, perform a destructive migration, then test

D. Change the schema only after permanently removing the application

Correct Answer: B. Expand, transition and migrate, then contract after old versions are gone

Question 8

Which measure is most directly a business outcome for an online checkout canary?

A. Container image size

B. Number of pipeline YAML lines

C. Successful payment completion rate

D. Number of deployment stages

Correct Answer: C. Successful payment completion rate

Question 9

What does RTO describe?

A. The acceptable amount of data loss measured in time

B. The maximum acceptable time to restore service

C. The time required to compile a source repository

D. The percentage of users included in a canary

Correct Answer: B. The maximum acceptable time to restore service

Question 10

Why is a small canary percentage not automatically safe?

A. Small canaries always require duplicate databases.

B. A small selected group may not represent affected users, and inadequate monitoring may miss harm.

C. Canaries cannot use feature flags.

D. Canaries prevent rollback by definition.

Correct Answer: B. A small selected group may not represent affected users, and inadequate monitoring may miss harm.