Continuous Delivery and Deployment Strategies

Introduction

Continuous delivery and continuous deployment extend continuous integration beyond the successful build. They address a practical organisational question: how can a tested software change move safely from a developer’s workstation into the hands of users? In a mature DevOps environment, releasing software is not an occasional event performed through a checklist. It is a repeatable, observable, controlled flow through development, testing, staging, and production.

Continuous delivery means that software is kept in a releasable state. Every change passes automated validation and can be released through a controlled decision. Continuous deployment goes further: every change that satisfies the defined quality criteria is released automatically to production. The distinction is therefore primarily about the release decision, not about whether automation exists.

A release strategy must balance speed, reliability, safety, cost, compliance, and user experience. This chapter explains approval gates, configuration management, feature flags, deployment patterns, rollback and roll-forward decisions, and the operational controls required for responsible automation.

Learning Objectives

By the end of this chapter, you should be able to:

• distinguish continuous delivery from continuous deployment;
• design a promotion flow across development, testing, staging, and production;
• explain approvals, quality gates, configuration management, and feature flags;
• compare rolling, blue-green, canary, and recreate deployment strategies;
• select appropriate rollback or roll-forward responses;
• apply observability, security, governance, and audit controls to releases; and
• analyse a practical release scenario and recommend a deployment approach.

1. From Integration to Release

Continuous integration (CI) verifies that changes can be integrated into a shared codebase. A typical CI pipeline compiles or packages the software, runs unit tests, performs static analysis, and produces an immutable build artifact. Continuous delivery (CD) takes that artifact through further environments using repeatable automation.

A useful principle is build once, promote many times. The artifact tested in staging should be the same artifact deployed to production. Rebuilding separately for each environment can introduce differences and makes it difficult to establish what was actually tested. Environment-specific values should be supplied through configuration rather than embedded in different builds.

A simplified release flow is:

Commit → Build → Unit tests → Security checks → Package artifact
       → Development → Integration tests → Staging → Approval or policy gate
       → Production deployment → Verification → Monitoring

The pipeline should stop when a required control fails. A failed test, vulnerable dependency, unavailable environment, or unsuccessful health check should prevent promotion unless an authorised, documented exception is made.

Continuous delivery and continuous deployment

Continuous deployment does not mean uncontrolled deployment. It requires stricter automation, reliable rollback or mitigation, clear ownership, and trustworthy production signals.

2. Release Pipeline Architecture

A release pipeline is a sequence of automated and human-controlled activities. Its stages should reflect risk rather than merely organisational departments.

Development environment

Development environments support rapid feedback. They may be ephemeral, meaning they are created for a branch or pull request and destroyed afterwards. Important controls include reproducible setup, seeded test data, secrets management, and isolation from production systems.

Testing environment

Testing environments validate integration, compatibility, performance, security, and resilience. Test results should be linked to a specific artifact version. Test data must not expose personal or confidential information without appropriate protection.

Staging environment

Staging should resemble production sufficiently to make release evidence meaningful. It should use production-like infrastructure, deployment mechanisms, configuration structure, and observability. Perfect identity with production is rarely economical, but major differences should be documented and assessed.

Production environment

Production promotion should include an explicit release record, artifact identifier, configuration version, approver or automated policy result, and post-deployment verification. Production access should be limited according to least privilege, with privileged actions logged.

Gates and approvals

A quality gate is an objective condition that must be satisfied before promotion. Examples include all critical tests passing, no unapproved high-severity vulnerabilities, successful infrastructure validation, and an acceptable change-failure risk.

An approval is a deliberate authorisation by a person or policy engine. Approval should not become a meaningless button. The approver needs relevant evidence: test results, change description, risk classification, deployment plan, monitoring plan, and rollback or mitigation instructions.

Useful approval models include:

• automated approval: a policy engine evaluates measurable conditions;
• peer approval: another engineer reviews the change;
• service-owner approval: the responsible owner accepts operational risk;
• business approval: a product or business representative confirms timing and impact; and
• emergency approval: an expedited process used for incidents, with retrospective review.

Separating code authorship from production approval can reduce conflicts of interest, especially for high-risk changes. However, excessive manual approval can create queues and encourage workarounds. The objective is informed control, not bureaucracy.

3. Configuration Management and Environment Consistency

Configuration includes values that influence application behaviour without changing the application artifact. Examples are database endpoints, timeout values, logging levels, regional settings, external service URLs, and feature-flag states.

Configuration should be:

1. versioned, so changes can be traced;
2. reviewed, so unintended changes are detected;
3. validated, so invalid values are rejected before deployment;
4. separated from secrets, because passwords and keys require stronger protection;
5. environment-aware, because development and production have different endpoints and access boundaries; and
6. reproducible, so a previous known-good state can be restored.

Configuration drift occurs when environments gradually differ from their declared state. Infrastructure as code, configuration repositories, immutable infrastructure, and automated reconciliation reduce drift. A configuration-management system should make the desired state explicit and detect unauthorised changes.

Secrets should not be stored in source code or ordinary pipeline logs. They should be held in an approved secrets manager, accessed through short-lived credentials where possible, rotated, and masked in output. A secret accidentally committed to a repository should be considered exposed; merely deleting the file does not remove it from history or invalidate the credential.

A practical configuration record might contain:

4. Feature Flags and Decoupled Release

A feature flag is a controlled runtime decision that determines whether functionality is enabled. Feature flags separate deployment—placing code in an environment—from release—making that capability available to users.

Common flag types include:

• release flags, used to hide incomplete functionality;
• operational flags, used to disable an expensive or unstable component;
• experiment flags, used to compare user experiences; and
• permission flags, used to control access by customer, role, region, or subscription.

Flags permit gradual exposure and rapid mitigation without redeploying the whole application. However, they add conditional paths and operational complexity. Every flag should have an owner, purpose, default state, expiry date, audit history, and removal plan. Permanent unused flags create technical debt and can produce untested combinations.

A flag decision should be safe by default. If the flag service is unavailable, the application should use a deliberately chosen fallback, such as disabling a non-essential feature. The correct fallback depends on safety, business continuity, and security requirements.

Feature flags must not replace proper authorisation. A client-side flag can hide a user-interface element, but server-side permission checks are still required to protect data and operations.

5. Deployment Strategies

A deployment strategy determines how the new version is introduced and how much risk is exposed at one time.

Recreate deployment

The existing version is stopped and the new version is started. This is simple but causes downtime and provides limited opportunity for comparison. It may be acceptable for internal tools or systems with planned maintenance windows.

Rolling deployment

A rolling deployment replaces instances incrementally. Some instances run the old version while others run the new version. Traffic is gradually shifted as new instances pass health checks.

Advantages include lower infrastructure cost and reduced blast radius. Limitations include temporary mixed-version operation, compatibility requirements, and more complex rollback. The old and new versions must generally be able to coexist with the same database and message formats.

A rolling deployment must define:

• batch size or replacement percentage;
• readiness and liveness conditions;
• maximum unavailable capacity;
• maximum extra capacity;
• pause intervals for observation; and
• automatic halt criteria.

Blue-green deployment

Blue-green deployment maintains two production-capable environments. The active environment, blue, serves users; the inactive environment, green, receives the new version. After validation, traffic is switched to green.

The major benefit is rapid traffic reversal: the previous environment remains available. The cost is duplicated infrastructure and the complexity of handling stateful components. Database changes must be backward compatible or managed through an expand-and-contract process:

1. expand the schema to support both versions;
2. deploy code that can use the expanded schema;
3. migrate or backfill data;
4. switch traffic or behaviour; and
5. contract by removing obsolete fields only after the old version is no longer needed.

A database rollback is not equivalent to an application rollback. Destructive schema changes may make reversal impossible without a separate data-recovery plan.

Canary deployment

A canary deployment sends a small proportion of traffic to the new version, observes results, and increases exposure in stages. For example, a service might receive 1%, 5%, 25%, 50%, and finally 100% of traffic, with pauses between stages.

Canary analysis should use predefined indicators such as error rate, latency, saturation, conversion, cancellation, or support contacts. It should compare the canary with a control group over an appropriate period and traffic volume. A small canary may not reveal rare failures, and a short observation period may miss delayed effects.

Canary deployments are especially valuable when the cost of exposing all users to a defect is high. They require reliable traffic routing, representative traffic, meaningful metrics, and an automatic or clearly assigned decision process.

Comparison of strategies

These strategies can be combined. For example, a canary may be implemented within a blue-green environment, or a rolling deployment may expose each batch through a feature flag.

6. Verification and Observability

A deployment is not successful merely because a process exits with status zero. Verification must establish that the service is healthy and that users are receiving acceptable outcomes.

Three complementary forms of validation are useful:

• synthetic checks: scripted requests verify critical paths;
• telemetry: metrics, logs, and traces show technical behaviour; and
• business indicators: measures such as completed orders, payment success, or customer complaints show user impact.

Important release indicators include:

• availability and error rate;
• latency percentiles, especially p95 and p99;
• resource saturation;
• queue depth and processing delay;
• dependency failures;
• security alerts; and
• business conversion or transaction success.

A service-level objective (SLO) defines a target level of reliability over a period. A release policy might state that promotion stops if the error rate exceeds a threshold for a specified observation window. Such thresholds must be chosen from historical behaviour and service risk, not invented without context.

Observability data must also be protected. Logs can contain personal data, tokens, customer identifiers, or commercially sensitive information. Retention, access, redaction, and deletion must follow organisational policy and applicable law.

7. Rollback, Roll-Forward, and Recovery

Rollback returns software or configuration to a previous known-good state. Roll-forward fixes the problem by deploying a new version. Neither is universally superior.

Rollback is attractive when the prior artifact remains compatible with current data and dependencies. It may be unsafe after irreversible schema changes, data migrations, external contract changes, or security updates. Roll-forward may be preferable when reverting would lose data, recreate a vulnerability, or cause incompatibility.

A release response should consider:

1. severity and user impact;
2. reversibility of the application and data changes;
3. availability of a known-good artifact;
4. dependency and contract compatibility;
5. time needed to diagnose and correct the defect; and
6. regulatory, security, and communication obligations.

An effective recovery plan includes a trigger, decision owner, technical procedure, verification steps, communication path, and post-incident review. Recovery should be tested through rehearsals or game days rather than assumed to work.

A useful design principle is backward compatibility during transition. APIs should tolerate old and new clients during a staged rollout. Events should use versioned schemas or compatible evolution rules. This makes both rolling deployment and roll-forward safer.

8. Governance, Security, and Responsible Automation

Automated release systems have significant authority. They can change production systems, access sensitive environments, and affect customers at scale. Governance should therefore include:

• separation of duties for high-risk changes;
• least-privilege service accounts;
• signed or verifiable artifacts;
• dependency and container scanning;
• audit trails for code, configuration, approvals, and deployments;
• protected branches and review requirements;
• policy-as-code for repeatable controls; and
• documented emergency-change procedures.

Compliance does not necessarily require manual deployment. Automated evidence can be stronger than informal human memory if it is complete, tamper-resistant, and linked to the exact change. Organisations should demonstrate who or what authorised a release, what tests ran, which artifact was deployed, and what happened afterwards.

AI-assisted tools may help generate tests, detect anomalous deployment behaviour, summarise incidents, or recommend approval decisions. They should not be treated as inherently trustworthy. Generated code and recommendations require review, and sensitive source code or operational data should not be sent to unapproved services. An automated model should not silently override a safety gate or conceal uncertainty.

9. Illustrative Case: Online Learning Platform

Background

An illustrative online learning platform serves students, instructors, and administrators. It is introducing a redesigned assignment-submission service. The service handles file uploads, plagiarism checks, grading events, and notifications. A failure during examination periods could prevent submissions and create significant institutional risk.

Problem

The organisation currently uses a manual weekend release. Releases are slow, configuration differs between staging and production, and the previous release required an emergency database restore. The team wants faster delivery without risking student submissions.

Analysis

The service has several characteristics:

• high business impact during peak periods;
• asynchronous dependencies, including scanning and notification services;
• data that must not be lost;
• a need for auditability; and
• a user population that can be divided by institution and traffic percentage.

A simple recreate deployment is inappropriate because downtime is unacceptable. A direct full release is also risky because the redesigned upload path has not experienced production-scale traffic.

Approach and solution

The team adopts the following design:

1. The application is built once and stored as an immutable, signed artifact.
2. Infrastructure and environment configuration are managed declaratively and reviewed.
3. Database changes follow expand-and-contract compatibility rules.
4. The redesigned path is protected by a server-side feature flag.
5. A blue-green environment is prepared, with production-like test data that contains no personal information.
6. Synthetic submissions and integration tests run before approval.
7. A canary begins with one institution and 5% of eligible traffic.
8. The release is paused for 30 minutes while error rate, upload latency, queue delay, grading completion, and notification success are compared with the control group.
9. The service owner approves expansion only if predefined thresholds remain acceptable.
10. If submission failures increase, the feature flag is disabled first. If infrastructure or application health fails, traffic returns to the previous environment.

The release record includes the artifact digest, configuration version, test evidence, approver, timestamps, metrics, and final decision.

Outcome

The staged approach identifies an incompatibility in a notification message schema during the canary. The upload path remains available, but notifications for a subset of submissions are delayed. The feature is disabled for additional users, the message contract is corrected, and the canary resumes after successful tests. No data is lost, and the issue is resolved without a full production rollback.

Learning points

The case demonstrates that deployment strategy is only one part of release safety. Compatibility, feature flags, observability, data protection, and decision authority work together. It also illustrates why rollback should not be the only recovery mechanism: disabling a capability and rolling forward can preserve more service than reverting the entire application.

10. Best Practices and Common Failure Modes

Best practices

• Promote the same immutable artifact across environments.
• Keep configuration versioned and secrets outside source code.
• Make deployments small, observable, and reversible where possible.
• Define automatic halt conditions before starting a progressive release.
• Use realistic staging and representative test data.
• Design database and API changes for compatibility.
• Give every feature flag an owner and removal date.
• Treat monitoring and recovery procedures as production code.
• Record deployment evidence for operational learning and compliance.
• Conduct blameless post-incident reviews that result in specific improvements.

Common failure modes

Manual configuration changes: These cause drift and make incidents difficult to reproduce. Use declarative configuration and reviewable changes.

Approval without evidence: A human click does not reduce risk if the approver cannot see test and operational evidence.

Rollback assumed to be automatic: Data changes and external dependencies can make reversal unsafe. Test recovery and define roll-forward options.

Too many feature flags: Unmanaged flags create combinatorial testing problems and obscure the system’s actual behaviour.

Canary without representative traffic: A canary that receives only harmless traffic can produce false confidence.

Metrics without decision thresholds: Observability is not operational control unless someone knows what signal requires a halt or mitigation.

Deploying during peak risk without justification: Release timing should reflect customer impact, staffing, dependency availability, and recovery readiness.

Contemporary Developments

Modern delivery practices increasingly combine progressive delivery, policy-as-code, GitOps, and supply-chain security. GitOps uses a version-controlled desired state and automated reconciliation, making infrastructure and application state auditable. Progressive delivery tools can automate canary analysis and feature-flag changes. Software supply-chain frameworks emphasise provenance, signed artifacts, dependency transparency, and build isolation.

These developments do not eliminate engineering judgement. Automation can accelerate an incorrect decision, and a policy can encode an unsuitable threshold. Teams must periodically review release rules against real incidents, customer outcomes, and changing regulatory expectations.

Chapter Summary

Continuous delivery keeps software in a releasable condition through automated testing, controlled promotion, and operational verification. Continuous deployment automatically makes the production release decision when defined policies are satisfied. Both depend on immutable artifacts, reliable configuration management, secure secrets handling, and evidence-based gates.

Rolling deployments reduce infrastructure cost but require compatibility between versions. Blue-green deployments support rapid traffic switching at greater infrastructure cost. Canary deployments reduce exposure by increasing traffic gradually and observing technical and business indicators. Feature flags separate deployment from release and provide mitigation options, but they require lifecycle management.

Rollback restores a prior state, whereas roll-forward corrects the defect with a new release. Data, API, and dependency compatibility determine whether rollback is safe. Responsible release automation also requires least privilege, auditability, security controls, privacy protection, and tested recovery procedures. The best strategy is the one that matches business impact, technical architecture, observability maturity, and organisational risk tolerance.

Key Terms

Continuous delivery: Practice of keeping software tested and ready for controlled production release.

Continuous deployment: Practice of automatically releasing every qualifying change to production.

Artifact: A versioned, built software package produced by the pipeline.

Promotion: Moving an artifact or configuration from one environment to another.

Quality gate: A condition that must be satisfied before pipeline progression.

Approval: Explicit authorisation for a release or high-risk action.

Configuration drift: Unintended difference between declared and actual environment state.

Infrastructure as code: Declarative management of infrastructure through versioned files.

Feature flag: Runtime control that changes functionality exposure without necessarily redeploying.

Rolling deployment: Incremental replacement of instances with a new version.

Blue-green deployment: Deployment using two production-capable environments and a traffic switch.

Canary deployment: Gradual exposure of a new version to a limited traffic group.

Readiness check: Test determining whether an instance can receive traffic.

Observability: Ability to understand system behaviour through metrics, logs, and traces.

Rollback: Restoration of a previous known-good version or state.

Roll-forward: Correction of a problem through a subsequent release.

Progressive delivery: Controlled, gradual exposure of a change using automation and observation.

Policy as code: Machine-readable rules that automate governance and compliance decisions.

Expand-and-contract: Database migration pattern that preserves compatibility during transition.

Change failure: A deployment-related event that causes degraded service, remediation, or rollback.

Review Questions

Short Answer Questions

1. What is the principal difference between continuous delivery and continuous deployment?
2. Why is the build-once-promote-many-times principle valuable?
3. What problem does configuration management address?
4. State one advantage and one limitation of canary deployment.
5. Why can a database change make application rollback unsafe?

Long Answer Questions

1. Explain how approvals and quality gates should work together in a production pipeline.
2. Compare rolling, blue-green, and canary deployment strategies, including their risks and suitable contexts.
3. Discuss the benefits and operational risks of feature flags.
4. Explain how observability supports progressive delivery and recovery decisions.
5. Describe the governance and security controls needed for responsible release automation.

Analytical / Application Questions

1. A payment service must remain available, uses a shared database, and has limited production observability. Recommend a release strategy and justify your choice.
2. During a canary release, latency remains normal but payment completion falls by 8%. What evidence would you examine, and would you expand, halt, disable a feature, roll back, or roll forward?
3. Design a release plan for a service requiring a non-backward-compatible schema change. Identify the migration stages, compatibility risks, approval evidence, and recovery options.

Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes continuous deployment from continuous delivery?

A. Continuous deployment eliminates automated testing.

B. Continuous deployment automatically releases qualifying changes to production.

C. Continuous delivery never uses production environments.

D. Continuous delivery requires rebuilding the artifact for every environment.

Correct Answer: B. Continuous deployment automatically releases qualifying changes to production.

Question 2

What does the principle “build once, promote many times” primarily achieve?

A. It ensures each environment receives a newly compiled artifact.

B. It removes the need for configuration management.

C. It reduces differences between the artifact tested and the artifact released.

D. It prevents the use of automated tests.

Correct Answer: C. It reduces differences between the artifact tested and the artifact released.

Question 3

Which deployment strategy maintains two production-capable environments and switches traffic between them?

A. Recreate

B. Rolling

C. Blue-green

D. Batch-only

Correct Answer: C. Blue-green

Question 4

What is a major requirement for a safe rolling deployment?

A. The old and new versions must be able to coexist during the transition.

B. All existing instances must stop before any new instance starts.

C. The database must always be deleted between versions.

D. No health checks should be used.

Correct Answer: A. The old and new versions must be able to coexist during the transition.

Question 5

What is the primary purpose of a feature flag?

A. To encrypt all application data automatically.

B. To separate deployment of code from exposure of functionality.

C. To replace server-side authorisation.

D. To compile source code at runtime.

Correct Answer: B. To separate deployment of code from exposure of functionality.

Question 6

Which situation most strongly supports using a canary deployment?

A. A system where downtime is already planned and acceptable.

B. A low-risk documentation-only change.

C. A high-impact service where gradual exposure and comparison are valuable.

D. A system with no monitoring or traffic-routing capability.

Correct Answer: C. A high-impact service where gradual exposure and comparison are valuable.

Question 7

Why should secrets be kept out of ordinary source repositories?

A. Repositories cannot store text files.

B. Secrets require controlled access, auditing, and rotation.

C. Secrets make unit tests impossible.

D. Configuration never differs between environments.

Correct Answer: B. Secrets require controlled access, auditing, and rotation.

Question 8

Which statement about rollback is correct?

A. Rollback is always safe after a destructive database migration.

B. Rollback and roll-forward mean exactly the same thing.

C. Rollback may be unsafe when data or external contracts are no longer compatible.

D. Rollback requires deleting all production records.

Correct Answer: C. Rollback may be unsafe when data or external contracts are no longer compatible.

Question 9

Which control best supports auditability of an automated release?

A. An undocumented manual change after deployment.

B. A record linking the artifact, configuration, tests, authorisation, and outcome.

C. A shared administrator password.

D. Disabling deployment logs to protect privacy.

Correct Answer: B. A record linking the artifact, configuration, tests, authorisation, and outcome.

Question 10

In an expand-and-contract database migration, what occurs first?

A. Obsolete fields are immediately deleted.

B. The schema is expanded to support the old and new application versions.

C. All application traffic is permanently stopped.

D. The database is replaced without testing.

Correct Answer: B. The schema is expanded to support the old and new application versions.