Continuous Integration, Delivery and Release Engineering

Introduction

Modern software is delivered through a sequence of changes rather than a single final release. A developer modifies source code, automated systems build and test it, security tools inspect it, an artifact is created, and that artifact is promoted through environments until it reaches users. This discipline is known as continuous integration, continuous delivery, and release engineering.

For full-stack developers, CI/CD is not merely a collection of deployment commands. It is a socio-technical system that connects version control, automated verification, infrastructure, databases, security, operations, and organizational decision-making. Heroku and AWS provide different implementation mechanisms, but the underlying engineering principles are largely the same: produce repeatable builds, preserve traceability, reduce deployment risk, and recover quickly when a release fails.

A useful distinction is:

• Continuous integration (CI): Developers frequently integrate changes into a shared branch, where automated builds and tests provide rapid feedback.
• Continuous delivery: The application is maintained in a releasable state, and a validated build can be deployed to production through a controlled action.
• Continuous deployment: Every change that passes the required automated controls is deployed automatically to production.
• Release engineering: The broader practice of designing, governing, automating, observing, and improving the path from source change to operational release.

The objective is not to deploy as frequently as possible. The objective is to make delivery sufficiently safe, observable, and reversible that the organization can choose an appropriate release frequency.

Learning Objectives

After completing this chapter, you should be able to:

1. Design a CI/CD pipeline for building, testing, scanning, packaging, and deploying a full-stack application.
2. Explain trunk-based development and distinguish artifacts from source code and environments.
3. Apply environment promotion, deployment approvals, and separation of duties.
4. Compare feature flags, blue-green releases, canary releases, rolling deployments, and rollback strategies.
5. Plan database migrations that preserve compatibility during deployment.
6. Select useful delivery and reliability metrics.
7. Evaluate the security, governance, ethical, and operational consequences of release decisions.

1. The Release Pipeline as a Control System

A release pipeline transforms an input—usually a commit or pull request—into an operational change. A simplified pipeline contains the following stages:

Commit → Build → Unit test → Static analysis → Security scan
        → Package artifact → Integration test → Deploy to test
        → Acceptance test → Approval or policy gate → Production
        → Observe → Release decision or rollback

Each stage should answer a specific question:

A pipeline is a control system because it measures conditions, applies decisions, and changes system state. A test failure should prevent promotion; a high error rate after release should trigger investigation or rollback. This perspective discourages pipelines that merely execute scripts without enforcing meaningful controls.

Pipeline reproducibility

A build is reproducible when the same source revision and declared inputs produce functionally equivalent output. Reproducibility requires, where practical:

• A locked dependency graph.
• Explicit runtime and operating-system versions.
• Versioned build scripts.
• No dependence on a developer's local machine state.
• Controlled environment variables and credentials.
• Immutable or content-addressed artifacts.

A pipeline should record the commit identifier, artifact identifier, dependency versions, test results, scanner versions, and deployment target. This creates provenance, allowing an organization to answer: What code is running? Who approved it? Which tests ran? Which dependencies were included?

2. Trunk-Based Development and Change Integration

In trunk-based development, developers integrate small changes frequently into a central branch, often called main or trunk. Short-lived branches and pull requests may still be used for review, but branches should not remain isolated for weeks or months.

The practice rests on several principles:

1. The trunk should remain buildable and deployable.
2. Changes should be small enough to review and revert.
3. Automated checks should run before integration.
4. Incomplete functionality should be hidden with feature flags rather than maintained in long-lived branches.
5. Integration conflicts should be discovered early.

Long-lived branches create integration debt. As branches diverge, each team makes assumptions about interfaces, database schemas, configuration, and behavior. Merging later may require complex conflict resolution and can produce failures that were difficult to attribute.

Trunk-based development is not equivalent to merging every change directly into production. A mature workflow may use pull requests, mandatory reviews, automated checks, protected branches, and progressive release controls. The important property is that integration occurs frequently and the trunk remains trustworthy.

Commit and pull-request standards

A professional repository commonly requires:

• Small, cohesive commits.
• Automated formatting and linting.
• Unit and integration tests appropriate to the change.
• Review by at least one qualified contributor.
• No secrets or credentials in source control.
• A linked issue, change description, or risk statement.
• Clear ownership for production-impacting components.

Review is not a substitute for automation. Humans are good at assessing design, domain meaning, and maintainability; machines are better at repeating deterministic checks quickly.

3. Building, Testing, Scanning, and Packaging

Build and test layers

Testing should be layered according to feedback speed and scope:

• Unit tests verify individual functions or modules.
• Component tests verify a service with selected dependencies.
• Integration tests verify interactions among application components, databases, queues, or external services.
• Contract tests verify that service interfaces remain compatible.
• End-to-end tests verify important user journeys through the deployed system.
• Smoke tests provide a short post-deployment check that the application is reachable and fundamentally functional.

A large end-to-end suite can provide valuable confidence, but it is usually slower and more brittle than unit testing. A balanced test strategy places fast, deterministic tests early and reserves slower tests for changes that require them.

Security and quality scanning

A pipeline should treat security as an engineering control rather than a final inspection. Common checks include:

• SAST: Static application security testing examines source or bytecode.
• Software composition analysis: Identifies vulnerable or disallowed dependencies.
• Secret scanning: Detects credentials, tokens, and private keys.
• Container-image scanning: Checks operating-system and library packages in images.
• Infrastructure-as-code scanning: Finds risky cloud or network configuration.
• DAST: Dynamic application security testing examines a running application.

Not every finding should block every deployment. Blocking policy should consider severity, exploitability, exposure, compensating controls, and whether the finding is false positive. However, suppressions must be documented, time-limited, and owned; otherwise the pipeline gradually becomes permissive without accountability.

Artifacts

An artifact is a versioned output that can be deployed. Examples include a container image, a compiled package, a Heroku slug, or a static front-end bundle. Source code is an input to the build; the artifact is the tested output.

The same artifact should be promoted from testing to staging and production whenever possible. Rebuilding separately for each environment introduces uncertainty because dependency resolution, timestamps, compiler versions, or build-time configuration may differ.

Configuration that varies by environment—such as database URLs, API keys, and feature-flag settings—should be supplied at runtime or through a controlled configuration system. Secrets should be stored in a secret manager or platform configuration facility, not embedded in artifacts or logs.

Heroku and AWS implementation patterns

On Heroku, a pipeline can connect applications representing development, staging, and production. A successful build may be promoted between Heroku apps, while configuration variables remain environment-specific. Review Apps can create temporary environments for pull requests, but they require controls for data access, cost, and cleanup.

On AWS, common implementations use a source repository, a build service, an artifact repository, and a deployment service. For example, a pipeline may build a Docker image, store it in Amazon Elastic Container Registry, and deploy it to Amazon ECS, EKS, or another runtime. AWS CodePipeline, CodeBuild, CodeDeploy, and related services can implement these stages, although third-party CI systems can use the same architecture.

The platform changes the commands and integration points, not the fundamental questions: Is the artifact traceable? Is the environment configured safely? Can the release be observed and reversed?

4. Environments and Promotion

An environment is a runtime context with its own infrastructure, configuration, data, access rules, and operational characteristics. Typical environments are development, test, staging, and production.

Promotion means moving a validated artifact and its release metadata from one environment to another. Promotion should not mean copying unreviewed source files or manually rebuilding the application.

A useful environment model includes:

• Development: Fast experimentation; limited production data.
• Test or integration: Automated verification of service interactions.
• Staging: Production-like infrastructure and configuration for acceptance testing.
• Production: Real users, strict access controls, monitoring, and change governance.

Perfect parity between staging and production may be expensive, but important behavior should be similar: runtime version, network rules, database engine, service contracts, and deployment mechanism. Differences must be known and documented.

Environment isolation is also a security boundary. Production credentials should not be available to ordinary pull-request jobs. Test environments should use synthetic or appropriately anonymized data. Access should follow least privilege, and deployment identities should be short-lived where practical.

Approval controls

Deployment approvals are decision points, not ceremonial clicks. An approval policy may require:

• Successful automated tests and scans.
• A designated reviewer for production changes.
• Evidence that a database migration is backward compatible.
• A change record or incident-risk assessment.
• A maintenance window for high-risk work.
• Confirmation that rollback or mitigation is available.

Separation of duties can reduce the risk that one person introduces and independently approves a sensitive change. At the same time, excessive approvals create delay and encourage bypass behavior. Controls should be proportional to risk and supported by clear ownership.

5. Release Strategies

Feature flags

A feature flag separates code deployment from feature exposure. Code can be deployed while a feature remains disabled. Flags support gradual rollout, testing with selected users, and rapid mitigation without rebuilding the application.

Flags introduce their own risks. A flag can become stale, multiply conditional paths, or expose inconsistent behavior across services. Each flag should have an owner, purpose, default state, expiration date, and testing plan. Security flags must fail safely; for example, disabling an authorization check must never be an acceptable fallback.

Rolling deployment

A rolling deployment gradually replaces instances running the old version with instances running the new version. It reduces the number of simultaneously changed instances but requires compatibility between versions during the transition. It is common in container orchestration platforms.

Blue-green deployment

Blue-green deployment maintains two production-like environments. The currently active environment is blue; the new version is deployed and tested in green. Traffic is switched to green when readiness conditions are met. If the new version fails, traffic can be directed back to blue, provided blue remains operational and data changes are compatible.

Blue-green deployment can provide a fast application rollback, but it may require double infrastructure capacity. It does not automatically solve database rollback, background jobs, cache changes, or external side effects.

Canary release

A canary release sends a small proportion of production traffic to a new version before wider rollout. Operators compare error rate, latency, resource use, and business indicators between the canary and baseline. Traffic may be increased in stages.

A canary is meaningful only when traffic is representative and telemetry is sufficiently granular. A small canary that receives no important workflows can appear healthy while the release is defective.

Release decision metrics

Useful technical signals include request error rate, latency percentiles, saturation, restart frequency, queue depth, and database performance. Business signals may include checkout completion, successful login, subscription activation, or support contacts.

A simple error-rate calculation is:

The variables must refer to the same time interval and population of requests. A worked example: if 180 of 12,000 requests fail during a 10-minute canary window, the error rate is 180 ÷ 12,000 × 100% = 1.5%. This value is meaningful only when compared with a baseline and interpreted alongside traffic volume, endpoint mix, and severity.

6. Rollback, Roll-Forward, and Recovery

A rollback restores a previously known application version or disables the newly released behavior. A roll-forward fixes the defect in a new release. The choice depends on whether the previous version remains compatible with current data and external systems.

A practical recovery plan includes:

1. Detect the failure through alerts or user reports.
2. Confirm whether the release is the likely cause.
3. Stop or pause further rollout.
4. Reduce harm, for example by disabling a feature flag.
5. Roll back or roll forward using a tested procedure.
6. Verify service health and data integrity.
7. Communicate status to stakeholders.
8. Record a blameless review and corrective actions.

Rollback is not always safe. A release may have changed a database schema, emitted irreversible messages, charged a payment provider, or altered external records. Therefore, rollback must be designed before deployment rather than improvised during an incident.

A useful operational measure is mean time to restore service, often abbreviated MTTR. It is calculated over a defined set of incidents as the average elapsed time from service-impact start to restoration. Organizations must define start and end points consistently; otherwise comparisons are misleading.

7. Database Migration Safety

Databases make release engineering more difficult because data persists across application versions. A safe migration usually follows an expand-and-contract pattern.

Expand-and-contract sequence

1. Expand: Add new tables, columns, indexes, or structures without removing the old interface.
2. Dual compatibility: Deploy application code that can read the old and new representations, or write both when necessary.
3. Backfill: Transform existing data in controlled batches, monitoring load and correctness.
4. Migrate reads and writes: Switch application behavior to the new representation.
5. Contract: Remove obsolete columns, code paths, or indexes only after all old application versions and jobs are gone.

For example, renaming full_name to display_name should not begin by dropping full_name. First add display_name, deploy code that reads both, backfill values, write the new field, verify completeness, and only then remove the old field in a later release.

Migration design should consider:

• Lock duration and table size.
• Transaction behavior and database engine limitations.
• Index construction cost.
• Replication lag.
• Concurrent application versions.
• Backfill idempotence.
• Recovery if the migration stops halfway.
• Privacy and retention requirements.

A migration is idempotent when safely repeating it does not produce an incorrect additional effect. Idempotence is especially valuable for retryable deployment jobs and interrupted backfills.

Database backups are necessary but not sufficient. A backup is useful only if restoration is tested, recovery objectives are understood, and the restored data is verified.

8. A Worked Pipeline Design

Consider a Node.js API with a React front end and PostgreSQL database. The application is stored in Git, packaged as a container, and deployed to AWS ECS. The same design could be adapted to Heroku by replacing the build and deployment integration.

Pipeline definition

1. A pull request triggers dependency installation using the lockfile, formatting checks, unit tests, and secret scanning.
2. A merge to main triggers a clean build and creates an artifact tagged with the commit identifier.
3. Static analysis, dependency scanning, and container scanning run against the artifact.
4. Integration tests run against an ephemeral PostgreSQL instance.
5. The artifact is stored in a registry with provenance metadata.
6. The artifact is deployed to a test environment.
7. Smoke and end-to-end tests verify core workflows.
8. A production approval gate checks test evidence, migration compatibility, ownership, and deployment risk.
9. ECS deploys the artifact using a canary or blue-green strategy.
10. Monitoring compares release health with the previous version.
11. The pipeline either increases traffic, pauses for investigation, or restores the previous task set.

Example promotion record

The exact threshold should be agreed before release and interpreted against a baseline. A threshold without a measurement window, sample-size consideration, and responsible operator is not a complete policy.

9. Illustrative Case: Checkout Service Release

Background

A fictional online retailer operates a checkout service on Heroku and a payment integration hosted by a third party. The team wants to introduce an optional express-checkout feature without risking existing purchases.

Problem

The feature changes the checkout API, adds a database field, and modifies payment-provider interaction. A direct production deployment could expose customers to failed payments or inconsistent orders.

Analysis

The team identifies three compatibility requirements:

• Old application instances must continue to process orders while new instances are introduced.
• The database must support both old and new application versions.
• The payment provider must not receive duplicate charges if a request is retried.

Approach

The team first adds a nullable express_method column. The application continues to operate without using it. A second release writes the field only when the feature flag is enabled and uses an idempotency key for payment requests. The code is deployed to staging and tested with a payment sandbox.

In production, the feature remains disabled. A small, internal-user cohort receives the feature through a flag. The team observes payment success, duplicate-charge reports, checkout latency, and application errors. The rollout proceeds to 10%, 50%, and then all eligible users only after each stage meets its predeclared conditions.

If errors increase, the flag is disabled first. If the application version itself is defective, the Heroku release is rolled back, provided the database remains compatible. The old column is not removed until all old code paths and operational jobs have been retired.

Outcome

The feature reaches users without interrupting existing checkout. The staged approach limits exposure, while the additive migration and payment idempotency protect against the most serious failure modes.

Learning points

• Deployment and feature exposure can be separate decisions.
• Database compatibility determines whether application rollback is safe.
• External side effects require idempotency and reconciliation, not merely code rollback.
• Release gates should include business indicators, not only infrastructure metrics.

10. Governance, Ethics, and Professional Practice

Release engineering affects users and society. An uncontrolled deployment can expose personal data, deny access to essential services, or create financial harm. Professional practice therefore includes:

• Protecting personal and confidential data in logs, test systems, and artifacts.
• Applying least privilege to developers, pipelines, and deployment identities.
• Recording approvals and changes without falsifying evidence.
• Preserving accessibility and localization during feature rollout.
• Avoiding experimentation that users could not reasonably expect, especially in sensitive contexts.
• Maintaining an audit trail for regulated systems.
• Providing incident communication that is accurate, timely, and respectful.

Automated tools and AI-assisted coding can increase development speed, but generated code must undergo the same testing, security review, licensing review, and ownership assessment as human-written code. AI systems may reproduce insecure patterns, expose confidential prompts or source material, or generate tests that appear comprehensive while missing domain risks. Automation changes the speed of review; it does not remove professional accountability.

11. Contemporary Developments and Limitations

Current release practices increasingly emphasize software supply-chain security, signed artifacts, provenance attestations, policy-as-code, ephemeral environments, and progressive delivery controlled by telemetry. Organizations also combine observability data with automated rollback systems.

These techniques have limitations. More pipeline stages can increase maintenance burden. Canary analysis can produce false confidence when metrics are incomplete. Feature flags can create combinatorial test complexity. Ephemeral environments can increase cloud cost. Automated rollback can worsen an incident if the previous version is incompatible with a destructive migration or if the monitoring signal is noisy.

Accordingly, automation should be designed with explicit failure modes, human escalation, auditability, and tested recovery procedures. The strongest pipeline is not the one with the most tools; it is the one that provides reliable evidence and supports sound decisions.

Best-Practice Checklist

• Keep the trunk buildable and integrate small changes frequently.
• Build once and promote the same immutable artifact.
• Lock dependencies and record provenance.
• Run tests and security scans at appropriate pipeline stages.
• Separate configuration and secrets from artifacts.
• Use production-like staging for important behavior.
• Define owners, approvals, and abort conditions before release.
• Use feature flags deliberately and remove stale flags.
• Prefer additive, backward-compatible database changes.
• Design rollback and roll-forward procedures before deployment.
• Monitor technical and business indicators.
• Test restoration, not only backup creation.
• Review incidents without blame and implement corrective actions.

Chapter Summary

Continuous integration brings changes together frequently and validates them automatically. Continuous delivery keeps validated software ready for controlled production release, while continuous deployment automates that final transition when policy allows. Release engineering encompasses the technical and organizational controls required to make these practices reliable.

A mature pipeline builds, tests, scans, packages, promotes, deploys, observes, and supports recovery. Trunk-based development reduces integration debt when supported by protected branches and automated checks. Versioned artifacts provide traceability and should be promoted rather than rebuilt. Feature flags, rolling releases, blue-green deployments, and canary releases provide different ways to limit exposure, but each has operational and data-related limitations.

Safe releases depend especially on backward-compatible database migrations, explicit approval policies, least-privilege access, meaningful telemetry, and tested rollback or roll-forward procedures. Effective release engineering balances speed with evidence, safety, governance, and responsibility to users.

Key Terms

Continuous integration: Frequent integration of code changes with automated validation.

Continuous delivery: Practice of keeping software deployable through an automated and controlled process.

Continuous deployment: Automatic production deployment after required controls pass.

Release engineering: Design and operation of the system that moves software from source to users.

Trunk-based development: Frequent integration of small changes into a central branch.

Artifact: A versioned, deployable output produced by a build.

Provenance: Evidence describing how an artifact was produced and validated.

Environment promotion: Moving a validated artifact through controlled runtime environments.

Feature flag: A runtime control that changes feature exposure without rebuilding software.

Blue-green deployment: Release method using two production-like environments and a traffic switch.

Canary release: Gradual release to a small, monitored portion of production traffic.

Rolling deployment: Gradual replacement of old instances with new instances.

Rollback: Restoring a prior application version or behavior.

Roll-forward: Correcting a defect through a subsequent release.

Expand-and-contract: Database migration pattern that preserves compatibility across versions.

Idempotence: Property that repeating an operation does not create an incorrect additional effect.

SAST: Static analysis of application code for security weaknesses.

Deployment gate: A condition or approval that must be satisfied before promotion.

MTTR: Mean time to restore service after an incident or failure.

Policy as code: Machine-enforced rules expressed in versioned code or configuration.

Review Questions

Short Answer Questions

1. Distinguish continuous delivery from continuous deployment.
2. Why should a pipeline promote the same artifact across environments?
3. What problem does trunk-based development address?
4. State the main steps in an expand-and-contract database migration.
5. Give two examples of evidence that should be recorded for a production release.

Long Answer Questions

1. Explain how a full-stack CI/CD pipeline can integrate building, testing, security scanning, packaging, and deployment.
2. Compare feature flags, blue-green deployment, and canary release, including one limitation of each.
3. Explain why application rollback may be unsafe after a database migration.
4. Discuss how approval controls and separation of duties can improve production governance without creating excessive delivery delay.
5. Design a release process for a payment-related feature, addressing testing, observability, idempotency, feature exposure, and recovery.

Analytical / Application Questions

1. During a canary window, 240 of 16,000 requests fail. Calculate the error rate and explain what additional evidence is needed before deciding whether to abort the rollout.
2. A new application version requires a non-null database column, but old instances will run for several minutes during deployment. Analyze the risk and propose a compatible migration sequence.
3. A blue-green deployment appears healthy according to infrastructure metrics, but checkout completion falls sharply. Explain why the release should be paused and identify the operational responses you would consider.

Multiple Choice Questions with Answers

Question 1

Which statement best describes continuous delivery?

A. Every commit is automatically deployed to production without review

B. Software is kept in a deployable state through automated and controlled processes

C. Developers work only on long-lived feature branches

D. Production changes are made manually without recorded evidence

Correct Answer: B. Software is kept in a deployable state through automated and controlled processes

Question 2

What is the principal advantage of promoting the same artifact through environments?

A. It guarantees that the application has no defects

B. It avoids the need for environment-specific configuration

C. It reduces differences caused by rebuilding for each environment

D. It eliminates the need for integration testing

Correct Answer: C. It reduces differences caused by rebuilding for each environment

Question 3

Which practice is most consistent with trunk-based development?

A. Keeping branches isolated until the end of a release quarter

B. Integrating small changes frequently into a central branch

C. Disabling automated tests to speed up merging

D. Deploying every unreviewed commit directly to production

Correct Answer: B. Integrating small changes frequently into a central branch

Question 4

Which release method directs a small proportion of production traffic to a new version first?

A. Canary release

B. Big-bang release

C. Cold backup

D. Dependency locking

Correct Answer: A. Canary release

Question 5

Why are feature flags useful during release?

A. They permanently remove the need for testing

B. They separate deployment of code from exposure of functionality

C. They encrypt all application traffic

D. They automatically migrate every database schema

Correct Answer: B. They separate deployment of code from exposure of functionality

Question 6

What is the first general step in an expand-and-contract migration?

A. Immediately drop the old column

B. Disable all database backups

C. Add compatible structures without removing the old interface

D. Delete old application instances before changing the schema

Correct Answer: C. Add compatible structures without removing the old interface

Question 7

A canary receives 240 failed requests out of 16,000 total requests. What is the error rate?

A. 0.15%

B. 1.5%

C. 6.67%

D. 15%

Correct Answer: B. 1.5%

Question 8

Which statement about rollback is correct?

A. Rollback always reverses external side effects such as completed payments

B. Rollback can be unsafe when data or external systems are incompatible with the old version

C. Rollback eliminates the need for monitoring

D. Rollback means deleting the production database and restoring source code

Correct Answer: B. Rollback can be unsafe when data or external systems are incompatible with the old version

Question 9

Which control most directly helps prevent credentials from being exposed in source code?

A. Secret scanning and managed secret storage

B. Increasing front-end bundle size

C. Using longer feature-flag names

D. Disabling integration tests

Correct Answer: A. Secret scanning and managed secret storage

Question 10

What is a key purpose of a deployment approval gate?

A. To replace all automated tests with managerial judgment

B. To provide a documented decision point based on evidence and risk

C. To ensure that every deployment occurs during an outage

D. To prevent all frequent releases

Correct Answer: B. To provide a documented decision point based on evidence and risk