Continuous Integration and Continuous Delivery

Introduction

Modern software systems are changed frequently. New features, security fixes, configuration changes, and defect corrections may be released several times each day. This speed creates a coordination problem: every change must be built, integrated with other work, tested, packaged, approved, and deployed reliably. Continuous Integration and Continuous Delivery (CI/CD) provide a disciplined approach to solving this problem.

CI/CD is not simply a collection of automation tools. It is a set of engineering practices, cultural expectations, and control mechanisms that make software delivery repeatable and observable. A basic pipeline converts a source-code change into a tested and deployable artifact. A mature pipeline can then promote that artifact through environments while preserving traceability, enforcing approvals, and supporting rapid recovery when a release causes harm.

In DevOps, the pipeline is a socio-technical system. Source control, build servers, test frameworks, artifact repositories, deployment platforms, monitoring, security controls, and human decision-making all contribute to its performance. Automation reduces manual error, but it does not eliminate the need for sound design, risk assessment, or accountability.

Learning Objectives

By the end of this chapter, you should be able to:

• distinguish Continuous Integration, Continuous Delivery, and Continuous Deployment;
• describe the stages of a basic CI/CD pipeline;
• design appropriate build, integration, testing, and artifact-management practices;
• explain deployment approvals and the separation of build from release;
• compare rolling, blue-green, canary, and feature-flag release strategies;
• select and apply suitable rollback procedures; and
• evaluate a pipeline using practical quality, delivery, and reliability measures.

1. The Foundations of CI/CD

Continuous Integration

Continuous Integration is the practice of integrating developers’ changes into a shared mainline frequently, followed by automated verification. A developer typically creates a small change, commits it to version control, and opens a pull request or merge request. Automated checks then compile the software, run tests, inspect code quality, and identify security or dependency problems.

The central idea is that integration risk is reduced by integrating often. Long-lived branches allow changes to diverge and make conflicts difficult to detect. Frequent integration exposes incompatibilities while the changed code and its context are still understood by the team.

CI depends on several conditions:

1. The source code and relevant configuration are stored in version control.
2. The project can be built in a repeatable manner.
3. Tests can run automatically in a clean environment.
4. The team treats a failing mainline as a high-priority problem.
5. Changes are sufficiently small to be reviewed and diagnosed.

CI is not achieved merely because a team owns a build server. If builds are unreliable, tests are ignored, or developers integrate only occasionally, the essential practice is absent.

Continuous Delivery and Continuous Deployment

Continuous Delivery extends CI so that every change that passes the required pipeline stages is kept in a deployable state. The software may require a manual business or operational approval before production release, but the technical steps needed for deployment are automated and rehearsed.

Continuous Deployment goes further: every change that passes the defined automated controls is released to production automatically. Continuous Deployment therefore requires a high degree of confidence in automated testing, monitoring, security controls, and recovery procedures.

Continuous Delivery is often the appropriate target for organizations beginning their DevOps journey. It separates technical readiness from the business decision to release. This allows an organization to improve automation without immediately removing all human governance.

2. A Basic Pipeline Model

A pipeline is a sequence of automated and controlled activities. A representative flow is:

Commit → Validate → Build → Unit test → Package artifact
       → Integration/system test → Security and quality checks
       → Publish artifact → Approval → Deploy → Verify → Promote or roll back

The stages are logically distinct even when a particular tool combines them. A useful design makes each stage’s purpose, inputs, outputs, failure conditions, and ownership explicit.

Stage 1: Source and Change Validation

The pipeline begins with a commit or merge request. Basic validation may include syntax checking, formatting, linting, branch-policy checks, and validation of configuration files. These checks are fast and should run early because rapid feedback reduces the cost of correction.

A pipeline should identify the exact revision being processed. A commit identifier, branch, author, timestamp, and pipeline run identifier should be recorded. Builds that cannot be associated with a precise source revision create traceability and audit problems.

Stage 2: Build

The build transforms source code into executable or deployable output. Examples include compiled binaries, container images, packaged libraries, mobile applications, and infrastructure modules. Build steps should be defined as code wherever practical, rather than depending on undocumented settings on a particular server.

A reproducible build produces the same intended output from the same source revision and controlled inputs. Perfect bit-for-bit reproducibility may be difficult, but teams should control compiler versions, dependency versions, operating-system images, build scripts, and environment variables. Dependencies should be locked or otherwise constrained to known versions.

A build should fail when essential inputs are unavailable, compilation fails, required checks fail, or the resulting package is invalid. It should not silently produce a partial or unverified artifact.

Stage 3: Testing

Testing is layered because different tests provide different balances of speed, scope, and realism.

• Unit tests examine a small component in isolation and are usually fast.
• Component or service tests examine a service with selected real or simulated dependencies.
• Integration tests examine interactions between components, databases, message brokers, or external interfaces.
• System or end-to-end tests examine a complete user or business flow.
• Acceptance tests assess whether stated business requirements are satisfied.
• Security tests include static analysis, dependency scanning, secret detection, dynamic testing, and configuration checks.
• Performance tests examine response time, throughput, capacity, or resource behavior under defined conditions.

A common mistake is to equate a high percentage of code coverage with high quality. Coverage indicates which code paths were executed by tests; it does not demonstrate that assertions are meaningful or that important behaviors were selected. Tests should therefore be judged by relevance, reliability, diagnostic value, and maintenance cost as well as coverage.

Tests that depend on timing, shared state, or unstable external services may be flaky. A flaky test sometimes passes and sometimes fails without a relevant software change. Flakiness damages trust in the pipeline and should be investigated rather than routinely ignored. A temporary quarantine may be necessary, but quarantined tests must have an owner and removal target.

Stage 4: Artifact Packaging and Management

An artifact is a versioned output of a build that can be stored, retrieved, tested, and deployed. Examples include a container image, a Java archive, a Python package, a binary installer, or a signed infrastructure bundle.

A strong practice is “build once, promote many.” The artifact tested in a pre-production environment should be the same artifact promoted to production. Rebuilding separately for production can introduce unnoticed differences. Environment-specific values should normally be supplied through configuration or secrets management rather than by changing the application package.

An artifact repository should support:

• immutable versioning;
• access control and retention policies;
• metadata such as source revision, build number, dependency list, and test results;
• integrity verification, such as checksums or signatures;
• vulnerability and license information; and
• traceability from production deployment back to source code.

A version such as service-2.4.1 is more useful than an ambiguous label such as latest. The latter may change meaning over time and can make rollback or investigation unsafe. A digest, checksum, or unique build identifier should be recorded for deployment.

Stage 5: Deployment

Deployment places an artifact into an environment and connects it to required infrastructure, configuration, data stores, and services. Deployment automation should be idempotent where feasible: running the same deployment operation more than once should produce the intended state rather than duplicate or corrupt resources.

Deployment configuration should distinguish between:

• application code, which is packaged and versioned;
• environment configuration, such as endpoints and scaling values;
• secrets, which require protected storage and controlled access; and
• infrastructure, which may itself be managed as versioned code.

The deployment process should record who or what initiated it, which artifact was used, which environment was targeted, the result, and relevant logs. Manual changes made directly to production create configuration drift and complicate later deployments.

Stage 6: Verification

Deployment is not complete when a command succeeds. Verification should determine whether the service is healthy and whether it performs acceptably. Smoke tests may check basic availability. Deeper checks may examine business transactions, error rates, latency, logs, queues, database connectivity, and resource consumption.

Observability supports verification. Logs provide event detail, metrics show measured behavior, and traces help follow requests across services. Health checks should distinguish between a process being alive and a service being ready to handle traffic.

3. Approvals and Governance

Automation does not mean that every decision should be automatic. A deployment approval is a controlled decision to promote a known artifact to a specified environment. Approvals may be required because of business timing, regulatory obligations, customer impact, segregation of duties, or operational risk.

Effective approval controls should state:

• what artifact is being approved;
• what evidence is available, including test and security results;
• which environment and release window are involved;
• who is authorized to approve;
• how the approval is recorded; and
• what conditions require rejection or escalation.

An approval should not be a ceremonial click. The approver needs sufficient information to understand risk. Excessive approvals can create bottlenecks and encourage rubber-stamping. Risk-based governance is preferable: low-risk changes may use automated controls, while high-risk changes may require additional review, a maintenance window, or a rollback rehearsal.

Separation of duties can reduce the risk of unauthorized change, but it should be designed carefully. Excessive separation can slow delivery without improving safety. Automated policy checks, protected branches, signed commits, restricted production access, and auditable deployment records can provide strong control while preserving flow.

4. Release Strategies

A deployment changes the technical environment; a release makes functionality available to users. These events can be separated through release strategies and feature controls.

Rolling Release

A rolling release replaces or updates instances gradually. Some instances run the new version while others continue running the old version. Traffic is shifted as new instances pass health checks.

Advantages include modest infrastructure requirements and gradual exposure. Limitations include temporary coexistence of versions, compatibility requirements, and more complex rollback when database or message formats change.

Blue-Green Release

In a blue-green release, two substantially similar environments exist. One serves production traffic, while the other receives the new version. After verification, traffic is switched to the new environment.

This strategy permits a rapid traffic reversal, provided the previous environment remains available and data changes are compatible. Its main cost is operating duplicate capacity, at least temporarily.

Canary Release

A canary release sends a small proportion of traffic to the new version. The team compares error rates, latency, resource behavior, and business indicators with the established version. Exposure increases only when evidence supports progression.

Canaries are especially useful when real production behavior cannot be fully represented by pre-production tests. They require reliable observability, representative traffic, and clearly defined promotion thresholds.

Feature Flags

A feature flag allows functionality to be enabled or disabled through configuration. This separates code deployment from user-visible release and can support internal testing, targeted rollout, or rapid disabling of a defective feature.

Flags introduce operational complexity. They must have owners, documented purpose, secure administration, testing coverage, and removal dates. Permanent unused flags become hidden technical debt.

5. Rollback and Recovery

Rollback is the process of restoring a previously known-good application or configuration state. Recovery is broader: it includes rollback, forward-fix deployment, data restoration, traffic redirection, and other actions that return the service to an acceptable condition.

A rollback plan should answer:

1. What evidence triggers rollback?
2. Who or what can authorize it?
3. Which artifact is the known-good version?
4. How is traffic returned or the feature disabled?
5. What happens to database and message changes?
6. How will recovery be verified?
7. How will users and stakeholders be informed?

Application rollback is often straightforward when artifacts are immutable. Data rollback is more difficult. A database migration that removes information may not be safely reversible. Backward-compatible, expand-and-contract migrations are therefore commonly preferred:

1. expand the schema without breaking the old application;
2. deploy code that can use both old and new structures;
3. migrate or backfill data;
4. switch reads and writes to the new structure; and
5. remove obsolete structures only after rollback is no longer required.

A forward fix may be safer than rollback when data has been transformed or when old and new versions are incompatible. The correct objective is not to reverse every technical action; it is to restore reliable service while protecting data integrity.

6. Pipeline Design Principles

A basic pipeline should be designed around the following principles:

• Fast feedback: run quick, high-value checks early.
• Fail safely: stop promotion when essential controls fail.
• Repeatability: define steps and environments consistently.
• Immutability: do not modify an artifact after it has been tested.
• Least privilege: give pipeline identities only required permissions.
• Traceability: connect requirements, commits, builds, artifacts, deployments, and incidents.
• Observability: make pipeline and production behavior visible.
• Small changes: reduce diagnosis and rollback complexity.
• Recovery readiness: rehearse rollback and restoration procedures.
• Pipeline security: protect credentials, runners, dependencies, and build outputs.

A pipeline itself is production-critical infrastructure. It should be monitored for queue time, execution failures, flaky tests, dependency outages, and unauthorized changes. Secrets should not be written to source code or ordinary logs. Build agents should be patched and isolated appropriately, and third-party actions or plugins should be assessed for supply-chain risk.

7. Metrics and Improvement

Useful delivery measures include:

• deployment frequency;
• lead time from accepted change to production;
• change failure rate;
• time to restore service after a failure;
• pipeline duration and queue time;
• test failure and flakiness rates; and
• proportion of deployments requiring manual intervention.

These metrics should be interpreted together. Increasing deployment frequency while change failure rate also increases may indicate that speed is being purchased at the expense of reliability. A team should avoid using metrics to punish individuals, because this encourages concealment and gaming. Metrics are most valuable when used to identify system constraints and guide improvement.

8. Applied Case: Northstar Library Service

Background

Northstar Library Service is an illustrative case involving a small organization that provides a web service for searching and reserving books. Its five developers release manually once every two months. A developer builds the application on a local computer, sends a package to an operations colleague, and supplies deployment instructions by email.

Problem

The organization experiences inconsistent builds, forgotten configuration steps, and long periods in which changes cannot be integrated safely. A previous release caused reservation failures because a database change was incompatible with the deployed application. The organization wants more frequent releases without weakening control.

Analysis

The main problems are not a lack of a particular tool. They are weak version traceability, environment differences, insufficient automated tests, and the absence of a tested recovery procedure. The team decides first to improve repeatability rather than immediately adopting automatic production deployment.

Approach

The team implements the following pipeline:

1. All source code, database migration scripts, and deployment definitions are stored in version control.
2. Every merge request runs formatting, static analysis, unit tests, and dependency checks.
3. A clean build creates a uniquely identified container image containing the source revision.
4. Integration tests run against a temporary database.
5. The image is stored in an artifact registry and is not rebuilt for later environments.
6. The image is deployed automatically to a test environment.
7. System tests verify searching, reservation, cancellation, and authentication.
8. A product owner and operations representative approve production promotion.
9. Production uses a canary deployment beginning with 10% of traffic.
10. Error rate, reservation success, and response latency are monitored for 30 minutes.
11. If thresholds are exceeded, traffic returns to the previous version; otherwise, exposure increases to 50% and then 100%.

The database process is redesigned using an expand-and-contract approach. New columns are introduced before application code depends on them, and destructive changes are postponed until the old version is no longer needed.

Outcome

The organization can release smaller changes weekly and can identify the exact source revision and artifact in production. The canary strategy limits exposure, while the preserved previous version provides a rapid application rollback. The team still requires manual approval, but the approval is evidence-based rather than dependent on informal email instructions.

Learning Points

The case demonstrates that CI/CD is a progression of capabilities. Source control, repeatable builds, automated tests, immutable artifacts, observable deployment, and recovery planning must reinforce one another. A sophisticated release strategy cannot compensate for an untrustworthy build or inadequate testing.

9. Limitations and Contemporary Considerations

CI/CD does not guarantee defect-free software. Automated tests may omit important scenarios, monitoring may fail to detect a subtle business problem, and an apparently safe change may interact with an external dependency. Automation can also accelerate the release of harmful code if governance is weak.

Organizations should integrate security into the pipeline through secure coding checks, dependency analysis, secret scanning, signed artifacts, restricted credentials, and review of infrastructure changes. Regulatory environments may require retention of approval records, evidence of testing, access reviews, and controlled production changes. Privacy-sensitive systems must ensure that test data is appropriately anonymized or synthetic.

Artificial intelligence can assist with test generation, log summarization, code review suggestions, and anomaly detection. These uses require human validation because generated tests may be superficial, summaries may omit important evidence, and models may reproduce insecure patterns. AI-generated code and pipeline configuration should be treated as untrusted changes until reviewed, tested, and scanned.

Cloud platforms, containers, infrastructure as code, policy as code, software bills of materials, and progressive delivery have made modern pipelines more capable. They have also increased the number of components that must be governed. The appropriate design remains proportional to system risk, team capability, and business need.

Chapter Summary

Continuous Integration reduces integration risk by integrating changes frequently and verifying them automatically. Continuous Delivery keeps accepted changes deployable, while Continuous Deployment releases them automatically when controls pass. A basic pipeline commonly includes source validation, build, testing, artifact publication, approval, deployment, verification, and promotion or recovery.

Reliable pipelines use repeatable builds, immutable versioned artifacts, layered tests, protected secrets, traceable records, and observable environments. The build-once-promote-many principle prevents tested software from being silently changed before production. Approvals should be risk-based and evidence-driven rather than ceremonial.

Rolling, blue-green, canary, and feature-flag strategies control how functionality reaches users. Rollback planning must consider application state, configuration, database migrations, and data integrity. Finally, CI/CD is an organizational capability as well as a technical workflow: its success depends on collaboration, disciplined change management, security, monitoring, and continuous improvement.

Key Terms

Continuous Integration: Frequent integration of source changes with automated verification.

Continuous Delivery: Practice of keeping every accepted change in a deployable state.

Continuous Deployment: Automatic production release of changes that pass defined controls.

Pipeline: An automated and controlled sequence for building, testing, packaging, and releasing software.

Build: The transformation of source code and inputs into executable or deployable output.

Artifact: A versioned output of a build that can be stored and deployed.

Artifact repository: A controlled store for versioned build outputs and metadata.

Immutable artifact: An artifact that is not modified after publication.

Unit test: A test of a small software component in isolation.

Integration test: A test of interactions between components or external dependencies.

Flaky test: A test whose result changes unpredictably without a relevant code change.

Deployment: The technical placement of software into an environment.

Release: The act of making deployed functionality available to users.

Canary release: Gradual exposure of a new version to a small proportion of traffic.

Blue-green deployment: Traffic switching between two substantially equivalent environments.

Feature flag: A control that enables or disables functionality independently of deployment.

Rollback: Restoration of a previously known-good application or configuration state.

Observability: The ability to understand system behavior through logs, metrics, and traces.

Infrastructure as code: Management of infrastructure through versioned, reviewable definitions.

Change failure rate: The proportion of deployments that cause failure, degradation, or required remediation.

Review Questions

Short Answer Questions

1. What is the principal purpose of Continuous Integration?
2. How does Continuous Delivery differ from Continuous Deployment?
3. Why should a team build an artifact once and promote it across environments?
4. Name three types of automated tests used in a CI/CD pipeline.
5. Why can database changes make rollback difficult?

Long Answer Questions

1. Explain the stages of a basic CI/CD pipeline and describe the purpose of each stage.
2. Compare rolling, blue-green, and canary release strategies, including their benefits and limitations.
3. Discuss how artifact repositories support security, traceability, and reliable rollback.
4. Explain how deployment approvals can provide governance without creating unnecessary delivery bottlenecks.
5. Describe the principles required to make a CI/CD pipeline repeatable and trustworthy.

Analytical / Application Questions

1. A team has fast builds but frequent production failures caused by configuration differences. Diagnose the likely pipeline weaknesses and propose a prioritized improvement plan.
2. A service uses a database migration that removes a column required by the previous application version. Which release and migration approach should be used to preserve recovery options? Justify your answer.
3. A canary release shows normal technical error rates but a significant fall in successful customer transactions. Should promotion continue? Explain what evidence and actions are appropriate.

Multiple Choice Questions with Answers

Question 1

Which statement best describes Continuous Delivery?

A. Every change is automatically released to production without review

B. Every accepted change is maintained in a deployable state

C. Developers integrate code only at the end of a project

D. Production systems are changed manually by operations staff

Correct Answer: B. Every accepted change is maintained in a deployable state

Question 2

What is the main advantage of the build-once-promote-many principle?

A. It allows production artifacts to be edited directly

B. It removes the need for automated testing

C. It ensures the tested artifact is the artifact released

D. It guarantees that database changes are reversible

Correct Answer: C. It ensures the tested artifact is the artifact released

Question 3

Which test primarily examines interactions between a service and a database?

A. Unit test

B. Integration test

C. Formatting check

D. Branch-policy check

Correct Answer: B. Integration test

Question 4

Which release strategy gradually exposes a new version to a small percentage of users or traffic?

A. Canary release

B. Big-bang release

C. Offline compilation

D. Static packaging

Correct Answer: A. Canary release

Question 5

Why is the label latest risky for production artifact selection?

A. It prevents source code compilation

B. It cannot be stored in an artifact repository

C. Its meaning may change over time

D. It always identifies a database migration

Correct Answer: C. Its meaning may change over time

Question 6

Which practice best supports rapid application rollback?

A. Rebuilding the application manually during an incident

B. Keeping an immutable, known-good previous artifact

C. Deleting all previous artifacts immediately

D. Changing source code directly on the production server

Correct Answer: B. Keeping an immutable, known-good previous artifact

Question 7

What is a major limitation of code coverage as a quality measure?

A. Coverage prevents tests from running

B. Coverage proves that all requirements are correct

C. Coverage does not show whether assertions are meaningful

D. Coverage can only be measured after production deployment

Correct Answer: C. Coverage does not show whether assertions are meaningful

Question 8

What is the purpose of an approval in a controlled CI/CD pipeline?

A. To identify the compiler version automatically

B. To authorize promotion of a known artifact using available evidence

C. To replace all automated tests

D. To make an artifact mutable

Correct Answer: B. To authorize promotion of a known artifact using available evidence

Question 9

Which database migration approach best preserves compatibility during a gradual release?

A. Delete the old schema before deploying new code

B. Use an expand-and-contract sequence

C. Change the schema manually after every request

D. Disable version control for migration files

Correct Answer: B. Use an expand-and-contract sequence

Question 10

Which combination is most directly associated with observability?

A. Logs, metrics, and traces

B. Branches, tags, and licenses

C. Compilers, editors, and keyboards

D. Approvals, invoices, and contracts

Correct Answer: A. Logs, metrics, and traces