Practical CI/CD Capstone

Introduction

A CI/CD capstone integrates the practices studied throughout a DevOps program into one demonstrable engineering system. The goal is not merely to create a pipeline that executes commands. A useful pipeline provides repeatable evidence that source code can be compiled, tested, packaged, released, deployed, observed, and safely recovered when a change causes harm.

This chapter presents an illustrative capstone for a small web application called TaskBoard. The application exposes a health endpoint and allows users to create and list tasks. The implementation technology is not important; the same design can be applied to Java, Python, JavaScript, Go, or .NET applications. The example uses generic CI/CD terminology so that learners can adapt it to GitHub Actions, GitLab CI, Jenkins, Azure DevOps, or another platform.

The capstone has six quality goals:

1. Repeatability: the same source revision produces the same build artifact.
2. Fast feedback: developers learn about errors early.
3. Traceability: every deployed version can be linked to source, tests, and approvals.
4. Security: credentials, dependencies, artifacts, and deployment permissions are controlled.
5. Observability: operators can determine whether the service is healthy and useful.
6. Recoverability: a failed release can be reversed using a tested procedure.

Learning Objectives

By the end of this chapter, a learner should be able to:

• design an end-to-end CI/CD workflow for a small application;
• distinguish continuous integration, continuous delivery, and continuous deployment;
• implement build, test, package, deployment, and verification stages;
• document pipeline assumptions, controls, and operating procedures;
• apply least privilege, secret management, dependency scanning, and artifact integrity controls;
• select meaningful delivery, reliability, and quality metrics;
• perform a controlled rollback and explain its risks;
• demonstrate the pipeline using evidence rather than unsupported claims.

1. From Source Change to Running Service

Continuous integration (CI) is the practice of frequently integrating changes into a shared codebase and automatically validating them. Continuous delivery means that validated changes are kept in a releasable state, usually with a human approval before production. Continuous deployment automatically promotes eligible changes to production.

These terms describe capability, not a particular product. A YAML file is not automatically a CI/CD system. The system includes source control, build environments, artifact storage, deployment targets, credentials, tests, monitoring, governance, and people who respond to failures.

A suitable pipeline separates activities according to their purpose:

A central design principle is build once, promote many times. Rebuilding separately for testing and production can produce different outputs because dependencies, timestamps, compiler versions, or network resources may differ. Instead, the pipeline creates one versioned artifact, records its identity, and promotes that exact artifact through environments.

2. Capstone Architecture and Repository Design

The illustrative repository contains the following elements:

TaskBoard/
├── src/                    application source
├── tests/unit/             fast isolated tests
├── tests/integration/      tests involving supporting services
├── Dockerfile              image construction instructions
├── compose.test.yml        local test dependencies
├── ci/pipeline.yml         pipeline definition
├── deploy/                 Kubernetes or platform manifests
├── scripts/smoke-test.sh   post-deployment verification
├── SECURITY.md             vulnerability reporting process
└── README.md               setup and operating documentation

A branch policy may require pull requests, successful automated checks, review by at least one other person, and protection against direct pushes to the protected branch. Small teams should avoid excessive process, but they should not confuse speed with the absence of controls.

A useful version convention is semantic versioning for public releases: MAJOR.MINOR.PATCH. A breaking interface change increases MAJOR; backward-compatible functionality increases MINOR; a backward-compatible defect correction increases PATCH. Internal teams may instead use a commit identifier, but the deployed artifact must still have a unique, searchable identity.

Pipeline environments

A minimal capstone can use three environments:

• Test: created or reset automatically for automated checks.
• Staging: a production-like environment used for acceptance and smoke tests.
• Production: the user-facing environment, protected by approval or an equivalent policy.

Environment configuration should be externalized. The artifact should not contain production passwords or environment-specific source code. Configuration values may be injected through a secret manager, workload identity, or protected environment variables.

3. Implementing the Pipeline

3.1 Validation and build

The first jobs should be inexpensive. They may check formatting, linting, dependency manifests, configuration syntax, and required documentation. A build should run in a clean, pinned environment rather than relying on tools installed manually on a long-lived server.

A conceptual pipeline is:

stages: [validate, test, package, deploy_staging, verify, release, deploy_production]

validate:
  commands:
    - format-check
    - lint
    - dependency-audit

test:
  commands:
    - unit-test
    - integration-test

package:
  commands:
    - build
    - create-image
    - scan-image
    - publish-image

deploy_staging:
  needs: [package]
  commands:
    - deploy artifact_digest

verify:
  needs: [deploy_staging]
  commands:
    - smoke-test
    - contract-test

release:
  needs: [verify]
  approval: required

deploy_production:
  needs: [release]
  commands:
    - deploy artifact_digest
    - smoke-test

This is illustrative rather than vendor-specific syntax. A real system must use the selected platform’s valid syntax and secure execution model. The important property is the dependency relationship: deployment cannot occur until the artifact has passed the required checks, and production cannot receive an unverified or newly rebuilt artifact.

3.2 Testing strategy

Tests should be layered. Unit tests are fast and isolate functions or modules. Integration tests verify interactions with databases, queues, or external services. End-to-end tests validate a user journey but are usually slower and more fragile. Smoke tests are deliberately small checks that establish whether a deployment is basically usable.

For TaskBoard, a smoke test might:

1. request /health and expect a successful response;
2. create a task through the API;
3. retrieve the task;
4. verify that the response contains the expected identifier and title;
5. delete or clean up the test data.

A passing test suite is not proof that the system has no defects. Tests sample behavior. Their value depends on coverage of important risks, test data quality, determinism, and maintenance. Coverage percentage can identify untested code, but it should not be treated as a complete quality measure.

3.3 Packaging and artifact integrity

The package stage should produce an immutable image or binary. A container image should use a small trusted base image, pin dependencies where practical, run as a non-root user, and exclude unnecessary build tools and source secrets.

An artifact registry should retain:

• repository and commit identifier;
• build timestamp and pipeline run identifier;
• dependency and vulnerability results;
• image digest or package checksum;
• build provenance, where supported.

A tag such as latest is convenient for experimentation but unsafe as the sole production reference because it can move. Production deployment should refer to a unique version or digest. If a checksum is used, its algorithm and encoding must be recorded.

4. Secure Delivery

Security is a property of the complete delivery system, not a single scanning job. The pipeline should apply the principle of least privilege: each job receives only the permissions required for its task. A test job generally should not be able to deploy to production.

Core controls

• Store secrets in a secret manager or protected CI variables, never in source code.
• Mask secret values in logs and prevent commands from printing them.
• Use short-lived credentials or workload identity where supported.
• Separate credentials for test, staging, and production.
• Require review for changes to deployment configuration.
• Scan source dependencies, container images, infrastructure definitions, and secrets.
• Pin or constrain action/plugin versions and review third-party actions.
• Protect the artifact registry and restrict who can overwrite releases.
• Record approvals, deployments, and rollback events.

Security scanners produce findings, not automatic truth. A vulnerability may be unreachable, mitigated, or incorrectly classified; conversely, a clean scan does not prove security. The team should define a risk-based policy. For example, a critical vulnerability in a runtime dependency may block release, while a low-risk development-only finding may require tracking rather than immediate failure.

Supply-chain security is especially important because pipeline components themselves execute with authority. A malicious or compromised plugin can steal credentials or alter artifacts. Approved action versions, dependency review, isolated runners, provenance records, and restricted tokens reduce this risk.

Privacy must also be considered. Test data should not contain unnecessary personal information. Production data should not be copied into developer environments without authorization, minimization, and appropriate protection.

5. Deployment and Safe Release Patterns

A deployment changes the running environment; a release makes functionality available to users. Separating these ideas enables controlled activation. Feature flags, for example, allow code to be deployed while a feature remains disabled until it is evaluated.

Rolling deployment

Instances are replaced gradually. It is efficient but can temporarily run old and new versions together. Therefore, database and API changes should be backward compatible during the transition.

Blue-green deployment

Two environments exist: the active blue environment and the inactive green environment. The new version is deployed and verified in green, after which traffic is switched. Rollback can often switch traffic back to blue. The approach is simple to reason about but may require capacity for two environments.

Canary deployment

A small fraction of traffic is directed to the new version. Metrics are compared with the stable version before expansion. Canary release reduces exposure but requires reliable routing, sufficient traffic, and carefully selected success criteria.

For a certificate-level capstone, blue-green or a controlled rolling deployment is usually easier to demonstrate. The design should state its health criteria, approval point, maximum deployment time, and rollback trigger.

6. Monitoring, Measurement, and Evidence

Monitoring should answer three questions: Is the service available? Is it performing acceptably? Is it producing correct business behavior? Logs, metrics, traces, and synthetic tests provide complementary evidence.

Useful service-level indicators include request success rate, latency, throughput, resource saturation, and queue depth. A health endpoint should distinguish process liveness from dependency readiness. A process may be running while its database connection is unusable.

The four commonly used delivery metrics are deployment frequency, lead time for changes, change failure rate, and time to restore service. Definitions must be explicit. For example:

Here, “deployment causing failure or rollback” must be defined by the organization, and the numerator and denominator must cover the same period. The formula is a percentage, so the ratio is dimensionless. If 2 of 20 deployments require rollback or urgent remediation, the rate is (2 ÷ 20) × 100% = 10%. A high deployment frequency is not beneficial if it is achieved by accepting frequent failures.

Recovery performance can be described by mean time to restore (MTTR):

Time must be measured consistently, such as minutes from confirmed service impact to restoration. “Mean” can hide severe outliers, so teams should also inspect median and high-percentile restoration times where data volume permits.

A capstone demonstration should show evidence such as:

• a pull request and review;
• a successful pipeline run;
• test and scan reports;
• the artifact digest;
• staging deployment and smoke-test output;
• approval record;
• production health dashboard;
• a simulated rollback and its verification.

7. Rollback Design

Rollback is not merely pressing an undo button. It is a planned operational action with eligibility conditions. The team must know which version is safe, how traffic is redirected, whether data changes are reversible, and who is authorized to act.

Application rollback is easiest when artifacts are immutable and database migrations are backward compatible. Destructive schema changes create special risk. A safer migration sequence is often: add a nullable field, deploy code that can use both old and new structures, migrate data, then remove obsolete structures in a later release.

A practical rollback runbook contains:

1. detection signal and decision authority;
2. communication channel and incident record;
3. last known good artifact identifier;
4. exact rollback command or procedure;
5. database compatibility assessment;
6. smoke tests and monitoring checks;
7. confirmation that traffic and background workers use the intended version;
8. follow-up investigation and corrective action.

Rollback should be tested in staging. A rollback that exists only in documentation may fail because of missing permissions, expired credentials, or overlooked state.

8. Illustrative Case: TaskBoard Release Failure

Background

A small team maintains TaskBoard for an internal support department. The team has four developers and previously deployed by manually copying files to a virtual machine. Releases were difficult to reproduce, and the team could not reliably identify which version was running.

Problem

A new task-filtering feature passed a developer’s manual test but caused an error when the database contained a null priority value. The release also included a dependency with a known high-severity vulnerability. Monitoring detected increased server errors only after users reported them.

Analysis

The team identified four control weaknesses:

• no automated integration test for null priority values;
• no dependency audit that blocked high-severity findings;
• no immutable artifact identity;
• no documented rollback procedure.

The problem was not simply that one developer made a coding mistake. The delivery system lacked feedback and recovery mechanisms.

Approach and solution

The team introduced pull requests and a protected main branch. The pipeline added formatting, unit tests, integration tests using a disposable database, dependency scanning, and image scanning. The package stage published an image identified by a digest. Staging deployment used that digest, followed by a smoke test and a test of the filtering workflow.

Production deployment required approval from the service owner. The application was deployed using a rolling strategy, with the previous digest retained. A dashboard displayed request error rate, latency, and health status. The team wrote and rehearsed a rollback runbook.

The dependency finding was assessed. Because the vulnerable library was used at runtime and a fixed version was available, the pipeline blocked the release until the dependency was upgraded. The null-value defect was fixed and covered by an integration test.

Outcome

The next release produced one traceable artifact, passed automated checks, and was promoted to staging before approval. During a later rehearsal, the team intentionally deployed a faulty configuration. The smoke test failed, production promotion was prevented, and the previous staging version was restored. The exercise exposed a missing permission, which was corrected before a real incident occurred.

Learning points

The case demonstrates that quality is systemic. Automated tests, security controls, artifact traceability, monitoring, approval, and rollback reinforce one another. A pipeline is successful when it makes the safe path easier and failure visible—not when it merely reports a green status.

9. Documentation and Demonstration

A professional capstone repository should include a README explaining prerequisites, local execution, test commands, architecture, environment variables, and known limitations. A pipeline document should describe triggers, stages, permissions, retention, approval rules, and failure handling. A release runbook should explain deployment and rollback in operational language.

The final demonstration should follow a coherent narrative:

1. show a small change and its review;
2. run the pipeline and explain each gate;
3. identify the immutable artifact;
4. deploy to staging;
5. execute automated verification;
6. show approval and production promotion;
7. display health and delivery evidence;
8. simulate failure and execute rollback;
9. explain security decisions, limitations, and future improvements.

A strong demonstration does not hide failed experiments. It distinguishes expected failures, such as a deliberately rejected vulnerability or failed smoke test, from uncontrolled failures. This is evidence of engineering maturity.

10. Limitations and Contemporary Practice

No pipeline eliminates operational risk. Tests may miss unanticipated behavior; scanners may generate false positives; monitoring may be incomplete; and deployment automation can amplify a defective change. Human judgment remains necessary for risk acceptance, incident response, architecture, and ethical decisions.

AI-assisted coding and test generation can accelerate development, but generated code must undergo review, testing, license assessment, and security analysis. Confidential source code or personal data should not be sent to an AI service without authorization. AI output is an input to engineering judgment, not proof of correctness.

Modern practices increasingly emphasize software supply-chain provenance, policy as code, ephemeral environments, progressive delivery, observability-driven development, and automated evidence collection. These practices should be adopted according to risk and organizational capability rather than fashion. A small capstone benefits more from clear artifact identity and a tested rollback than from a complex platform that nobody understands.

Best-Practice Checklist

Before declaring the capstone complete, verify that:

• source changes require review and protected-branch checks;
• builds run in a clean, reproducible environment;
• unit and integration tests execute automatically;
• dependencies, secrets, and artifacts are scanned;
• production uses an immutable artifact reference;
• credentials are least-privileged and environment-specific;
• staging verification precedes production release;
• approvals and deployment records are retained;
• health, error, and performance signals are visible;
• rollback has a named owner and has been rehearsed;
• documentation enables another person to operate the system.

Chapter Summary

A practical CI/CD capstone combines source control, automated validation, testing, security, packaging, deployment, monitoring, governance, and recovery. The most reliable design builds one immutable artifact and promotes it through controlled environments. Testing must be layered, security must include the pipeline supply chain, and deployment must be paired with observable success criteria. Metrics such as change failure rate and MTTR are useful only when their definitions and measurement periods are consistent. A rollback plan must address application state, database compatibility, permissions, communication, and verification. The final demonstration should present evidence of repeatability, traceability, security, operational health, and recovery rather than simply showing a successful pipeline screen.

Key Terms

Continuous integration: Frequent integration of source changes with automated validation.

Continuous delivery: Keeping validated changes ready for release through a controlled process.

Continuous deployment: Automatically releasing eligible changes to production.

Artifact: A build output, such as a package, binary, or container image.

Artifact digest: A content-derived identifier that precisely identifies an artifact.

Immutable artifact: An artifact whose contents cannot change after publication.

Pipeline gate: A condition that must pass before a later stage can proceed.

Smoke test: A small test checking whether a deployed service is basically usable.

Integration test: A test of interactions among application components or services.

Deployment: Installing a version into a runtime environment.

Release: Making deployed functionality available to users.

Blue-green deployment: Running two environments and switching traffic between them.

Canary deployment: Gradually exposing a new version to a limited traffic segment.

Rollback: Restoring a previously known-good version or configuration.

Least privilege: Granting only the permissions required for a task.

Software supply chain: The people, tools, dependencies, processes, and artifacts used to create software.

Observability: The ability to infer system condition from logs, metrics, traces, and tests.

Change failure rate: The percentage of deployments that cause failure or require remediation.

MTTR: Mean time to restore service after an incident.

Provenance: Evidence describing how, where, and from what inputs an artifact was produced.

Review Questions

Short Answer Questions

1. What is the difference between continuous delivery and continuous deployment?
2. Why is building once and promoting the same artifact preferable to rebuilding for each environment?
3. Name two types of automated tests and state what each is intended to validate.
4. What is the purpose of an artifact digest?
5. Why should a rollback procedure be rehearsed?

Long Answer Questions

1. Design a CI/CD pipeline for a small web application and explain the purpose of each stage.
2. Explain how least privilege and secret management reduce the security risks of a delivery pipeline.
3. Compare rolling, blue-green, and canary deployment strategies, including one strength and one limitation of each.
4. Explain why database migrations can make application rollback difficult and propose a safer migration sequence.
5. Discuss how monitoring, tests, approvals, and artifact traceability work together during a production release.

Analytical / Application Questions

1. A team has 30 deployments in a month, of which 3 cause customer-visible incidents and require remediation. Calculate the change failure rate and explain one limitation of this metric.
2. A staging smoke test fails after deployment, but the artifact passed all unit tests. Identify at least three plausible causes and describe the next diagnostic actions.
3. Design a rollback runbook for a service using a rolling deployment and a database migration. Identify the most important decision points and risks.

Multiple Choice Questions with Answers

Question 1

Which principle ensures that the exact tested output is promoted to production?

A. Build once and promote many times

B. Rebuild separately in every environment

C. Use the latest mutable tag

D. Test only after production deployment

Correct Answer: A. Build once and promote many times

Question 2

What is the primary purpose of a smoke test?

A. To prove that every code path is defect-free

B. To check whether a deployment is basically usable

C. To measure developer productivity

D. To replace all integration tests

Correct Answer: B. To check whether a deployment is basically usable

Question 3

Which credential design best follows least privilege?

A. One administrator token shared by every pipeline job

B. A production token stored in source code

C. Separate short-lived permissions limited to each job

D. A permanent token printed in deployment logs

Correct Answer: C. Separate short-lived permissions limited to each job

Question 4

A team has 2 failed deployments out of 20 total deployments. What is the change failure rate?

A. 2%

B. 5%

C. 10%

D. 20%

Correct Answer: C. 10%

Question 5

Which deployment strategy directs a small fraction of traffic to a new version first?

A. Canary deployment

B. Big-bang deployment

C. Clean-room deployment

D. Static deployment

Correct Answer: A. Canary deployment

Question 6

Why is the tag latest unsafe as the sole production reference?

A. It cannot be stored in an artifact registry

B. It may point to different contents over time

C. It always refers to a vulnerable image

D. It prevents automated testing

Correct Answer: B. It may point to different contents over time

Question 7

Which test most directly checks interaction with a database?

A. Unit test

B. Integration test

C. Formatting check

D. License scan

Correct Answer: B. Integration test

Question 8

What does MTTR measure in the chapter’s context?

A. Average time to restore service after incidents

B. Average time to write a new feature

C. Total number of monthly deployments

D. Percentage of tested source lines

Correct Answer: A. Average time to restore service after incidents

Question 9

Which database migration approach best supports rollback?

A. Immediately delete the old column before deployment

B. Deploy an incompatible schema and application together

C. Use backward-compatible additions before later removal

D. Copy production data into every developer environment

Correct Answer: C. Use backward-compatible additions before later removal

Question 10

Which evidence most strongly demonstrates artifact traceability?

A. A screenshot of a green pipeline only

B. A mutable image tag with no commit record

C. A digest linked to the commit, tests, and deployment record

D. A verbal statement that the release was tested

Correct Answer: C. A digest linked to the commit, tests, and deployment record