Pipeline Concepts, Architecture, and Terminology

Introduction

A CI/CD pipeline is an automated path through which a software change moves from source control to validation, packaging, and—when authorized—release. Pipelines provide more than automation scripts. They express the organization’s delivery process: what must happen, in what order, under which conditions, using which evidence and controls.

A useful distinction is that continuous integration (CI) focuses on frequently integrating and validating changes, whereas continuous delivery keeps software in a releasable condition and makes production release a controlled decision. Continuous deployment goes further by automatically releasing qualifying changes to production. These practices depend on a pipeline architecture that is understandable, repeatable, observable, and secure.

This chapter explains the principal pipeline concepts and the relationships among them. Terminology varies across GitHub Actions, GitLab CI/CD, Jenkins, Azure Pipelines, CircleCI, and other platforms; the underlying ideas, however, are broadly transferable.

Learning Objectives

By the end of this chapter, you should be able to:

• explain pipelines, stages, jobs, runners, agents, and triggers;
• distinguish artifacts from dependencies and explain how data moves between jobs;
• describe environments, gates, approvals, and deployment controls;
• explain pipeline-as-code and common CI/CD architecture models;
• interpret a pipeline design and identify practical security, reliability, and governance concerns.

1. The Pipeline as a Delivery Model

A pipeline represents a sequence—or, more accurately, a directed graph—of automated activities. A simple pipeline may perform the following operations:

1. obtain source code;
2. install dependencies;
3. compile or build the application;
4. run unit and integration tests;
5. perform quality and security checks;
6. publish a versioned artifact;
7. deploy to one or more environments;
8. verify the deployment and record the result.

The sequence is not necessarily linear. Independent tests may run in parallel, while a deployment job may wait for several earlier jobs. Thus, the pipeline is best understood as a graph of nodes and relationships rather than merely a list of commands.

A pipeline normally has three kinds of state:

• source state: the commit, branch, tag, or pull request being processed;
• execution state: whether jobs are queued, running, passed, failed, skipped, or cancelled;
• delivery state: which artifact version has been promoted to which environment.

A strong pipeline makes these states visible and traceable. A release engineer should be able to identify the source revision, tools, tests, artifact, approvals, and environment associated with a deployment.

Figure: A general pipeline flow from source change to deployment and feedback.

2. Core Pipeline Terminology

Pipelines and stages

A pipeline is the complete configured workflow. A stage is a logical phase within it, such as Build, Test, Security Scan, or Deploy. Stages improve readability and often express broad ordering. For example, all test jobs may depend on successful build jobs.

A stage is not necessarily a single machine or a single command. It can contain several jobs that execute concurrently. Platform terminology differs: some systems treat stages as explicit containers, while others use a graph of jobs without a formal stage layer.

Jobs and steps

A job is an executable unit scheduled by the CI/CD platform. It usually runs on one runner or agent and has a defined workspace, environment variables, commands, and timeout. A job may contain multiple steps, such as checking out code, restoring a cache, running tests, and uploading results.

A step is generally smaller than a job and shares the job’s workspace and execution context. Separating work into jobs creates scheduling and dependency boundaries; separating commands into steps mainly improves organization and logging.

Runners and agents

A runner or agent is the execution worker that performs a job. It may be:

• hosted, provisioned and maintained by the CI/CD provider; or
• self-hosted, operated by the organization on a virtual machine, physical server, container, or Kubernetes-based system.

Workers are commonly selected by labels or capabilities such as linux, windows, docker, or gpu. A worker should be treated as disposable unless the design explicitly requires persistence. Disposable execution reduces contamination between builds and makes results more reproducible.

Self-hosted workers provide control over networking, software, and specialized hardware, but introduce patching, capacity, isolation, and credential-management responsibilities. They should not automatically receive broad production privileges.

Triggers

A trigger is an event or rule that starts, schedules, or approves pipeline execution. Common triggers include:

• a push to a branch;
• creation or update of a pull request;
• a version tag or release event;
• a scheduled time;
• a manual action;
• completion of another pipeline;
• an external webhook or API request.

Trigger design is a governance decision. Pull-request pipelines should provide rapid feedback, while production deployment should usually require stronger conditions than a routine source push. Scheduled security scans and dependency updates can operate independently of application changes.

Artifacts

An artifact is a file or collection of files produced by a pipeline and retained for later use. Examples include a compiled binary, container image, package, test report, code-coverage report, or deployment manifest.

A release artifact should be identifiable by version or digest and should be promoted rather than rebuilt for every environment. Rebuilding can cause the supposedly same release to differ because of changing dependencies, timestamps, compiler versions, or external inputs. Artifact retention periods should reflect operational, legal, and audit needs.

Dependencies and data flow

A dependency expresses a relationship in which one job or pipeline requires another job’s completion, output, or status. Dependencies determine ordering and often control data transfer. For example, a package job depends on successful compilation and tests; a deployment job depends on the published package.

Two meanings should be distinguished:

1. software dependencies are libraries or services required by the application;
2. pipeline dependencies are relationships among jobs, stages, or pipelines.

Caches may accelerate installation of software dependencies, but a cache is not a trusted release artifact. Caches can be invalidated, replaced, or restored incorrectly and should not be the sole source of deployable output.

Environments

An environment is a named operational context in which software runs. Typical environments include development, test, staging, and production. Environments may differ in infrastructure, data, network access, scale, monitoring, and approval requirements.

Environment configuration should be externalized where possible. The same artifact should be promoted across environments while environment-specific values—such as endpoints or feature settings—are injected securely. Production secrets must not be stored in source repositories or printed in logs.

Gates and approvals

A gate is a condition that must be satisfied before a pipeline proceeds. Gates may be automatic or human-controlled. Examples include successful tests, vulnerability thresholds, change-window rules, service-health checks, or approval by an authorized person.

A manual approval is a gate, but not every gate should be manual. Excessive approval steps create queues and encourage bypassing controls. Conversely, high-risk production changes may require separation of duties, documented approval, and a rollback plan. Effective gates are explicit, auditable, proportionate to risk, and difficult to circumvent accidentally.

3. Pipeline-as-Code

Pipeline-as-code stores workflow definitions in a version-controlled repository, usually beside application code or in a dedicated platform repository. The definition specifies triggers, jobs, workers, commands, dependencies, permissions, artifacts, and deployment rules.

Its principal benefits are:

• reviewability: changes can be reviewed through pull requests;
• reproducibility: a historical commit identifies the workflow used;
• consistency: teams can reuse templates and standards;
• disaster recovery: the configuration is not trapped in a server interface;
• automation: pipeline changes can be tested and promoted like other code.

Pipeline-as-code also creates risks. A malicious change to the workflow can alter tests, exfiltrate secrets, or deploy unauthorized software. Therefore, workflow files require code review, least-privilege credentials, restricted write access, pinned or controlled action versions, and careful treatment of contributions from forks. Secrets should be supplied by a managed secret store and masked in logs; masking is not a substitute for preventing secret exposure.

A mature definition separates reusable workflow logic from application-specific values. Templates and shared components reduce duplication, but excessive abstraction can make a pipeline difficult to understand. The appropriate goal is governed reuse, not maximum indirection.

4. Common CI/CD Architecture Models

Single-server or centralized model

In a centralized model, one CI server coordinates source retrieval, execution, reporting, and sometimes deployment. It is simple to start and may suit a small team. Its weaknesses include a potential single point of failure, limited scalability, and broad concentration of credentials.

Controller–worker model

A controller schedules jobs while multiple workers execute them. This model improves parallelism and allows different operating systems or toolchains. The controller should ideally orchestrate rather than execute untrusted build commands. Worker isolation, queue management, and capacity planning become important.

Distributed or elastic runner model

Workers are created on demand as virtual machines, containers, or pods. Elastic capacity reduces idle infrastructure and supports bursty workloads. However, startup latency, image maintenance, network access, and ephemeral storage must be managed.

Multi-stage promotion model

In a promotion model, a versioned artifact passes through environments: for example, test → staging → production. The artifact is not rebuilt between stages. Automated verification may occur at each stage, with an approval or policy gate before production.

Monorepo and polyrepo models

A monorepo contains many services or components in one repository. It can simplify cross-component changes but may require path-based triggers, selective testing, and sophisticated dependency analysis. Polyrepo architecture places services in separate repositories, enabling autonomy but requiring coordination for shared libraries, integration testing, and release compatibility.

Trunk-based and branch-oriented pipelines

Trunk-based development integrates small changes frequently into a main branch, generally favoring short-lived branches and rapid validation. A branch-oriented model may maintain long-lived release or development branches. Long-lived branches can support regulated release cycles, but they increase merge divergence and integration risk. Neither model is universally correct; the choice should reflect product risk, team structure, release policy, and deployment capability.

5. Designing a Pipeline Graph

Pipeline design should begin with the delivery objective rather than with a particular tool. Identify the change event, required evidence, deployable output, target environments, and rollback mechanism. Then represent work as jobs with explicit dependencies.

A useful design principle is fail fast without sacrificing necessary evidence. Cheap syntax and unit checks can run early. Independent security and integration checks can run in parallel. Expensive end-to-end tests may run after a build is known to be valid. Nevertheless, security checks should not be postponed indefinitely merely because they are slower.

A pipeline should answer these questions:

• What exact source revision is being built?
• Which worker image and tool versions are used?
• What causes execution?
• Which jobs may run concurrently?
• What artifact is produced, and where is it stored?
• What evidence is required for promotion?
• Who or what can deploy to each environment?
• How is failure reported and recovery performed?

Worked example: identifying parallelism

Suppose a pipeline has a 4-minute build, then three independent jobs: unit tests of 6 minutes, security scanning of 5 minutes, and packaging of 3 minutes. If they run in parallel and deployment takes 2 minutes after all three finish, the ideal execution time after the build is the duration of the longest parallel job, not their sum:

Substituting the values gives:

This is an idealized lower-bound calculation. It assumes sufficient workers, no queue time, no setup overhead, and no retries. If the jobs were executed sequentially, the corresponding time would be 4 + 6 + 5 + 3 + 2 = 20 minutes. The example illustrates why explicit dependency graphs and adequate runner capacity matter.

6. Applied Case: Illustrative Online Retail Pipeline

Background

An illustrative online retailer operates a web application, an API, and a containerized payment service. Developers frequently submit pull requests, but production releases have historically been manual and difficult to audit.

Problem

The organization has experienced three problems: tests are inconsistent between developers’ machines, deployments rebuild software separately for staging and production, and production credentials are embedded in deployment scripts.

Analysis

The existing process lacks a controlled artifact boundary, makes environment differences difficult to identify, and gives the deployment process more privilege than necessary. Manual operations also make it difficult to determine which source revision is running in production.

Approach

The team designs a pipeline with these stages:

1. Validation: linting, dependency checks, and unit tests on every pull request.
2. Build: creation of a container image tagged with the commit identifier.
3. Verification: integration tests and vulnerability scanning against that image.
4. Publication: push of the image to a protected registry, identified by immutable digest.
5. Staging deployment: automatic deployment using staging credentials.
6. Production gate: automated health checks plus approval from an authorized release role.
7. Production deployment: promotion of the same image digest, followed by smoke tests and monitoring.

The workflow is stored in version control. Workers are ephemeral. Secrets are held in a managed secret store, and the production deployment identity can deploy only the approved service and namespace. Rollback uses the previously known-good image digest rather than rebuilding source code.

Outcome

The organization gains traceability from commit to running image, consistent execution environments, and a clear separation between testing and production authorization. The deployment still contains a human gate because payment-service changes are considered high risk, but routine validation is automated and parallelized.

Learning points

The key improvement is not simply “more automation.” It is the combination of immutable artifacts, explicit dependencies, controlled environments, least privilege, observable outcomes, and a recovery procedure. Automation without these properties can increase the speed of an unsafe process.

7. Challenges, Governance, and Contemporary Practice

Pipelines can fail for reasons beyond application defects. Queue congestion, unavailable workers, expired certificates, registry outages, flaky tests, and incorrect environment configuration all affect delivery. Teams should distinguish product failure from pipeline failure and monitor both.

Flaky tests deserve special attention. Automatically retrying a failed test may reduce false alarms, but repeated retries can conceal real defects. Flakiness should be measured, assigned to an owner, and removed rather than normalized.

Supply-chain security is now central to pipeline governance. Organizations should verify dependencies, restrict untrusted actions, scan images and packages, generate provenance or attestation where supported, and protect artifact repositories from unauthorized replacement. Build logs and audit records should avoid exposing personal data, tokens, or customer information.

Artificial intelligence can assist with test generation, log summarization, failure classification, and configuration suggestions. It should not be granted unrestricted authority to modify production workflows or approve high-risk deployments. AI-generated pipeline changes require the same review, testing, security analysis, and accountability as human-authored changes.

Important contemporary practices include deployment strategies such as blue-green and canary release, feature flags, infrastructure-as-code, policy-as-code, and progressive delivery. These are extensions of pipeline thinking: delivery becomes a controlled feedback system rather than a single large event. They also introduce complexity and should be adopted when their risk-reduction benefits justify the operational cost.

8. Best-Practice Checklist

• Keep jobs focused and dependencies explicit.
• Prefer immutable, versioned artifacts and promote them unchanged.
• Use ephemeral workers where practical.
• Separate build, test, publication, and deployment permissions.
• Store pipeline definitions in version control.
• Protect workflow files and review changes.
• Use least-privilege, short-lived credentials where possible.
• Make logs, test results, artifacts, and approvals traceable.
• Use parallelism deliberately, but account for worker capacity.
• Treat flaky tests, unstable environments, and slow queues as engineering problems.
• Define rollback or forward-recovery procedures before production release.
• Match gates and approvals to business and technical risk.

Chapter Summary

A CI/CD pipeline is an automated, observable workflow that moves a source change toward a validated and controlled release. Pipelines contain stages, jobs, and steps; runners or agents execute jobs; triggers initiate work; dependencies determine ordering and data flow; artifacts carry build outputs; environments provide operational contexts; and gates control promotion. Pipeline-as-code improves reviewability, reproducibility, and recovery, but must be protected because workflow definitions can control sensitive operations. Common architectures include centralized, controller–worker, elastic-runner, promotion-based, monorepo, polyrepo, trunk-based, and branch-oriented models. Effective pipeline design combines automation with immutable artifacts, least privilege, explicit evidence, governance, observability, and recovery planning.

Key Terms

Pipeline: An automated workflow that validates, packages, and may deploy software.

Stage: A logical phase of pipeline execution.

Job: A schedulable unit of work executed by a runner or agent.

Step: An individual command or action within a job.

Runner: A worker that executes pipeline jobs.

Agent: Another common term for a pipeline execution worker.

Trigger: An event or rule that starts or schedules pipeline execution.

Artifact: A retained output produced for later use or promotion.

Dependency: A required relationship among software components or pipeline tasks.

Environment: An operational context such as test, staging, or production.

Gate: A condition that must be satisfied before progression.

Approval: An authorized human or policy decision allowing progression.

Pipeline-as-code: Storing workflow definitions in version control.

Immutable artifact: A release output that is not modified after publication.

Promotion: Moving the same artifact through successive environments.

Ephemeral worker: A short-lived execution worker discarded after a job or run.

Monorepo: A repository containing multiple components or services.

Polyrepo: An arrangement in which components are maintained in separate repositories.

Canary release: A release initially exposed to a limited proportion of users or traffic.

Review Questions

Short Answer Questions

1. What is the relationship between a pipeline, a stage, a job, and a step?
2. Why should a deployable artifact normally be promoted rather than rebuilt for each environment?
3. What is the difference between a runner and a trigger?
4. Give two examples of pipeline gates.
5. State two benefits and one risk of pipeline-as-code.

Long Answer Questions

1. Explain how artifacts, dependencies, and environments work together in a multi-stage promotion pipeline.
2. Compare hosted and self-hosted runners, including their operational and security implications.
3. Describe a controller–worker architecture and explain why worker isolation matters.
4. Discuss how trunk-based and branch-oriented development influence pipeline design.
5. Explain how least privilege, approvals, auditability, and rollback contribute to safe production delivery.

Analytical / Application Questions

1. Design a pipeline for a small web application that must run tests on pull requests and deploy to production only after staging verification. Identify its triggers, jobs, artifacts, dependencies, environments, and gates.
2. A team rebuilds its application independently for test and production. Analyze the risks and propose a corrected artifact strategy.
3. A pipeline has a long queue, flaky integration tests, and a self-hosted worker with production credentials. Identify the highest-priority risks and recommend a staged improvement plan.

Multiple Choice Questions with Answers

Question 1

Which item is the best definition of a pipeline artifact?

A. A rule that starts a workflow

B. A retained output produced for later use or promotion

C. A machine that executes a job

D. A condition requiring approval

Correct Answer: B. A retained output produced for later use or promotion

Question 2

What is the primary role of a runner or agent?

A. To store source-code history

B. To approve production changes

C. To execute pipeline jobs

D. To define application requirements

Correct Answer: C. To execute pipeline jobs

Question 3

Which practice best supports reproducible promotion across environments?

A. Rebuild the application separately in every environment

B. Promote the same immutable artifact

C. Use an unversioned cache as the release source

D. Copy commands manually between servers

Correct Answer: B. Promote the same immutable artifact

Question 4

Which of the following is a pipeline trigger?

A. A pull-request update

B. A compiled binary

C. A production database

D. A test assertion

Correct Answer: A. A pull-request update

Question 5

What does a pipeline dependency primarily express?

A. A visual color scheme for jobs

B. A relationship controlling ordering or required output

C. A password for a deployment account

D. A business-service level agreement

Correct Answer: B. A relationship controlling ordering or required output

Question 6

Which statement about pipeline-as-code is correct?

A. It prevents all workflow security risks

B. It stores workflow definitions in version control

C. It requires every job to run on one server

D. It eliminates the need for code review

Correct Answer: B. It stores workflow definitions in version control

Question 7

A build takes 4 minutes, parallel tests take 6, 5, and 3 minutes, and deployment takes 2 minutes. Under ideal conditions, what is the total time?

A. 12 minutes

B. 15 minutes

C. 20 minutes

D. 24 minutes

Correct Answer: A. 12 minutes

Question 8

Which is an appropriate use of a gate?

A. Allowing any unreviewed code to deploy automatically

B. Requiring health checks before production promotion

C. Replacing all automated tests with manual approval

D. Disabling audit records after deployment

Correct Answer: B. Requiring health checks before production promotion

Question 9

What is a principal advantage of ephemeral workers?

A. They preserve all previous workspaces permanently

B. They reduce contamination between builds

C. They require production credentials by default

D. They eliminate the need for capacity planning

Correct Answer: B. They reduce contamination between builds

Question 10

Which architecture uses a coordinating controller and multiple execution workers?

A. Controller–worker architecture

B. Single-user desktop architecture

C. Manual release architecture

D. Artifact-only architecture

Correct Answer: A. Controller–worker architecture