4. Build Engineering, Dependencies and Artifact Management

Introduction and Context

Continuous integration and continuous deployment are often described as automation practices, but automation alone does not make delivery reliable. A pipeline can execute rapidly and still produce software that cannot be reproduced, traced, secured, or promoted confidently between environments. The foundation of dependable delivery is build engineering: the disciplined design of the processes, tools, inputs, outputs, and controls that transform source code into deployable artifacts.

A build is more than compilation. It may include dependency resolution, code generation, testing, packaging, container construction, signing, metadata generation, and publication to an artifact repository. Each stage introduces possible variation. A changing dependency, an unrecorded compiler version, an external network call, a mutable container tag, or an uncontrolled build script can cause two executions of apparently identical source code to produce different results.

This chapter examines how DevOps teams create reproducible and hermetic builds, manage dependencies and caches, optimize build performance, generate software bills of materials (SBOMs), and govern binaries, packages, containers, and repositories across environments. The subject is both technical and managerial. Build decisions affect security, auditability, developer productivity, release risk, cost, and the organization’s ability to respond to vulnerabilities.

Learning Objectives

By the end of this chapter, learners should be able to:

1. Distinguish reproducible, deterministic, and hermetic builds.

2. Explain how dependency resolution, versioning, lockfiles, and transitive dependencies affect CI/CD reliability.

3. Design dependency, cache, and artifact strategies for multi-stage delivery pipelines.

4. Evaluate build optimization techniques without weakening correctness or security.

5. Explain the purpose and limitations of SBOMs and artifact provenance.

6. Define governance policies for packages, binaries, containers, repositories, and promotions.

7. Analyze build failures and supply-chain risks in realistic organizational settings.

8. Select appropriate controls for trust, traceability, retention, and deployment promotion.

Build Engineering as a Delivery Discipline

Build engineering concerns the controlled transformation of human-authored inputs into software outputs. Inputs may include source files, build scripts, compilers, operating-system libraries, package-manager metadata, credentials, environment variables, and external services. Outputs may include executable binaries, libraries, archives, container images, deployment manifests, and test reports.

A useful abstraction is:

Artifact = f(source, dependencies, tools, configuration, environment, time)

The objective of mature build engineering is to make the relevant inputs explicit and controlled. If an output changes, the team should be able to determine which input changed. If the same source and declared inputs are used again, the result should be identical or sufficiently equivalent for the intended purpose.

A build is reproducible when independent executions using the same declared inputs produce the same output, or produce outputs whose meaningful content is equivalent. Reproducibility supports debugging, audit, incident investigation, and trust in release artifacts.

A deterministic build produces the same result for the same inputs. In practice, timestamps, file ordering, random identifiers, locale settings, compiler behavior, and generated metadata can undermine determinism. Reproducibility is the broader engineering property; determinism is one of its important mechanisms.

A hermetic build prevents undeclared external influences from affecting the result. A hermetic build typically controls network access, tool versions, environment variables, filesystem inputs, and operating-system assumptions. Hermeticity is not identical to reproducibility: a build may be reproducible by convention while still depending on an uncontrolled external service. Hermeticity provides stronger isolation and makes the build’s dependency boundary explicit.

These properties should be treated as levels of assurance rather than slogans. A local build that uses a developer’s home directory, globally installed tools, and the current time is weakly controlled. A CI build using a pinned toolchain, a lockfile, an isolated workspace, and a controlled dependency mirror is substantially stronger. A highly mature build additionally records provenance, signs outputs, and permits an independent party to reconstruct the artifact.

Dependencies: The Hidden Input to Every Build

A dependency is any external component required to build, test, package, or execute software. Direct dependencies are explicitly requested by a project. Transitive dependencies are brought in by those direct dependencies. Build dependencies, test dependencies, and runtime dependencies may have different risk and lifecycle characteristics.

Dependency management begins with declaration. A project should state its dependencies in machine-readable manifests and should distinguish acceptable version ranges from exact versions. A range such as `^2.4` can improve access to compatible fixes but may also produce different results at different times. Exact versions improve repeatability but require a deliberate update process. Lockfiles record the resolved dependency graph, including versions and often integrity hashes. They should normally be reviewed and committed to source control for applications and deployable services.

Versioning schemes such as semantic versioning provide useful expectations about compatibility, but they do not guarantee that a package is safe or that its maintainers follow the rules consistently. A minor version can introduce an operational problem, and a package with a patch release can still contain a vulnerability. Dependency policy therefore requires both version discipline and security assessment.

A dependency graph should be considered an organizational asset. Teams need to know:

• which components are directly and transitively included;
• which licenses apply;
• who owns or maintains each component;
• whether a component is supported;
• whether it contains known vulnerabilities;
• whether it is present at build time, runtime, or both; and
• which released artifacts contain it.

Dependency confusion and typosquatting illustrate why source selection matters. A package manager may retrieve a malicious public package when an internal package name is not properly scoped or when repository precedence is ambiguous. Organizations should use trusted registries, namespace controls, allowlists where appropriate, and explicit repository configuration.

Reproducible and Hermetic Build Design

A reproducible build begins with a complete description of inputs. Source code should be identified by an immutable commit or source archive. Dependencies should be locked and verified by checksums. Compilers, interpreters, package managers, and operating-system images should be versioned. Build configuration should be stored with the project or referenced through a version-controlled, reviewable system.

Time is a frequent source of variation. Generated files may contain the current timestamp, archive entries may be ordered differently, and certificates or metadata may include creation dates. Where possible, builds should use a controlled build timestamp, stable file ordering, normalized paths, and deterministic archive settings. Randomness should be seeded or removed from release output unless it is required for security.

Environment isolation is equally important. Builds should not silently depend on a developer’s globally installed package, an undeclared environment variable, or a file left in a workspace by an earlier job. Ephemeral CI workers, isolated workspaces, containerized build environments, and explicit environment declarations reduce such hidden dependencies.

Network access deserves particular attention. A build that downloads packages during execution is exposed to outages, repository compromise, and changing upstream content. A stronger pattern is to resolve dependencies through an approved internal proxy or mirror, verify integrity, and, for sensitive releases, make all required inputs available before the build begins. Network restrictions also reduce the risk that a compromised build script exfiltrates secrets.

Hermeticity is often implemented through a combination of:

1. pinned base images and toolchains;

2. lockfiles and cryptographic checksums;

3. isolated workers and clean workspaces;

4. controlled or disabled network access;

5. explicit environment and filesystem inputs;

6. deterministic packaging settings; and

7. recorded provenance describing how the artifact was created.

Hermeticity has a cost. Fully isolated builds may require internal mirrors, additional storage, and more maintenance. The appropriate level depends on risk. A regulated payment service, a widely distributed software product, and a short-lived internal prototype should not necessarily have identical controls.

Build Systems, Dependency Graphs and Caching

Modern build systems represent work as a directed acyclic graph. Nodes represent actions such as compiling a module, generating code, running tests, or building an image. Edges represent dependencies between actions. This model enables incremental execution: if an input has not changed, its previous result may be reused.

Caching can substantially reduce pipeline duration, but it is a correctness mechanism as well as a performance mechanism. A cache key must represent every input capable of changing the output. If a compiler version, build flag, operating-system library, or dependency lockfile is omitted from the key, stale output may be reused incorrectly.

Common cache types include:

• dependency caches, which store downloaded packages;
• compilation caches, which reuse object files or compiled modules;
• task-output caches, which reuse completed build actions; and
• container layer caches, which reuse unchanged image layers.

Caches should be treated as disposable and untrusted from a correctness perspective. The authoritative result is a newly verified artifact, not an assumption that a cached result is valid. Cache poisoning is a security concern: an attacker who can write to a shared cache may influence later builds. Access controls, namespace separation, integrity checks, and cautious cache permissions are therefore necessary.

Build optimization should begin with measurement. Useful metrics include total pipeline duration, queue time, execution time, cache hit rate, critical-path duration, dependency-download time, and failure-retry rate. Parallel execution reduces elapsed time only when workers, licenses, memory, and I/O are sufficient. Excessive parallelism can increase cost or create resource contention.

Other optimization techniques include modular builds, test selection based on dependency impact, prebuilt toolchain images, remote execution, artifact reuse, and parallel stages. However, optimization must not bypass essential tests, weaken isolation, or create different behavior between validation and release builds. A fast but unreliable pipeline is not efficient; it transfers cost to debugging and production incidents.

From Build Outputs to Governed Artifacts

An artifact is a versioned output intended for reuse, deployment, distribution, or evidence. Examples include a Java archive, Python wheel, operating-system package, compiled binary, Helm chart, Terraform module, or container image. Source repositories store inputs; artifact repositories store outputs. This separation allows teams to promote a tested output rather than rebuilding it for each environment.

A robust artifact identity should be immutable and unambiguous. A human-readable version such as `2.7.1` is useful, but a content digest is stronger because it identifies the exact bytes. Container tags such as `latest` or `production` are convenient pointers, not immutable identities. Deployment systems should record the digest actually deployed.

Artifact repositories generally provide storage, metadata, access control, retention policies, vulnerability scanning, and replication. They may host multiple formats, including container images and language-specific packages. Repository design should reflect organizational boundaries and promotion practices. Development snapshots, release candidates, and production-approved artifacts should not be mixed without clear status metadata.

The preferred promotion model is often:

build once → test → scan → sign → approve → promote the same artifact

Rebuilding separately for testing, staging, and production weakens traceability because the outputs may differ. Environment-specific configuration should normally be supplied at deployment time, while the application artifact remains unchanged.

Artifact metadata should include, where appropriate, version, digest, source revision, build timestamp, toolchain information, dependency information, test status, scan results, owner, license data, and approval history. This metadata enables release decisions and incident response.

SBOMs, Provenance and Supply-Chain Assurance

A software bill of materials is a structured inventory of components contained in or required by a software artifact. An SBOM may describe packages, libraries, versions, relationships, licenses, and suppliers. Common exchange formats include SPDX and CycloneDX. The format matters less than the accuracy, completeness, and operational use of the information.

SBOM generation should occur as close as possible to the build process and should be associated with the exact artifact digest. An SBOM for source code alone may not reflect generated components, operating-system packages, or the final container image. Teams should therefore decide whether they need source, build, application, container, or combined SBOMs.

An SBOM is not a security certificate. It does not prove that a component is safe, that the artifact was built securely, or that all vulnerabilities are known. It is an inventory that supports vulnerability matching, license review, customer communication, and targeted response. Vulnerability data can be incomplete or incorrectly mapped, and exploitability depends on configuration and usage.

Provenance complements the SBOM. It records facts such as the source revision, builder identity, build process, parameters, dependencies, and resulting digest. Attestations can be signed and verified by deployment policy. A mature supply-chain process asks not only “What is in this artifact?” but also “How was it built, by which trusted process, and from which inputs?”

Governance Across Environments

Governance determines who may create, modify, promote, delete, or deploy artifacts. It also defines how long artifacts are retained, how exceptions are handled, and what evidence is required for release.

A practical governance model includes the following controls:

• immutable release repositories or immutable versions;
• role-based access control and separation of duties;
• signed artifacts or signed provenance for high-risk systems;
• vulnerability and license policies with documented exceptions;
• retention rules based on operational, legal, and recovery requirements;
• audit logs for publication, promotion, deletion, and deployment;
• controlled replication between regions or environments; and
• disaster-recovery procedures for repositories and signing keys.

Promotion should be policy-driven rather than dependent on informal communication. For example, a production deployment may require a successful test suite, an acceptable vulnerability status, a valid signature, an approved SBOM, and evidence that the artifact originated from the protected release pipeline.

Exceptions are inevitable, particularly when a vulnerability has no available fix or when a legacy component is temporarily required. An exception should identify the affected artifact, business justification, owner, compensating controls, expiry date, and review authority. Permanent undocumented exceptions convert governance into appearance rather than control.

An Illustrative Case: Rebuilding the Same Service Three Times

Background

Northstar Payments, a fictional financial-technology organization, operates a set of containerized transaction services. Its pipeline builds an image for development, rebuilds the application for staging, and rebuilds it again during production release. The organization has recently experienced inconsistent behavior between staging and production.

Situation and Problem

A production image contains a different transitive library version from the staging image, despite both being associated with release 8.4. A developer had updated a package mirror between the two builds. In addition, the production build used a newer base image because the Dockerfile referenced a floating tag. The team could not determine precisely which source and dependencies were included in the deployed image. A later vulnerability announcement required identifying all affected releases, but no reliable component inventory existed.

Analysis

The process violated several build-engineering principles. Dependency resolution was not locked consistently, the base image was mutable, and the artifact was rebuilt rather than promoted. The repository stored tags but did not enforce digest-based deployment. The absence of an SBOM and provenance record increased the cost of incident analysis. Although the pipeline was automated, it was not reproducible or hermetic.

Approach and Solution

Northstar introduced a protected release workflow. Dependencies were resolved through an internal registry and recorded in lockfiles with integrity hashes. The build image and operating-system packages were pinned by digest. Release workers used clean, ephemeral environments, and outbound network access was restricted after dependency preparation.

The pipeline now creates one release candidate, executes tests and security checks, generates an SBOM, signs the image and provenance, and publishes the result under an immutable digest. Staging and production deploy that same digest. Human-readable tags remain available for discovery, but deployment policy requires a digest. Promotion records include approvals, scan results, and the source revision.

The team also redesigned caching. Dependency caches are separated by project and toolchain version, and cache keys include the lockfile and build-image digest. Cache misses are monitored rather than treated as failures. Repository retention keeps approved releases and recent development artifacts while deleting obsolete snapshots according to policy.

Outcome

The organization can now identify the exact source, dependencies, toolchain, and policy decisions associated with a deployment. A vulnerability affecting a common library can be matched against SBOM records to identify potentially affected artifacts. Staging and production use the same bytes, reducing environment drift. Build duration initially increased because of dependency preparation and scanning, but subsequent caching and parallelization reduced the critical path.

Learning Points

The case illustrates that artifact promotion is safer than repeated rebuilding; immutable digests are stronger than mutable tags; SBOMs support response but do not replace security controls; and performance improvements must be designed around trustworthy cache keys and reproducible inputs.

Challenges, Risks and Common Mistakes

One common mistake is confusing a successful build with a trustworthy build. A pipeline can pass while silently retrieving an unreviewed package or using a changing base image. Another mistake is pinning only direct dependencies while allowing transitive dependencies to float. Teams also frequently cache package directories without including the lockfile or toolchain in the cache key.

Overly aggressive cleanup can remove the only copy of a release needed for rollback or legal investigation. Conversely, unlimited retention increases cost and may preserve vulnerable or sensitive artifacts indefinitely. Repository policies must balance recoverability, compliance, storage cost, and data exposure.

Security scanning creates its own challenges. False positives can produce alert fatigue, while false negatives arise from incomplete inventories or unavailable vulnerability data. A severity score alone does not establish exploitability. Decisions should consider exposure, reachability, compensating controls, fix availability, and business criticality.

Build systems may also leak secrets through logs, cached files, image layers, or package-manager configuration. Secrets should be injected only when needed, masked in logs, and excluded from final artifacts. Build permissions should be minimal, especially where a compromised dependency or script could execute arbitrary commands.

Professional and Ethical Considerations

Artifact governance affects customers, colleagues, and the public. Releasing software without understanding its components can expose users to avoidable security and licensing risks. Engineers have a professional responsibility to represent build evidence honestly and not claim reproducibility, testing, or compliance that has not been demonstrated.

Organizations should respect open-source licenses and preserve required notices. They should also protect private package contents, signing keys, customer data, and proprietary build metadata. Automated policy must be explainable: a team should know why an artifact was blocked and how a justified exception can be reviewed.

In regulated sectors, retention, auditability, segregation of duties, and traceability may be mandatory. Even where regulation does not prescribe a particular tool, the organization should be able to show what was built, from which inputs, by which process, and under whose authority it was released.

Current Developments and Emerging Practice

Build systems are increasingly adopting remote execution, content-addressable storage, signed attestations, and policy-as-code. These developments support scalable builds and stronger verification, but they also expand the trust boundary. Remote workers, caches, registries, and signing services must themselves be authenticated and monitored.

Software supply-chain frameworks increasingly emphasize provenance, verified source, protected build platforms, and continuous vulnerability response. Organizations are also moving toward risk-based dependency update automation, where routine updates are tested and proposed automatically while high-impact changes require deeper review.

Artificial intelligence can assist with dependency upgrade proposals, vulnerability triage, build-log diagnosis, and SBOM analysis. It should not be treated as an authority: generated recommendations require validation, and sensitive source, logs, and dependency data must be handled according to organizational policy. AI-generated code also becomes part of the same dependency, licensing, testing, and provenance obligations as other code.

Best Practices for Build and Artifact Operations

A mature organization should establish a small number of enforceable principles:

1. Declare and lock all meaningful build inputs.

2. Use immutable source references, dependency versions, base images, and release artifacts.

3. Build in clean, isolated, and preferably network-controlled environments.

4. Generate SBOMs and provenance for the exact artifact digest.

5. Build once and promote the same artifact across environments.

6. Design cache keys from the complete input set and protect cache writes.

7. Separate development, candidate, and production repositories or statuses.

8. Apply vulnerability, license, signature, and approval policies automatically where possible.

9. Retain sufficient artifacts and metadata for rollback, audit, and incident response.

10. Measure build performance, reliability, cache effectiveness, and policy exceptions.

These practices should be adapted to system criticality. The goal is not maximal process for its own sake, but evidence-based control of the risks introduced by software transformation and distribution.

Chapter Summary

Build engineering makes CI/CD dependable by controlling the transformation from source to deployable output. Reproducible builds use declared and stable inputs; hermetic builds additionally prevent undeclared external influences. Dependency manifests, lockfiles, checksums, trusted registries, and pinned toolchains reduce variation and supply-chain risk.

Caching and incremental execution improve performance when cache keys include all relevant inputs and cache contents are protected. Artifact repositories should preserve immutable, traceable outputs, while promotion of the same artifact across environments is safer than repeated rebuilding. SBOMs describe artifact contents, and provenance describes how artifacts were produced; together they strengthen vulnerability response and release assurance but do not guarantee safety.

Effective governance combines access control, signatures, policy checks, retention, audit trails, exception management, and recovery planning. Build optimization, security, compliance, and developer experience must be considered together. The central principle is simple: a release should be identifiable, reproducible, inspectable, and promoted through evidence-based controls.

Key Terms

Reproducible build: A build that produces the same or meaningfully equivalent output from the same declared inputs.

Deterministic build: A build whose output is fixed for a given set of inputs.

Hermetic build: A build isolated from undeclared external influences.

Dependency: An external component required to build, test, package, or run software.

Transitive dependency: A dependency introduced through another dependency.

Lockfile: A file recording resolved dependency versions and often integrity information.

Content digest: A cryptographic identifier derived from artifact content.

Artifact: A versioned output intended for reuse, deployment, or distribution.

Artifact repository: A controlled service that stores and manages build outputs.

Cache key: An identifier used to determine whether a prior build result can be reused.

Incremental build: A build that executes only work affected by changed inputs.

SBOM: A structured inventory of software components and relationships in an artifact.

Provenance: Metadata describing an artifact’s origin, inputs, builder, and build process.

Attestation: A signed statement asserting properties about an artifact or process.

Artifact promotion: Moving an already-built artifact through approval stages or environments.

Immutable artifact: An artifact whose content cannot change under its identity.

Dependency confusion: A supply-chain attack involving retrieval of an unintended package, often from a public registry.

Build cache poisoning: The insertion of malicious or incorrect data into a cache used by later builds.

Software supply chain: The people, processes, tools, dependencies, and infrastructure involved in producing software.

Review Questions

Short Answer Questions

1. What distinguishes a reproducible build from a hermetic build?

2. Why are lockfiles important for application builds?

3. What is the difference between an artifact tag and an artifact digest?

4. What information does an SBOM provide?

5. Why should cache keys include the toolchain version?

Long Answer Questions

1. Explain how an organization can design a hermetic build environment and discuss the costs of doing so.

2. Describe a complete dependency-management strategy for a large organization using multiple package ecosystems.

3. Discuss why building once and promoting the same artifact is generally safer than rebuilding for each environment.

4. Explain the relationship among SBOMs, provenance, signatures, vulnerability scanning, and deployment policy.

5. Analyze the governance requirements of an enterprise artifact repository, including access, retention, promotion, and auditability.

Analytical / Application Questions

1. A pipeline uses a dependency lockfile but produces different container images on different days. Identify at least four possible causes and propose diagnostic tests for each.

2. A team wants to improve build time by sharing one global cache across all projects. Evaluate the proposal, including performance benefits, correctness risks, and security controls.

3. A critical vulnerability is discovered in a transitive library. Explain how SBOMs, artifact digests, repository metadata, and deployment records could be used to determine the organization’s exposure and response priorities.

22. Multiple Choice Questions with Answers

Question 1

Which property most directly prevents undeclared network services or local files from influencing a build?

A. Incremental execution

B. Hermeticity

C. Semantic versioning

D. Artifact retention

Correct Answer: B. Hermeticity

Question 2

Why should a cache key include the compiler or toolchain version?

A. To make artifact tags shorter

B. To ensure old outputs are not reused with an incompatible toolchain

C. To eliminate the need for lockfiles

D. To permit unrestricted network access

Correct Answer: B. To ensure old outputs are not reused with an incompatible toolchain

Question 3

Which practice provides the strongest assurance that staging and production receive identical software bytes?

A. Rebuilding from the same branch in each environment

B. Using the latest container tag

C. Promoting one immutable artifact by digest

D. Copying build scripts manually between environments

Correct Answer: C. Promoting one immutable artifact by digest

Question 4

What is the primary purpose of an SBOM?

A. To prove that software contains no vulnerabilities

B. To inventory components and their relationships in an artifact

C. To replace all software testing

D. To encrypt container images

Correct Answer: B. To inventory components and their relationships in an artifact

Question 5

Which example is a transitive dependency?

A. A library explicitly listed in the project manifest

B. A compiler installed on a build worker

C. A library required by a library directly selected by the project

D. A deployment approval record

Correct Answer: C. A library required by a library directly selected by the project

Question 6

A container is deployed using a mutable tag such as `latest`. What is the principal governance weakness?

A. The tag cannot be displayed in a repository

B. The tag may refer to different content at different times

C. The image cannot contain an SBOM

D. The container will never pass tests

Correct Answer: B. The tag may refer to different content at different times

Question 7

Which sequence best represents a controlled release flow?

A. Build, promote, test, rebuild, scan

B. Build, test, scan, sign, approve, promote the same artifact

C. Deploy, build, approve, delete, scan

D. Tag, rebuild in production, then test

Correct Answer: B. Build, test, scan, sign, approve, promote the same artifact

Question 8

What is a major risk of a cache whose key omits the dependency lockfile?

A. The repository will require more human approvals

B. A result based on an old dependency graph may be reused incorrectly

C. The build will become hermetic automatically

D. The artifact digest will become shorter

Correct Answer: B. A result based on an old dependency graph may be reused incorrectly

Question 9

Which statement best describes provenance?

A. It is only a list of software licenses

B. It records how and from which inputs an artifact was produced

C. It is a replacement for access control

D. It is a mutable deployment tag

Correct Answer: B. It records how and from which inputs an artifact was produced

Question 10

A vulnerability has no available fix, but a service must temporarily use the affected library. What is the most appropriate governance response?

A. Ignore the finding permanently

B. Delete all evidence of the dependency

C. Document a time-bound exception with ownership and compensating controls

D. Disable vulnerability scanning

Correct Answer: C. Document a time-bound exception with ownership and compensating controls