8. CI/CD, GitOps and Software Supply Chains

Introduction and Context

Cloud-native systems are not delivered as isolated application binaries. They are assembled from source code, dependencies, container images, infrastructure definitions, configuration, policies, and operational procedures. Consequently, deployment is no longer merely the final technical step after development. It is an engineered control system that must move changes rapidly while preserving reliability, security, auditability, and the ability to recover.

Continuous integration and continuous delivery or deployment (CI/CD) provide the automation model for this movement. GitOps extends the model by treating declarative system configuration in version control as the authoritative desired state and by using automated reconciliation to make the running environment conform to that state. Software supply-chain security adds another dimension: the organisation must know what it is building, where components originated, how they were transformed, whether they were tampered with, and under what conditions they may be deployed.

For advanced Kubernetes practitioners, these ideas are inseparable. A Kubernetes manifest can be syntactically correct but operationally unsafe; an image can pass functional tests but contain a vulnerable dependency; a pipeline can deploy successfully but provide no reliable evidence of what was released. This chapter examines delivery as a socio-technical architecture involving people, repositories, automation, policy, platforms, and measurable outcomes.

Learning Objectives

By the end of this chapter, the learner should be able to:

1. Distinguish continuous integration, continuous delivery, and continuous deployment.

2. Design a Kubernetes-oriented CI/CD pipeline with appropriate quality and security gates.

3. Explain GitOps principles, reconciliation, drift detection, and repository structures.

4. Select progressive delivery strategies according to risk, speed, and recovery requirements.

5. Describe the role of registries, SBOMs, digital signatures, attestations, and provenance.

6. Evaluate delivery performance using deployment frequency, lead time, failure rate, recovery time, and related indicators.

7. Identify governance, security, ethical, and operational risks in software supply chains.

From Code Change to Trusted Release

Continuous integration is the practice of integrating small changes frequently into a shared codebase and validating them through automated builds and tests. Its purpose is not simply to run a build server. It reduces integration uncertainty by discovering compilation errors, interface conflicts, failing tests, and quality regressions near the time they are introduced.

Continuous delivery extends this practice by keeping software in a releasable condition. A successful pipeline produces a versioned, deployable artifact, but production release may still require an explicit business or operational decision. Continuous deployment removes that manual release decision for changes that satisfy defined controls. The distinction is therefore one of organisational policy and risk tolerance, not merely tooling.

A useful delivery chain is:

```text

Commit -> Build -> Unit test -> Package -> Scan -> Sign ->

Integration test -> Deploy to environment -> Verify -> Promote or roll back

```

The artifact should be built once and promoted between environments rather than rebuilt separately for each environment. Rebuilding can introduce variation and makes it difficult to prove that the tested artifact is the artifact ultimately deployed. Environment-specific values should normally be supplied through controlled configuration, secrets management, or deployment overlays, while the immutable application artifact remains constant.

Pipeline stages are controls with different purposes. Fast static checks and unit tests provide early feedback. Integration and contract tests evaluate interactions with databases, APIs, queues, and other services. Security checks identify vulnerable dependencies, insecure container configurations, exposed secrets, and policy violations. Deployment verification evaluates runtime behaviour rather than merely the success of an API call to the Kubernetes control plane.

A mature pipeline is not necessarily one with the greatest number of stages. Excessive gates can create queues, encourage bypasses, and reduce learning speed. The appropriate question is whether each control addresses a material risk, provides actionable feedback, and operates at a suitable point in the delivery process.

The GitOps Operating Model

GitOps uses a version-controlled repository as the declarative source of truth for the desired state of an environment. A controller running in or near the Kubernetes cluster observes that desired state and reconciles the actual state toward it. This differs from a conventional push pipeline in which an external system connects to a cluster and issues imperative deployment commands.

Four principles are central. First, the desired state is declarative: it describes what should exist rather than listing every command required to create it. Second, the desired state is versioned and reviewable. Third, an automated agent continuously pulls or observes changes and reconciles them. Fourth, the system provides feedback when actual state diverges from desired state.

Reconciliation is continuous rather than a one-time event. If an operator changes a Deployment directly using a command-line tool, the cluster may temporarily differ from Git. The controller can report drift and, depending on policy, restore the Git-defined state. This creates consistency and auditability, although it also means that emergency changes must be handled through an explicit break-glass process and subsequently recorded in version control.

A common architecture separates application source repositories from environment repositories. The application repository contains source, tests, build instructions, and perhaps a base deployment definition. The environment repository contains the versions and configuration approved for development, staging, or production. A successful CI pipeline publishes an image and proposes an update to the appropriate environment repository. Review, automated policy checks, and merge controls then determine whether the deployment controller will reconcile the change.

Repository design involves trade-offs. A monorepo can simplify coordinated changes and visibility, whereas multiple repositories can strengthen ownership boundaries and reduce accidental access. Environment repositories should not become unreviewable collections of generated files. Templates, Helm charts, Kustomize overlays, or other mechanisms should remain understandable, and rendered output should be validated before promotion.

GitOps does not mean that Git alone is a security control. Repository access, branch protection, commit signing, secrets handling, controller permissions, and audit logging remain necessary. Nor does GitOps eliminate deployment orchestration; it changes the authority model from an imperative external actor to a declarative, continuously reconciling system.

Delivery Architecture for Kubernetes

A robust architecture commonly includes the following logical elements:

• A source repository containing application and infrastructure code.
• A CI runner operating with short-lived credentials and isolated build environments.
• An artifact registry for container images, packages, charts, and metadata.
• Test and security services, including scanners and policy engines.
• An environment repository describing approved desired states.
• A deployment or GitOps controller operating with narrowly scoped cluster permissions.
• Observability systems that provide logs, metrics, traces, events, and release annotations.
• An approval and audit system for exceptional or high-risk changes.

The separation between build and deploy privileges is important. A build runner should not automatically have broad production cluster credentials. Ideally, CI publishes an artifact and changes declarative configuration; a controller in the target environment performs deployment using an identity limited to that cluster and namespace scope. This reduces the impact of a compromised pipeline runner.

Secrets require special treatment. Plain-text credentials must not be stored in source repositories or embedded in container images. External secret managers, encrypted repository mechanisms, or Kubernetes-integrated secret operators can supply secrets at runtime. Secret rotation, access logging, and revocation must be designed rather than assumed.

Testing as Evidence for Promotion

Testing in delivery pipelines is a layered evidence model. Unit tests provide rapid feedback about local logic. Component tests evaluate a service with controlled dependencies. Integration tests examine real interactions. Contract tests verify that service providers and consumers agree on interfaces. End-to-end tests assess critical user journeys but are usually slower and more fragile.

Infrastructure and policy tests are equally relevant in Kubernetes. Manifest validation can detect schema errors, invalid resource requests, or prohibited settings. Policy tests may reject privileged containers, unapproved registries, missing labels, absent probes, or workloads lacking network restrictions. Chaos and resilience tests can examine behaviour under dependency failure, pod disruption, latency, or node loss.

No single test suite proves production safety. Test selection should be risk-based. A low-risk documentation change need not wait for the same extensive test sequence as a payment-processing change, but the classification itself must be governed and auditable. Flaky tests are particularly damaging because they weaken trust in the pipeline and encourage retries without investigation.

Release verification should include health and business signals. Kubernetes readiness indicates whether a pod is prepared to receive traffic; it does not prove that transactions are correct. Useful post-deployment checks may include error rate, latency, saturation, queue depth, successful business operations, and comparison with the previous version.

Progressive Delivery and Deployment Choices

Deployment strategy determines how exposure, risk, and reversibility are balanced.

A rolling update replaces instances gradually. It is simple and resource-efficient, but the old and new versions coexist and may interact with shared state. Readiness probes, compatible APIs, and careful database migration design are essential.

A blue-green deployment maintains two environments. Traffic switches from the blue version to the green version after validation. Rollback can be rapid because the previous environment remains available, although the approach may require substantially more capacity and careful handling of persistent data.

A canary deployment exposes a small proportion of traffic to the new version before increasing exposure. It is valuable when real production behaviour is difficult to reproduce in tests. Automated analysis can compare error rate, latency, resource use, and business indicators between cohorts.

Feature flags separate code deployment from feature activation. They allow a capability to be enabled for selected users or disabled without rebuilding the image. Flags introduce operational debt, privacy concerns, and configuration complexity; they require ownership, expiration dates, and testing of both states.

Shadow traffic sends copies of requests to a candidate version without using its responses for real users. It can reveal performance and compatibility problems, but side effects must be prevented and sensitive data must be handled appropriately.

The choice should be based on risk and recovery objectives. A strategy is not “safe” merely because it is automated. Important measures include deployment frequency, lead time from approved change to production, change failure rate, mean time to recovery, rollback duration, blast radius, and the percentage of deployments requiring manual intervention. These indicators should be interpreted together: increasing deployment frequency while failure rate and recovery time rise may represent deterioration rather than improvement.

Software Supply-Chain Assurance

A software supply chain includes developers, source repositories, dependencies, build tools, CI runners, registries, deployment systems, and runtime environments. Each link may be attacked or may introduce accidental risk. Supply-chain assurance therefore concerns both cybersecurity and evidence.

An artifact registry stores immutable or controlled versions of images and packages. Tags such as `latest` are convenient but ambiguous; production systems should refer to immutable digests or tightly governed version identifiers. Retention, replication, malware scanning, access control, and provenance metadata are registry responsibilities.

A software bill of materials (SBOM) is an inventory of components in an artifact, including their versions and relationships where supported. SBOMs assist vulnerability response, licensing review, incident investigation, and customer disclosure. They are not a guarantee that software is secure: inventories can be incomplete, vulnerability databases can contain errors, and exploitability depends on context.

Digital signing provides a mechanism for verifying that an artifact or attestation was produced by an authorised identity and has not been altered. Verification is meaningful only when trust roots, key management, identity binding, and revocation procedures are defined. A signature proves origin and integrity claims; it does not prove that the code is free from defects.

Provenance records how an artifact was built: source revision, builder identity, build parameters, dependencies, and timestamps. Attestations can also record test results, scan outcomes, or policy decisions. Together, SBOMs, signatures, and provenance enable a policy such as: only deploy an image signed by an approved builder, linked to a reviewed source revision, built in an isolated environment, and accompanied by acceptable scan evidence.

Build security includes dependency pinning, lock files, reproducible or hermetic builds where feasible, isolated runners, protected build definitions, and restricted network access. A compromised dependency or build plugin can affect many downstream applications. Organisations should distinguish between a vulnerability that is present in an inventory and one that is reachable and exploitable in the deployed service.

Policy, Governance, and Responsible Automation

Policy-as-code expresses organisational rules in machine-evaluable form. Examples include requiring approved base images, prohibiting privileged execution, enforcing resource limits, checking image signatures, or preventing production deployment outside an authorised window. Policies should produce understandable explanations and provide an exception process with an owner, justification, expiry, and compensating controls.

Governance must balance standardisation with delivery autonomy. Platform teams can provide paved roads—approved pipeline templates, registries, deployment patterns, observability defaults, and secure identity integration—while product teams retain responsibility for application behaviour and service outcomes.

Regulatory obligations may require evidence of access control, separation of duties, change approval, retention, vulnerability management, or incident response. Auditability should not be reduced to storing every log indefinitely. Records must be reliable, appropriately retained, protected from unauthorised alteration, and usable for reconstruction of a decision.

Ethical concerns include the use of automated risk scoring, collection of developer activity data, and deployment decisions that affect users without adequate review. Metrics should improve systems rather than become instruments for punitive individual ranking. Supply-chain transparency also raises confidentiality questions: SBOMs and provenance may reveal internal components or infrastructure details and should be shared according to an appropriate disclosure model.

Illustrative Case Study: A Financial Services Platform

Background

A fictional financial services organisation, Northbridge Payments, operates a Kubernetes platform for merchant transaction APIs. Releases are made by a central operations team using manually prepared manifests. The organisation deploys approximately once a month, and emergency changes are common because teams lack confidence in the process.

Situation and Problem

A new API release passes unit tests but introduces a latency increase under production load. Because the image is rebuilt separately for staging and production, the team cannot conclusively establish whether the tested binary is the production binary. Rollback requires locating an earlier image tag and manually editing manifests. The organisation also lacks an SBOM and cannot quickly determine whether a newly disclosed dependency vulnerability affects its services.

Analysis

The principal problems are not simply slow pipelines. They include mutable artifact identification, weak separation of duties, inadequate runtime verification, and absent supply-chain evidence. The deployment process is push-oriented and depends on individual operational knowledge. Monitoring is available, but release-specific dashboards and automated thresholds are not.

Approach

Northbridge introduces an immutable registry workflow. CI builds one image, records its digest, generates an SBOM, signs the image, and produces provenance linked to the reviewed source revision. Automated checks include unit, contract, integration, manifest, dependency, and container configuration tests. A successful pipeline creates a pull request in an environment repository rather than directly modifying the production cluster.

A GitOps controller reconciles the approved repository state. Production releases use a canary strategy: five percent of traffic is routed to the new version, and an analysis service compares error rate, latency, saturation, and successful transaction rate with the stable cohort. Promotion proceeds in stages only if thresholds remain within defined limits. A failed analysis stops promotion and restores traffic to the stable version. Database changes follow an expand-and-contract pattern so that old and new application versions can coexist.

Policy enforcement rejects unsigned images, unapproved registries, missing resource limits, and production changes without required review. Emergency access is retained but generates an alert and requires a subsequent repository change explaining the intervention.

Outcome

The organisation gains a reproducible relationship between source, artifact, deployment, and runtime evidence. Releases become smaller and more frequent, while the canary process limits exposure when a defect escapes pre-production testing. Vulnerability response improves because affected components can be located from SBOM records. The system does not eliminate failures, but it reduces blast radius and makes recovery more predictable.

Learning Points

The case demonstrates that delivery performance depends on architecture, not only on pipeline speed. Immutable artifacts, declarative reconciliation, progressive exposure, observable verification, and supply-chain evidence reinforce one another. A fast deployment without these controls may increase operational risk rather than organisational capability.

Challenges and Common Failure Modes

A frequent mistake is treating CI/CD as a tool procurement exercise. Installing a pipeline product does not resolve unclear ownership, poor tests, insecure credentials, or unsuitable release policies. Another mistake is placing all controls at the end of the pipeline, where failures are expensive and feedback is slow.

GitOps implementations can fail when repositories contain secrets, generated configuration is impossible to review, or controllers receive excessive permissions. Drift can also be misunderstood: some runtime changes are legitimate and should be represented in the desired-state model, while others indicate unauthorised activity.

Progressive delivery can create false confidence when traffic percentages do not represent meaningful user segments, when metrics are too coarse, or when rollback does not reverse an irreversible database or external side effect. Automated rollback thresholds must account for normal variation and must be tested.

Security scanning can produce overwhelming numbers of findings. Risk-based prioritisation should consider exploitability, reachability, exposure, available mitigations, and business impact. Suppressing findings without expiry or ownership converts noise into hidden risk. Finally, metrics can be gamed: increasing deployment count through trivial changes does not demonstrate improved delivery capability.

Emerging Directions

Supply-chain frameworks increasingly emphasise verifiable build levels, signed attestations, dependency transparency, and continuous verification rather than one-time approval. Policy engines are becoming integrated into admission control, registries, and pipeline systems. Platform engineering is packaging secure delivery capabilities as reusable internal products.

Artificial intelligence is being applied to test generation, anomaly detection, dependency analysis, and pipeline failure diagnosis. These uses require human oversight because generated tests may miss important behaviour, models may produce incorrect explanations, and automated remediation can amplify an error. Another developing direction is deployment intelligence that combines release metadata with observability data to identify risky changes, but such systems should support—not replace—engineering judgement.

Best-Practice Design Principles

A professionally governed delivery system generally follows these principles:

• Build once, promote immutable artifacts, and deploy by digest.
• Keep desired state declarative, versioned, reviewable, and reconciled.
• Separate build, approval, and deployment privileges.
• Use short-lived credentials and least-privilege identities.
• Generate SBOMs and provenance, and verify signatures at deployment.
• Place fast, actionable checks early and risk-based checks before promotion.
• Prefer progressive exposure for high-impact or uncertain changes.
• Define rollback and recovery before release, including data migration behaviour.
• Measure speed, stability, recovery, and user outcomes together.
• Make exceptions explicit, time-bound, observable, and accountable.

Chapter Summary

CI/CD is an engineered pathway from source change to verified release. Continuous integration reduces integration uncertainty; continuous delivery maintains deployability; continuous deployment automates release when controls are satisfied. GitOps adds declarative desired state, version control, and continuous reconciliation, while preserving the need for secure identities, review, and operational governance.

Kubernetes delivery should combine layered testing, policy gates, immutable artifacts, progressive delivery, and strong observability. Software supply-chain assurance relies on registries, SBOMs, signatures, provenance, isolated builds, and verifiable promotion rules. Deployment strategies must be evaluated through lead time, deployment frequency, change failure rate, recovery time, blast radius, and business outcomes rather than speed alone. The central professional challenge is to create a system that is fast because it is controlled, observable, and recoverable—not fast because safeguards have been removed.

Key Terms

Continuous integration: Frequent integration of code changes accompanied by automated validation.

Continuous delivery: Maintaining software in a deployable state with release commonly requiring an explicit decision.

Continuous deployment: Automatically releasing changes that satisfy defined controls.

GitOps: An operating model that uses versioned declarative state and automated reconciliation for delivery and operations.

Desired state: The declared configuration that a system is intended to maintain.

Reconciliation: The process of comparing actual state with desired state and correcting differences.

Progressive delivery: Gradually exposing a change while evaluating evidence before wider promotion.

Canary deployment: Releasing a change to a limited traffic cohort before full deployment.

Blue-green deployment: Operating two release environments and switching traffic between them.

Feature flag: A runtime control that activates or deactivates functionality independently of deployment.

Artifact registry: A controlled repository for images, packages, charts, and related metadata.

SBOM: A structured inventory of software components contained in an artifact.

Digital signature: Cryptographic evidence supporting artifact integrity and signer identity.

Provenance: Evidence describing the source, process, environment, and inputs used to produce an artifact.

Attestation: A signed statement about an artifact, such as test or scan results.

Policy-as-code: Machine-evaluable expression of organisational or security rules.

Drift: A difference between declared desired state and the actual running state.

Change failure rate: The proportion of deployments that cause incidents, rollback, or remediation.

Mean time to recovery: The average time required to restore acceptable service after failure.

Blast radius: The extent of users, services, or resources affected by a change or failure.

Review Questions

Short Answer Questions

1. Distinguish continuous delivery from continuous deployment.

2. Why should a tested container image be promoted rather than rebuilt for production?

3. What is reconciliation in a GitOps system?

4. What information does an SBOM provide, and what does it not prove?

5. Why are readiness probes insufficient as the only release verification mechanism?

Long Answer Questions

1. Explain how a Kubernetes CI/CD architecture can separate build, approval, and deployment responsibilities.

2. Discuss the principles of GitOps and analyse its advantages and limitations compared with imperative push deployment.

3. Compare rolling, blue-green, canary, and feature-flag-based delivery in terms of risk, capacity, observability, and rollback.

4. Explain how SBOMs, signatures, provenance, and attestations combine to improve software supply-chain assurance.

5. Develop a governance model for policy-as-code that balances platform standardisation with product-team autonomy.

Analytical / Application Questions

1. A canary release shows normal infrastructure metrics but a 3 percent decline in successful payments. Should promotion continue? Design the evidence and decision process.

2. An organisation has high deployment frequency but also frequent emergency rollbacks and long recovery times. Analyse the likely weaknesses and propose measurable interventions.

3. A GitOps controller repeatedly restores a manually applied production change. Explain the source of the conflict and design a safe break-glass and reconciliation procedure.

22. Multiple Choice Questions with Answers

Question 1

Which practice best preserves the relationship between tested and deployed software?

A. Rebuilding the image separately in each environment

B. Promoting one immutable image identified by digest

C. Using the latest tag for all environments

D. Copying binaries manually to production

Correct Answer: B. Promoting one immutable image identified by digest

Question 2

What is the defining operational feature of GitOps?

A. Manual execution of deployment commands from a laptop

B. Continuous reconciliation of actual state toward versioned desired state

C. Storing passwords in a protected Git branch

D. Running all tests only after production deployment

Correct Answer: B. Continuous reconciliation of actual state toward versioned desired state

Question 3

Which statement about an SBOM is most accurate?

A. It guarantees that all listed components are exploitable

B. It replaces vulnerability testing

C. It inventories software components and supports impact analysis

D. It proves that an artifact was built by an authorised system

Correct Answer: C. It inventories software components and supports impact analysis

Question 4

A team wants to limit user exposure while obtaining real production evidence about a release. Which strategy is most appropriate?

A. Canary deployment

B. Big-bang replacement

C. Manual manifest editing

D. Rebuilding the image in production

Correct Answer: A. Canary deployment

Question 5

Why should a CI runner generally not receive broad production cluster credentials?

A. It prevents unit tests from running

B. It reduces the impact of a compromised build environment

C. It makes artifacts immutable automatically

D. It eliminates the need for deployment verification

Correct Answer: B. It reduces the impact of a compromised build environment

Question 6

Which evidence most directly describes how an artifact was produced?

A. A readiness probe

B. A provenance record

C. A service-level objective

D. A traffic percentage

Correct Answer: B. A provenance record

Question 7

A deployment passes Kubernetes readiness checks but business transaction success falls significantly. What is the best interpretation?

A. The release is safe because the pods are ready

B. Readiness checks are sufficient for business validation

C. Runtime and business-level verification has detected a release problem

D. The registry must be deleted

Correct Answer: C. Runtime and business-level verification has detected a release problem

Question 8

Which metric combination gives the most balanced view of delivery performance?

A. Deployment count alone

B. Pipeline duration alone

C. Lead time, deployment frequency, failure rate, and recovery time

D. Number of Git commits per developer

Correct Answer: C. Lead time, deployment frequency, failure rate, and recovery time

Question 9

What is a sound policy-as-code exception practice?

A. Allowing permanent undocumented bypasses

B. Disabling all policies during incidents

C. Recording an owner, justification, expiry, and compensating controls

D. Ignoring policy failures in production namespaces

Correct Answer: C. Recording an owner, justification, expiry, and compensating controls

Question 10

Why can database migrations complicate rollback?

A. Databases cannot be monitored

B. Irreversible schema or data changes may not support the previous application version

C. Canary deployments never use databases

D. SBOMs prevent schema changes

Correct Answer: B. Irreversible schema or data changes may not support the previous application version