Continuous Testing, DevSecOps and Software Supply Chains

Introduction

Modern software is delivered as a continuously changing service rather than as a single product released at the end of a long project. Source code, infrastructure definitions, container images, third-party libraries, cloud services, configuration, and data-processing components all contribute to the delivered system. Consequently, quality and security cannot be inspected only immediately before release. They must be designed into the delivery process and evaluated repeatedly across the software life cycle.

Continuous testing is the systematic execution of appropriate tests throughout development, integration, deployment, and operation. DevSecOps extends DevOps by making security a shared responsibility and by embedding security controls into engineering workflows. Software supply-chain security addresses risks in the external and internal components from which software is assembled, including dependencies, build tools, registries, credentials, and distribution channels.

These ideas are closely related. A pipeline that runs unit tests but permits a compromised dependency is not trustworthy. A security scanner that reports thousands of unprioritized findings may slow delivery without reducing risk. Effective practice therefore combines technical controls, risk analysis, automation, human judgment, and governance.

Learning Objectives

After studying this chapter, you should be able to:

1. Explain the purpose and appropriate scope of unit, integration, system, acceptance, performance, and security testing.
2. Design a test strategy that provides rapid feedback without sacrificing meaningful coverage.
3. Apply shift-left and shift-right security across the software life cycle.
4. Use threat modeling, secrets management, and security gates in a delivery pipeline.
5. Explain software bills of materials (SBOMs), dependency controls, provenance, and artifact integrity.
6. Evaluate trade-offs, limitations, and governance obligations in a DevSecOps implementation.

1. Continuous Testing as a Quality Strategy

Testing is evidence gathering. A test executes a system or component under specified conditions and compares observed behavior with an expected result. Continuous testing does not mean testing every possible behavior after every commit. It means selecting and automating tests so that useful evidence is produced at suitable points in the pipeline and during operation.

A practical pipeline commonly contains these stages:

The stages are not rigidly sequential in every organization. Fast tests may run on each pull request; slower tests may run on a merge, nightly schedule, or release candidate. The objective is an economically sensible feedback loop: defects should be found as early as practical, while production behavior must still be observed because test environments are incomplete representations of reality.

Test levels and test purposes

Unit tests verify a small unit, usually a function, class, or module, in isolation from external systems. Dependencies are often replaced with test doubles such as stubs, fakes, or mocks. Unit tests are fast and precise, making them valuable for local development and pull-request validation. However, excessive mocking can produce tests that pass while the real database, framework, or network interaction is broken.

Integration tests verify interactions between components. Examples include an application communicating with a real database, a service publishing to a message broker, or an API validating authentication middleware. Integration tests reveal contract, serialization, configuration, and transaction errors that unit tests cannot detect.

System or end-to-end tests exercise a complete product through interfaces close to those used by customers. They provide strong evidence about business workflows but are slower, more fragile, and more expensive to diagnose. They should be reserved for important journeys rather than used as the only form of testing.

Acceptance tests establish whether the system satisfies stated business or user requirements. They may be automated or manual and should be expressed in observable outcomes rather than implementation details.

Performance tests assess characteristics such as response time, throughput, resource utilization, scalability, and stability. Load testing applies expected demand; stress testing exceeds expected demand to identify failure behavior; spike testing examines abrupt demand changes; and endurance testing observes long-running operation.

Security tests include static application security testing (SAST), dynamic application security testing (DAST), software composition analysis (SCA), infrastructure-as-code scanning, container scanning, fuzz testing, penetration testing, and configuration verification. No single tool proves that a system is secure.

Test doubles and the test pyramid

The test pyramid is a heuristic: maintain many fast, focused tests; fewer integration tests; and a small number of broad end-to-end tests. The exact proportions depend on the architecture. A service with substantial business logic may need many unit tests, whereas a data-intensive system may require more integration tests.

Test doubles are useful when a dependency is slow, unavailable, nondeterministic, or costly. They become harmful when they model an idealized dependency rather than its real behavior. A balanced strategy uses contract tests to verify that independently developed services agree on schemas, status codes, authentication expectations, and failure semantics.

Coverage percentage is only a measurement of executed structural paths, not proof of correctness. A test suite can execute every line while asserting almost nothing meaningful. Teams should combine coverage with mutation testing, defect escape rates, test duration, flakiness, and assessment of critical requirements.

2. Building Tests into a Delivery Pipeline

A pipeline should fail for a reason that a team can understand and act upon. Tests must therefore be deterministic, isolated where appropriate, observable, and versioned with the application. Test data should be reproducible without exposing personal or confidential information.

A useful pipeline policy distinguishes blocking from advisory controls. A failed test that indicates broken functionality should block promotion. A newly published vulnerability may initially be advisory while its exploitability and applicability are investigated, although critical findings in an internet-facing component may require an immediate block. Exceptions should have an owner, rationale, expiry date, and compensating control.

A simplified quality gate may require:

• successful compilation and reproducible dependency resolution;
• passing unit, integration, and critical acceptance tests;
• no unapproved critical security findings;
• signed or otherwise verifiable build artifacts;
• an SBOM associated with the release;
• deployment smoke tests and rollback readiness.

Pipelines should avoid treating security as a separate final department. Security specialists remain essential for threat analysis and complex assessments, but developers, operations engineers, and product owners all influence risk. A security result should be presented with affected component, evidence, severity, exploitability, remediation guidance, and due date—not merely a scanner score.

Performance testing in pipelines

Performance testing requires controlled workload and environment. A useful metric is throughput, often measured in requests per second:

The units are requests per second when elapsed time is measured in seconds. This relationship assumes that completed requests are counted consistently and that the measurement interval is defined. For example, 12,000 completed requests in 60 seconds gives 200 requests per second. Throughput alone is insufficient: a system could achieve high throughput while individual users experience unacceptable latency. Percentiles such as p95 and p99 are therefore commonly reported. A p95 latency of 400 ms means that 95% of measured requests completed in 400 ms or less; it does not mean that every request completed in 400 ms.

Performance results are valid only in relation to workload, data volume, hardware, network conditions, and application version. A test environment that is substantially smaller or differently configured than production may reveal trends but cannot guarantee production performance.

3. Shift-Left and Shift-Right Security

Shift-left security moves security activities earlier in the life cycle. Examples include secure coding education, threat modeling during design, dependency checks during pull requests, secret detection before commit, and security tests in continuous integration. Early remediation is often less disruptive because architecture and code are still changing.

Shift-right security continues security assurance after deployment. It includes runtime monitoring, web application firewalls, intrusion detection, vulnerability management, incident response, audit logging, canary releases, runtime application self-protection, and periodic penetration testing. Shift-right practices recognize that unknown vulnerabilities, operational misconfigurations, and attacks may only become visible in real use.

These approaches are complementary, not competing. Shifting everything left is impossible because production conditions are not fully knowable in advance. Shifting everything right is expensive and dangerous because defects reach users before they are detected.

A mature security feedback loop is:

1. Identify assets, actors, trust boundaries, and threats.
2. Prevent or reduce risk through design and implementation controls.
3. Detect suspicious behavior through logs, telemetry, and alerts.
4. Respond using rehearsed containment and recovery procedures.
5. Learn from incidents and update designs, tests, and controls.

4. Threat Modeling

Threat modeling is a structured way to identify what can go wrong and decide how to reduce the risk. It should be performed when a system is designed and revisited when architecture, data, integrations, or threat conditions change.

A practical method is:

• Scope: describe the system, users, assets, data flows, and assumptions.
• Decompose: draw components and trust boundaries; identify entry points and privileged operations.
• Identify threats: consider spoofing, tampering, repudiation, information disclosure, denial of service, and elevation of privilege, as well as threats specific to the domain.
• Prioritize: assess likelihood, impact, exploitability, and existing controls.
• Mitigate: change architecture, permissions, validation, encryption, monitoring, or operational procedures.
• Validate: convert important mitigations into requirements and tests.

A threat is not the same as a vulnerability. A threat is a potential harmful event or actor; a vulnerability is a weakness that can be exploited; risk combines the possibility and consequence of harm. Qualitative scales such as low, medium, high, and critical are useful for communication but should be defined consistently. A vulnerability score from a standardized system is evidence for prioritization, not an automatic business decision.

Threat modeling should include supply-chain threats: a malicious package, compromised build runner, stolen signing key, poisoned container image, or dependency confusion attack. The analysis should ask not only whether the application code is correct, but also whether the code and artifacts came from an authorized and traceable process.

5. Secrets Management and Identity

A secret is sensitive authentication or authorization material, such as a password, API token, private key, database credential, or signing key. Secrets must not be placed in source code, public issue trackers, container images, build logs, or unencrypted configuration files.

A robust secrets practice includes:

• storing secrets in a dedicated secrets manager or protected platform facility;
• granting the minimum required permission to each workload;
• using short-lived, automatically rotated credentials where feasible;
• separating development, testing, staging, and production secrets;
• preventing secret values from appearing in logs and error messages;
• scanning commits and artifacts for accidental disclosure;
• revoking and replacing a secret immediately when exposure is suspected;
• auditing access and testing recovery procedures.

Encryption at rest does not solve excessive access. A secret manager is useful only if its access policies, identity integration, audit logs, backup, and recovery arrangements are correctly designed. Developers should generally receive access through authorized local development mechanisms rather than copies of production credentials.

6. Software Supply-Chain Security

A software supply chain includes all people, processes, systems, and components involved in creating and delivering software. It may include operating-system packages, language libraries, build plugins, CI runners, container bases, infrastructure modules, artifact repositories, deployment tools, and signing services.

Supply-chain attacks exploit trust relationships. Examples include typosquatted packages with misleading names, dependency confusion in which a public package is selected instead of a private one, compromised maintainer accounts, malicious updates, vulnerable transitive dependencies, and tampered build artifacts.

Dependency controls

Effective dependency management begins with an inventory and a policy. Teams should prefer maintained packages, verify provenance, pin or constrain versions appropriately, review transitive dependencies, remove unused libraries, and update regularly. Pinning improves reproducibility, but it can also preserve vulnerabilities indefinitely; therefore, pinning must be combined with an update process and vulnerability monitoring.

Package lock files record resolved versions and integrity information. They improve repeatability but are not a complete trust mechanism. An attacker who compromises the package source or a developer account may still cause a malicious version to be accepted. Private registries, namespace controls, allowlists, isolated builds, and review of dependency changes provide additional protection.

SBOMs

A software bill of materials is a machine-readable inventory of software components and their relationships. It can identify direct and transitive dependencies, versions, licenses, suppliers, and sometimes hashes or package identifiers. Common exchange formats include SPDX and CycloneDX.

An SBOM supports vulnerability response, license review, procurement, customer communication, and incident analysis. Its value depends on completeness, accuracy, freshness, and association with the exact artifact deployed. An SBOM that describes source code but not the built container is incomplete for container risk management. It should be generated during the build, stored with release metadata, and updated when the artifact changes.

An SBOM does not certify that software is safe. It tells an organization what is believed to be present. Vulnerability databases may contain false positives, missing records, or vulnerabilities that are not exploitable in a particular configuration. Security teams must combine SBOM data with reachability, exposure, runtime configuration, exploit availability, and compensating controls.

Provenance, signing, and reproducibility

Provenance records where, when, and how an artifact was built, including source revision, builder identity, dependencies, and build parameters. Artifact signing provides a way to verify that an artifact was signed by an expected identity and was not altered after signing. Verification must occur at the point of use; signing an artifact without enforcing verification provides little protection.

Reproducible builds seek to produce the same artifact from the same source and declared inputs. Perfect reproducibility can be difficult because timestamps, nondeterministic ordering, external services, and compiler behavior may vary. Even partial reproducibility improves investigation and trust. Build environments should be isolated, minimally privileged, monitored, and protected from untrusted pull-request code when they handle signing credentials.

7. A DevSecOps Reference Workflow

Consider a pull request that changes a payment API. The workflow may proceed as follows:

1. The developer runs formatting, unit tests, secret detection, and dependency checks locally.
2. The repository validates branch protection, reviews, and signed commits where required.
3. The CI runner uses a trusted, ephemeral environment and retrieves dependencies through an approved registry.
4. The pipeline runs unit tests, SAST, dependency analysis, infrastructure scanning, and contract tests.
5. A build produces an immutable container image and an SBOM. The image is scanned and signed.
6. Integration and system tests run in an isolated environment with synthetic data.
7. Threat-model changes are reviewed if a new endpoint or trust boundary is introduced.
8. A deployment policy verifies image signature, provenance, vulnerability thresholds, and environment approvals.
9. A canary deployment exposes the release to a small proportion of traffic while health, error, latency, and security signals are monitored.
10. If evidence is acceptable, rollout continues; otherwise, automated rollback and incident procedures are activated.

This workflow illustrates that a pipeline is a control system, not merely a sequence of scripts. It must protect its own credentials, record decisions, support rollback, and make evidence available for audit.

8. Illustrative Case: The Northstar Retail Platform

Background. Northstar Retail operates a full-stack web application with a React front end, an API service, a relational database, and several payment and shipping integrations. Releases occur weekly, but emergency dependency updates are common.

Problem. A vulnerable transitive library was discovered in production. The team could not quickly determine which services used it, whether the vulnerable code path was reachable, or which deployed image contained the library. In addition, a database password had previously appeared in a CI log.

Analysis. The team mapped the build and deployment process and found three weaknesses: dependency versions were not centrally inventoried, images were rebuilt differently in separate environments, and CI jobs had broad access to long-lived credentials. Tests focused on unit behavior and did not verify image provenance or deployment configuration.

Approach. Northstar introduced lock-file review, an approved internal registry, dependency and secret scanning, and automated SBOM generation for every release image. Each image received a digest and signature. Production deployment verified the signature and allowed only images built by the protected CI workflow. The team replaced static cloud credentials with short-lived workload identities, added integration tests against a disposable database, and created a threat model for payment flows. A vulnerability policy prioritized internet-exposed, exploitable components while documenting exceptions.

Outcome. When a later library vulnerability was announced, Northstar queried its SBOM inventory, identified affected images, checked whether the vulnerable functionality was used, and rebuilt only necessary services. Runtime monitoring and a canary deployment provided additional assurance. The new process did not eliminate vulnerabilities, but it reduced uncertainty and shortened the path from disclosure to informed remediation.

Learning points. Inventory is a prerequisite for response. Controls must protect the pipeline as well as the application. Automation is most useful when it produces traceable evidence and actionable decisions. Finally, security maturity is demonstrated by repeated capability, not by a single successful audit.

9. Limitations, Governance, and Ethics

Automation can generate false positives, false negatives, flaky tests, and alert fatigue. A scanner may report a vulnerable library that is unreachable, while a business-logic flaw may evade automated analysis. Teams should measure the quality of findings, tune rules carefully, and retain expert review for high-impact systems.

Testing with production-like data raises privacy obligations. Synthetic or de-identified data should be preferred, access should be restricted, and retention should be limited. Logs and telemetry may themselves contain personal information or security-sensitive details. Data protection laws, contractual requirements, accessibility obligations, licensing terms, and sector-specific standards may apply to both the application and its dependencies.

Security controls can also create ethical concerns. Excessive monitoring may infringe employee privacy; automated blocking may disadvantage users if not tested for accessibility; and aggressive vulnerability disclosure without coordination may expose customers. Governance should define accountability, proportionality, transparency, retention, and appeal or exception processes.

Artificial intelligence can assist with test generation, code review, log triage, and vulnerability explanation. However, generated tests may encode incorrect assumptions, generated code may introduce insecure patterns, and confidential source or telemetry may be disclosed to an external service. AI output must be reviewed, tested, tracked, and governed like any other untrusted contribution.

10. Contemporary Best Practices

Current professional practice increasingly emphasizes secure-by-design architecture, identity-based access, ephemeral build environments, policy as code, artifact attestations, continuous vulnerability intelligence, and runtime observability. Standards and industry initiatives such as NIST secure software development guidance, OWASP testing and application-security resources, SLSA provenance concepts, SPDX, and CycloneDX provide useful reference points. They should be adapted to organizational context rather than adopted as checklists without analysis.

A high-performing team typically:

• defines security and quality requirements before implementation;
• keeps feedback fast for common changes and reserves deep tests for appropriate stages;
• treats test code, pipeline code, and infrastructure code as production code;
• maintains an authoritative inventory of artifacts and dependencies;
• verifies provenance and signatures at deployment;
• uses least privilege and short-lived credentials;
• monitors both technical signals and business impact;
• rehearses rollback, incident response, and vulnerability response;
• reviews metrics for improvement rather than using them to punish teams.

Chapter Summary

Continuous testing provides evidence throughout the software life cycle. Unit tests are fast and focused; integration and contract tests verify collaboration between components; system and acceptance tests assess complete user outcomes; performance tests evaluate behavior under defined workloads; and security tests identify classes of weaknesses without guaranteeing security.

DevSecOps integrates security into design, development, delivery, and operation. Shift-left practices reduce the cost of discovering defects early, while shift-right practices address runtime uncertainty and attack detection. Threat modeling connects assets, trust boundaries, threats, mitigations, and testable requirements. Secrets management requires protected storage, least privilege, rotation, auditing, and rapid revocation.

Software supply-chain security depends on dependency governance, SBOMs, provenance, signing, reproducible or controlled builds, and protected pipelines. An SBOM improves visibility but is not a security certificate. Effective programs combine automation with risk-based judgment, privacy-aware governance, operational monitoring, and practiced response.

Key Terms

Continuous testing: Repeated, automated and manual testing across the software life cycle.

Unit test: A test of a small software unit in relative isolation.

Integration test: A test of interactions between components or external systems.

System test: A test of an integrated product or service as a whole.

Contract test: A test that verifies an agreed interface or interaction between components.

Performance test: A test of response, throughput, scalability, resource use, or stability.

SAST: Static application security testing that analyzes code or related artifacts without executing the application.

DAST: Dynamic application security testing performed against a running application.

SCA: Software composition analysis of third-party and open-source components.

Shift-left security: Performing security activities earlier in the development life cycle.

Shift-right security: Continuing security detection and assurance after deployment.

Threat modeling: Structured identification and mitigation of threats to a system.

Secret: Sensitive authentication, authorization, encryption, or signing material.

Software supply chain: People, processes, tools, components, and systems used to build and deliver software.

SBOM: A machine-readable inventory of software components and their relationships.

Provenance: Evidence describing the origin and build process of an artifact.

Artifact signing: Cryptographic association of an artifact with an authorized signer.

Dependency confusion: An attack that causes a public package to be selected instead of an intended private package.

Canary deployment: Gradual release to a small population before wider rollout.

Policy as code: Machine-enforced rules represented and versioned as code.

Review Questions

5 Short Answer Questions

1. What is the principal difference between a unit test and an integration test?
2. Why is test coverage percentage insufficient as the sole measure of test quality?
3. Define shift-left security and give two examples.
4. What information does an SBOM provide?
5. Why should production secrets not be stored in source code or container images?

5 Long Answer Questions

1. Explain how the test pyramid can improve feedback speed, and discuss situations in which its proportions may need to change.
2. Describe a complete threat-modeling process for a web API that handles sensitive customer data.
3. Compare SAST, DAST, SCA, and penetration testing, including the strengths and limitations of each.
4. Explain how dependency pinning, vulnerability monitoring, provenance, and artifact signing work together.
5. Discuss the governance and privacy issues that arise when security monitoring and testing use production-like data.

3 Analytical / Application Questions

1. A team has many passing end-to-end tests but frequent production failures caused by database schema changes. Analyze the likely weaknesses in its testing strategy and propose a revised pipeline.
2. A critical vulnerability is reported in a transitive package, but the organization cannot identify which deployed services contain it. Design an immediate response and a longer-term supply-chain improvement plan.
3. Design security controls for a CI pipeline that builds and signs container images from untrusted pull requests. Explain how you would protect credentials, isolate execution, verify artifacts, and handle exceptions.

Multiple Choice Questions with Answers

Question 1

Which test type primarily verifies a small function or module in isolation?

A. Unit test

B. System test

C. Endurance test

D. Acceptance test

Correct Answer: A. Unit test

Question 2

What is the main purpose of a contract test?

A. To measure CPU utilization under stress

B. To verify an agreed interface between components

C. To scan a container for malware

D. To replace all end-to-end tests

Correct Answer: B. To verify an agreed interface between components

Question 3

Which activity is an example of shift-left security?

A. Reviewing runtime alerts after an incident

B. Conducting threat modeling during system design

C. Rolling back a canary deployment

D. Investigating a production intrusion

Correct Answer: B. Conducting threat modeling during system design

Question 4

What does an SBOM primarily provide?

A. A guarantee that an application has no vulnerabilities

B. A list of software components and their relationships

C. A replacement for penetration testing

D. A database password rotation schedule

Correct Answer: B. A list of software components and their relationships

Question 5

Why is an SBOM alone insufficient to determine whether a vulnerability is exploitable?

A. It cannot identify any component versions

B. It does not by itself establish reachability, configuration, or exposure

C. It is always limited to direct dependencies

D. It only applies to hardware inventories

Correct Answer: B. It does not by itself establish reachability, configuration, or exposure

Question 6

A pipeline records 12,000 completed requests in 60 seconds. What is its throughput?

A. 20 requests per second

B. 60 requests per second

C. 200 requests per second

D. 720,000 requests per second

Correct Answer: C. 200 requests per second

Question 7

Which practice best protects a CI pipeline from accidental exposure of credentials?

A. Embedding production passwords in the repository

B. Printing tokens in build logs for debugging

C. Using short-lived identities and a protected secrets manager

D. Sharing one permanent administrator token across jobs

Correct Answer: C. Using short-lived identities and a protected secrets manager

Question 8

What does artifact provenance describe?

A. The visual design of a user interface

B. Where and how an artifact was built

C. The number of users currently online

D. The legal owner of a database record

Correct Answer: B. Where and how an artifact was built

Question 9

What is a canary deployment?

A. A release exposed initially to a small population

B. A deployment that never reaches production

C. A test that only checks source-code formatting

D. A permanent duplicate of every service

Correct Answer: A. A release exposed initially to a small population

Question 10

Which statement best describes shift-right security?

A. It prohibits all testing after deployment

B. It moves security exclusively into design reviews

C. It uses runtime monitoring and response to address operational threats

D. It replaces dependency management with manual testing

Correct Answer: C. It uses runtime monitoring and response to address operational threats