DevOps, DevSecOps and Software Delivery

Introduction

Modern full-stack development is not complete when source code compiles or when a feature works on a developer’s computer. Software must be integrated, tested, secured, packaged, released, observed and improved in a reliable manner. DevOps provides the cultural, organizational and technical practices that connect development and operations. DevSecOps extends this approach by embedding security throughout the software life cycle rather than treating security as a final inspection.

For full-stack developers, these practices affect everyday work: branching and merging, automated tests, pull requests, container images, deployment pipelines, feature flags, logging, vulnerability management and incident response. Effective software delivery is therefore a socio-technical capability. Tools are important, but they cannot compensate for unclear ownership, unsafe release practices or poor system design.

This chapter examines version control, trunk-based development, automated testing, continuous integration and delivery, artifact management, feature flags, release strategies, software supply-chain security and delivery-performance metrics.

Learning Objectives

By the end of this chapter, students should be able to:

1. Explain DevOps and DevSecOps and distinguish continuous integration, continuous delivery and continuous deployment.
2. Apply Git-based version control and trunk-based development practices.
3. Design an automated testing and CI/CD pipeline.
4. Explain artifact repositories, reproducible builds and software provenance.
5. Select suitable feature-flag and release strategies.
6. Identify software supply-chain threats and appropriate controls.
7. Calculate and interpret delivery-performance metrics.
8. Evaluate trade-offs involving speed, reliability, security, cost and governance.

DevOps as a Software Delivery System

DevOps is an approach to delivering and operating software through collaboration, automation, feedback and shared responsibility. It reduces the separation between people who build software and people who run it. The goal is not merely faster deployment; it is the ability to deliver valuable changes safely and repeatedly.

A useful conceptual model is a continuous loop:

Plan → Code → Build → Test → Secure → Release → Deploy → Operate → Monitor → Learn

The loop has several characteristics:

• Small batches: Changes are kept small enough to review, test and, if necessary, reverse.
• Automation: Repetitive validation and delivery tasks are performed consistently by machines.
• Fast feedback: Developers receive information about defects, security problems and operational behavior quickly.
• Shared ownership: Teams remain responsible for software in production, not only for source code.
• Continuous improvement: Data from delivery and operations informs process improvement.

DevOps does not mean that every organization must deploy many times per day. Deployment frequency should be appropriate to risk, architecture, regulation and user needs. A financial system and a low-risk marketing website may require different release controls, although both can benefit from automation and reliable testing.

DevSecOps

DevSecOps incorporates security into planning, coding, building, testing, releasing and operating. Security activities include threat modeling, secure coding, dependency analysis, secret detection, infrastructure scanning, dynamic testing, access control, logging and incident response.

The principle of “shift left” encourages earlier security work, but security must also be “shift right”: production monitoring, vulnerability response, abuse detection and recovery remain necessary. Early detection is usually cheaper and less disruptive, but no stage can establish that a complex system is absolutely secure.

A mature DevSecOps process uses risk-based controls. A critical authentication service may require stronger review and staged deployment than a non-sensitive user-interface change. Excessive controls applied indiscriminately can create bypass behavior and slow feedback; insufficient controls create avoidable exposure.

Version Control and Trunk-Based Development

Version control records changes to source code and related files. Git represents development as a history of commits connected through branches and references. A commit should describe one coherent change and should be associated with an understandable message. Repository history supports collaboration, auditing, rollback analysis and release reconstruction.

Important practices include:

• Keep the repository buildable and testable.
• Review changes through pull requests or an equivalent mechanism.
• Protect the main branch from unreviewed direct changes.
• Never commit passwords, private keys or production tokens.
• Use .gitignore and secret-scanning controls appropriately.
• Write commits that are small, focused and reversible.
• Tag releases with immutable, meaningful identifiers.

Trunk-based development

In trunk-based development, developers integrate frequently into a shared main branch, often called the trunk. Some teams commit directly under strong automated checks; others use short-lived branches and pull requests. The essential property is that branches exist for a short time and do not become parallel development lines lasting weeks or months.

Long-lived branches increase merge conflict, integration risk and uncertainty about what has been tested together. Trunk-based development reduces these problems, but it requires a healthy main branch, fast automated tests and techniques such as feature flags for incomplete work.

A practical workflow is:

1. Update the local repository from the trunk.
2. Create a small short-lived branch when review is required.
3. Implement a focused change with tests.
4. Run local checks and create a pull request.
5. Obtain review and pass automated checks.
6. Merge promptly, preferably using a controlled strategy.
7. Delete the branch after integration.

Rebasing and merging are technical choices rather than universal rules. Rebasing can produce a linear local history but rewrites commit identities; merging preserves explicit branch integration. Teams should choose a policy that supports traceability and teach it clearly.

Automated Testing and Quality Gates

Automated tests provide repeatable evidence about software behavior. They do not prove that software has no defects. A testing strategy should combine different levels and types of evidence.

The familiar “test pyramid” is a useful heuristic: many fast unit and component tests, fewer integration tests and a small number of valuable end-to-end tests. It is not a law. A distributed system may need substantial integration and contract testing because important behavior occurs between services.

Quality gates should be explicit. Examples include compilation, linting, unit-test success, dependency policy, secret scanning and minimum review requirements. A gate should be connected to a meaningful risk. A high code-coverage percentage alone is not proof of quality because tests may execute lines without checking important outcomes.

Test isolation, deterministic data, controlled clocks and reliable cleanup reduce flaky tests. A flaky test is one that sometimes passes and sometimes fails without a relevant code change. Flakiness consumes engineering time and weakens trust in the pipeline; it should be measured and repaired rather than routinely ignored.

Continuous Integration, Delivery and Deployment

Continuous integration (CI) means that changes are integrated into a shared codebase frequently and automatically validated. A CI pipeline commonly performs source checkout, dependency installation, compilation, static analysis, unit testing, packaging and security checks.

Continuous delivery means that software is kept in a releasable state. Production release may still require a human decision, business approval or a scheduled window.

Continuous deployment means that every change that passes the required automated controls is automatically deployed to production. Continuous deployment is a stronger condition than continuous delivery and is appropriate only when the organization has sufficient confidence, observability and recovery capability.

A pipeline should fail safely. It should stop before publishing or deploying when critical checks fail, while making diagnostic information accessible. Pipelines should be treated as production software: they need version control, code review, access control, monitoring and maintenance.

A representative pipeline is:

1. Validate: Check formatting, syntax, policy and commit metadata.
2. Build: Compile or bundle the application in a controlled environment.
3. Test: Run unit, integration, contract and selected end-to-end tests.
4. Analyze: Run static application security testing, dependency checks and secret detection.
5. Package: Produce a versioned artifact.
6. Verify: Test the artifact in an environment similar to production.
7. Release: Approve or automatically promote the artifact.
8. Deploy: Use an automated, auditable deployment mechanism.
9. Observe: Check health, performance, logs, traces and security signals.
10. Recover: Roll back, roll forward or disable a feature when required.

Pipelines should avoid rebuilding source separately for each environment. The preferred pattern is to build once, identify the resulting artifact, and promote that same artifact through environments. Environment-specific configuration should be supplied securely at deployment time.

Artifact Management and Reproducible Builds

An artifact is a deployable output such as a JavaScript bundle, container image, mobile package, binary, library or infrastructure package. An artifact repository stores and distributes these outputs together with metadata such as version, checksum, build time, source revision and dependency information.

A reliable artifact policy should provide:

• Immutable release versions.
• Access control and retention rules.
• Checksums or cryptographic digests.
• Provenance linking an artifact to source and build process.
• Separation between development snapshots and production releases.
• Malware and vulnerability scanning where appropriate.
• Promotion rather than uncontrolled copying.

A reproducible build produces the same intended output from the same source, dependency versions, build instructions and relevant inputs. In practice, timestamps, unordered files, external network access and compiler variation can make bit-for-bit reproducibility difficult. Pinning dependencies, using controlled build images, recording build metadata and limiting network access improve reproducibility.

A software bill of materials (SBOM) describes components included in an artifact. Formats such as SPDX and CycloneDX are commonly used for component inventories. An SBOM does not itself make software secure; it improves the ability to identify affected products when a component vulnerability is announced.

Feature Flags and Release Strategies

A feature flag is a runtime-controlled decision that determines whether a feature is available. Flags can separate deployment from release: code may be deployed while the user-visible feature remains disabled.

Useful flag categories include:

• Release flags: Temporarily hide incomplete or newly deployed functionality.
• Experiment flags: Assign users to variants for controlled evaluation.
• Operational flags: Disable expensive or risky behavior during an incident.
• Permission flags: Enable functionality for particular roles or tenants.

Flags introduce complexity. Each flag requires an owner, purpose, default behavior, security review and removal date. Stale flags create hidden branches in behavior and should be retired promptly. A flag must not be used as a substitute for authorization: access decisions require a secure, server-side policy where appropriate.

Common release strategies are:

Rollback is not always safe. Database schema changes can be irreversible, and a previous application version may not understand new data. Expand-and-contract migration is safer: first add backward-compatible structures, then deploy code using them, migrate data, and remove obsolete structures only after the old version is no longer needed.

Software Supply-Chain Security

The software supply chain includes source repositories, developers, dependencies, package registries, build systems, CI runners, artifact repositories, deployment tools and cloud identities. An attacker may target any link rather than the application’s own code.

Threats include malicious or compromised dependencies, dependency confusion, typosquatting, abandoned packages, poisoned build environments, stolen CI credentials, unsigned artifacts and unauthorized pipeline modification.

Controls should be layered:

1. Source integrity: Branch protection, code review, signed commits where justified and strong account authentication.
2. Dependency governance: Lockfiles, approved registries, version policies, vulnerability monitoring and review of transitive dependencies.
3. Build isolation: Ephemeral runners, least-privilege credentials, restricted network access and separated trust zones.
4. Artifact integrity: Digests, signatures, provenance and protected repositories.
5. Deployment authorization: Short-lived credentials, environment approvals and auditable promotion.
6. Runtime defense: Minimal images, non-root execution, network restrictions, monitoring and rapid patching.

A vulnerability scanner is an aid to risk management, not an automatic decision-maker. Severity, exploitability, exposure, compensating controls and business impact must be considered. Organizations should define response times for critical findings and document accepted risks.

Secrets should be injected from a managed secret store or protected CI variables, not stored in source code or ordinary logs. If a secret is exposed, removing the file from the latest commit is insufficient: the credential must be revoked or rotated, and historical copies must be assessed.

Delivery-Performance Metrics

Metrics should support learning rather than encourage gaming. Four widely used delivery measures are:

• Deployment frequency: How often an organization successfully deploys to a target environment during a period.
• Lead time for changes: Time from a change entering the development process to its successful deployment.
• Change failure rate: Proportion of deployments that cause a production failure, rollback, hotfix or other defined remediation.
• Failed deployment recovery time: Time required to restore acceptable service after a deployment-related failure.

If D is the number of deployments in a period and T is the duration of that period, deployment frequency is:

If N_f is the number of deployments requiring defined remediation and N is the total number of deployments, change failure rate is:

The denominator and failure definition must be stated. For example, counting only incidents that trigger a rollback produces a different metric from counting all deployment-caused defects.

For recovery events with durations r₁, r₂, …, r_n, the mean recovery time is:

This mean can be distorted by an extreme incident. Median and percentile values may provide additional insight. Time units must be consistent, and the start and end events must be defined precisely.

Worked example

During one month, a team performs 24 production deployments. Three deployments require rollback or an urgent corrective change. Recovery times for four deployment incidents are 30, 60, 45 and 105 minutes.

Deployment frequency is 24 deployments per month. Change failure rate is:

CFR = (3 ÷ 24) × 100% = 12.5%

Mean recovery time is:

MTTR = (30 + 60 + 45 + 105) ÷ 4 = 240 ÷ 4 = 60 minutes.

These values do not by themselves prove that the team is effective. They should be interpreted with service reliability, customer impact, security outcomes, delivery scope and working conditions. Increasing deployment frequency by making changes smaller may be beneficial; increasing it by bypassing tests is not.

A related reliability model is availability. If U is total service time and D is unavailable time during the same observation period, then:

This formula assumes that uptime and downtime are measured consistently and that the observation window is specified. Availability alone does not describe latency, correctness, data loss or security.

An Illustrative Case: Northstar Learning Platform

Background

Northstar is a fictional university learning platform with a React-based web client, an API service, background workers and a relational database. Its team has eight developers and releases approximately once every three weeks.

Problem

The team uses long-lived feature branches. Integration occurs near the end of each release cycle, producing merge conflicts and late test failures. Production deployments are manual. A dependency vulnerability was discovered weeks after it was publicly reported, and a previous rollback failed because the database schema had changed incompatibly.

Analysis

The main problems are not simply a lack of tools. The process has large batches, weak feedback, unclear artifact identity, insufficient supply-chain visibility and an unsafe data migration strategy. A new pipeline alone would not solve these issues unless the team changed its development practices.

Approach

Northstar adopts short-lived branches and a protected trunk. Every pull request runs formatting, unit tests, type checks, secret scanning and dependency analysis. A successful merge builds one container image identified by an immutable digest. The image is stored in a protected registry with an SBOM and provenance metadata.

The pipeline deploys the same image to a test environment, runs integration and contract tests, and then promotes it to production through a canary release. A feature flag initially exposes the new assignment workflow to internal staff and then to 5% of users. Error rate, latency and task-completion measures are monitored. The flag has an owner and removal date.

The database migration follows expand-and-contract. A nullable column is added first, application code writes both old and new representations, data is backfilled, reads are switched, and obsolete structures are removed only after rollback is no longer required. Production credentials are replaced with short-lived workload identities, and critical dependency alerts are assigned an explicit response target.

Outcome

After several release cycles, integration occurs daily rather than at the end of a long branch period. The team can identify which source revision produced a deployed artifact and can disable the new feature without redeploying. The organization still experiences defects, but detection occurs earlier and the blast radius is smaller. The team uses delivery metrics alongside incident reviews rather than treating a single target as a performance quota.

Learning Points

1. Trunk-based development depends on fast feedback and a healthy main branch.
2. Build-once-and-promote improves artifact identity and release confidence.
3. Feature flags reduce release risk but create maintenance and security obligations.
4. Database compatibility must be designed into rollback and deployment strategy.
5. DevSecOps requires controls across source, dependencies, builds, artifacts and runtime.
6. Metrics are useful only when definitions and incentives are transparent.

Professional, Ethical and Governance Considerations

Delivery systems possess substantial power. A compromised CI account may modify production software, while an incorrectly configured feature flag may expose private information. Least privilege, separation of duties, audit trails and emergency access procedures are therefore governance requirements as well as technical practices.

Teams must protect personal data in test environments. Production data should not be copied casually into development systems; masking, synthetic data and controlled access are preferable. Logs and telemetry may themselves contain identifiers, tokens or sensitive user behavior. Retention, access and deletion policies should follow applicable law, contractual obligations and institutional policy.

Automation also affects employment and accountability. Automated approval must not become an excuse to eliminate responsible ownership. Teams should know who can authorize emergency changes, who receives alerts and who communicates with affected users. Blameless incident analysis focuses on system conditions and corrective learning, while still maintaining appropriate accountability for intentional misconduct.

AI coding assistants can accelerate implementation, test generation and documentation, but generated code may contain vulnerabilities, licensing problems, hallucinated APIs or insecure data handling. AI output requires the same review, testing, dependency analysis and authorization as human-written code. Confidential source code and credentials should not be submitted to an AI service without an approved data-governance basis.

Contemporary Developments and Best Practices

Current delivery practice increasingly emphasizes:

• Platform engineering: Internal platforms provide reusable, governed paths for building and deploying services.
• Infrastructure as code: Infrastructure definitions are reviewed, tested and versioned alongside application changes.
• Observability: Metrics, logs and distributed traces are correlated to explain system behavior.
• Policy as code: Rules for images, identities, deployment environments and dependencies are automatically evaluated.
• Provenance and attestations: Build systems record how an artifact was produced and which inputs were used.
• Progressive delivery: Canary analysis and automated rollback use operational evidence during release.
• Developer experience measurement: Teams examine cognitive load, pipeline waiting time and tool usability, not only deployment counts.

A practical best-practice checklist is:

1. Keep changes small and integrate frequently.
2. Protect the trunk and make failures visible.
3. Maintain a balanced, reliable automated test suite.
4. Build once and promote immutable artifacts.
5. Scan and inventory dependencies continuously.
6. Use least privilege and rotate exposed credentials immediately.
7. Separate deployment from release when it reduces risk.
8. Make database changes backward compatible during transition.
9. Monitor user impact and maintain tested recovery procedures.
10. Define metrics carefully and use them for improvement, not punishment.

Challenges and Limitations

DevOps practices can fail when organizations adopt tools without changing incentives or responsibilities. A large pipeline may be slow, fragile and expensive. Excessive approval gates may encourage workarounds, while insufficient controls may create security and reliability incidents. Trunk-based development is difficult when tests take hours or when teams cannot isolate incomplete behavior. Feature flags can become a complex configuration system. Canary releases require meaningful traffic segmentation and trustworthy monitoring.

Metrics also have limitations. Deployment frequency can increase through trivial changes; low change failure rate can result from under-reporting; mean recovery time can hide severe outliers; and lead time can be measured from inconsistent starting points. Metrics must therefore be interpreted as indicators of a system, not as complete measures of engineering value.

Chapter Summary

DevOps connects development, operations and organizational learning through collaboration, automation, rapid feedback and shared ownership. DevSecOps extends this system by embedding security across the software life cycle. Version control and short-lived trunk-based workflows reduce integration risk. Automated tests provide repeatable evidence, while CI validates changes and continuous delivery keeps software releasable. Continuous deployment goes further by automatically releasing validated changes.

Artifact repositories, immutable identifiers, SBOMs and provenance improve traceability and supply-chain response. Feature flags separate deployment from release but require ownership and retirement. Rolling, blue-green, canary and related strategies manage release risk in different ways. Safe database migration is essential to reliable rollback. Deployment frequency, lead time, change failure rate and recovery time help teams evaluate delivery, provided that definitions and incentives are sound. Effective practice balances speed with security, reliability, privacy, governance and human accountability.

Key Terms

Artifact: A deployable output such as a container image, binary, package or bundle.

Artifact repository: A controlled system for storing and distributing versioned build outputs.

Canary release: A release initially exposed to a small, selected population.

Change failure rate: The percentage of deployments requiring defined remediation.

Continuous delivery: Keeping software in a state where it can be released reliably.

Continuous deployment: Automatically deploying each change that passes required controls.

Continuous integration: Frequent integration of changes with automated validation.

DevOps: A socio-technical approach joining development, operations and continuous improvement.

DevSecOps: DevOps with security practices embedded throughout the life cycle.

Feature flag: A runtime control that changes feature behavior or availability.

Lead time for changes: Time from a defined change-start event to successful deployment.

MTTR: Mean time to recover from defined service or deployment failures.

Provenance: Evidence describing an artifact’s origin, inputs and build process.

Reproducible build: A build designed to produce equivalent output from equivalent inputs.

SBOM: Software bill of materials listing components in an artifact.

Supply chain: The people, processes, tools and components involved in producing and delivering software.

Trunk-based development: Frequent integration into a shared main branch using short-lived branches or direct commits.

Vulnerability: A weakness that can be exploited to compromise confidentiality, integrity or availability.

Review Questions

Short Answer Questions

1. What is the principal difference between continuous delivery and continuous deployment?
2. Why does trunk-based development favor short-lived branches?
3. State two purposes of an artifact repository.
4. What is a feature flag, and why should stale flags be removed?
5. Name the four commonly used delivery-performance measures discussed in this chapter.

Long Answer Questions

1. Explain how a CI/CD pipeline can incorporate DevSecOps controls from source commit through production monitoring.
2. Compare rolling, blue-green and canary release strategies, including an appropriate use case and limitation for each.
3. Discuss how reproducible builds, SBOMs and provenance contribute to software supply-chain security.
4. Explain why database backward compatibility is important for rollback and describe expand-and-contract migration.
5. Critically evaluate the claim that a higher deployment frequency always indicates better software delivery.

Analytical / Application Questions

1. A team makes 40 deployments in a quarter, of which 6 require rollback or urgent remediation. Calculate its change failure rate and explain what additional information is needed before judging performance.
2. Design a release plan for a new payment feature in a full-stack application. Justify your choices of tests, feature flags, deployment strategy, monitoring and rollback approach.
3. A CI runner has access to source code, production credentials and unrestricted internet access. Analyze the risks and propose a revised supply-chain security design.

Multiple Choice Questions with Answers

Question 1

Which practice is most characteristic of trunk-based development?

A. Maintaining feature branches for several months

B. Integrating small changes frequently into a shared main branch

C. Merging only immediately before a major release

D. Avoiding automated tests until after deployment

Correct Answer: B. Integrating small changes frequently into a shared main branch

Question 2

What is the primary distinction between continuous delivery and continuous deployment?

A. Continuous delivery excludes automated testing

B. Continuous deployment keeps software releasable but never releases it

C. Continuous deployment automatically releases changes that pass required controls

D. Continuous delivery applies only to infrastructure code

Correct Answer: C. Continuous deployment automatically releases changes that pass required controls

Question 3

Which artifact practice best supports reliable promotion across environments?

A. Rebuilding independently in every environment

B. Using an immutable artifact identified by a digest

C. Copying unversioned files manually to each server

D. Allowing production dependencies to float to latest versions

Correct Answer: B. Using an immutable artifact identified by a digest

Question 4

A feature flag primarily separates which two activities?

A. Compilation and linking

B. Authentication and authorization

C. Deployment and release

D. Unit testing and code formatting

Correct Answer: C. Deployment and release

Question 5

A canary release is designed primarily to:

A. Expose a new version to all users immediately

B. Test a release with a limited population before wider exposure

C. Eliminate the need for monitoring

D. Guarantee that rollback is unnecessary

Correct Answer: B. Test a release with a limited population before wider exposure

Question 6

Which control most directly improves software supply-chain traceability?

A. Deleting build logs after every deployment

B. Using an SBOM and recording build provenance

C. Sharing one permanent administrator password

D. Disabling dependency updates permanently

Correct Answer: B. Using an SBOM and recording build provenance

Question 7

A team performs 20 deployments and 2 require defined remediation. What is its change failure rate?

A. 2%

B. 5%

C. 10%

D. 20%

Correct Answer: C. 10%

Question 8

Which testing approach is generally most suitable for verifying a service-to-service interface agreement?

A. Contract testing

B. Snapshot deletion testing

C. Manual production editing

D. Code formatting

Correct Answer: A. Contract testing

Question 9

Why is expand-and-contract migration useful?

A. It guarantees that no database backup is needed

B. It supports compatibility between old and new application versions during transition

C. It prevents all schema changes

D. It requires the old application to remain permanently deployed

Correct Answer: B. It supports compatibility between old and new application versions during transition

Question 10

Which statement about delivery metrics is most accurate?

A. A single metric fully measures engineering effectiveness

B. Deployment frequency should be increased regardless of risk

C. Metrics require clear definitions and should support learning rather than gaming

D. Change failure rate is meaningful without defining what counts as failure

Correct Answer: C. Metrics require clear definitions and should support learning rather than gaming