Agile and DevOps Foundations

Introduction and Context

Software is no longer a peripheral support function. It is part of how organisations design products, deliver services, communicate with customers, manage operations and compete. Banks, hospitals, manufacturers, retailers and public agencies increasingly depend on software that changes frequently and operates continuously. This environment creates a central management and engineering challenge: how can an organisation deliver useful change quickly without sacrificing quality, security, reliability or regulatory control?

Agile and DevOps are influential responses to this challenge. Agile is primarily an approach to developing products and managing work under conditions of changing requirements and incomplete knowledge. DevOps extends the focus across the entire value stream, connecting software development, infrastructure, security, operations and business stakeholders. The two are related but not identical. Agile does not automatically produce continuous delivery, and DevOps is not simply a collection of automation tools.

This chapter introduces the historical background, concepts, terminology, benefits and limitations of Agile and DevOps. It also compares them with traditional development and IT operations, explains their relationship, and illustrates their practical use through an applied case.

Learning Objectives

By the end of this chapter, you should be able to:

1. Define Agile, DevOps and essential related terminology.
2. Explain the historical conditions that led to Agile and DevOps.
3. Distinguish Agile practices from DevOps practices and explain their relationship.
4. Compare iterative and incremental approaches with traditional sequential development.
5. Identify business and technical drivers, benefits, risks and limitations.
6. Explain the importance of culture, automation, feedback, measurement, security and governance.
7. Apply these concepts to a realistic software-delivery scenario.

Why Traditional Approaches Became Difficult

A traditional software project often followed a sequential or “waterfall” pattern: requirements were specified, architecture and design were completed, software was coded, testing was performed, and the product was released to operations. The stages might overlap in practice, but responsibility and approval generally moved from one specialist group to another.

This approach can be appropriate when requirements are stable, technology is well understood and change is expensive or tightly controlled. For example, some safety-critical or regulated systems require extensive up-front analysis, formal documentation and controlled release procedures.

However, sequential approaches can create difficulties when requirements are uncertain. Customers may not know what they need until they see a working product. Long intervals between specification and delivery make feedback expensive. Defects discovered late may require redesign of earlier work. Development teams may optimise for completing code while operations teams are later expected to deploy and support a system they had little influence over. This creates organisational “handoffs,” queues and competing objectives.

The issue is not that planning, documentation or specialist roles are inherently wrong. The deeper problem is that a process designed for predictable work may perform poorly when work is complex, uncertain and subject to rapid change.

The Emergence of Agile

In the 1980s and 1990s, many software practitioners explored iterative, incremental and lightweight methods. These included evolutionary development, rapid application development, Scrum, extreme programming and adaptive software development. Although these methods differed, they commonly emphasised short development cycles, frequent feedback, close customer involvement and adaptation.

In 2001, representatives of several lightweight methods published the Manifesto for Agile Software Development. Its four value statements are:

• Individuals and interactions over processes and tools.
• Working software over comprehensive documentation.
• Customer collaboration over contract negotiation.
• Responding to change over following a plan.

The wording does not mean that the item on the right has no value. Agile gives greater priority to the item on the left when a trade-off is necessary. Thus, Agile does not reject documentation, contracts, planning or tools. It rejects treating them as more important than collaboration, usable outcomes, customer value and learning.

Agile is best understood as a family of values, principles and practices rather than a single prescriptive method. Scrum, Kanban and Extreme Programming are different approaches that can be used in an Agile environment.

Core Agile Ideas

Iterative and incremental delivery

An iteration is a repeated development cycle in which a team plans, builds, integrates and reviews work. Incremental delivery means that each cycle adds usable capability to the product. Iteration supports learning; increments provide tangible value.

A team developing an online booking service might first deliver account registration, then search, then booking, then payment. Each increment should be sufficiently integrated and tested to provide evidence about whether the product is moving in the right direction.

Customer value and prioritisation

Agile teams prioritise a product backlog: an ordered list of desired product changes, fixes, technical improvements and other work. The product owner or equivalent decision-maker helps order work according to value, risk, urgency, dependencies and learning potential.

Prioritisation is not the same as choosing only easy features. A technically difficult item may be valuable because it reduces a major risk. Conversely, a frequently requested feature may have little value if it does not solve an important customer problem.

Empirical process control

Many Agile approaches rely on empiricism: decisions are based on observation and evidence rather than assumptions alone. Three recurring activities are:

• Transparency: work, quality and progress are visible.
• Inspection: the product and process are examined regularly.
• Adaptation: plans or practices change when evidence shows that change is needed.

This approach is particularly useful where the correct solution cannot be known in advance.

Cross-functional collaboration

A cross-functional team contains, or can access, the skills needed to produce a valuable increment. It may include software engineers, testers, designers, analysts and subject-matter experts. Cross-functional work reduces waiting for specialist handoffs and encourages shared responsibility for outcomes.

Technical excellence

Agile is not merely a scheduling technique. Practices such as automated testing, version control, continuous integration, refactoring, simple design and clear coding standards help a team respond safely to change. Without technical discipline, rapid iterations may accumulate defects and technical debt.

Common Agile Frameworks and Practices

Scrum

Scrum organises work into fixed-length iterations called Sprints, commonly lasting two to four weeks. Typical accountabilities include the Product Owner, Scrum Master and Developers. Events include Sprint Planning, the Daily Scrum, Sprint Review and Sprint Retrospective. Scrum uses artefacts such as the Product Backlog, Sprint Backlog and Increment.

Scrum provides a structure for inspection and adaptation, but it does not guarantee agility. A team can perform Scrum events mechanically while retaining slow approvals, poor quality and command-and-control behaviour.

Kanban

Kanban is a method for improving the flow of work. Work items are visualised on a board and limits are placed on work in progress (WIP). Limiting WIP discourages starting more work than the system can finish and exposes bottlenecks.

Kanban is often useful for support, maintenance and operational work, where priorities arrive continuously rather than in fixed batches. It can also complement Scrum.

Extreme Programming

Extreme Programming, or XP, emphasises engineering practices including test-driven development, pair programming, continuous integration, refactoring, small releases and close customer collaboration. Its contribution is especially important because it connects Agile values with software quality practices.

The Emergence of DevOps

DevOps developed from the recognition that development and IT operations could not be managed as entirely separate optimisation problems. Development teams were often rewarded for delivering new functionality, while operations teams were rewarded for stability and risk reduction. When software crossed the boundary between them, disagreements, delays and deployment failures could result.

The DevOps movement drew on Agile development, systems administration, lean thinking, infrastructure automation, continuous integration and continuous delivery. It became prominent in the late 2000s and early 2010s through practitioner communities, conferences and research into high-performing technology organisations.

DevOps is commonly defined as a set of cultural philosophies, practices and tools that increases an organisation’s ability to deliver applications and services at high velocity, while improving reliability and security. The definition is useful, but DevOps should not be reduced to a job title, a department or a software product.

The central idea is end-to-end responsibility: the people who design and build a service work closely with those who deploy, operate, secure and improve it. Feedback from production informs development, and operational requirements influence design from the beginning.

The DevOps Value Stream

A value stream is the sequence of activities required to deliver value from an idea to a customer or user. In software, it may include discovery, prioritisation, design, coding, review, building, testing, security assessment, packaging, deployment, operation, monitoring and learning.

A bottleneck in any stage can delay the entire stream. Producing more code does not improve performance if testing, approval or deployment remains a constraint. DevOps therefore examines the whole system rather than measuring only the productivity of one department.

Figure: A simplified DevOps value stream. Feedback from operation informs planning and subsequent development.

Essential DevOps Practices

Version control

Version control records changes to source code, configuration, infrastructure definitions and sometimes documentation. It supports collaboration, review, rollback and traceability. A distributed system such as Git allows local work to be committed and synchronised with shared repositories.

Continuous integration

Continuous integration (CI) means that developers integrate changes frequently into a shared codebase, where automated build and test activities provide rapid feedback. CI reduces the risk of discovering integration problems after a long period of isolated development.

CI is not merely a build server. It requires small changes, a shared repository, repeatable builds, automated tests and a team expectation that a broken main branch is repaired promptly.

Continuous delivery and continuous deployment

Continuous delivery means that software is maintained in a releasable state through automated validation and controlled processes. A release may still require a human decision or business approval. Continuous deployment goes further: every change that passes the required automated checks is automatically deployed to production.

These terms are related but should not be confused. Continuous delivery is a capability; continuous deployment is a release policy or operating practice. An organisation can practise continuous delivery without deploying every change automatically.

Infrastructure as code

Infrastructure as code (IaC) represents infrastructure configuration in version-controlled, machine-readable files. Servers, networks, permissions and cloud resources can then be provisioned consistently and reviewed like software. IaC reduces configuration drift, but it also introduces risks: an incorrect change can affect many environments, and secrets must not be exposed in source repositories.

Monitoring and observability

Monitoring collects and reports information about system behaviour, such as availability, latency, resource consumption and error rates. Observability is the broader capability to infer internal system conditions from externally visible outputs, commonly including metrics, logs and traces.

Effective monitoring is connected to action. An alert should identify a meaningful condition and provide enough context for an appropriate response. Excessive or poorly designed alerts create fatigue and can hide serious incidents.

Continuous security and compliance

DevOps increasingly incorporates security throughout the lifecycle. This approach is often called DevSecOps, although security should be a shared responsibility rather than an isolated suffix. Practices include threat modelling, dependency scanning, secure coding, secret management, infrastructure checks, access control, audit trails and automated policy tests.

Automation does not remove accountability. Legal requirements, privacy obligations, segregation of duties and human review may remain necessary, especially in regulated environments.

Agile and DevOps: Relationship and Distinction

Agile primarily addresses how teams discover and build valuable software under changing conditions. DevOps addresses how an organisation enables the reliable flow of that software into operation and learns from its use. Agile often focuses on the development team and product increment; DevOps expands attention to the complete service lifecycle.

Their relationship can be summarised as follows:

Agile can exist without mature DevOps. A team may work in short Sprints but still deploy manually every six months. DevOps can also be attempted without genuine Agile product management, producing highly automated delivery of poorly prioritised work. The strongest results occur when adaptive product management, technical excellence and operational responsibility reinforce one another.

Business Drivers and Benefits

Organisations adopt Agile and DevOps for several interconnected reasons:

• Changing customer expectations: Users expect frequent improvements and responsive digital services.
• Competitive pressure: Digital competitors can alter products rapidly.
• Shorter feedback cycles: Early use reveals whether an idea solves a real problem.
• Reduced cost of delay: Valuable changes can reach users sooner.
• Operational resilience: Automation, testing and observability can reduce avoidable failures.
• Scalability: Standardised pipelines and infrastructure help teams manage growing systems.
• Employee effectiveness: Removing repetitive handoffs can improve focus and ownership.
• Risk reduction: Small changes are generally easier to understand and reverse than large releases.

These benefits are not automatic. A rapid release process can increase harm if quality, security or governance are ignored. The relevant objective is not maximum speed but sustainable, evidence-based flow of valuable and trustworthy change.

Measuring Outcomes

Metrics should support learning rather than encourage local optimisation. Useful categories include:

• Delivery throughput: frequency of successful releases or completed valuable work.
• Lead time: time from a change being ready or committed until it is delivered, provided the starting point is defined clearly.
• Change failure rate: proportion of changes that require remediation, rollback or result in an incident.
• Time to restore service: elapsed time between a service-impacting failure and restoration.
• Reliability and user outcomes: availability, latency, task completion, satisfaction and business results.

A metric requires a defined population, time period and measurement rule. For example, if 4 of 100 production changes require rollback, the change failure rate is 4%, assuming each change is counted once and the observation period and inclusion criteria are specified. Metrics should not be used to rank teams without context; gaming, unsafe shortcuts and concealment can follow.

Comparison with Traditional IT Operations

Traditional IT operations often emphasise stability through scheduled releases, manual controls, separate environments and specialist approval. These practices may be rational in environments where failure has severe consequences. However, manual procedures can be inconsistent, slow and difficult to audit when they are undocumented.

DevOps does not mean eliminating operations. It changes how operational expertise is applied. Operations concerns—capacity, recoverability, security, supportability and service-level objectives—are considered during design. Automated controls and repeatable procedures can make change both faster and safer. Human judgement remains important for unusual incidents, major risk decisions and governance.

The contrast is therefore not “traditional equals bad, DevOps equals good.” The meaningful comparison is between ad hoc, siloed work and a controlled system that combines automation, collaboration, feedback and appropriate oversight.

Illustrative Case: Northstar Retail Platform

Background

Northstar Retail, a fictional medium-sized retailer, operated an online store using a twelve-year-old application. The development team delivered a large release every four months. Operations deployed releases manually on weekends, and customer support often discovered problems before engineering did.

Problem

Release preparation required extensive coordination. Test environments differed from production, deployment instructions were partly manual, and database changes were difficult to reverse. A major release introduced a payment defect that affected some customers for several hours. Development argued that requirements had changed late; operations argued that the release had not been adequately tested.

Analysis

An initial assessment identified several system-level causes:

1. Requirements were batched for months, delaying customer feedback.
2. Developers worked in long-lived branches, so integration problems appeared late.
3. Regression testing was mostly manual.
4. Infrastructure configuration was not version controlled.
5. Monitoring showed server health but not key customer transactions.
6. Development and operations had different objectives and few shared measures.

The organisation also recognised constraints. Payment and privacy obligations required approval records, access controls and evidence of testing. Fully automatic deployment to all customers was therefore not an immediate objective.

Approach

Northstar formed a cross-functional product team including developers, testers, an operations engineer, a security adviser and a product representative. It divided the roadmap into smaller customer outcomes and introduced a prioritised backlog.

The team then:

• stored application and infrastructure definitions in version control;
• adopted small, frequent changes and peer review;
• created an automated build and test pipeline;
• added database migration checks and rollback procedures;
• used a production-like test environment;
• introduced monitoring for payment success, error rates and response time;
• conducted joint incident reviews without blaming individuals;
• required human approval before high-risk production releases.

The team first practised continuous delivery: every accepted change was automatically built, tested and prepared for release, but selected releases still required approval. Later, low-risk changes were automatically deployed using progressive exposure, such as a small percentage of users followed by observation.

Outcome

After several months, Northstar reduced the size of releases and detected integration defects earlier. Deployment became more repeatable, and support staff received clearer operational information. The organisation did not eliminate incidents, but it reduced the time required to detect and restore service. It also learned that automation alone was insufficient: unclear product priorities and weak ownership could still create waste.

Learning points

The case demonstrates that Agile and DevOps are organisational changes as well as technical changes. Small batches, automated checks, shared responsibility and production feedback improved the system. Governance was not removed; it was embedded in repeatable controls. The outcome depended on addressing process, architecture, culture and measurement together.

Limitations, Risks and Misconceptions

“Agile means no planning”

Agile uses planning continuously and at multiple horizons. The difference is that plans are treated as hypotheses that can be revised as evidence changes. Eliminating planning creates confusion rather than agility.

“DevOps means developers do everything”

DevOps promotes shared responsibility, not the disappearance of specialist expertise. Security engineers, database specialists, operations professionals and compliance experts remain valuable. The goal is collaboration and integrated work, not role erasure.

Automation can scale bad decisions

An automated pipeline can repeatedly deploy defective software, expose secrets or create insecure infrastructure. Controls must be designed, tested and monitored. Automation should reduce avoidable variation while preserving appropriate decision rights.

Local optimisation

A team that maximises coding output may increase testing queues. A security group that approves nothing may reduce security risk in one narrow sense while making unsafe workarounds more likely. DevOps requires system-level measures and cooperation across constraints.

Cognitive load and burnout

Continuous change can become exhausting if teams support too many services, receive excessive alerts or lack time for maintenance. Sustainable pace, clear ownership, usable platforms and realistic on-call arrangements are essential.

Not suitable in exactly the same form everywhere

A small web product, a medical device and a national infrastructure system have different risks and regulatory expectations. Agile and DevOps principles can be adapted, but practices must reflect safety, privacy, contractual and operational constraints.

Governance, Ethics and Professional Responsibility

Software delivery decisions affect privacy, safety, access to services and financial outcomes. Teams should consider:

• whether data collection is necessary, lawful and transparent;
• whether automated decisions create unfair outcomes;
• whether access controls protect confidential information;
• whether changes are traceable and reversible;
• whether services are accessible to people with disabilities;
• whether incident reporting is honest and timely;
• whether suppliers and open-source dependencies are managed responsibly.

DevOps can strengthen governance through audit trails, peer review, automated policy checks, reproducible builds and controlled permissions. Nevertheless, compliance should not be treated as a checklist detached from real risk. Evidence must demonstrate that controls operate effectively.

Artificial intelligence can assist with code generation, test creation, log analysis and incident support. It also introduces risks involving insecure generated code, inaccurate explanations, confidential data exposure, licensing and overreliance. AI-assisted work requires review, testing, secure data handling and clear accountability. An AI tool does not become the owner of a production decision.

Best-Practice Principles for Beginners

1. Start with a clear customer or user outcome.
2. Make work visible and limit excessive work in progress.
3. Keep changes small enough to understand and recover from.
4. Use version control for code, configuration and infrastructure definitions.
5. Build and test changes frequently.
6. Automate repeatable checks, but inspect the quality of the automation.
7. Design for monitoring, security, recovery and accessibility.
8. Treat operations as part of product design, not as a final handoff.
9. Measure outcomes and system flow rather than individual busyness.
10. Conduct blameless learning reviews after incidents.
11. Preserve necessary approvals, evidence and separation of duties.
12. Improve continuously through experiments and feedback.

Chapter Summary

Agile is a family of values, principles and practices for delivering valuable software through collaboration, short feedback cycles, iterative learning and adaptation. Its roots lie in dissatisfaction with rigid, sequential development in uncertain environments. Agile methods such as Scrum, Kanban and Extreme Programming differ in structure but share an emphasis on value, transparency and responsiveness.

DevOps extends adaptive thinking across development, operations, security and the wider service lifecycle. Its important practices include version control, continuous integration, continuous delivery, infrastructure as code, monitoring, observability and shared responsibility. Agile focuses strongly on building the right product and DevOps on building, delivering and operating it reliably; together they support a faster and safer value stream.

Neither approach is a universal recipe. Benefits depend on technical quality, culture, governance, appropriate measurement and sustainable working practices. Automation can reduce error and delay, but it can also amplify poor decisions. Professional DevOps therefore combines speed with security, reliability, ethics, compliance and accountability.

Key Terms

Agile: An adaptive approach to delivering value through collaboration, iteration, feedback and response to change.

Backlog: An ordered list of product work, such as features, defects, risks and technical improvements.

Continuous delivery: Maintaining software in a releasable state through repeatable automated validation and controlled release processes.

Continuous deployment: Automatically deploying every change that satisfies the required checks and policies.

Continuous integration: Frequently integrating code into a shared codebase with automated builds and tests.

DevOps: Cultural, organisational, technical and operational practices that improve the flow and reliability of software delivery.

Empiricism: Decision-making based on observation, evidence and adaptation rather than assumptions alone.

Infrastructure as code: Managing infrastructure through version-controlled, machine-readable definitions.

Increment: A usable addition to a product.

Iteration: A repeated cycle of planning, development, testing, review and learning.

Kanban: A flow-oriented method using visualisation and work-in-progress limits.

Observability: The ability to infer internal system conditions from outputs such as metrics, logs and traces.

Product Owner: In Scrum, the accountability for maximising product value and ordering the Product Backlog.

Refactoring: Improving internal code structure without changing its observable behaviour.

Scrum: An Agile framework based on accountabilities, events, artefacts and iterative delivery through Sprints.

Technical debt: The future cost created by expedient technical choices, incomplete work or deferred improvement.

Value stream: The sequence of activities through which an idea or change becomes value for a user or customer.

Work in progress: Work that has been started but is not yet completed.

Review Questions

Short Answer Questions

1. Define Agile and identify two of its central values.
2. What is the difference between continuous delivery and continuous deployment?
3. Explain why work-in-progress limits can improve flow.
4. Name three practices commonly associated with DevOps.
5. Why is production feedback important in a DevOps value stream?

Long Answer Questions

1. Compare sequential software development with iterative and incremental development. Discuss when each may be appropriate.
2. Explain how Agile and DevOps are related but distinct approaches.
3. Describe how continuous integration, automated testing and version control work together.
4. Discuss the benefits and limitations of infrastructure as code.
5. Explain why governance and security should be integrated into Agile and DevOps rather than treated only as final approval activities.

Analytical / Application Questions

1. A team completes a two-week Sprint but needs three months of manual testing before release. Analyse the barriers to flow and recommend a staged improvement plan.
2. An organisation reports that deployment frequency has increased, but customer complaints and service incidents have also increased. Which additional evidence and metrics should leaders examine, and why?
3. Design an initial Agile and DevOps approach for a regulated healthcare service. Identify which practices could be automated and which controls would still require human oversight.

Multiple Choice Questions with Answers

Question 1

Which statement best defines Agile?

A. A fixed sequence of software phases with no requirement changes

B. An adaptive approach that uses collaboration, feedback and iterative delivery

C. A tool used exclusively for infrastructure provisioning

D. A method for replacing all human decision-making with automation

Correct Answer: B. An adaptive approach that uses collaboration, feedback and iterative delivery

Question 2

What is the primary purpose of a product backlog?

A. To store production logs

B. To record employee attendance

C. To provide an ordered list of product work

D. To define the physical network layout

Correct Answer: C. To provide an ordered list of product work

Question 3

Which practice is most directly associated with continuous integration?

A. Integrating code frequently into a shared codebase with automated checks

B. Releasing software only once per year

C. Separating developers from testers permanently

D. Writing requirements after deployment

Correct Answer: A. Integrating code frequently into a shared codebase with automated checks

Question 4

What distinguishes continuous deployment from continuous delivery?

A. Continuous deployment prevents automated testing

B. Continuous deployment automatically releases changes that pass required checks

C. Continuous delivery applies only to infrastructure

D. Continuous delivery eliminates all approval processes

Correct Answer: B. Continuous deployment automatically releases changes that pass required checks

Question 5

What is the main purpose of limiting work in progress in Kanban?

A. To increase the number of simultaneously started tasks

B. To expose bottlenecks and improve the flow of completed work

C. To prevent teams from visualising work

D. To guarantee that every task has identical complexity

Correct Answer: B. To expose bottlenecks and improve the flow of completed work

Question 6

Which statement best describes DevOps?

A. A software licence for cloud computing

B. A replacement for all IT operations specialists

C. A set of cultural, technical and operational practices supporting reliable delivery

D. A project management method that prohibits production monitoring

Correct Answer: C. A set of cultural, technical and operational practices supporting reliable delivery

Question 7

Why is observability valuable?

A. It allows internal system conditions to be inferred from outputs such as logs, metrics and traces

B. It guarantees that software contains no defects

C. It removes the need for incident response

D. It means that source code never needs version control

Correct Answer: A. It allows internal system conditions to be inferred from outputs such as logs, metrics and traces

Question 8

Which is a limitation or risk of automation?

A. It always removes the need for governance

B. It can scale an insecure or incorrect decision if poorly designed

C. It prevents repeatability

D. It makes monitoring impossible

Correct Answer: B. It can scale an insecure or incorrect decision if poorly designed

Question 9

In the Northstar case, why was continuous delivery adopted before full continuous deployment?

A. The organisation wanted to avoid all automated testing

B. Regulatory and risk requirements required controlled human approval for selected releases

C. The team had no source-code repository

D. Operations refused to receive production feedback

Correct Answer: B. Regulatory and risk requirements required controlled human approval for selected releases

Question 10

Which principle best reflects responsible DevOps measurement?

A. Measure only individual developer busyness

B. Maximise deployment speed regardless of incidents

C. Use delivery, reliability and user-outcome evidence to support system-level learning

D. Avoid metrics because all measurement encourages improvement

Correct Answer: C. Use delivery, reliability and user-outcome evidence to support system-level learning