Agile Methods and Product-Oriented Delivery

Introduction and Context

Agile software engineering is an approach to developing and delivering software through short learning cycles, close stakeholder collaboration, frequent feedback, and continuous adaptation. It emerged partly as a response to the limitations of plan-driven development, in which requirements are specified early, implementation occurs over a long period, and usable software may not appear until late in the project.

Agile does not mean “without planning,” “without documentation,” or “without discipline.” Rather, it changes the timing and purpose of planning. Agile teams plan enough to make a useful next decision, deliver a small increment, evaluate evidence, and revise their direction when necessary.

This chapter examines Scrum, Kanban, Lean software development, Extreme Programming (XP), user stories, backlogs, estimation, retrospectives, iterative discovery, and the movement from project-oriented to product-oriented delivery. It also evaluates the limits of agile methods, including scaling difficulties, poor implementation, governance risks, and the danger of treating frameworks as rigid recipes.

Learning Objectives

By the end of this chapter, you should be able to:

• explain the values and principles underlying agile software development;
• distinguish Scrum, Kanban, Lean, and Extreme Programming;
• write and refine useful user stories and acceptance criteria;
• manage product backlogs and prioritize work using value, risk, and learning;
• apply relative estimation and interpret velocity and flow metrics responsibly;
• conduct effective reviews, retrospectives, and iterative discovery activities;
• compare project-oriented and product-oriented operating models;
• analyze scaling challenges, governance concerns, and ethical risks in agile delivery.

1. Agile Foundations

1.1 The Agile Manifesto and its interpretation

The Agile Manifesto values:

1. individuals and interactions over processes and tools;
2. working software over comprehensive documentation;
3. customer collaboration over contract negotiation;
4. responding to change over following a plan.

The wording does not reject the item on the right. Documentation, tools, contracts, and plans remain important. The phrase “over” expresses a preference when the two sides conflict. For example, a team should not allow an elaborate reporting process to prevent it from delivering and evaluating working software.

The associated principles emphasize early and continuous delivery, welcoming changing requirements, frequent delivery of valuable software, sustainable development, technical excellence, simplicity, self-organizing teams, and regular reflection. These principles describe an empirical approach: decisions are improved through observation and feedback rather than relying entirely on predictions made at the beginning.

1.2 Empirical process control

Many agile methods use three related ideas:

• Transparency: the work, quality, risks, and decisions are visible.
• Inspection: the product and process are examined frequently.
• Adaptation: plans, priorities, or methods are changed when evidence indicates that change is needed.

Empirical control is not random experimentation. It requires a clear goal, observable evidence, bounded changes, and accountability for outcomes.

1.3 Iterative and incremental development

An iteration is a repeated development cycle in which a team plans, builds, tests, and evaluates work. An increment is a usable addition to the product. Iterative work improves understanding over several cycles; incremental work adds product capability over several cycles. A strong agile process is both iterative and incremental.

A team may discover during testing that a proposed feature is confusing or technically expensive. Agile delivery allows this discovery to influence the next cycle. This is especially valuable when requirements are uncertain, users are difficult to predict, or technology is evolving.

2. Scrum

Scrum is a lightweight framework for developing, delivering, and sustaining complex products. It provides defined accountabilities, events, and artifacts, but it does not prescribe every engineering practice.

2.1 Scrum accountabilities

• Product Owner: maximizes product value by managing and ordering the Product Backlog. The Product Owner is not merely a requirements secretary; the role involves product decisions, stakeholder communication, and value management.
• Scrum Master: helps establish Scrum, coaches the team and organization, facilitates useful events, and helps remove impediments. The Scrum Master is not simply a project administrator.
• Developers: create a usable product increment each Sprint. Developers include all people needed to produce the increment, such as programmers, testers, designers, and analysts.

A Scrum Team is intended to be focused, cross-functional, and sufficiently self-managing to decide how work is performed.

2.2 Scrum artifacts and commitments

• Product Backlog: an ordered, evolving list of work needed to improve the product. Its commitment is the Product Goal.
• Sprint Backlog: the selected Product Backlog items, the plan for delivering them, and the Sprint Goal. Its commitment is the Sprint Goal.
• Increment: a usable, verified addition to the product. Its commitment is the Definition of Done.

The Definition of Done is a shared quality standard. It may include code review, automated tests, security checks, documentation, deployment readiness, and accessibility verification.

2.3 Scrum events

A Sprint is a fixed-length period, commonly two weeks, during which the team creates an increment. Scrum events include:

• Sprint Planning: establishes why the Sprint is valuable, what can be achieved, and how the work will be done.
• Daily Scrum: a short event for Developers to inspect progress toward the Sprint Goal and adapt their plan.
• Sprint Review: stakeholders and the Scrum Team inspect the outcome and discuss future adaptations.
• Sprint Retrospective: the team examines its people, interactions, process, and tools, then identifies improvements.

A review is not a ceremonial demonstration designed to defend a completed plan. It is a working session for product inspection and future decision-making. Similarly, a retrospective should not become a complaint meeting; it should produce specific, testable improvements.

3. Kanban, Lean, and Extreme Programming

3.1 Kanban

Kanban is a method for managing and improving the flow of knowledge work. Its common practices are:

1. visualize the workflow;
2. limit work in progress (WIP);
3. manage flow;
4. make policies explicit;
5. establish feedback loops;
6. improve collaboratively and evolve experimentally.

A Kanban board might contain columns such as Ready, Analysis, Development, Code Review, Testing, and Done. A work-in-progress limit restricts how many items may occupy a stage. If Testing has a limit of three and already contains three items, developers should help resolve testing work rather than start a fourth development item.

Two important flow measures are cycle time, the elapsed time from work starting to completion, and throughput, the number of items completed per unit of time. A frequently used relationship is Little’s Law:

Here, WIP is the average number of items in the system, throughput is the average completion rate, and average cycle time is measured in the same time unit used for throughput. The relationship assumes a reasonably stable system over the period being analyzed: items entering and leaving are approximately balanced, and averages are meaningful. It is not a guarantee that every individual item follows the average.

For example, if a team averages 6 completed items per week and average cycle time is 2 weeks, expected average WIP is:

Reducing WIP while maintaining throughput generally reduces average cycle time. However, simply imposing a low WIP limit without removing bottlenecks can cause waiting, conflict, or underutilization.

3.2 Lean software development

Lean thinking seeks to maximize customer value while minimizing waste and improving the system as a whole. Common forms of waste include partially completed work, excess features, waiting, handoffs, unnecessary motion or coordination, defects, task switching, and unused human capability.

Lean emphasizes small batches, fast feedback, built-in quality, pull-based work, respect for people, and continuous improvement. A key idea is that optimizing one local activity can harm the complete system. For example, maximizing the number of features coded by a development team may increase testing queues and delay customer value.

3.3 Extreme Programming

Extreme Programming provides disciplined engineering practices particularly suited to rapidly changing requirements. Practices include:

• test-driven development (TDD);
• pair programming or collaborative programming;
• continuous integration;
• refactoring;
• simple design;
• collective code ownership;
• small releases;
• sustainable pace;
• close customer involvement.

In TDD, a developer first writes a failing automated test, implements the smallest change that makes it pass, and then refactors while preserving behavior. This cycle is often called red–green–refactor. TDD is not a substitute for system-level testing, exploratory testing, security testing, or user research.

3.4 Comparing approaches

These approaches can be combined, but combining labels does not automatically create a coherent process. A team should understand why each practice is used and whether the practices reinforce one another.

4. Product Backlogs, User Stories, and Discovery

4.1 User stories

A user story is a concise statement of a user need, commonly written as:

As a [type of user], I want [capability], so that [benefit].

For example: “As a student, I want to save a draft application so that I can complete it later.” The format encourages attention to user type, desired capability, and purpose. It is not a complete specification. Conversation, examples, prototypes, constraints, and acceptance criteria provide the necessary detail.

A useful story is often evaluated using the INVEST heuristic:

• Independent: can be developed with limited dependence on other stories;
• Negotiable: expresses a need rather than prescribing every design detail;
• Valuable: provides value to a user or other stakeholder;
• Estimable: sufficiently understood to support estimation;
• Small: can be completed within a reasonable delivery period;
• Testable: has observable conditions of satisfaction.

INVEST is a heuristic, not a mathematical law. Some valuable work, such as platform modernization or security infrastructure, may not fit neatly into a user-facing story. Such work still requires a clear purpose and evidence of completion.

4.2 Acceptance criteria

Acceptance criteria describe conditions that must be satisfied for a story to be accepted. They should be observable and testable. For a password-reset story, criteria might include:

• the user can request a reset using a registered email address;
• the reset link expires after a defined period;
• the system does not reveal whether an email address is registered;
• the new password meets the stated security policy;
• the action is recorded in an audit log.

The final criterion demonstrates that non-functional requirements—such as security, privacy, performance, and accessibility—must be represented in backlog work rather than treated as optional extras.

4.3 Product backlog management

Backlog management includes discovery, clarification, ordering, splitting, estimation, and removal of obsolete items. Ordering should consider expected value, urgency, risk reduction, dependencies, learning, and cost of delay.

A backlog is not a contract promising that every item will be delivered. It is a current model of opportunities and needs. Items near the top normally contain more information than distant items. This reduces the cost of changing direction when evidence changes.

4.4 Iterative discovery

Product discovery is the process of learning what should be built and why. Useful activities include interviews, observation, analytics, usability tests, prototypes, technical spikes, experiments, and analysis of support requests.

Discovery and delivery should interact continuously. A team might prototype two checkout designs, test them with representative users, and then implement the more promising option. This is more responsible than investing months in a feature based only on internal assumptions.

A discovery hypothesis can be expressed as:

We believe that [proposed change] will cause [user behavior or outcome] for [target users]. We will consider it supported if [measurable evidence] occurs within [time period].

The hypothesis should identify evidence rather than merely describe activity. “We released the feature” is not evidence of value; increased successful task completion may be.

5. Estimation and Forecasting

Agile teams often use relative estimation rather than pretending to know exact hours early in uncertain work. Story points may represent a blended assessment of relative size, complexity, uncertainty, and effort. They are not universal units and should not be compared across teams.

A team may use planning poker. Team members independently select cards, discuss differences, clarify assumptions, and reach a shared estimate. The discussion is often more valuable than the number because it reveals hidden work and uncertainty.

5.1 Velocity

Velocity is the amount of estimated work completed to the team’s Definition of Done in a completed iteration. If a team completes 18, 22, 20, and 24 points in four comparable Sprints, its average velocity is:

Velocity can support local forecasting, but it is not a productivity score. Changing estimation scales, team membership, quality standards, or work type makes comparisons misleading. Management pressure to increase velocity commonly produces inflated estimates and damages trust.

If a backlog contains 84 points and the team’s recent average velocity is 21 points per Sprint, a simple forecast is:

This is an approximate forecast, not a promise. It assumes comparable future conditions, stable team capacity, and backlog estimates of reasonable quality. A range based on historical variation is usually more honest than a single date.

5.2 Flow-based forecasting

Kanban teams often use throughput and cycle-time distributions instead of story points. Forecasting should acknowledge variability. For example, a team that completes between 5 and 8 items per week should not promise exactly 7 items every week. Probabilistic forecasts using historical data and simulation can be more informative, but they remain dependent on the stability and representativeness of the data.

6. Feedback, Reviews, and Retrospectives

Feedback operates at several levels:

• Product feedback: Do users achieve their goals? Is the product valuable, usable, secure, and accessible?
• Technical feedback: Is the code maintainable, reliable, performant, and safe?
• Process feedback: Does the way of working support quality and sustainable delivery?
• Organizational feedback: Do funding, governance, architecture, and incentives support product outcomes?

A retrospective should be psychologically safe and evidence-based. A practical structure is:

1. establish the purpose and working agreement;
2. gather observations and data;
3. identify patterns and causes;
4. select one or two high-leverage improvements;
5. define an owner, action, and review point.

For example, if code reviews frequently wait for more than two days, an improvement might be a review WIP limit and a daily reviewer rotation. The team should inspect whether the change improves cycle time and quality rather than assuming that holding the meeting was sufficient.

7. From Projects to Products

A project-oriented model treats work as a temporary effort with a defined scope, budget, and deadline. A product-oriented model organizes persistent, cross-functional ownership around a product, service, or customer problem. Product teams are accountable not only for delivering outputs but also for outcomes such as adoption, successful task completion, retention, revenue, safety, or reduced support demand.

Neither model is universally appropriate. A bridge construction project, regulatory migration, or fixed infrastructure replacement may require strong project controls. A digital service operating in an uncertain market benefits from persistent product ownership. Many organizations use a hybrid model: governance establishes investment boundaries while product teams decide how to achieve outcomes.

Product orientation creates responsibilities. Teams must understand customer needs, business strategy, operational costs, security, privacy, inclusion, and the full product lifecycle. It must not become a way to assign permanent accountability without providing decision rights, resources, or realistic objectives.

8. Scaling Agile and Organizational Constraints

Scaling is not simply multiplying Scrum teams. Dependencies, architecture, shared services, compliance, funding, and organizational structure create coordination challenges. Common responses include:

• organizing teams around valuable product capabilities rather than technical silos;
• maintaining a shared product strategy and outcome measures;
• making dependencies visible and reducing them through architecture and team design;
• coordinating integration frequently;
• using communities of practice for shared learning;
• establishing lightweight portfolio prioritization;
• preserving team autonomy where local decisions are appropriate.

Frameworks marketed for large organizations may provide useful coordination patterns, but no framework can eliminate fundamental trade-offs. Excessive ceremonies can create bureaucracy, while insufficient coordination can produce inconsistent architecture and duplicated work.

A scaling design should be evaluated by outcomes: reduced delay, improved quality, clearer decisions, better user value, and sustainable work. The number of synchronized meetings is not evidence of agility.

9. Governance, Ethics, and Responsible Delivery

Agile teams must comply with legal, contractual, and organizational obligations. Important concerns include:

• privacy and lawful processing of personal data;
• accessibility for users with disabilities;
• security threat modeling and secure development;
• auditability and traceability where required;
• protection against discriminatory algorithms or exclusionary design;
• responsible use of telemetry and experimentation;
• honest communication of uncertainty and product limitations.

The claim that “agile teams only need working software” is dangerous if it is interpreted as permission to omit security evidence, accessibility tests, architectural records, or regulatory documentation. Documentation should be proportionate and useful, but quality and compliance cannot be deferred indefinitely.

AI-assisted coding and product analytics can support agile work by generating tests, summarizing feedback, detecting patterns, or accelerating prototypes. Risks include insecure generated code, license or intellectual-property problems, fabricated explanations, privacy leakage, biased data, and overconfidence in automated output. Human review, testing, data governance, and explicit accountability remain necessary.

10. Illustrative Case: Improving a University Course-Registration Service

Background

A university’s online registration service receives heavy traffic during enrollment periods. Students report confusing prerequisites, failed payments, and uncertainty about whether a course place has been reserved.

Problem

The service team previously operated through annual projects. Requirements were collected months before enrollment, development was organized by technical department, and testing occurred near release. The project frequently met its feature scope but still produced support calls and emergency fixes.

Analysis

Interviews and support data revealed three major problems: students could not understand eligibility rules, payment failures were poorly explained, and the system provided weak confirmation feedback. Technical analysis also found that the payment service and registration database had no reliable end-to-end test environment.

The team created a product goal focused on enabling students to complete registration confidently and securely. It formed a cross-functional product team containing a Product Owner, developers, a designer, a tester, an operations specialist, and a student-support representative.

Approach

The team used two-week Sprints, a visible Kanban board, WIP limits, automated integration tests, and weekly usability sessions. Initial backlog items included:

• “As a student, I want to see why I am not eligible for a course so that I can choose an appropriate alternative.”
• “As a student, I want clear payment status so that I know whether I need to retry.”
• “As an administrator, I want an audit trail of registration decisions so that disputes can be investigated.”

The team split large stories into thin vertical slices. A prototype of the eligibility explanation was tested with students before full implementation. Security and privacy criteria were included in the Definition of Done. The team reviewed not only completed features but also indicators such as successful registration completion, payment-related support calls, and accessibility defects.

Outcome

The first release improved clarity for a limited group of courses before the next enrollment period. Usability testing identified a misleading label, which was corrected before wider rollout. The team did not claim success merely because the release occurred; it compared behavioral and support data and continued improving the service.

Learning points

1. Product discovery reduced the risk of building technically correct but confusing functionality.
2. Cross-functional ownership exposed operational and user-support issues earlier.
3. Small releases made learning possible before full-scale commitment.
4. Security, privacy, accessibility, and auditability belonged inside delivery rather than at the end.
5. Outcome measures complemented, rather than replaced, engineering quality measures.

Challenges and Limitations of Agile

Agile methods can fail when leadership demands fixed scope and dates while also expecting continuous change without trade-offs. They can also fail when Product Owners lack decision authority, teams are overloaded with dependencies, technical debt is ignored, or retrospectives produce no action.

Agile is less effective when work is highly repeatable, requirements are governed by strict external specifications, or physical dependencies make frequent increments impractical. Even in these contexts, feedback, risk management, incremental verification, and collaborative planning may still be useful.

A further limitation is that local agility cannot compensate indefinitely for organizational dysfunction. If funding changes monthly, architecture is controlled by distant committees, or teams are measured by utilization rather than outcomes, adopting a board and daily meeting will not solve the underlying problem.

Best Practices

• Begin with a clear product or service outcome.
• Keep work items small enough to obtain feedback, but meaningful enough to deliver value.
• Make quality, security, accessibility, and compliance explicit.
• Limit work in progress and expose queues and bottlenecks.
• Use estimates for forecasting, never as measures of personal worth.
• Prefer evidence from users and systems over internal opinion.
• Maintain technical excellence through automation, refactoring, integration, and testing.
• Treat retrospectives as experiments with follow-up, not as rituals.
• Adapt the method to the context while preserving transparency, feedback, and accountability.
• Distinguish output—what was built—from outcome—what changed for users or the organization.

Chapter Summary

Agile software engineering is an empirical approach based on frequent delivery, collaboration, inspection, adaptation, and sustainable quality. Scrum organizes work through accountabilities, artifacts, Sprints, and feedback events. Kanban emphasizes visualization, WIP limits, and flow; Lean focuses on value, waste reduction, and system improvement; and Extreme Programming strengthens technical feedback through practices such as TDD, continuous integration, refactoring, and small releases.

User stories and acceptance criteria help express needs, while product backlogs evolve as discovery produces new evidence. Relative estimation and velocity can support local forecasting, but they should not be used as productivity rankings. Flow metrics such as throughput and cycle time offer another forecasting perspective, and Little’s Law connects average WIP, throughput, and cycle time under stable-system assumptions.

Product-oriented delivery gives persistent teams responsibility for customer and organizational outcomes rather than only temporary project scope. It supports learning but requires decision rights, quality discipline, responsible governance, and suitable measures. Scaling agile requires reducing dependencies and coordinating product strategy without replacing team autonomy with bureaucracy. Ultimately, agile is not a collection of ceremonies; it is a disciplined way of organizing learning, delivery, quality, and accountability under uncertainty.

Key Terms

Agile development: An adaptive approach emphasizing feedback, collaboration, frequent delivery, and response to change.

Empirical process control: Decision-making based on transparency, inspection, and adaptation.

Scrum: A lightweight framework for developing and sustaining complex products.

Product Owner: The Scrum accountability responsible for maximizing product value and ordering the Product Backlog.

Scrum Master: The Scrum accountability that supports effective Scrum and helps remove impediments.

Sprint: A fixed-length Scrum cycle for creating a usable increment.

Product Backlog: An ordered, evolving list of work and opportunities for a product.

Definition of Done: A shared standard describing when work is complete and usable.

Kanban: A method for visualizing and improving the flow of knowledge work.

Work in progress: Work that has started but is not yet complete.

Lean software development: An approach focused on value, waste reduction, flow, and continuous improvement.

Extreme Programming: An agile method emphasizing technical excellence and rapid feedback.

User story: A concise expression of a user need and its intended benefit.

Acceptance criteria: Observable conditions that must be satisfied for work to be accepted.

Velocity: The amount of estimated work completed in a comparable iteration.

Cycle time: Elapsed time from the start of work to its completion.

Throughput: The number of work items completed per unit of time.

Product-oriented delivery: Persistent ownership of a product and its outcomes by a cross-functional team.

Retrospective: A structured reflection used to improve a team’s process and working conditions.

Iterative discovery: Repeated learning about user needs, solution options, and product outcomes.

Review Questions

Short Answer Questions

1. What do transparency, inspection, and adaptation mean in empirical process control?
2. Distinguish a Product Backlog from a Sprint Backlog.
3. State two purposes of acceptance criteria.
4. Why should velocity not be used to compare teams?
5. What is the purpose of a WIP limit in Kanban?

Long Answer Questions

1. Compare Scrum, Kanban, Lean, and Extreme Programming, identifying the problem each approach primarily addresses.
2. Explain how user stories, acceptance criteria, and iterative discovery work together in product development.
3. Discuss the differences between project-oriented and product-oriented delivery models.
4. Explain how a retrospective can produce measurable process improvement rather than becoming a ceremonial meeting.
5. Evaluate the claim that agile methods reduce the need for documentation, governance, and planning.

Analytical / Application Questions

1. A team completes 16, 20, 18, and 22 story points in four Sprints. Calculate its average velocity and explain two reasons why the resulting forecast should remain approximate.
2. A Kanban system has an average throughput of 8 items per week and an average cycle time of 1.5 weeks. Estimate average WIP using Little’s Law, state the assumptions, and explain what could happen if the system is not stable.
3. Design an agile delivery approach for a public-health appointment service. Identify a product goal, two user stories, relevant acceptance criteria, feedback activities, and at least three ethical or governance concerns.

Multiple Choice Questions with Answers

Question 1

Which Scrum artifact is committed to the Product Goal?

A. Sprint Backlog

B. Product Backlog

C. Increment

D. Definition of Done

Correct Answer: B. Product Backlog

Question 2

A Kanban team averages 5 items per week and an average cycle time of 2 weeks. Under stable-system assumptions, what is its average WIP?

A. 2.5 items

B. 7 items

C. 10 items

D. 25 items

Correct Answer: C. 10 items

Question 3

Which practice most directly limits the amount of unfinished work in a Kanban workflow?

A. Increasing estimation points

B. Setting WIP limits

C. Lengthening review meetings

D. Removing acceptance criteria

Correct Answer: B. Setting WIP limits

Question 4

Which statement best describes a user story?

A. A complete technical design specification

B. A fixed contract that cannot change

C. A concise expression of a user need and intended benefit

D. A numerical measure of developer productivity

Correct Answer: C. A concise expression of a user need and intended benefit

Question 5

What is the primary purpose of a Sprint Retrospective?

A. To approve the annual budget

B. To inspect and improve the team’s way of working

C. To replace all automated tests

D. To demonstrate the product to external customers

Correct Answer: B. To inspect and improve the team’s way of working

Question 6

A team completes 18, 22, 20, and 24 points over four comparable Sprints. What is its average velocity?

A. 20 points per Sprint

B. 21 points per Sprint

C. 22 points per Sprint

D. 84 points per Sprint

Correct Answer: B. 21 points per Sprint

Question 7

Which Extreme Programming practice involves writing a failing test before implementing the behavior?

A. Pair rotation

B. Test-driven development

C. Portfolio planning

D. Work item aging

Correct Answer: B. Test-driven development

Question 8

Which measure is most directly associated with product outcomes?

A. Number of planning meetings held

B. Number of lines of code written

C. Successful completion of the user’s intended task

D. Number of story points assigned

Correct Answer: C. Successful completion of the user’s intended task

Question 9

Why is a Definition of Done important?

A. It guarantees that all future requirements are known

B. It provides a shared standard for usable, complete work

C. It eliminates the need for testing

D. It ranks developers by speed

Correct Answer: B. It provides a shared standard for usable, complete work

Question 10

Which statement best distinguishes product-oriented delivery from project-oriented delivery?

A. Product teams never plan work

B. Project teams always use waterfall methods

C. Product teams maintain ongoing responsibility for product outcomes

D. Product teams do not need governance

Correct Answer: C. Product teams maintain ongoing responsibility for product outcomes