Test Planning and Quality Process

Introduction

Software testing is most effective when it is treated as a planned quality activity rather than as a final inspection performed immediately before release. Test planning connects business objectives, requirements, architecture, development practices, people, tools, environments, and release decisions. It defines what will be tested, how testing will be performed, who will perform it, what evidence will be produced, and what conditions must be satisfied before testing or release can proceed.

In full-stack development, planning must address multiple layers: user interfaces, browser behavior, client-side logic, APIs, authentication, business rules, databases, external services, deployment configuration, and operational concerns. Unit tests provide fast feedback about isolated functions and components. Integration tests examine interactions among modules and services. System and end-to-end tests evaluate complete user journeys, often with tools such as Selenium. A quality process coordinates these levels so that testing is neither duplicated unnecessarily nor omitted at important boundaries.

This chapter presents a practical and analytical approach to test strategies, test plans, acceptance criteria, test cases, test data, environments, entry and exit criteria, requirements traceability, risk assessment, estimation, and defect management. It considers both traditional lifecycle models and Agile delivery.

Learning Objectives

By the end of this chapter, students should be able to:

1. Distinguish a test strategy from a test plan and select suitable testing levels.
2. Convert requirements into measurable acceptance criteria and test cases.
3. Design representative, controlled, and ethical test data.
4. Specify test environments, entry criteria, and exit criteria.
5. Construct a requirements traceability matrix.
6. Assess testing risks and estimate testing effort.
7. Manage defects through a controlled workflow and use evidence to support decisions.
8. Adapt quality processes to Agile, iterative, and traditional development models.

1. Quality as a Planned Engineering Process

Quality is not identical to the absence of discovered defects. A product may contain no currently known defect and still be unsuitable because it is insecure, inaccessible, slow, difficult to maintain, or inconsistent with stakeholder needs. A useful quality process therefore considers both verification and validation.

• Verification asks whether the product is being built correctly according to specified requirements, designs, and standards.
• Validation asks whether the correct product is being built for its intended users and context.

Testing provides evidence for both questions, but it cannot prove that a system contains no defects. Testing is sample-based: only selected inputs, states, environments, and sequences are exercised. Good planning improves the value of that sample by prioritizing important behavior and risk.

Quality assurance includes preventive activities such as standards, reviews, coding conventions, training, and process improvement. Quality control includes evaluative activities such as executing tests, inspecting results, and reporting defects. Testing is primarily a quality-control activity, although test planning also contributes to prevention by exposing ambiguous requirements and design weaknesses early.

Test strategy and test plan

A test strategy is the high-level approach governing testing across a product, program, or organization. It may define testing principles, levels, automation policy, risk priorities, environments, roles, tools, and quality gates.

A test plan applies that strategy to a particular release, sprint, feature, or project. It normally identifies scope, objectives, schedule, resources, test items, test techniques, data, environments, entry and exit criteria, reporting, and risks.

The distinction is important. A strategy might state that critical business rules require automated unit and API tests and that every release requires security regression testing. The release test plan then identifies the specific business rules, scripts, datasets, environment, people, dates, and evidence needed for version 4.2.

2. Selecting a Test Approach

A test approach should be based on product risk, architecture, delivery model, regulatory obligations, and available evidence. Common dimensions include the following.

Test levels

Unit testing examines a small, isolated unit such as a function, class, or component. Dependencies are often replaced with mocks, stubs, or fakes. Unit tests are fast and precise but may not reveal interface, configuration, database, or browser problems.

Integration testing evaluates interactions between components. Examples include a service communicating with a database, a front end calling an API, or an application publishing an event to a message broker. Integration tests may use real dependencies, containers, or controlled test doubles.

System testing evaluates the integrated application against system requirements. It may include complete workflows, cross-browser behavior, authorization, error handling, and non-functional tests.

Acceptance testing determines whether the product is acceptable to customers, product owners, or other authorized stakeholders. User acceptance testing is not simply a repeat of technical testing; it evaluates fitness for business use.

A full-stack test strategy should avoid relying exclusively on slow end-to-end tests. A balanced suite usually places many fast tests at the unit and service levels, fewer integration tests at important boundaries, and a focused set of end-to-end tests for high-value user journeys. This is often called the test pyramid, although its exact proportions are not a universal law.

Test types

Functional testing checks what the system does. Non-functional testing checks qualities of how it behaves. Important types include:

• Regression testing: confirms that changes have not broken previously working behavior.
• Smoke testing: a small, broad set of checks determining whether a build is stable enough for deeper testing.
• Sanity testing: focused checks of a changed area or correction.
• Performance testing: examines response time, throughput, resource usage, and behavior under load.
• Security testing: examines authentication, authorization, input handling, session management, and exposure of sensitive data.
• Usability and accessibility testing: evaluates learnability, clarity, keyboard operation, screen-reader compatibility, and other user needs.
• Compatibility testing: examines browsers, operating systems, devices, screen sizes, and relevant configurations.
• Exploratory testing: combines learning, test design, and execution in a time-boxed activity. It is structured investigation, not unplanned clicking.

3. From Requirements to Acceptance Criteria

Poor requirements create expensive testing problems. A requirement should describe an observable outcome, its conditions, and relevant constraints. Acceptance criteria convert a requirement into conditions that can be evaluated objectively.

For example, “Users should be able to reset their password easily” is incomplete. A stronger statement might be:

When a registered user submits a valid email address, the system sends a single-use reset link that expires after 30 minutes. The link permits a password satisfying the stated policy and cannot be reused after completion.

This statement supports test conditions involving a valid account, an unknown account, link expiry, reuse, password policy, duplicate requests, and privacy-preserving responses.

A common Agile format is:

• Given a precondition,
• When an action occurs,
• Then an observable result follows.

Example:

• Given a registered user is on the password-reset page,
• When the user submits a valid email address,
• Then the system confirms the request without exposing whether the address exists and sends an appropriate reset message.

Acceptance criteria should be testable, necessary, unambiguous, consistent, and measurable. Avoid vague words such as “quickly,” “user-friendly,” or “secure” unless they are accompanied by measurable conditions.

4. Test Conditions, Cases, and Procedures

A test condition is an item or behavior to be tested, such as “account locks after the fifth consecutive failed login.” A test case specifies inputs, preconditions, steps, expected results, and often a priority and traceability identifier. A test procedure may group cases into an executable sequence, including setup and cleanup.

A practical test case contains:

Test design techniques improve coverage without requiring every possible input. Equivalence partitioning divides inputs into groups expected to behave similarly; one or more representatives are selected from each group. Boundary value analysis focuses on edges because defects frequently occur at limits. Decision-table testing represents combinations of conditions and actions. State-transition testing examines behavior as an object moves among states. Pairwise testing reduces combinations while ensuring that selected pairs of parameter values occur together.

Worked example: boundary analysis

Suppose a username must contain 3 to 20 characters inclusive. Relevant partitions are:

• Invalid: fewer than 3 characters
• Valid: 3 through 20 characters
• Invalid: more than 20 characters

Useful boundary values include 2, 3, 4, 19, 20, and 21 characters. These values are not sufficient for all testing, but they efficiently examine the acceptance boundaries. The test must also consider allowed characters, whitespace, normalization, duplicate names, and security-related inputs.

5. Test Data Management

Test data is part of the test design, not an afterthought. Data should represent normal, boundary, invalid, exceptional, and security-sensitive conditions. It must also be repeatable enough to support reliable diagnosis.

Common data categories include:

• Positive data: expected valid inputs.
• Negative data: malformed, missing, unauthorized, or otherwise invalid inputs.
• Boundary data: values at, just below, and just above limits.
• Reference data: stable products, regions, roles, currencies, or configuration values.
• Stateful data: accounts, orders, or records prepared in specific lifecycle states.
• Volume data: sufficiently large datasets for performance and pagination testing.
• Fault data: unavailable services, timeouts, duplicate messages, and corrupted files.

Production data should not be copied casually into test environments. It may contain personal, financial, health, or confidential information. Appropriate controls include synthetic data generation, masking, tokenization, minimization, access control, retention limits, and audit logs. Masking must preserve useful relationships where necessary; independently replacing values can destroy referential integrity and reduce test validity.

Tests should declare their data dependencies and clean up safely. Shared mutable data often creates order-dependent tests: a test passes alone but fails after another test changes the same account. Isolated fixtures, unique identifiers, transactional rollback, and controlled reset procedures reduce this risk.

6. Environments and Configuration

A test environment is more than a server. It includes application version, operating system, browser, database schema, network, services, credentials, feature flags, queues, certificates, test data, monitoring, and deployment configuration.

A useful environment specification records:

1. Purpose and owner.
2. Hardware or cloud resources.
3. Software and dependency versions.
4. Database schema and seed data.
5. External-service simulations or contracts.
6. Secrets-management method.
7. Browser and device matrix.
8. Logging, monitoring, and time-zone settings.
9. Reset and backup procedures.
10. Known differences from production.

Environment parity is desirable but not always economically possible. Differences must be documented and assessed. For example, a local database may not reproduce production indexing behavior, while a mocked payment provider cannot reveal problems in the real provider’s error codes. Contract tests, representative staging environments, and limited production-like validation can reduce these gaps.

Selenium-based browser tests require additional control over browser versions, WebDriver or Selenium Manager configuration, viewport size, time zone, locale, permissions, and test accounts. Stable locators, explicit synchronization, and independent test setup are essential. A browser test should wait for a meaningful condition rather than use arbitrary delays, because fixed sleeps make tests slow and unreliable.

7. Entry and Exit Criteria

Entry criteria define conditions that must be met before a test activity begins. Examples include an approved build, deployed application, available test data, configured environment, reviewed requirements, and resolved blocking infrastructure defects.

Exit criteria define conditions for concluding a test activity. They should combine evidence and risk judgment, not merely elapsed time. Examples include completion of planned high-priority tests, acceptable results for critical workflows, no open critical defects, documented residual risks, and stakeholder approval.

Exit criteria are not identical to “zero defects.” Some defects may be accepted because they have low impact, a workaround, or a planned correction. Conversely, a release may be unacceptable even with few defects if a critical requirement remains untested.

A quality gate should identify:

• required evidence;
• thresholds or qualitative decision rules;
• decision authority;
• exceptions process; and
• communication of residual risk.

Metrics such as pass rate must be interpreted carefully. If many low-value tests pass while a critical scenario fails, the aggregate pass rate can be misleading. Tests blocked by environment failure should not automatically be counted as passed or failed; they should be reported as blocked with cause and impact.

8. Requirements Traceability

A requirements traceability matrix (RTM) links requirements to design elements, test cases, execution results, and defects. It supports coverage analysis, impact assessment, auditability, and change management.

A compact RTM may contain:

Traceability can be forward, from requirement to test, and backward, from test or defect to requirement. A missing test link indicates a potential coverage gap. A test without a business or technical rationale may be unnecessary or obsolete.

Coverage is often summarized as:

This is a reporting measure, not proof of quality. It assumes that requirements are clearly identified and that “mapped” means more than merely having a test name; the test must meaningfully address the requirement. Coverage should also be considered by risk, priority, test type, and execution status.

9. Risk-Based Testing

Risk-based testing allocates effort according to the likelihood and consequence of failure. A simple qualitative model classifies each risk by likelihood and impact as low, medium, or high. A quantitative approximation can use an exposure score:

Here, probability is a value between 0 and 1 when expressed as a proportion, and impact must be represented on a defined scale. The multiplication is useful for ranking only when the scales are consistently defined; it is not a prediction of financial loss unless impact is measured in monetary units.

For example, suppose a payment authorization failure is assigned probability 0.20 and impact 5 on an agreed five-point impact scale. Its exposure score is 1.0. A profile-image display defect assigned probability 0.60 and impact 1 has a score of 0.6. The payment risk receives greater testing priority despite its lower probability because its impact is substantially higher.

Risk analysis should consider:

• business criticality;
• complexity and change size;
• novelty of technology;
• integration count;
• defect history;
• security and privacy consequences;
• regulatory obligations;
• detectability of failure; and
• operational recoverability.

A risk register should record the risk, cause, event, consequence, likelihood, impact, exposure, mitigation, owner, trigger, and residual risk. Testing is one response; others include design changes, monitoring, redundancy, training, or accepting the risk with explicit authority.

10. Estimating Testing Effort

Testing estimates support staffing and scheduling, but they are uncertain forecasts rather than guarantees. A basic decomposition is:

Each component should be estimated using historical data, expert judgment, analogous projects, or task decomposition. A test case count alone is weak because cases vary greatly in complexity.

For a simple illustration, assume a release requires 40 hours of test design, 16 hours of environment and data preparation, 60 hours of first execution, 24 hours of retesting, and 10 hours of reporting. The estimated total is:

This estimate assumes the listed work is complete, resources are available, and no major scope change occurs. It should include a documented uncertainty allowance rather than disguising uncertainty as precision. Automation may reduce repeated execution effort but introduces development, maintenance, review, infrastructure, and failure-diagnosis costs. Therefore, “automate everything” is not a sound economic rule.

11. Defect Management Workflow

A defect is a deviation between an expected condition and an observed condition, supported by evidence. A defect report should enable another person to reproduce, understand, prioritize, and diagnose the problem.

A strong report includes:

• unique identifier;
• concise title;
• environment and build;
• preconditions and test data;
• reproducible steps;
• expected and actual results;
• frequency or reproducibility;
• screenshots, logs, network traces, or response payloads;
• severity and priority rationale;
• related requirement or test;
• reporter and owner.

Severity describes technical or business impact. Priority describes the order in which the issue should be addressed. A cosmetic defect can have low severity but high priority before a public launch; a severe defect in an unused experimental feature may receive lower immediate priority. They are related but not interchangeable.

A typical workflow is:

1. New: reported with initial evidence.
2. Triaged: reviewed for validity, severity, priority, duplication, and ownership.
3. Assigned: given to a responsible developer or team.
4. In progress: investigation and correction are underway.
5. Resolved: a proposed correction is available.
6. Ready for retest: deployed to a suitable environment.
7. Verified or reopened: testing confirms the correction or demonstrates continued failure.
8. Closed, deferred, rejected, or duplicated: final disposition is recorded with rationale.

Defect management should be blameless and evidence-based. The goal is to improve the product and process, not to punish individuals. Root-cause analysis may examine unclear requirements, inadequate review, missing automated checks, environment differences, or process weaknesses. Useful measures include defect aging, reopen rate, escape rate, severity distribution, and time to resolution, but metrics can be distorted when teams optimize the number of tickets rather than actual quality.

12. Agile and Traditional Quality Processes

In a traditional sequential model, test planning may produce a comprehensive plan early, with formal baselines, review gates, and acceptance phases. This can support contractual, regulated, or highly predictable work, but late discovery of requirement or integration problems is a significant risk.

Agile development distributes planning and testing across iterations. The team refines acceptance criteria, tests during development, integrates continuously, and reviews working software with stakeholders. The plan remains valuable, but it is progressively elaborated and updated as knowledge changes. A “definition of ready” may describe conditions for starting work, while a “definition of done” may require code review, automated tests, acceptance checks, and defect decisions.

Neither model removes the need for strategy. Agile does not mean undocumented testing, and a traditional plan does not justify postponing all testing until the end. Effective teams combine:

• early clarification of examples and acceptance criteria;
• automated checks in the continuous-integration pipeline;
• focused exploratory testing;
• risk-based regression selection;
• traceability appropriate to the domain; and
• transparent release decisions.

13. Case Study: Illustrative Online Grocery Platform

Background

An illustrative university project team is developing an online grocery platform. Customers browse products, place orders, pay through an external provider, and receive delivery updates. The application has a React front end, a REST API, a relational database, and a message-based notification service.

Problem

During a staging demonstration, orders occasionally show “paid” in the user interface while the order service remains in “pending.” The issue is difficult to reproduce. A browser test passes when executed alone but fails after a previous test has used the same customer account.

Analysis

The team maps requirements to tests and identifies two risks: payment callbacks may arrive late or more than once, and shared test data creates state contamination. The original test plan covered the successful payment path but not duplicate callbacks, delayed callbacks, service retries, or independent data setup.

Approach and solution

The team revises acceptance criteria to state that payment notifications are idempotent: processing the same provider event more than once must not create duplicate charges or transitions. Integration tests use a provider simulator capable of sending duplicates, delays, and failures. The database records the provider event identifier with a uniqueness constraint. API tests verify correct state transitions, while a small Selenium suite verifies the customer-visible order status.

Test fixtures generate a unique customer and order for each test. A cleanup job removes data created by the test run. The RTM links the payment requirement to unit, integration, API, and browser tests. Exit criteria require all high-risk payment tests to pass, no unresolved critical payment defects, and explicit product-owner acceptance of documented residual risks.

Outcome

The team reproduces the defect, corrects duplicate-event handling, and removes the order-dependent browser failure. The release is approved after evidence shows that delayed and repeated notifications produce one consistent order state.

Learning points

The case demonstrates that an end-to-end failure may originate in event processing, data isolation, or environment behavior. A layered strategy gives faster diagnosis than relying only on browser tests. Traceability exposes missing scenarios, while risk analysis directs effort toward payment integrity rather than superficial interface coverage.

14. Limitations, Governance, and Contemporary Practice

No test plan can anticipate every future state or interaction. Automated tests can become brittle, especially when they depend on implementation details or unstable user-interface locators. Tests may also pass while monitoring, deployment, accessibility, or real-world usage remains inadequate. Test results therefore require professional judgment.

Governance is especially important where systems process personal data, make consequential decisions, or operate in regulated sectors. Teams should define access controls, retention periods, auditability, segregation of duties where appropriate, and approval authority for release exceptions. Test data must comply with privacy obligations, and logs should avoid unnecessary sensitive values.

AI-assisted test generation can suggest cases, data, locators, or defect summaries. It does not replace human review. Generated tests may encode incorrect assumptions, miss domain risks, expose confidential information, or create large volumes of low-value checks. Teams should validate generated artifacts, protect proprietary data, and retain accountability for quality decisions.

Best practices include:

1. Start with risk and user outcomes, not merely with available tools.
2. Make acceptance criteria observable and measurable.
3. Test at the lowest suitable level and reserve end-to-end tests for critical journeys.
4. Keep test data isolated, reproducible, privacy-safe, and representative.
5. Treat environments and configuration as controlled test assets.
6. Use traceability to expose gaps and unnecessary tests.
7. Define entry and exit criteria before execution begins.
8. Report defects with evidence and distinguish severity from priority.
9. Review metrics for meaning, not just numerical improvement.
10. Update the strategy when architecture, risk, or delivery conditions change.

Chapter Summary

Test planning is the disciplined connection between requirements, risk, test design, execution, evidence, and release decisions. A test strategy defines the broad approach, while a test plan applies it to a product or release. Effective planning selects appropriate test levels and types, converts requirements into measurable acceptance criteria, and uses techniques such as equivalence partitioning, boundary analysis, decision tables, and state-transition testing.

Reliable testing depends on controlled test data, documented environments, explicit entry and exit criteria, and traceability from requirements to tests and defects. Risk-based testing prioritizes effort according to likelihood and impact. Estimation should decompose work and acknowledge uncertainty. Defect management requires reproducible evidence, clear workflow, appropriate severity and priority, and a blameless improvement focus.

Agile teams continuously refine and execute testing, whereas traditional models may use more formal baselines and gates; both require deliberate quality planning. In full-stack systems, layered tests, integration controls, isolated data, and focused Selenium tests provide stronger evidence than an undifferentiated collection of end-to-end scripts.

Key Terms

Test strategy: High-level policy and approach governing testing.

Test plan: Release- or project-specific description of testing activities and controls.

Verification: Evaluation of whether the product conforms to specified requirements.

Validation: Evaluation of whether the product satisfies intended user and business needs.

Acceptance criteria: Conditions that must be satisfied for a requirement to be accepted.

Test condition: A behavior, rule, or attribute selected for testing.

Test case: Defined inputs, conditions, steps, and expected results for a test.

Regression testing: Testing to detect unintended effects of changes.

Integration testing: Testing interactions among components or services.

Test data: Inputs and states used to execute tests.

Test environment: The complete technical and configuration context in which testing occurs.

Entry criteria: Conditions required before a test activity begins.

Exit criteria: Conditions used to decide whether a test activity may conclude.

Requirements traceability matrix: A mapping between requirements and related tests, results, and defects.

Risk-based testing: Allocation of testing effort according to risk.

Severity: Degree of impact caused by a defect.

Priority: Urgency or business order assigned to defect correction.

Defect workflow: Controlled lifecycle for reporting, investigating, correcting, and verifying defects.

Exploratory testing: Simultaneous learning, test design, and execution guided by a purpose.

Test pyramid: A model favoring many fast lower-level tests, fewer integration tests, and a focused set of end-to-end tests.

Review Questions

Short Answer Questions

1. Distinguish between a test strategy and a test plan.
2. What is the difference between verification and validation?
3. Why should test data be isolated between automated tests?
4. Define entry criteria and exit criteria.
5. Distinguish defect severity from defect priority.

Long Answer Questions

1. Explain how acceptance criteria can be transformed into test conditions and test cases. Include suitable examples.
2. Describe the role of a requirements traceability matrix in planning, execution, and release decisions.
3. Compare unit, integration, system, and acceptance testing in a full-stack application.
4. Explain how risk-based testing influences test scope, prioritization, and resource allocation.
5. Discuss how testing should be organized differently in Agile and traditional development models.

Analytical / Application Questions

1. A login requirement states that an account is locked after five consecutive failed attempts and unlocked after 30 minutes. Design boundary and state-transition test conditions, including relevant test data.
2. A team reports that 95% of tests pass, but the only failed test concerns payment authorization. Analyze whether release should proceed and identify additional evidence required.
3. Design a defect-management and traceability approach for a Selenium test that intermittently fails because the browser displays stale order data. Identify possible causes, evidence to collect, and corrective actions.

Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes a test strategy from a test plan?

A. A strategy contains individual test steps, while a plan contains source code.

B. A strategy defines the broad testing approach, while a plan applies it to a specific release or project.

C. A strategy is used only for manual testing, while a plan is used only for automation.

D. A strategy reports defects, while a plan fixes defects.

Correct Answer: B. A strategy defines the broad testing approach, while a plan applies it to a specific release or project.

Question 2

What is the primary purpose of acceptance criteria?

A. To specify the programming language used by developers

B. To define observable conditions for deciding whether a requirement is satisfied

C. To replace all integration tests

D. To identify the individual responsible for every defect

Correct Answer: B. To define observable conditions for deciding whether a requirement is satisfied

Question 3

Which technique is most directly focused on values at and around input limits?

A. Boundary value analysis

B. Random guessing

C. Code compilation

D. Defect triage

Correct Answer: A. Boundary value analysis

Question 4

What is a major purpose of a requirements traceability matrix?

A. To increase server memory

B. To link requirements with tests, results, and defects

C. To generate production passwords

D. To replace acceptance criteria

Correct Answer: B. To link requirements with tests, results, and defects

Question 5

A risk has probability 0.20 and impact score 5. Using risk exposure = probability × impact, what is its exposure score?

A. 0.04

B. 0.25

C. 1.00

D. 25.00

Correct Answer: C. 1.00

Question 6

Which practice best reduces order-dependent automated tests?

A. Using the same mutable account for every test

B. Adding arbitrary delays after every step

C. Creating isolated fixtures and controlled cleanup

D. Disabling failure reporting

Correct Answer: C. Creating isolated fixtures and controlled cleanup

Question 7

What does a defect’s severity primarily describe?

A. The technical or business impact of the defect

B. The number of developers on the project

C. The age of the test environment

D. The number of steps in the test case

Correct Answer: A. The technical or business impact of the defect

Question 8

Which is an appropriate exit criterion?

A. Testing has continued for exactly one hour

B. All tests, regardless of relevance, have been deleted

C. High-priority tests are complete and critical defects are resolved or explicitly accepted

D. The aggregate pass rate is reported without considering failed critical tests

Correct Answer: C. High-priority tests are complete and critical defects are resolved or explicitly accepted

Question 9

Why is a focused set of Selenium end-to-end tests generally preferred to testing every behavior through the browser?

A. Browser tests cannot test user journeys

B. End-to-end tests are often slower and more brittle than lower-level tests

C. Selenium can execute only one test per release

D. Unit tests always require a browser

Correct Answer: B. End-to-end tests are often slower and more brittle than lower-level tests

Question 10

Which statement about AI-assisted test generation is most accurate?

A. Generated tests never require review

B. AI tools eliminate the need for risk analysis

C. Generated artifacts should be reviewed for correctness, coverage, privacy, and relevance

D. AI-generated tests are valid only when they replace all manual testing

Correct Answer: C. Generated artifacts should be reviewed for correctness, coverage, privacy, and relevance