Software Testing and Verification

Introduction

Software testing is the systematic evaluation of a software product to discover defects, provide information about quality, and reduce the risks associated with its use. Verification asks whether the product is being built correctly according to specified requirements, designs, standards, and coding conventions. Validation asks whether the correct product has been built for its users and operational context. Testing is a major verification and validation activity, although reviews, static analysis, inspections, and formal reasoning also contribute.

Testing cannot prove that a program contains no defects. It can demonstrate that selected behaviours work under selected conditions and can provide evidence about residual risk. Effective testing therefore depends on disciplined planning, representative test data, traceability, risk analysis, appropriate tools, and clear communication of results.

This chapter develops the practical foundations needed to design and conduct unit, integration, system, regression, acceptance, and non-functional testing. It also explains test plans, test cases, coverage, test doubles, automation, defect tracking, and professional testing practice.

Learning Objectives

By the end of this chapter, a student should be able to:

1. Distinguish verification from validation and explain the role of testing.
2. Select suitable testing levels, techniques, and test conditions.
3. Prepare a test plan and well-structured test cases.
4. Apply equivalence partitioning, boundary-value analysis, decision-table testing, and state-transition testing.
5. Explain code coverage and interpret coverage measures responsibly.
6. Use test doubles in unit and integration testing.
7. Design regression, acceptance, and non-functional tests.
8. Describe an automated testing pipeline and a professional defect workflow.
9. Evaluate testing results, limitations, risks, and ethical responsibilities.

1. Verification, Validation, and Quality

Verification examines work products against stated specifications. Examples include checking whether an interface implements the approved design, whether a function returns the required data type, and whether source code follows a security standard. Validation evaluates fitness for intended use. A system may satisfy its written specification yet fail validation if users cannot understand it or if it does not support their real workflow.

Testing is usually described as a dynamic activity because software is executed. Static testing, by contrast, examines artefacts without execution. Code review, requirements inspection, static security analysis, and model checking are examples of static activities. A mature quality process combines both forms.

Important testing principles include:

• Testing shows the presence of defects, not their absence.
• Exhaustive testing is generally impossible for non-trivial systems.
• Testing should begin as early as practical; defects found late are often more expensive to correct.
• Defects tend to cluster in a small number of components.
• Repeating the same tests eventually provides diminishing benefit, a phenomenon called the pesticide paradox.
• A test strategy must be adapted to the product, risks, technology, and stakeholders.
• A product that is technically correct may still fail if it does not meet user needs.

Quality is multidimensional. Functional suitability, reliability, usability, performance efficiency, security, maintainability, compatibility, and portability may all matter. Testing priorities should reflect business impact and risk rather than merely the number of test cases executed.

2. Testing Levels and Test Objects

Unit testing

Unit testing examines the smallest practical testable component, such as a function, class, or module. The purpose is to isolate local logic and provide fast feedback. Unit tests should normally be deterministic, repeatable, and independent of external databases, networks, clocks, or file systems. Developers commonly create and maintain them as part of implementation.

Integration testing

Integration testing examines interactions among components, services, databases, devices, or external systems. It can reveal interface mismatches, incorrect data formats, timing problems, transaction errors, and configuration faults. Integration may proceed incrementally, for example from a few related components to larger subsystems, or through a broader system integration test.

System testing

System testing evaluates the complete integrated product against functional and non-functional requirements. It is often performed in an environment resembling production. System tests may cover business workflows, error handling, security controls, performance, recovery, and interoperability.

Regression testing

Regression testing checks that changed software has not unintentionally damaged existing behaviour. It is not a separate level so much as a purpose that can apply to unit, integration, system, and acceptance tests. A regression suite should be risk-based and maintained; obsolete tests can create false confidence or unnecessary execution cost.

Acceptance testing

Acceptance testing determines whether the product is acceptable to its customer, sponsor, regulator, or operational users. User acceptance tests normally express realistic business scenarios rather than internal implementation details. Contract acceptance, operational acceptance, regulatory acceptance, and alpha or beta testing are possible forms.

A useful test hierarchy is sometimes described as a pyramid: many fast unit tests, fewer integration tests, and a smaller number of broad end-to-end tests. The exact proportions are not universal, but the principle is useful: tests should provide rapid, informative feedback at reasonable cost.

3. Test Planning and Traceability

A test plan defines the scope, strategy, resources, schedule, environment, responsibilities, risks, entry criteria, exit criteria, and reporting arrangements for testing. It should answer questions such as:

• What product and requirements are in scope?
• Which quality risks are most important?
• What levels and techniques will be used?
• What data, tools, environments, and test accounts are required?
• Who will design, execute, review, and approve tests?
• What constitutes a blocked, failed, passed, or incomplete test?
• When is testing sufficient for the current release decision?

Entry criteria might include an available build, approved requirements, configured test data, and a stable environment. Exit criteria might include completion of critical tests, resolution or acceptance of high-severity defects, acceptable performance results, and a documented residual-risk decision. Exit criteria are not a claim of perfection; they are a controlled basis for release.

Requirements traceability links requirements to design elements, test conditions, test cases, and defects. A simple traceability matrix can identify untested requirements and show the impact of a requirement change. For safety-critical, financial, medical, or regulated systems, traceability may be an audit obligation.

Test case structure

A useful test case contains:

• unique identifier and title;
• requirement or risk reference;
• preconditions and environment;
• input data and setup;
• numbered execution steps;
• expected results, including observable state changes;
• actual result and status;
• evidence, such as logs or screenshots;
• tester, date, build, and related defect identifier.

Expected results must be precise. “The system works” is not testable; “the transaction is stored with status Pending and a confirmation identifier is displayed” is observable and verifiable.

4. Test Design Techniques

Equivalence partitioning

Equivalence partitioning divides input values into groups expected to behave similarly. One or more representatives are selected from each group. For an age field accepting integer values from 18 through 65, useful partitions include values below 18, values from 18 to 65, and values above 65. The technique reduces redundant tests, but the assumption that values in a partition behave alike must be reviewed.

Boundary-value analysis

Defects often occur at limits. For the same valid range, important values include 17, 18, 19, 64, 65, and 66. Boundary tests should consider inclusive versus exclusive conditions, data type limits, empty values, maximum lengths, and transitions between states.

Decision-table testing

Decision tables represent combinations of conditions and resulting actions. They are useful when business rules contain several interacting conditions, such as eligibility based on membership, payment status, and order value. Redundant or impossible combinations should be identified rather than blindly implemented as tests.

State-transition testing

State-transition testing examines valid and invalid changes between states. An order might move from New to Paid, Packed, Shipped, and Delivered, while cancellation may be allowed only before shipment. Tests should cover valid transitions, prohibited transitions, repeated events, and recovery after failure.

Use-case and scenario testing

Scenario tests follow realistic user goals across system boundaries. They are valuable for discovering integration and usability problems that isolated unit tests cannot reveal. Both successful and alternative or exception flows should be included.

Exploratory testing

Exploratory testing combines learning, test design, and execution. The tester investigates the product while recording observations and may use a time-limited charter, such as “explore account recovery with invalid, expired, and repeated tokens.” It is structured investigation, not unplanned or undocumented clicking.

5. Coverage Measures

Coverage measures indicate which selected elements have been exercised. They are useful indicators, not direct measures of correctness.

Statement coverage is commonly expressed as:

Branch coverage is commonly expressed as:

Here, a statement is an executable program statement, and a decision outcome is a possible result of a decision, such as True or False. These simple ratios assume that the numerator and denominator are measured consistently and that excluded, unreachable, or generated code is explicitly treated. A high percentage does not prove that assertions are correct, inputs are representative, security properties hold, or requirements are complete.

For example, if 90 of 100 relevant statements execute, statement coverage is 90%. If a conditional has two outcomes and tests execute only the True outcome, branch coverage for that decision is 50%, even if the statement inside the True branch is executed repeatedly. Coverage should be combined with requirements coverage, risk coverage, mutation testing where suitable, code review, and defect history.

6. Test Doubles and Isolation

A test double is a replacement for a real collaborator used to control, observe, or simplify a test.

Terminology varies between frameworks, and one object may have more than one role. Doubles improve speed and determinism, but excessive mocking can make tests mirror implementation rather than behaviour. Important integration paths still require tests against realistic components and environments.

A unit test should generally follow Arrange–Act–Assert: arrange data and doubles, act by invoking the unit, and assert the observable result. Tests should avoid shared mutable state, hidden ordering dependencies, and reliance on real time. Where time is relevant, inject a clock or use a controlled time source.

7. Automation and Continuous Testing

Automated tests are executable checks run by a tool or pipeline. Automation is particularly effective for repeatable regression tests, data-driven tests, API checks, static analysis, and performance baselines. It does not eliminate the need for human judgment, exploratory testing, usability evaluation, or stakeholder acceptance.

A continuous integration pipeline may perform the following sequence:

1. obtain the source revision and dependencies;
2. compile or package the product;
3. run linting, static analysis, and unit tests;
4. publish results and coverage;
5. run integration and API tests;
6. deploy to a controlled test environment;
7. execute system, security, or performance checks;
8. make results visible and apply release gates.

Tests should be reliable. A flaky test sometimes passes and sometimes fails without a relevant product change. Flakiness may arise from race conditions, uncontrolled time, shared environments, random data, network dependency, or insufficient cleanup. Flaky tests should be quarantined only with ownership and a repair plan; permanently ignoring them weakens confidence.

Automation should be selected using a cost–benefit perspective. Consider execution frequency, stability, maintenance effort, defect impact, data setup, tool support, and the cost of a false pass or false failure. Automated tests are software and require version control, review, refactoring, and defect management.

8. Non-Functional Testing

Non-functional testing examines qualities and constraints rather than only business functions.

• Performance testing measures response time, throughput, resource use, and scalability under specified workloads.
• Load testing evaluates expected operational demand.
• Stress testing examines behaviour beyond expected demand and the quality of degradation or recovery.
• Security testing examines authentication, authorization, confidentiality, integrity, input handling, logging, and resistance to relevant threats.
• Usability and accessibility testing examines learnability, error prevention, keyboard operation, screen-reader compatibility, contrast, and understandable feedback.
• Reliability and recovery testing examines failure rates, restart behaviour, backup restoration, and resilience.
• Compatibility testing examines browsers, operating systems, devices, protocols, and external versions.
• Installation and configuration testing examines deployment, upgrade, rollback, and environment-specific settings.

A performance requirement should be measurable and contextual. For example, “95% of search requests shall complete within 2 seconds for 500 concurrent users under the defined dataset and network conditions” is more testable than “the search shall be fast.” Results must record hardware, software version, workload, dataset, measurement method, and environmental conditions.

9. Defect Management

A defect report should allow another person to reproduce and understand the problem. It normally includes a concise title, product version, environment, preconditions, steps to reproduce, expected result, actual result, evidence, severity, priority, and reporter. Severity describes technical or user impact; priority describes urgency and ordering. A severe defect is not automatically the highest priority if it affects a rarely used feature, while a moderate defect may require urgent correction before a public event.

A typical workflow is New, Triaged, Assigned, In Progress, Fixed, Ready for Retest, Verified, and Closed. Other states such as Reopened, Rejected, Duplicate, Deferred, or Cannot Reproduce may be needed. Retesting confirms the correction for the original failure; regression testing checks related and previously working behaviour.

Defect reports should be factual and respectful. They should describe observable behaviour rather than blame individuals. Trend analysis can reveal recurring causes, unstable components, escape defects, and process weaknesses. Root-cause analysis may identify unclear requirements, inadequate review, incorrect assumptions, insufficient test data, or environmental differences.

10. Applied Case: Illustrative Online Pharmacy Order System

Background

Consider an illustrative online pharmacy system that permits customers to search products, upload prescriptions, pay for eligible orders, and receive delivery updates. Because medication orders involve privacy, safety, and regulatory obligations, both functional and non-functional risks are significant.

Problem

During a pilot release, some orders were marked Paid although the payment provider had returned a timeout. A second problem allowed a cancelled order to proceed to packing when cancellation and warehouse processing occurred nearly simultaneously.

Analysis

The first defect involved an integration boundary and ambiguous handling of uncertain payment outcomes. The second involved a race condition in a state transition. Simple happy-path system tests had not represented either condition. The risks included financial inconsistency, incorrect fulfilment, customer harm, and audit difficulty.

Approach and solution

The team added unit tests for payment-result mapping, including success, decline, timeout, duplicate callback, and malformed response. A stub payment provider supplied controlled responses, while integration tests used a sandbox provider. Contract tests checked that request and response schemas remained compatible.

A state-transition table was created for New, Payment Pending, Paid, Cancelled, Packed, and Shipped. Tests covered valid transitions, prohibited transitions, repeated cancellation, and simultaneous cancellation and packing requests. The implementation used an atomic state update so that only one valid transition could succeed. An idempotency key prevented duplicate payment callbacks from creating duplicate effects.

Acceptance tests used realistic workflows involving prescription approval, payment, cancellation, notification, and audit records. Security tests checked that one customer could not access another customer’s prescription. Performance tests measured search and order submission under the expected peak workload. The release pipeline ran unit and API tests on every change and broader tests before deployment.

Outcome

The revised tests detected the original defects before production release, exposed an additional missing audit event, and provided traceability from safety-related requirements to tests. The project accepted a known limitation: the external payment sandbox did not reproduce all provider outages, so operational monitoring and a documented manual recovery procedure were added.

Learning points

This case illustrates that testing should be risk-driven, not merely feature-count-driven. Integration boundaries, state transitions, concurrency, idempotency, security, and recovery require deliberate test design. Test doubles provide control, but realistic integration tests and production monitoring remain necessary.

11. Challenges, Ethics, and Contemporary Practice

Testing is constrained by time, budget, incomplete requirements, unavailable environments, changing dependencies, and the impossibility of testing every combination. A team must communicate uncertainty rather than report misleading certainty. A green pipeline can reflect weak tests, incorrect assertions, or omitted requirements.

Test data may contain personal, financial, medical, or confidential information. Production data should not be copied casually into test environments. Data should be minimised, anonymised or synthetically generated, access-controlled, retained only as necessary, and securely destroyed. Testers must also avoid creating security vulnerabilities through careless credentials, exposed logs, or unsafe attack testing.

AI-assisted tools can generate test ideas, input data, code scaffolding, or summaries. They can also generate incorrect assertions, miss domain risks, reproduce bias, leak confidential information, or create tests that pass without meaningful verification. Human review, approved data policies, traceability, and independent execution remain essential. AI-generated tests should be treated as proposals requiring validation, not as evidence by themselves.

Contemporary practice increasingly includes shift-left activities such as contract review and static analysis before coding, and shift-right activities such as production observability, canary releases, feature flags, synthetic monitoring, and controlled fault injection. These practices extend quality feedback across the software life cycle but do not replace disciplined pre-release testing.

Best-Practice Checklist

• Derive tests from requirements, risks, architecture, and user workflows.
• Make expected results observable and unambiguous.
• Use equivalence classes and boundaries to avoid arbitrary test selection.
• Keep unit tests fast and isolated, while preserving meaningful integration tests.
• Measure relevant coverage and explain exclusions.
• Automate stable, repetitive checks and review the automation itself.
• Test failure handling, recovery, security, accessibility, and data protection.
• Control test environments, accounts, clocks, randomness, and external services.
• Track defects with reproducible evidence and clear severity and priority.
• Combine automated checks with exploratory testing and stakeholder acceptance.
• Use exit criteria and residual-risk decisions rather than relying on test counts alone.

Chapter Summary

Software testing provides evidence about product behaviour and risk; it does not prove the absence of defects. Verification checks conformance to specified work products, while validation checks fitness for intended use. Unit, integration, system, regression, and acceptance testing address different scopes and purposes. Test plans establish scope, responsibilities, environments, risks, and completion criteria, while traceability links requirements to tests and defects.

Equivalence partitioning, boundary-value analysis, decision tables, state-transition testing, scenario testing, and exploratory testing help create effective test conditions. Statement and branch coverage are useful indicators but must not be confused with correctness. Test doubles support isolation, and automation provides rapid repeatable feedback when maintained responsibly. Non-functional testing addresses performance, security, usability, accessibility, reliability, compatibility, and recovery. Defect tracking, ethical data handling, human review, and risk-based release decisions are essential professional practices.

Key Terms

Verification: Checking whether a work product conforms to specified requirements or standards.

Validation: Evaluating whether a product meets user needs and intended use.

Test case: A documented set of conditions, inputs, actions, and expected results.

Test plan: A document defining testing scope, strategy, resources, risks, and completion criteria.

Unit testing: Testing an isolated function, class, or module.

Integration testing: Testing interactions among components or external systems.

System testing: Testing the complete integrated product.

Regression testing: Testing to detect unintended effects of change.

Acceptance testing: Testing performed to determine whether stakeholders accept the product.

Equivalence partitioning: Dividing inputs into groups expected to behave similarly.

Boundary-value analysis: Testing values at and around limits.

Decision table: A structured representation of condition combinations and actions.

State-transition testing: Testing permitted and prohibited changes between states.

Coverage: A measure of exercised program elements or specified test elements.

Test double: A replacement collaborator used to control or observe a test.

Stub: A test double that supplies controlled responses.

Mock: A test double used to verify expected interactions.

Flaky test: A test whose result changes unpredictably without a relevant product change.

Severity: The impact of a defect on the system or users.

Priority: The urgency or order in which a defect should be addressed.

Review Questions

Short Answer Questions

1. Distinguish verification from validation.
2. What is the purpose of a test plan?
3. Define regression testing and explain when it is used.
4. What is a test double? Name two types.
5. Why does high code coverage not prove software correctness?

Long Answer Questions

1. Explain the differences among unit, integration, system, and acceptance testing, using an example system.
2. Describe how equivalence partitioning and boundary-value analysis can be applied to a constrained input field.
3. Explain the contents of a professional test case and the importance of precise expected results.
4. Discuss the benefits and limitations of test automation in a continuous integration pipeline.
5. Explain how non-functional testing contributes to software quality, with examples of at least four non-functional qualities.

Analytical / Application Questions

1. A registration form accepts usernames containing 6–20 characters. Design equivalence classes and boundary-value tests, including invalid and valid cases.
2. An order may move through New, Paid, Packed, Shipped, and Delivered, but cancellation is allowed only before Packing. Design a state-transition test strategy and identify two prohibited transitions.
3. A release has high unit-test coverage but repeated production failures at the payment-provider boundary. Analyse why the coverage result was insufficient and propose a revised test strategy.

Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes verification from validation?

A. Verification checks conformance to specifications, while validation checks fitness for intended use.

B. Verification is performed only by customers, while validation is performed only by developers.

C. Verification applies only to performance, while validation applies only to security.

D. Verification occurs after release, while validation occurs before coding.

Correct Answer: A. Verification checks conformance to specifications, while validation checks fitness for intended use.

Question 2

Which testing level primarily examines an isolated function or class?

A. Acceptance testing

B. Unit testing

C. System testing

D. Operational testing

Correct Answer: B. Unit testing

Question 3

For an input field accepting integers from 18 through 65 inclusive, which value is a boundary test just below the lower valid boundary?

A. 18

B. 19

C. 17

D. 65

Correct Answer: C. 17

Question 4

What is the principal purpose of a stub?

A. To measure branch coverage

B. To supply controlled responses from a collaborator

C. To replace the entire production system permanently

D. To assign defect priority

Correct Answer: B. To supply controlled responses from a collaborator

Question 5

If 80 of 100 relevant statements are executed, what is the statement coverage?

A. 20%

B. 50%

C. 80%

D. 125%

Correct Answer: C. 80%

Question 6

Which activity is most directly associated with acceptance testing?

A. Checking whether stakeholders can approve realistic business workflows

B. Checking whether a private helper method has branch coverage

C. Measuring compiler memory consumption

D. Replacing all external services with mocks

Correct Answer: A. Checking whether stakeholders can approve realistic business workflows

Question 7

What does a flaky test do?

A. Always fails because the requirement is wrong

B. Produces inconsistent results without a relevant product change

C. Tests only invalid input

D. Measures only system throughput

Correct Answer: B. Produces inconsistent results without a relevant product change

Question 8

Which technique is especially suitable for combinations of business conditions and resulting actions?

A. Decision-table testing

B. Random deletion testing

C. Syntax highlighting

D. Version tagging

Correct Answer: A. Decision-table testing

Question 9

Which defect-report field describes the urgency or order in which a defect should be addressed?

A. Severity

B. Priority

C. Precondition

D. Coverage

Correct Answer: B. Priority

Question 10

Which is a limitation of high branch coverage?

A. It guarantees that all requirements are correct.

B. It proves that users will find the system usable.

C. It does not guarantee correct assertions or representative test data.

D. It prevents the need for integration testing.

Correct Answer: C. It does not guarantee correct assertions or representative test data.