Software Testing and Quality Engineering

Introduction

Software testing is the systematic evaluation of software to find defects, reduce technical and business risk, and provide evidence that a system is fit for use. Quality engineering is broader than testing: it integrates prevention, detection, measurement, automation, operational feedback, and continuous improvement across the software life cycle. A modern engineer therefore asks not only “Does the program work?” but also “How do we know?”, “Under which conditions?”, “What risks remain?”, and “How efficiently can evidence be produced and maintained?”

Testing cannot prove the complete absence of defects in a non-trivial program. It can, however, provide increasingly strong evidence through carefully selected tests, reviews, static analysis, observability, and risk-based reasoning. This chapter presents a practical testing strategy for contemporary software teams, including unit, integration, contract, system, acceptance, property-based, mutation, and exploratory testing.

Learning Objectives

By the end of this chapter, a student should be able to:

• distinguish verification, validation, testing, quality assurance, and quality engineering;
• design a layered automated test strategy;
• select and control test doubles;
• construct unit, integration, contract, system, and acceptance tests;
• use property-based, mutation, and exploratory testing appropriately;
• interpret coverage and static-analysis evidence without overclaiming its meaning;
• prioritize testing using risk, impact, likelihood, and detectability;
• evaluate test suites for maintainability, reliability, speed, and diagnostic value;
• recognize ethical, security, privacy, accessibility, and regulatory responsibilities in quality engineering.

1. Foundations of Software Quality

1.1 Verification and validation

Verification asks whether software has been built according to specified requirements and design. Reviews, static analysis, unit tests, and traceability checks support verification. Validation asks whether the resulting system solves the users’ real problem. Acceptance testing, usability evaluation, operational trials, and stakeholder feedback support validation.

A requirement such as “the system shall process a payment” is incomplete unless quality attributes are also considered. Relevant attributes may include correctness, performance, security, reliability, usability, accessibility, maintainability, portability, and observability. Testing should therefore be derived from both functional requirements and quality risks.

Quality assurance emphasizes planned processes that prevent defects, such as coding standards, design reviews, continuous integration, and definition-of-done criteria. Quality control emphasizes detecting nonconformance through inspection and testing. Quality engineering combines both perspectives and treats quality as a responsibility of the entire delivery system rather than a final activity assigned only to testers.

1.2 The testing principles

Several principles guide professional testing:

1. Testing shows the presence, not the absence, of defects. Passing tests increase confidence but do not establish perfection.
2. Exhaustive testing is usually impossible. Input spaces, states, environments, and timing combinations are often too large.
3. Early testing is economical. Defects found near their origin are generally easier to diagnose and repair than defects found after integration or release.
4. Defects cluster. A small number of components often contain a large proportion of failures, so historical and architectural risk should influence test effort.
5. The pesticide paradox applies. A fixed test suite eventually stops finding new defect classes; tests must evolve.
6. Tests are context dependent. A safety-critical controller, a social application, and a scientific simulation require different evidence.
7. A system with no known defects may still be useless. Conformance does not guarantee that the product meets user needs.

1.3 Test oracles and testability

A test oracle determines whether an observed result is acceptable. It may be a precise expected value, a reference implementation, an invariant, a business rule, a metamorphic relation, or human judgment. Oracle problems arise when correct output is difficult to calculate, as in machine learning, simulations, or image processing.

Testability is the ease with which software can be controlled, observed, isolated, and evaluated. Dependency injection, deterministic clocks, stable identifiers, structured logs, event tracing, and small cohesive modules improve testability. Hidden global state, uncontrolled randomness, real network dependencies, and time-dependent behavior reduce it.

2. A Layered Test Strategy

A useful test strategy distributes evidence across levels. The commonly used pyramid places many fast, focused unit tests at the base, fewer integration tests in the middle, and a smaller number of broad system or user-interface tests at the top. The pyramid is a heuristic, not a law. For distributed systems, contract and component tests may be especially important; for safety-critical systems, formal analysis and hardware-in-the-loop testing may be required.

The strategy should map risks to test levels. A tax calculation may need precise unit tests, integration tests with payment and accounting systems, acceptance tests for legal rules, and mutation testing of boundary conditions. A user-interface color preference may require accessibility and usability evaluation rather than extensive service-level testing.

2.1 Test case design

Good tests partition the input space rather than selecting arbitrary examples. Common techniques include:

• Equivalence partitioning: divide inputs into groups expected to behave similarly.
• Boundary-value analysis: test values at, just below, and just above limits.
• Decision-table testing: enumerate combinations of conditions and actions.
• State-transition testing: test valid and invalid transitions between states.
• Pairwise testing: cover interactions between pairs of factors when exhaustive combinations are too numerous.
• Traceability: link tests to requirements, risks, defects, or properties.

A test should normally have a clear fixture, action, and observable assertion. It should fail for the intended reason, communicate its purpose, and avoid unnecessary dependence on implementation details.

3. Unit Testing and Test Doubles

A unit is the smallest meaningful testable behavior, such as a function, class, or cohesive module. Unit tests should be deterministic, isolated where isolation adds value, fast enough for frequent execution, and precise when they fail.

Consider a shipping-cost function with the rule: orders below 50 monetary units incur a cost of 8; orders of 50 or more qualify for free shipping. Important tests include values 0, 49.99, 50, and 50.01, together with invalid negative values if the domain forbids them. The boundary at 50 is more informative than many randomly chosen values.

3.1 Test doubles

A test double substitutes for a real collaborator:

• Dummy: passed only to satisfy an interface.
• Stub: returns prepared answers.
• Spy: records calls for later inspection.
• Mock: verifies expected interactions, often during the test.
• Fake: a simpler working implementation, such as an in-memory repository.

Mocks can reveal incorrect collaboration protocols, but excessive interaction assertions make tests brittle during harmless refactoring. Prefer state or observable business outcomes when those provide sufficient evidence. Use a fake when its behavior is faithful enough for the purpose; do not assume an in-memory database reproduces transaction isolation, query planning, or vendor-specific behavior.

3.2 Managing time, randomness, and external effects

Inject a clock rather than reading the operating-system clock directly. Inject a random-number generator or seed it when reproducibility is important. Encapsulate file, network, email, payment, and message-broker effects behind interfaces. Tests should explicitly control retries, timeouts, and failure responses. A test that sometimes passes and sometimes fails without code changes is flaky; flaky tests consume engineering time and damage trust in automation.

4. Integration, Contract, System, and Acceptance Testing

4.1 Integration testing

Integration tests evaluate interactions among real components. They are essential where defects arise from serialization, database schemas, authentication, transaction boundaries, configuration, concurrency, or infrastructure behavior. Test environments should be isolated and reproducible where possible. Containers, disposable databases, migration checks, and seeded data can improve repeatability.

An integration test should expose which boundary it exercises. A failure saying only “checkout failed” is less useful than one identifying a rejected schema field, an authorization response, or a transaction rollback.

4.2 Contract testing

Contract testing verifies an agreement between a consumer and a provider. A consumer-driven contract may specify an HTTP method, path, request fields, response status, and response schema. The provider verifies that it can satisfy the published contract. This catches incompatible changes earlier than a complete end-to-end test suite and is particularly valuable when teams deploy services independently.

Contracts must include meaningful semantics, not merely syntactic shape. A field that is present but interpreted in the wrong currency or time zone is still a contract failure. Contracts also require versioning, compatibility policy, and ownership. They do not replace provider unit tests or realistic integration tests.

4.3 System and acceptance testing

System tests evaluate a complete deployed product or a substantial slice of it. They should include realistic authentication, persistence, messaging, configuration, and failure behavior. Because broad tests are expensive to diagnose, keep scenarios business-oriented and limit unnecessary duplication of lower-level tests.

Acceptance tests express conditions under which stakeholders accept a feature. They may be written as examples in a behavior-driven style: given an account is active, when a valid payment is submitted, then an order is recorded and a receipt is issued. Acceptance criteria should be observable, unambiguous, and linked to business value. They should also include negative, security, accessibility, and operational conditions where relevant.

5. Property-Based, Mutation, and Exploratory Testing

5.1 Property-based testing

Example-based testing checks selected examples. Property-based testing generates many inputs and checks general properties that should remain true. For a sorting function, possible properties include that the output is ordered, contains exactly the same multiset of elements as the input, and has the same length.

A property must be meaningful and independent enough to detect defects. Testing a function against a reference implementation can be powerful, but if both implementations share the same defect, the oracle is compromised. Generators should produce valid, invalid, boundary, and structurally complex values. Shrinking reduces a failing generated input to a smaller counterexample, improving diagnosis.

Metamorphic testing is useful when an exact oracle is difficult. If a query result is invariant under reordering of independent input records, then reordering should not change the result. If a transformation is known to change output predictably, the relation can be tested even without knowing the absolute correct output.

5.2 Mutation testing

Mutation testing evaluates the sensitivity of a test suite by making small artificial changes, called mutants, such as replacing > with ≥, deleting a condition, or changing a return value. A mutant is killed if at least one test fails; it survives if all tests pass.

The mutation score is commonly defined as:

where K is the number of killed mutants, T is the total number of generated mutants, and E is the number of equivalent or otherwise excluded mutants. The formula assumes that excluded mutants have been identified consistently. Equivalent mutants change implementation without changing externally observable behavior and cannot reasonably be killed by a correct test.

Mutation score is evidence about fault-detection strength, not a direct probability that production code is correct. Mutation operators should be relevant to the language and domain, and surviving mutants should be reviewed rather than automatically treated as defects in the test suite.

5.3 Exploratory testing

Exploratory testing combines learning, test design, and execution. A tester may use a time-boxed charter such as: “Explore account recovery with expired links, repeated requests, unusual encodings, and interrupted network access.” The tester records actions, observations, risks, and defects. Session-based management gives exploratory work structure without reducing it to scripted repetition.

Exploratory testing is especially effective for usability, accessibility, emergent interactions, error recovery, and areas where requirements are incomplete. It complements automation; it does not excuse the absence of regression tests for stable, repeatable behavior.

6. Coverage, Static Analysis, and Quality Evidence

Coverage measures which selected structural elements were exercised. Statement coverage records executed statements; branch coverage records evaluated decision outcomes; condition coverage considers individual Boolean conditions; path coverage concerns execution paths and is generally impractical for large systems.

A basic statement-coverage percentage is:

The denominator must be defined, including whether generated, unreachable, or excluded code is counted. Coverage is meaningful only in relation to assertion quality and risk. A test may execute a statement without checking its result. Therefore, high coverage can coexist with weak testing, while lower coverage may be acceptable for generated or low-risk code if important behavior is otherwise evidenced.

Static analysis examines source or compiled artifacts without executing the program. It can detect type errors, unreachable code, suspicious data flow, security patterns, duplicated logic, dependency vulnerabilities, and style violations. Tools produce false positives and false negatives; teams should configure rules, review findings, suppress exceptions with justification, and prevent only findings whose cost and reliability are understood.

Other useful evidence includes defect escape rate, flaky-test rate, test duration, mean time to diagnose, change-failure rate, recovery time, performance thresholds, security scan results, and accessibility conformance. Metrics should support decisions, not become targets that encourage gaming.

7. Risk-Based Quality Engineering

Testing effort should be proportional to risk. A practical risk model considers the probability of failure and the consequence if failure occurs. A simple risk priority number can be used as a prioritization aid:

where S is severity, L is likelihood, and D is detectability difficulty, each assigned on an explicitly defined ordinal scale. Because ordinal scores are not physical measurements, the product is a ranking aid rather than a probability or monetary loss. Teams should supplement it with expert judgment, threat modeling, regulatory obligations, and actual incident data.

High-risk areas may require multiple independent forms of evidence: unit and property tests, integration tests, security analysis, code review, operational monitoring, and acceptance testing. Low-risk areas may need a smaller sample. Risk assessment should be revisited when architecture, dependencies, data, users, or regulations change.

A release decision should consider residual risk, not merely a pass rate. Questions include: Which requirements lack evidence? Which failures are undiagnosed? Are known defects accepted by an accountable owner? Can the system be rolled back? Are monitoring and incident-response arrangements adequate?

8. A Continuous Testing Workflow

A professional workflow commonly includes the following activities:

1. Identify business, technical, security, safety, privacy, and operational risks.
2. Define observable acceptance criteria and testable quality attributes.
3. Design unit and property tests during implementation.
4. Run static analysis and fast tests on every change.
5. Run integration and contract tests in continuous integration.
6. Deploy to a representative environment for system and acceptance tests.
7. Perform exploratory, accessibility, performance, and security testing according to risk.
8. Record defects with reproducible steps, expected behavior, actual behavior, impact, and evidence.
9. Review failures and improve both code and tests.
10. Monitor production, use safe release techniques, and feed operational learning back into development.

Tests should be versioned with the code and reviewed like production code. Parallel execution can reduce duration, but only if tests are isolated and data collisions are controlled. Test environments should avoid real personal data unless lawful, necessary, protected, and appropriately governed.

9. Illustrative Case: Testing a Digital Wallet

Background

Consider a fictional digital-wallet service used to add funds, transfer money, and display balances. It integrates with an identity provider, a payment gateway, a ledger database, and a notification service.

Problem

An earlier release showed an incorrect balance after a payment gateway timeout. The payment provider had accepted the transaction, but the wallet retried the request and recorded two credits. The existing suite had high statement coverage but few failure-path tests and no contract test for idempotency.

Analysis

The major risks were duplicate financial effects, inconsistent ledger state, incorrect authorization, and disclosure of transaction information. The team mapped tests to boundaries and failure modes rather than merely adding more happy-path examples.

Approach and solution

Unit tests were added for idempotency-key handling, balance invariants, rounding rules, and retry decisions. Property-based tests generated sequences of successful, failed, and repeated requests and checked that a single idempotency key could create at most one ledger credit. A fake payment gateway simulated timeouts and delayed responses.

Integration tests used a real database to verify transaction isolation and unique constraints. Contract tests specified that the payment provider must return an unambiguous transaction identifier and support idempotency keys. System tests exercised the deployed service with delayed gateway responses. Acceptance tests checked that users saw a pending state rather than a misleading completed state.

Mutation testing changed comparison operators and removed duplicate-request checks. Surviving mutants exposed missing boundary assertions. Exploratory testers investigated repeated clicks, device loss, clock changes, localization, accessibility, and interrupted connectivity. Static analysis identified an unsafe log statement that could have exposed payment identifiers.

Outcome

The revised evidence did not prove that all financial defects were impossible, but it substantially reduced known risk. A database uniqueness constraint provided defense in depth, monitoring detected duplicate-provider references, and a reconciliation job compared provider and ledger records. The release process also required an explicit risk-owner approval for unresolved payment defects.

Learning points

The case illustrates that coverage alone is insufficient, external contracts must include semantics, failure behavior deserves first-class testing, and quality engineering combines prevention, automation, independent exploration, and production controls.

10. Limitations, Ethics, and Contemporary Practice

Automation can create false confidence when tests assert implementation details, use unrealistic fixtures, or are routinely quarantined after becoming flaky. Artificial intelligence can assist with test generation, input synthesis, log clustering, mutation suggestions, and failure summarization, but generated tests require human review. AI-generated tests may reproduce specification errors, miss domain risks, expose confidential data, or produce assertions that merely restate implementation.

Security testing must consider authorization boundaries, injection, secrets, dependency risk, abuse cases, and privacy. Accessibility testing should combine automated checks with keyboard, screen-reader, contrast, zoom, and cognitive-usability evaluation. Regulated or safety-related systems may require documented traceability, independence of review, validated tools, controlled configuration, and legally mandated evidence. Testing personal data must follow data-minimization, retention, consent, and access-control requirements.

Contemporary quality engineering increasingly includes production verification: canary releases, feature flags, synthetic monitoring, distributed tracing, resilience experiments, and automated rollback. These practices must be carefully controlled. Deliberately injecting faults into production without authorization can harm users and violate governance obligations.

Best Practices

• Test behavior and risk, not implementation volume alone.
• Keep unit tests fast and deterministic.
• Use real integrations where fakes would hide important behavior.
• Treat contracts as versioned, semantic agreements.
• Include negative paths, boundaries, concurrency, security, accessibility, and recovery.
• Measure coverage and mutation score as evidence, never as proof.
• Investigate flaky tests immediately.
• Make failures diagnosable through clear assertions, logs, and traces.
• Review generated tests and static-analysis findings critically.
• Link release decisions to explicit residual-risk ownership.

Chapter Summary

Software testing supplies evidence about software behavior, while quality engineering integrates testing with prevention, analysis, governance, and operational learning. Unit tests provide fast feedback on focused behavior; integration tests expose boundary faults; contract tests protect independently evolving services; system and acceptance tests evaluate complete business outcomes. Property-based testing explores broad input spaces through general properties, mutation testing evaluates whether tests can detect realistic changes, and exploratory testing finds unexpected usability and interaction problems.

Coverage and static analysis are valuable but incomplete indicators. Risk-based strategies prioritize effort according to impact, likelihood, detectability, regulation, and operational context. Effective teams control test doubles, manage nondeterminism, design meaningful oracles, investigate flaky tests, and combine automated and human evidence. Ethical practice requires protection of personal data, accessibility, security, accountable release decisions, and careful governance of AI-assisted testing and production experiments.

Key Terms

Acceptance testing: Evaluation of whether a product satisfies stakeholder and business acceptance criteria.

Branch coverage: The proportion of decision outcomes exercised by tests.

Contract testing: Verification that communicating components satisfy an agreed interaction contract.

Exploratory testing: Simultaneous learning, test design, and execution guided by a testing charter.

Fake: A simplified but functioning substitute for a real dependency.

Flaky test: A test whose result changes without a relevant product change.

Integration testing: Testing of interactions among two or more components.

Mutation testing: Evaluation of test-suite sensitivity using deliberately modified program versions.

Oracle: A mechanism or rule used to determine whether an observed result is correct.

Property-based testing: Testing general properties over many generated inputs.

Quality assurance: Planned activities intended to prevent defects and improve development processes.

Quality engineering: A life-cycle approach combining prevention, testing, measurement, and operational quality improvement.

Risk-based testing: Allocation of test effort according to the importance and likelihood of failure.

Static analysis: Examination of software artifacts without executing the program.

System testing: Evaluation of a complete system or substantial deployed product slice.

Test double: A substitute used to control or observe a dependency during testing.

Test oracle problem: Difficulty of determining the correct output for an input.

Unit testing: Testing of a focused software unit in isolation or near isolation.

Review Questions

Short Answer Questions

1. Distinguish verification from validation.
2. Why can high statement coverage coexist with weak testing?
3. What is the difference between a stub, a spy, and a mock?
4. What does a surviving mutant indicate?
5. Why are contracts important in independently deployed services?

Long Answer Questions

1. Explain a layered test strategy and discuss when the conventional test pyramid may need adaptation.
2. Describe property-based testing, including generators, properties, shrinking, and oracle limitations.
3. Explain how risk-based testing can guide the selection of test levels and quality evidence.
4. Compare integration, contract, system, and acceptance testing in terms of purpose, cost, and diagnostic value.
5. Discuss the ethical and governance considerations involved in testing systems that process personal or financial data.

Analytical / Application Questions

1. Design a test strategy for an online examination system. Identify major risks and assign suitable unit, integration, contract, system, acceptance, exploratory, security, and accessibility tests.
2. A project reports 95% statement coverage but has frequent production failures involving retries and time zones. Analyze why the metric is insufficient and propose additional evidence.
3. A mutation run creates 120 mutants, of which 78 are killed and 12 are judged equivalent or excluded. Calculate the mutation score and explain what the result does and does not establish.

Multiple Choice Questions with Answers

Question 1

Which testing activity primarily verifies agreement between independently developed services?

A. Contract testing

B. Mutation testing

C. Exploratory testing

D. Statement coverage analysis

Correct Answer: A. Contract testing

Question 2

A test double that records interactions for later inspection is called a:

A. Dummy

B. Spy

C. Stub

D. Fake

Correct Answer: B. Spy

Question 3

What is the main purpose of shrinking in property-based testing?

A. To increase code coverage automatically

B. To reduce a failing generated input to a simpler counterexample

C. To replace all integration tests

D. To remove equivalent mutants

Correct Answer: B. To reduce a failing generated input to a simpler counterexample

Question 4

A mutation run creates 120 mutants; 78 are killed and 12 are excluded as equivalent. What is the mutation score?

A. 65.0%

B. 70.0%

C. 72.2%

D. 75.0%

Correct Answer: C. 72.2%

Question 5

Which statement about high statement coverage is most accurate?

A. It proves that all requirements are satisfied

B. It proves that no defects remain

C. It shows execution of statements but not necessarily adequate assertions

D. It makes exploratory testing unnecessary

Correct Answer: C. It shows execution of statements but not necessarily adequate assertions

Question 6

Which technique is most directly suited to testing values just below, at, and just above a limit?

A. Boundary-value analysis

B. Pairwise testing

C. Static analysis

D. Contract testing

Correct Answer: A. Boundary-value analysis

Question 7

Why should a payment retry test include idempotency behavior?

A. To ensure repeated requests cannot create unintended duplicate effects

B. To maximize statement coverage only

C. To avoid using a database in tests

D. To replace authentication testing

Correct Answer: A. To ensure repeated requests cannot create unintended duplicate effects

Question 8

Which practice most directly improves reproducibility of time-dependent tests?

A. Reading the system clock in every assertion

B. Injecting a controllable clock

C. Adding random delays

D. Running tests only manually

Correct Answer: B. Injecting a controllable clock

Question 9

In risk-based testing, a high-severity feature should generally receive:

A. Less testing because it is difficult

B. Testing effort proportional to its risk and required assurance

C. Only user-interface tests

D. No testing if coverage exceeds 80%

Correct Answer: B. Testing effort proportional to its risk and required assurance

Question 10

Which activity combines learning, test design, and execution around a time-boxed charter?

A. Exploratory testing

B. Static typing

C. Contract publication

D. Database migration

Correct Answer: A. Exploratory testing