Test Design Techniques and Practical Validation

Introduction and Context

Software testing is not merely the execution of test cases. It is a disciplined process of deriving test conditions, selecting representative inputs, observing system behaviour, and evaluating evidence against specified quality criteria. At postgraduate level, test design must be understood as a problem of risk-based evidence generation under limited time, budget, information, and computational resources.

A system may accept an enormous or even infinite number of possible inputs. Exhaustive testing is therefore generally infeasible. Test design techniques reduce this space while attempting to preserve fault-detection capability. The principal families are:

• Specification-based techniques, which derive tests from requirements, business rules, interfaces, and models without relying on internal implementation.
• Structure-based techniques, which derive tests from the source code, architecture, control flow, data flow, or other internal structures.
• Experience-based techniques, which use tester expertise, historical defect knowledge, intuition, and exploratory investigation.

No individual technique is sufficient. Specification-based testing may miss implementation paths that requirements do not describe; structural coverage may exercise code without checking whether the output is correct; and experience-based testing may be inconsistent between testers. Effective validation combines these techniques according to product risk, system complexity, failure consequences, and available evidence.

Learning Objectives

By the end of this chapter, the reader should be able to:

1. Explain the purpose, strengths, and limitations of specification-based, structure-based, and experience-based test design.

2. Apply equivalence partitioning and boundary value analysis to input domains.

3. Construct decision tables and state-transition tests for rule-driven and event-driven systems.

4. Select suitable combinatorial testing strategies for interacting parameters.

5. Derive and interpret control-flow coverage measures.

6. Conduct disciplined exploratory testing using explicit missions and recorded evidence.

7. Combine techniques in a risk-based validation strategy and evaluate residual risk.

1. Test Design as Evidence Engineering

A test case consists of preconditions, inputs or events, execution conditions, expected results, and an oracle. The test oracle is the mechanism used to determine whether observed behaviour is correct. It may be a formal specification, a reference implementation, an invariant, a conservation rule, a domain expert, or a metamorphic relation.

Test design involves four linked decisions:

1. What to test: features, risks, interfaces, paths, states, data combinations, or quality attributes.

2. How to select tests: partitioning, boundary analysis, model coverage, structural coverage, random sampling, or exploratory investigation.

3. What constitutes success: expected outputs, state changes, messages, logs, performance thresholds, security properties, or absence of unacceptable side effects.

4. When evidence is sufficient: a release criterion based on risk, coverage, defect trends, and unresolved uncertainty.

A high coverage percentage is not equivalent to high quality. Coverage measures describe what has been exercised, not necessarily whether the most important behaviours were correct. For example, a test may execute a branch but fail to verify the data returned to the user. Coverage should therefore be interpreted alongside requirements traceability, defect history, operational risk, and oracle quality.

Important terminology

A test condition is an item or event that can be verified. A test basis is the body of information from which tests are derived. A test suite is a collection of test cases. A test data set is the data used during execution. Test adequacy is the extent to which a test set satisfies a stated criterion. Fault detection is the discovery of an incorrect implementation; a failure is an observable deviation from expected behaviour.

2. Specification-Based Test Design

Specification-based techniques are often called black-box techniques because they do not require knowledge of internal code. They are particularly valuable for acceptance testing, system testing, API validation, and testing independently developed components.

2.1 Equivalence Partitioning

Equivalence partitioning divides an input or output domain into classes for which the system is expected to behave similarly. One or more representatives are selected from each class. The technique reduces test volume by assuming that all members of a valid class and all members of an invalid class are handled by the same logic.

Partitions should be based on externally observable rules, not merely on data type. Typical partitions include:

• valid and invalid values;
• values within and outside an allowed range;
• supported and unsupported formats;
• authenticated and unauthenticated users;
• existing, missing, and duplicated records;
• normal, exceptional, and error conditions.

Partitions should be mutually distinguishable in expected behaviour. If two values trigger different rules, they should not be placed in the same class.

Example. Suppose an examination mark must be an integer from 0 through 100, inclusive. A basic partitioning scheme is:

• less than 0: invalid;
• 0 through 100: valid;
• greater than 100: invalid;
• non-integer or non-numeric input: invalid format.

Representative tests could use −1, 50, 101, and “A”. Partitioning alone may not expose errors at 0 or 100; that is the purpose of boundary analysis.

Equivalence partitioning is an assumption about fault distribution. It is efficient when failures are likely to be associated with business categories, but weaker when faults are concentrated at particular values or arise from interactions among fields.

2.2 Boundary Value Analysis

Boundary value analysis targets values at the edges of equivalence partitions, where programming errors frequently occur. For an inclusive interval [a, b], typical numerical tests include a−1, a, a+1, b−1, b, and b+1, subject to domain constraints. For an exclusive interval (a, b), the tests must reflect the representation and smallest meaningful increment.

Worked example. A password length must be between 8 and 20 characters inclusive. Boundary tests are:

• 7: just below the lower boundary;
• 8: lower boundary;
• 9: just above the lower boundary;
• 19: just below the upper boundary;
• 20: upper boundary;
• 21: just above the upper boundary.

For a system that counts Unicode characters, the test designer must clarify whether “length” means code points, grapheme clusters, or bytes. A password containing an accented or composite character may expose an implementation mismatch. Thus, boundary analysis includes semantic boundaries, not only numeric ones.

Robust boundary value analysis includes invalid values outside the limits. Worst-case boundary analysis may combine boundary values across several variables, but the number of combinations can grow rapidly. Boundary selection should therefore be risk-based.

2.3 Decision Table Testing

Decision tables represent combinations of conditions and the actions that follow. They are useful when behaviour depends on several business rules. A table contains condition stubs, action stubs, and rules represented by columns.

Consider a loan application with conditions:

• C1: applicant is an existing customer;
• C2: credit score is at least 700;
• C3: income verification is successful.

A simplified policy may state:

• existing customer and high score: approve if income is verified;
• new customer with high score and verified income: refer for manual review;
• low score: reject;
• missing income verification: request documents.

Each logically distinct rule becomes a test condition. “Don’t care” entries are useful when a condition does not affect an action, but they must not conceal contradictions. Before testing, the table should be checked for completeness, mutually exclusive rules, and impossible combinations.

Decision tables expose omissions such as an unhandled combination of a valid customer and a missing verification document. They are especially effective for pricing, authorization, eligibility, tax, and workflow rules. Their limitation is combinatorial growth: n binary conditions can theoretically yield 2^n combinations, although constraints and rule simplification often reduce the practical set.

2.4 State-Transition Testing

State-transition testing models a system as states, events, guards, actions, and transitions. It is appropriate when the response to an event depends on history. Examples include account lockout, order processing, payment authorization, network protocols, and user sessions.

A transition can be represented as:

`current state --event [guard] / action--> next state`

For a login service, possible states include Active, Suspended, and Locked. Three consecutive failed attempts may transition Active to Locked; a successful login may reset the failure count; an administrator may transition Locked to Active. Tests should cover:

• valid transitions;
• invalid events in each state;
• guards at their limits;
• repeated events;
• reset and recovery paths;
• sequences that distinguish history-dependent behaviour.

Coverage may be defined at the level of states, transitions, transition pairs, or selected paths. Transition-pair coverage is stronger than merely visiting each state because it tests interactions between successive transitions. State models must also address concurrency, timeouts, persistence, and recovery after system failure where these affect observable behaviour.

2.5 Use-Case and Scenario Testing

Use cases describe interactions among actors and a system. Tests should include the main success scenario, alternative flows, exception flows, authorization failures, cancellation, timeout, and recovery. A scenario test is valuable because it validates business intent across multiple components rather than checking isolated functions.

However, scenario tests can be expensive and may provide weak localization when they fail. They should be complemented by lower-level tests that isolate rules and boundaries.

2.6 Combinatorial Testing

Many failures arise from interactions among configuration parameters, such as browser, operating system, language, payment method, and user role. Testing every combination may be impossible. Combinatorial testing selects a covering array so that every combination of values for each selected parameter subset appears at least once.

In t-wise testing, every combination of values for any t parameters is covered. Pairwise testing is 2-wise testing; it covers all value pairs but not necessarily all triples. If there are five binary parameters, exhaustive testing requires 2^5 = 32 combinations. A pairwise suite may require substantially fewer tests, but its exact size depends on the number of parameters, value counts, constraints, and the chosen construction algorithm.

The strength t should be chosen from failure risk and interaction knowledge. Pairwise testing is not a guarantee against higher-order faults. Constraints must be encoded: for example, a mobile operating system may not support a desktop browser. Invalid combinations should not be generated merely to satisfy mathematical coverage.

3. Structure-Based Test Design

Structure-based techniques derive tests from implementation or design structures. They support unit, integration, and low-level verification and are valuable for detecting unreachable code, missing branches, faulty conditions, and data-use errors.

3.1 Control-Flow Graphs

A control-flow graph (CFG) represents a program using nodes for statements or basic blocks and directed edges for possible transfers of control. A test execution traces a path through the graph.

Common coverage criteria are:

• Statement coverage: every executable statement is executed at least once.
• Branch or decision coverage: every outcome of each decision is executed at least once.
• Condition coverage: each atomic Boolean condition takes both true and false outcomes, where feasible.
• Condition/decision coverage: decision outcomes and atomic condition outcomes are both covered.
• Modified condition/decision coverage (MC/DC): each condition is shown to independently affect the decision outcome, subject to the formal rules adopted by the organization or standard.
• Path coverage: selected paths are executed; complete path coverage is generally infeasible in programs with loops.

Decision coverage is not always equivalent to condition coverage. For `if (A && B)`, tests that make the decision true and false may leave B always true, so B has not been independently exercised.

3.2 Cyclomatic Complexity

For a connected control-flow graph, cyclomatic complexity is commonly calculated as:

`V(G) = E − N + 2`

where E is the number of directed edges and N is the number of nodes. If the graph has P connected components, the generalized form is:

`V(G) = E − N + 2P`.

For a structured, single-entry/single-exit routine, it is also commonly equal to the number of predicate nodes plus one. The measure estimates the number of linearly independent paths and can guide the minimum number of basis-path tests. It is not a direct measure of defect count, maintainability, or test quality.

Worked example. Suppose a connected CFG has 10 nodes and 12 edges. Then:

`V(G) = 12 − 10 + 2 = 4`.

A basis set may contain four linearly independent paths, but loops, infeasible paths, data dependencies, and environmental conditions may require additional tests. Conversely, four tests achieving basis-path coverage do not prove that all requirements are correct.

3.3 Data-Flow Testing

Data-flow testing examines the points where variables are defined, used, and possibly killed or invalidated. A definition-use pair is a path from a definition to a use without an intervening redefinition. Criteria include all-definitions and all-uses coverage. Such tests can expose variables used before initialization, values overwritten prematurely, and stale data after an error path.

Data-flow techniques are useful in safety-critical, financial, and stateful applications, but they may be costly for alias-rich languages, concurrent systems, and dynamic data structures.

4. Experience-Based and Exploratory Testing

Experience-based techniques use knowledge that may not be fully represented in specifications or models. They include error guessing, checklist-based testing, defect-based testing, and exploratory testing.

Error guessing asks what a competent developer or tester might get wrong: null values, duplicated submissions, clock changes, rounding, stale sessions, large files, interrupted network connections, and permission changes during an operation. Historical defect data should inform these guesses rather than relying only on intuition.

Exploratory testing is simultaneous learning, test design, execution, and interpretation. It is not unscripted random clicking. A disciplined exploratory session specifies:

• a mission or charter;
• timebox and scope;
• risks and hypotheses;
• test data and tools;
• observations and evidence;
• defects, questions, and follow-up tasks.

A tester may use tours such as the business-process tour, claims tour, configuration tour, or failure tour. Session-based test management can record mission, duration, interruptions, test ideas, defects, and coverage notes.

Exploration is particularly valuable when requirements are incomplete, interfaces are changing, or risks are emergent. Its limitations include reproducibility challenges, dependence on tester skill, and difficulty demonstrating systematic coverage. These limitations are reduced by recording actions, environment, data, screen captures, logs, and exact reproduction steps.

5. Integrating Techniques into a Validation Strategy

A mature test strategy maps risks to techniques rather than selecting a single universal method. A possible mapping is:

| Risk or question | Suitable techniques |

|---|---|

| Are input limits enforced? | Equivalence partitioning, boundary analysis |

| Are policy combinations complete? | Decision tables, pairwise or t-wise testing |

| Does history affect behaviour correctly? | State-transition and scenario testing |

| Are all important implementation decisions exercised? | Branch, condition, MC/DC coverage |

| Are hidden usability and resilience problems present? | Exploratory and error-guessing techniques |

| Are configuration interactions safe? | Combinatorial testing |

Traceability should connect requirements to test conditions, test cases, results, defects, and risk decisions. Automation is appropriate for stable, repeatable checks such as regression, boundary suites, API contracts, and coverage measurement. Human investigation remains essential for usability, ambiguity, emergent behaviour, and oracle construction.

Test prioritization should consider business impact, likelihood, detectability, change volatility, technical complexity, and regulatory exposure. A low-frequency failure may deserve high priority if its consequence is catastrophic. Conversely, a frequently executed but low-impact cosmetic issue may receive lower release priority, subject to contractual and accessibility obligations.

6. Detailed Applied Case: Illustrative Digital Wallet Validation

Background

The following is an illustrative case. A digital wallet permits users to add funds, transfer money, and withdraw funds. It supports three account states: Active, Suspended, and Closed. Transfers require authentication, sufficient balance, a valid beneficiary, and compliance screening.

Problem

An earlier release produced three incidents: transfers were accepted at an exact daily limit when policy required rejection, suspended accounts could initiate transfers through an API, and a timeout after debit occasionally led users to retry and create duplicate transfers.

Analysis

The team identified several interacting risks:

1. Numeric boundaries: daily transfer limit and available balance.

2. Decision rules: account state, authentication, screening, beneficiary status, and amount.

3. State transitions: Pending, Completed, Failed, and Reversed transfer states.

4. Idempotency: repeated requests with the same client transaction identifier.

5. Interface differences: mobile application and public API.

Approach

First, equivalence partitions were created for transfer amount: zero or negative, within limit, exactly at limit, above limit, and invalid format. Boundary tests covered limit−smallest unit, limit, and limit+smallest unit. The smallest unit was explicitly defined as one cent rather than an arbitrary floating-point increment.

Second, a decision table combined account eligibility, authentication, beneficiary validity, screening result, and balance sufficiency. Rules were reviewed with compliance staff to remove impossible combinations and identify an omitted rule for suspended accounts.

Third, a state model represented Pending, Completed, Failed, and Reversed. Tests included network timeout after debit, retry with the same identifier, retry with a new identifier, and reconciliation after service restart.

Fourth, pairwise testing was applied to supported device, operating system, language, and payment rail, with constraints excluding unsupported combinations. High-risk payment rails received additional three-wise tests.

Fifth, structural tests targeted authorization branches, exception handling, rollback logic, and idempotency checks. Branch coverage was measured, but the release gate also required evidence that every high-risk rule had a passing test and every critical defect had a regression test.

Finally, exploratory sessions used the charter “attempt to create duplicate or unauthorized transfers under interruption and changing account status.” Testers inspected client responses, server logs, ledger entries, audit events, and notifications.

Outcome

The combined approach detected the exact-limit comparison defect, the API authorization omission, and a race condition in idempotency handling. It also revealed that the user interface displayed “transfer failed” while reconciliation later completed the transfer. The team changed the contract to represent an indeterminate outcome and added a status-query operation.

Learning points

The case demonstrates that no single technique would have found all defects. Boundary analysis detected the limit problem, decision tables exposed authorization gaps, state testing addressed history and recovery, structural testing targeted implementation branches, and exploration revealed a cross-system race and misleading user communication. The strongest evidence came from combining model coverage, code coverage, business assertions, and operational observation.

7. Limitations, Governance, and Contemporary Practice

Test design is affected by incomplete requirements, changing environments, nondeterministic services, test-data privacy, third-party dependencies, and limited observability. Tests can also create false confidence when data is unrealistic, mocks do not reflect production behaviour, or the oracle is weak.

Sensitive personal and financial data should not be copied into test environments without authorization. Synthetic or properly de-identified data, access controls, retention limits, and audit trails are required. Security testing must avoid unauthorized scanning or destructive actions. Accessibility and inclusive design should be represented in partitions and scenarios, including keyboard access, screen readers, localization, and assistive technologies.

Regulated or safety-related systems may require prescribed traceability, independence of testing, evidence retention, and coverage criteria. The applicable standard depends on the domain; coverage metrics must not be selected without understanding the governing assurance objective.

Contemporary teams increasingly use model-based testing, property-based testing, mutation testing, service virtualization, observability platforms, and AI-assisted test generation. AI can propose partitions, generate data, summarize failures, and identify likely edge cases. However, generated tests may reproduce specification errors, create invalid combinations, or assert the implementation rather than the requirement. Human review, provenance, privacy controls, deterministic execution, and independent oracles remain necessary. Mutation testing can evaluate whether a suite detects seeded changes, but mutation scores are indicators of test sensitivity, not proof of correctness.

Best practice is to begin with risk and domain rules, make assumptions explicit, use multiple complementary techniques, automate stable checks, preserve reproducible evidence, review coverage gaps, and reassess tests whenever requirements, architecture, data, or operational threats change.

Chapter Summary

Test design is the systematic selection of tests that produce meaningful evidence about software quality. Specification-based techniques include equivalence partitioning, boundary value analysis, decision tables, state-transition testing, use-case testing, and combinatorial testing. They derive tests from externally visible behaviour and models.

Structure-based techniques use control flow and data flow to exercise implementation structures. Statement, branch, condition, MC/DC, path, and data-flow criteria answer different questions and should not be treated as interchangeable. Cyclomatic complexity can estimate independent paths using `V(G) = E − N + 2` for a connected graph, but it is not a complete quality measure.

Experience-based and exploratory testing address ambiguity, emergent risk, and defects omitted from formal models. Their value increases when missions, observations, evidence, and reproduction steps are recorded. A professional validation strategy combines techniques according to risk, traceability, regulatory needs, system architecture, and available oracles. Coverage is evidence of exercise, not proof of correctness. Ethical data handling, accessibility, security authorization, and human oversight of AI-assisted testing are integral to trustworthy validation.

Key Terms

Equivalence partitioning: Dividing an input or output domain into classes expected to behave similarly.

Boundary value analysis: Testing values at and immediately around domain boundaries.

Decision table: A tabular representation of condition combinations and resulting actions.

State-transition testing: Testing behaviour represented as states, events, guards, and transitions.

Combinatorial testing: Selecting tests to cover combinations of parameter values efficiently.

T-wise testing: Testing that covers every combination of values for each selected set of t parameters.

Control-flow graph: A graph representing possible execution transfers in a program.

Statement coverage: The proportion or set of executable statements exercised by tests.

Branch coverage: Exercise of each outcome of each decision.

Condition coverage: Exercise of true and false outcomes of atomic Boolean conditions.

MC/DC: Coverage demonstrating that each condition independently affects a decision outcome.

Cyclomatic complexity: A control-flow measure related to the number of linearly independent paths.

Data-flow testing: Testing definitions, uses, and lifetimes of data values.

Test oracle: A mechanism for deciding whether observed behaviour is correct.

Exploratory testing: Testing in which learning, test design, execution, and interpretation occur together.

Error guessing: Using experience and defect knowledge to anticipate likely failures.

Test adequacy: The degree to which a test set satisfies a stated criterion.

Traceability: Links among requirements, risks, tests, results, and defects.

Mutation testing: Evaluating test effectiveness by checking whether tests detect seeded code changes.

Residual risk: Risk remaining after planned validation and controls.

Review Questions

Short Answer Questions

1. What is the purpose of equivalence partitioning?

2. How does boundary value analysis complement equivalence partitioning?

3. Distinguish statement coverage from branch coverage.

4. What is a test oracle?

5. Why must constraints be considered in combinatorial testing?

Long Answer Questions

1. Explain how to construct and validate a decision table for a rule-intensive application.

2. Describe state-transition testing and discuss state, transition, transition-pair, and path coverage.

3. Compare specification-based, structure-based, and experience-based test design in terms of strengths and limitations.

4. Explain cyclomatic complexity, derive its graph formula, and discuss its appropriate use and limitations.

5. Develop a risk-based validation strategy for a regulated financial service using at least four test design techniques.

Analytical / Application Questions

1. A booking system accepts dates from today through 180 days from today inclusive. Design equivalence partitions and boundary tests, and identify assumptions that must be clarified.

2. An authorization decision depends on user role, account status, multi-factor authentication, and transaction risk. Design a decision-table strategy and explain how you would reduce redundant rules.

3. A payment request may be Pending, Completed, Failed, or Reversed, and network timeouts may occur after debit. Design a state-transition and exploratory testing approach for detecting duplicate charges and misleading status messages.

Multiple Choice Questions with Answers

Question 1

Which technique is most directly suited to testing values at the edges of an allowed input range?

A. Boundary value analysis

B. Mutation testing

C. Data-flow testing

D. Use-case modeling

Correct Answer: A. Boundary value analysis

Question 2

For an inclusive interval from 8 through 20, which value is just below the lower boundary?

A. 8

B. 9

C. 7

D. 20

Correct Answer: C. 7

Question 3

What is the primary purpose of a decision table?

A. Measure source-code execution time

B. Represent condition combinations and resulting actions

C. Calculate memory consumption

D. Replace all exploratory testing

Correct Answer: B. Represent condition combinations and resulting actions

Question 4

Which test design technique is most appropriate when behaviour depends on the sequence of prior events?

A. State-transition testing

B. Statement coverage

C. Equivalence partitioning only

D. Static formatting analysis

Correct Answer: A. State-transition testing

Question 5

In t-wise combinatorial testing, what does t represent?

A. The number of test environments

B. The number of defects expected

C. The size of parameter subsets whose value combinations are covered

D. The execution time limit

Correct Answer: C. The size of parameter subsets whose value combinations are covered

Question 6

A connected control-flow graph has 10 nodes and 12 edges. What is its cyclomatic complexity?

A. 2

B. 3

C. 4

D. 5

Correct Answer: C. 4

Question 7

Which coverage criterion requires each outcome of every decision to be exercised?

A. Statement coverage

B. Branch coverage

C. Function coverage

D. Requirement coverage

Correct Answer: B. Branch coverage

Question 8

What distinguishes disciplined exploratory testing from random clicking?

A. It never records results

B. It uses a mission, timebox, observations, and evidence

C. It requires complete path coverage

D. It excludes tester judgement

Correct Answer: B. It uses a mission, timebox, observations, and evidence

Question 9

Which is a principal limitation of high code coverage?

A. It proves every requirement is correct

B. It guarantees absence of defects

C. It may exercise code without validating the correctness of outputs

D. It cannot be measured automatically

Correct Answer: C. It may exercise code without validating the correctness of outputs

Question 10

Why should AI-generated test cases receive human review?

A. AI cannot produce any test data

B. Generated tests may encode specification errors or invalid combinations

C. Human review makes automation impossible

D. AI-generated tests always have duplicate options

Correct Answer: B. Generated tests may encode specification errors or invalid combinations