Testing, Verification and Resource Analytics

Introduction

Quantum programs combine ordinary classical software with probabilistic physical or simulated processes. A circuit may be syntactically valid yet produce an incorrect distribution, use excessive resources, or fail on a particular backend. Testing quantum software therefore requires more than checking whether a program runs without error. It requires evidence that the circuit implements its intended transformation, that measured results are statistically consistent with expectations, and that resource use is acceptable.

This chapter presents a practical verification workflow for diploma-level quantum programmers. It covers deterministic test cases, statistical assertions, invariants, property-based testing, cross-simulator comparison, debugging, uncertainty quantification, resource estimation, benchmarking, and reproducible technical reporting.

Learning Objectives

By the end of this chapter, a learner should be able to:

• distinguish testing, verification, validation, and benchmarking;
• construct deterministic and statistical tests for quantum circuits;
• use invariants and properties to detect implementation errors;
• compare state-vector, shot-based, noisy, and hardware results;
• quantify sampling uncertainty using confidence intervals;
• diagnose common circuit and measurement bugs;
• estimate qubits, gates, depth, shots, runtime, and cost;
• design reproducible experiments and technical reports.

1. Testing, Verification and Validation

Testing executes a program on selected inputs and checks its outputs. Verification asks whether the implementation conforms to a specified circuit, algorithm, or mathematical property. Validation asks whether the specification solves the intended practical problem. A circuit can be verified correctly but still be unsuitable for an application if the original specification was wrong.

Quantum results are often distributions rather than single values. If a circuit is expected to produce probabilities p(x), a test may compare observed frequencies with those probabilities. The observed frequency for outcome x is

where n_x is the number of shots producing x and N is the total number of shots. The quantity f(x) is an estimator of p(x), not usually an exact replacement for it.

A useful testing hierarchy is:

1. Syntax and construction tests: gates, registers, measurements, and parameters are created correctly.
2. Deterministic tests: circuits whose outcomes are known with certainty are checked exactly.
3. Statistical tests: sampled frequencies are compared with expected probabilities.
4. Metamorphic and property-based tests: general relationships are tested across many generated inputs.
5. Integration tests: the complete workflow, including transpilation and execution, is checked.
6. Performance tests: time, memory, depth, shots, and cost are measured.

A passing test provides evidence, not proof of correctness. Test quality depends on input coverage, oracle quality, simulator assumptions, and the realism of the noise model.

2. Deterministic Quantum Test Cases

Deterministic circuits are especially valuable because their expected measurement is known exactly. Examples include:

• applying X to |0⟩ and expecting 1;
• applying X twice and expecting 0;
• applying a reversible circuit followed by its inverse;
• preparing a computational-basis state and measuring in the same basis;
• applying a circuit to a basis state when its truth table is known.

For an ideal deterministic circuit, all N shots should produce the same bit string. If even one different result occurs on an ideal noiseless simulator, there is likely a construction, measurement, or interpretation error. On real hardware, a small number of deviations may be caused by noise.

A basic deterministic test should record:

• input state;
• circuit operations;
• measurement basis;
• expected bit string;
• simulator or backend;
• number of shots;
• observed counts;
• pass/fail rule.

For example, the circuit X|0⟩ produces |1⟩. With 1,000 ideal shots, the expected count is {1: 1000}. If the software returns {0: 1000}, possible causes include reversed gate order, incorrect register indexing, or a measurement-label convention.

Truth-table testing

Small reversible circuits can be tested against a complete truth table. For each computational-basis input |x⟩, execute the circuit and compare the output with the specification. This is practical for one- and two-qubit components, but becomes expensive because n qubits have 2ⁿ basis states.

Inverse-circuit testing

If U† is implemented as the inverse of U, then

where |ψ⟩ is any state in the tested input set. This identity assumes ideal gates and a correctly implemented inverse. In a simulator with noise, the result may only be approximately restored.

3. Statistical Assertions and Uncertainty

Most quantum algorithms produce random samples. A statistical assertion must specify both the expected result and an acceptable tolerance. For a binary event with probability p, the number of occurrences X in N independent shots follows a binomial distribution:

The estimated probability is p̂ = X/N. Its standard error is

When p is unknown, it is common to estimate the standard error using p̂:

These approximations are most reliable when Np̂ and N(1 − p̂) are not small. For rare events or small samples, Wilson or exact binomial intervals are preferable to a simple normal approximation.

An approximate 95% normal confidence interval is

The interval describes sampling uncertainty under the assumed model. It does not include systematic errors, calibration drift, or an incorrect circuit.

Worked example: a Hadamard measurement

A Hadamard gate applied to |0⟩ ideally gives equal probabilities for 0 and 1. Suppose N = 1,000 shots produce X = 523 ones. Then

p̂ = 523/1000 = 0.523

and

SE ≈ √(0.523 × 0.477/1000) ≈ 0.0158.

The approximate 95% interval is

0.523 ± 1.96 × 0.0158 ≈ 0.523 ± 0.031,

or approximately [0.492, 0.554]. The ideal value 0.5 lies inside this interval, so the result is compatible with ordinary shot noise.

For multiple outcomes, a test may compare the complete distribution using total variation distance:

where P is the reference distribution and Q is the observed or candidate distribution. DTV ranges from 0, identical distributions, to 1, maximally distinguishable distributions. It is useful for an overall comparison but does not identify which outcome caused the discrepancy.

Other measures include the squared error, chi-square statistics, and fidelity-like quantities. A chi-square test requires care: expected counts should generally not be too small, and multiple tests require adjustment to control false discoveries.

4. Invariants and Metamorphic Testing

An invariant is a property that should remain true throughout a computation or under a defined transformation. Invariants are powerful when an exact expected output is difficult to calculate.

Examples include:

• probabilities sum to one;
• a density matrix has trace one;
• a unitary circuit preserves state-vector norm;
• a permutation circuit maps basis states to basis states;
• a parity-preserving circuit preserves parity;
• probabilities are unchanged by a global phase;
• applying a circuit and its inverse restores the input.

For a normalized state |ψ⟩, the norm condition is

For a density matrix ρ, valid normalization requires

and physical validity additionally requires ρ to be positive semidefinite and Hermitian.

Metamorphic testing tests relations between executions rather than one fixed answer. If a circuit C is expected to be invariant under a qubit relabelling, execute both versions and compare the relabelled outputs. If a parameterized circuit has a known periodicity, compare results at θ and θ + 2π where appropriate. The relation must follow from the circuit’s mathematics; it should not be assumed merely because it seems plausible.

5. Property-Based Testing

Property-based testing generates many inputs and checks general properties. In quantum programming, generated inputs may include:

• random computational-basis states;
• random normalized state vectors for small systems;
• random parameter values within specified ranges;
• random gate sequences from a permitted gate set;
• random qubit permutations;
• random circuits with controlled depth.

A useful property is inverse consistency: generate a circuit U, construct U†, and test that U†U restores each generated input within a numerical tolerance. Another is equivalence after decomposition: compare a high-level gate with its low-level decomposition.

Numerical comparisons should use a tolerance because floating-point arithmetic introduces small errors. For vectors a and b, a common criterion is

where ||·||₂ is the Euclidean norm and ε is a documented tolerance. The tolerance must be large enough for numerical round-off but small enough to detect meaningful errors. It should not be selected after inspecting the test results.

Property-based testing requires shrinking: when a generated failing circuit is found, reduce it to a smaller circuit that still fails. A two-gate counterexample is easier to diagnose than a failure involving 100 random gates.

6. Cross-Simulator and Cross-Backend Comparison

Different execution models reveal different defects:

A robust comparison uses the same circuit, parameters, basis, shot count where relevant, and bit-string convention. Comparisons should be made at equivalent stages. A pre-transpilation circuit should not be compared directly with a heavily optimized hardware circuit without recording the transformation.

Possible comparison metrics include:

• exact amplitude difference for ideal state vectors;
• fidelity for pure states, F = |⟨ψ|φ⟩|²;
• total variation distance for distributions;
• expectation-value difference;
• success probability for the algorithm’s target event.

For pure normalized states, fidelity lies between 0 and 1. A global phase does not change physical predictions: |ψ⟩ and e^{iφ}|ψ⟩ produce the same measurement probabilities. Therefore, direct component-by-component comparison may incorrectly report a difference unless global phase is accounted for.

7. Debugging Quantum Circuits

Quantum debugging is difficult because measurement disturbs a state and because intermediate amplitudes are not directly observable on hardware. A disciplined strategy is therefore essential.

7.1 Reduce the failing example

First reduce the number of qubits, gates, parameters, and shots while preserving the failure. Determine whether the error appears in circuit construction, simulation, measurement, transpilation, or reporting.

7.2 Inspect state preparation and ordering

Common errors include:

• confusing qubit order with displayed bit-string order;
• measuring the wrong classical bit;
• applying a controlled gate to reversed control and target;
• using degrees where a library expects radians;
• omitting a reset between experiments;
• reusing stale parameters;
• interpreting little-endian output as big-endian output.

A basis-state test often exposes ordering errors immediately.

7.3 Check intermediate checkpoints

For an ideal simulator, inspect amplitudes or probabilities after logical blocks. For hardware, use equivalent diagnostic circuits, repeated measurements, calibration data, or tomography only when justified by the cost. Excessive instrumentation can change circuit depth and create a different experiment.

7.4 Separate algorithmic and hardware errors

Run the circuit in at least three forms when possible:

1. ideal state-vector simulation;
2. ideal shot-based simulation;
3. noisy simulation or hardware execution.

If the state-vector test fails, the algorithm or implementation is wrong. If only shot-based execution differs, investigate sampling or measurement handling. If only noisy or hardware execution differs, investigate noise, compilation, connectivity, and calibration.

8. Resource Analytics

Resource analytics quantifies what a circuit requires. Important quantities include logical qubits, physical qubits, gate counts, depth, shots, memory, runtime, and monetary cost.

Qubit count

The logical qubit count is the number of qubits specified by the algorithm. The physical qubit count includes qubits used by the device after mapping, routing, ancillas, and error-correction overhead. These values may differ substantially.

For a state-vector simulator using complex amplitudes, the number of amplitudes is 2ⁿ for n qubits. If each complex amplitude uses b bytes, the raw amplitude memory is

This excludes workspace, metadata, multiple buffers, and simulator overhead. With 16 bytes per complex double-precision amplitude, 30 qubits require approximately 16 × 2³⁰ bytes, or 16 GiB, before overhead.

Gate count and depth

Gate count is the number of operations, while circuit depth is the number of sequential layers after parallel operations are grouped. Depth depends on scheduling and hardware constraints. Two circuits with equal gate counts can have different depths and runtimes.

Report gate counts by type where useful: one-qubit gates, two-qubit gates, measurements, resets, barriers, and non-native operations. Two-qubit gates often dominate error and execution time on current hardware.

Shot requirements

For estimating a binary probability with approximate standard error δ, the worst-case binomial variance occurs at p = 0.5. Thus

To target an approximate standard error no greater than δ, a conservative requirement is

This is a sampling estimate, not a guarantee against hardware bias. For δ = 0.01, the bound gives N ≥ 2,500 shots.

Runtime and cost

A transparent runtime model separates queue, compilation, execution, and post-processing:

If a provider charges price c per shot and the experiment uses N shots, a simplified execution cost is

If pricing is per circuit, per runtime unit, or includes minimum charges, the provider’s billing specification must replace this simplified model. Estimates must state whether queue time and failed jobs are included.

9. Benchmarking Methodology

A benchmark is meaningful only when its conditions are controlled. Record:

• software and library versions;
• hardware or simulator configuration;
• circuit source and commit identifier;
• qubit count, gate counts, and depth;
• optimization and transpilation settings;
• shot count and random seeds;
• noise model and calibration timestamp;
• wall-clock and device execution time;
• memory and cost;
• accuracy metric and pass criteria.

Use repeated runs to distinguish variation from a single unusually fast or slow execution. Report median and spread when runtimes are noisy. Do not compare a simulator’s wall-clock time with hardware queue time as though they were the same performance measure.

A useful benchmark table contains one row per configuration and columns for circuit size, depth, shots, runtime, memory, accuracy, and cost. Graphs should include units and identify whether values are measured, simulated, or estimated.

10. Reproducible Technical Reports

A reproducible report allows another person to recreate the result. It should contain:

1. Purpose and hypothesis: what is being tested and why.
2. Specification: expected circuit behaviour and acceptance criteria.
3. Environment: versions, operating system, backend, and configuration.
4. Method: circuit construction, shots, seeds, noise assumptions, and metrics.
5. Results: counts, probabilities, intervals, resource data, and failures.
6. Analysis: interpretation, uncertainty, anomalies, and limitations.
7. Artifacts: source code, circuit diagrams, configuration files, and raw data.
8. Conclusion: whether the stated criteria were met.

Random seeds improve repeatability for simulators but do not make hardware outcomes deterministic. A seed should therefore be recorded as an experimental parameter, not presented as a substitute for statistical analysis.

11. Illustrative Case: Verifying a Two-Qubit Bell-State Routine

Background

A development team implements a routine intended to prepare the Bell state

The routine applies H to qubit 0 and then CNOT with qubit 0 as control and qubit 1 as target, followed by measurement.

Problem

An initial hardware run reports approximately 50% 00 and 50% 01, rather than 00 and 11. The team must determine whether the circuit, measurement mapping, or hardware caused the discrepancy.

Analysis and approach

First, a state-vector simulation checks the ideal state. The amplitudes should be 1/√2 for |00⟩ and |11⟩, with zero amplitude for |01⟩ and |10⟩. Second, an ideal shot-based simulation checks that only 00 and 11 occur. Third, the team verifies qubit and classical-bit ordering by testing X on each qubit separately.

The inverse test applies the preparation circuit followed by its inverse to several basis states. The original states must be restored in the ideal simulator. The team then runs a noisy simulation using measured two-qubit error rates and compares its distribution with hardware data using total variation distance.

Outcome

The ideal state-vector and shot-based tests pass. The single-qubit ordering tests reveal that the displayed bit string reverses the physical qubit order. After correcting the interpretation, the hardware data is approximately 50% 00 and 50% 11, although with a small population in 01 and 10 caused by noise. The report distinguishes the original software interpretation error from the remaining device error.

Learning points

• A distribution failure may be a representation error rather than a quantum-gate error.
• Small deterministic tests should precede expensive hardware experiments.
• Cross-model comparison separates algorithmic, statistical, and hardware effects.
• Reports should preserve raw counts rather than only rounded percentages.

12. Limitations, Governance and Best Practices

Testing cannot prove correctness for arbitrary large quantum programs through finite examples. State-vector simulation is limited by exponential memory, while hardware testing is affected by drift, noise, access constraints, and provider changes. Statistical tests can produce false positives or false negatives, especially when many assertions are run without correction.

In professional settings, preserve audit trails for circuit versions, backend identities, calibration times, and data transformations. Do not report simulator results as hardware evidence. Cost estimates should state pricing assumptions, and sensitive application data should not be placed in external quantum services without appropriate governance and contractual review.

Recommended practice is to maintain a layered test suite: deterministic unit tests, property-based tests, statistical distribution tests, cross-simulator tests, and a small set of hardware acceptance tests. Use explicit tolerances, independent reference implementations where feasible, version-controlled artifacts, and automated continuous integration for small circuits. Treat resource analytics as part of design rather than an activity performed only after implementation.

Chapter Summary

Quantum software quality requires more than successful execution. Deterministic tests validate simple known behaviours; statistical assertions address sampled distributions; invariants and metamorphic properties test general relationships; and property-based methods explore broad input spaces. Cross-simulator comparison helps separate algorithmic defects from sampling, noise, compilation, and hardware effects.

Probability estimates carry sampling uncertainty, which should be reported with suitable confidence intervals or error bounds. Debugging should reduce failures, inspect ordering and measurement conventions, and compare ideal, noisy, and hardware execution. Resource analytics records logical and physical qubits, gate counts, depth, memory, shots, runtime, and cost. Reproducible reports document environments, assumptions, raw results, metrics, and limitations.

Key Terms

Testing: Executing software and checking its outputs against criteria.

Verification: Establishing conformity with a specified design or property.

Validation: Establishing that a system meets its intended practical purpose.

Deterministic test: A test with an exactly known ideal outcome.

Statistical assertion: A test that accepts results within a probability-based tolerance.

Shot: One execution and measurement of a quantum circuit.

Invariant: A property that remains true under specified operations.

Metamorphic testing: Testing relationships between related executions.

Property-based testing: Testing general properties over generated inputs.

State-vector simulator: A simulator representing complex amplitudes of a pure state.

Density matrix: A representation supporting mixed states and noise models.

Total variation distance: Half the sum of absolute differences between two distributions.

Fidelity: A measure of similarity between quantum states.

Logical qubit: A qubit specified by the algorithm.

Physical qubit: A device qubit used to execute a mapped circuit.

Circuit depth: Number of sequential operation layers.

Sampling uncertainty: Variation caused by a finite number of shots.

Transpilation: Transformation of a circuit for a target device or gate set.

Reproducibility: Ability to recreate a result using documented methods and artifacts.

Review Questions

Short Answer Questions

1. Distinguish testing, verification, and validation in quantum software.
2. Why are deterministic circuits useful in a quantum test suite?
3. What information should be recorded for a statistical assertion?
4. Explain the difference between logical and physical qubits.
5. Why can bit-string ordering produce an apparent Bell-state failure?

Long Answer Questions

1. Explain how inverse-circuit testing and invariants can be combined to verify a quantum component.
2. Describe a cross-simulator workflow for separating algorithmic errors from hardware noise.
3. Explain the assumptions and limitations of the approximate binomial confidence interval.
4. Discuss the difference between gate count, circuit depth, and runtime.
5. Design the contents of a reproducible technical report for a hardware experiment.

Analytical / Application Questions

1. A binary experiment produces 523 successes in 1,000 shots. Calculate the estimated probability and an approximate 95% confidence interval, and interpret the result against an expected probability of 0.5.
2. A state-vector simulator uses 16 bytes per complex amplitude. Estimate raw amplitude memory for 28 qubits and explain why actual memory may be larger.
3. A Bell-state circuit passes an ideal state-vector test but fails on hardware. Propose a staged diagnostic plan that distinguishes ordering, sampling, compilation, and physical noise.

Multiple Choice Questions with Answers

Question 1

Which test is most appropriate for detecting a reversed measurement-bit convention?

A. A large random circuit benchmark

B. A simple basis-state test on individual qubits

C. A cost estimate using provider pricing

D. A confidence interval with no measurements

Correct Answer: B. A simple basis-state test on individual qubits

Question 2

For N shots and X occurrences of a binary event, which quantity estimates its probability?

A. N/X

B. X + N

C. X/N

D. X − N

Correct Answer: C. X/N

Question 3

What does total variation distance equal for two identical probability distributions?

A. 0

B. 0.5

C. 1

D. It is undefined

Correct Answer: A. 0

Question 4

Which expression represents inverse consistency in an ideal circuit?

A. U + U† = I

B. U†U|ψ⟩ = |ψ⟩

C. U|ψ⟩ = 0

D. U† = U for every U

Correct Answer: B. U†U|ψ⟩ = |ψ⟩

Question 5

What is the raw number of amplitudes in an n-qubit state vector?

A. n²

B. 2n

C. n/2

D. 2ⁿ

Correct Answer: D. 2ⁿ

Question 6

Which execution model directly represents mixed states and many noise channels?

A. Density-matrix simulation

B. A truth table only

C. A syntax checker

D. A gate-count report

Correct Answer: A. Density-matrix simulation

Question 7

Using the conservative bound N ≥ 1/(4δ²), how many shots are required for δ = 0.01?

A. 25

B. 250

C. 2,500

D. 25,000

Correct Answer: C. 2,500

Question 8

Which quantity counts sequential layers rather than total operations?

A. Shot count

B. Circuit depth

C. Logical-qubit count

D. Fidelity

Correct Answer: B. Circuit depth

Question 9

Why should a hardware result not be reported as deterministic merely because a simulator used a fixed seed?

A. Seeds eliminate all hardware noise

B. Hardware outcomes remain probabilistic and affected by device conditions

C. A fixed seed changes the number of qubits

D. Hardware cannot perform measurements

Correct Answer: B. Hardware outcomes remain probabilistic and affected by device conditions

Question 10

Which item is essential for a reproducible technical report?

A. Only the final rounded percentages

B. An undocumented backend chosen after execution

C. Software versions, circuit source, shots, and configuration

D. A claim that all noise is negligible

Correct Answer: C. Software versions, circuit source, shots, and configuration