Verification, Validation and Performance Evaluation

Introduction

A major project is not complete when a circuit has been assembled, software has been compiled, or a prototype appears to operate. It is complete only when credible evidence shows that the implementation satisfies its stated requirements, solves the intended stakeholder problem, performs acceptably under realistic conditions, and has known limitations. Verification, validation and performance evaluation provide the evidence for that conclusion.

These activities are closely related but distinct. Verification asks, “Was the system built according to its specifications?” Validation asks, “Was the correct system built for its intended users and context?” Performance evaluation measures how well the system performs, while testing is the planned execution of experiments used to obtain evidence. A technically impressive project may fail validation if it addresses the wrong need; conversely, a useful concept may fail verification because its implementation does not meet safety, accuracy or interface requirements.

For electrical engineering projects, evaluation may involve electrical measurements, control-system responses, embedded-software behaviour, communication reliability, energy consumption, thermal performance, electromagnetic compatibility, safety and user acceptance. A rigorous evaluation therefore combines quantitative measurements with justified qualitative observations.

Learning Objectives

By the end of this chapter, a student should be able to:

1. Distinguish verification, validation, testing and performance evaluation.
2. Convert project requirements into measurable acceptance criteria.
3. Design repeatable and statistically defensible experiments.
4. Select appropriate technical, reliability and usability metrics.
5. Compare a project with a baseline or prior solution fairly.
6. Diagnose failures using structured methods.
7. Interpret uncertainty, limitations and threats to validity.
8. Report evidence ethically and make defensible conclusions.

1. Verification and Validation as Complementary Activities

Verification is primarily a conformance activity. It examines whether design outputs comply with requirements, drawings, algorithms, standards and interface definitions. Examples include checking whether a power supply produces the specified voltage, whether a printed circuit board follows the approved schematic, or whether firmware implements the defined communication protocol.

Validation is primarily a fitness-for-purpose activity. It examines whether the completed system meets stakeholder needs in the intended operating environment. A battery-monitoring system may be verified to measure voltage accurately but fail validation if its display is unreadable in sunlight or if its alarm is too late for the operator to respond.

A useful project evidence chain is:

Stakeholder need → system requirement → design specification → implementation → test evidence → acceptance decision.

Each arrow can introduce error. A stakeholder need may be misunderstood; a requirement may be vague; a test may not represent actual use; or data may be recorded selectively. Consequently, verification and validation should begin during requirements analysis rather than being postponed until the final demonstration.

1.1 Requirement traceability

A traceability matrix links each requirement to its design element, test method, measured result and acceptance decision. Requirements should be written so that they are testable. “The system shall be reliable” is inadequate unless reliability is defined in measurable terms, such as successful operation for a specified duration, number of cycles or environmental range.

Requirements may be verified by inspection, analysis, demonstration or test. Inspection is appropriate for dimensions, labelling and workmanship. Analysis is appropriate for calculated ratings or simulation results. Demonstration shows operation under a specified condition. Test obtains evidence through controlled measurement.

2. Designing a Rigorous Evaluation

A credible evaluation begins with an evaluation question. Examples include:

• Does the controller maintain speed within the required tolerance under load changes?
• Does the proposed converter achieve higher efficiency than the existing design?
• Can the monitoring system detect an over-temperature condition before damage occurs?

For each question, define the independent variable, dependent variable, controlled variables, measurement instrument, sample size, operating conditions and decision rule. The independent variable is deliberately changed; the dependent variable is measured; controlled variables are held constant to prevent confounding.

2.1 Test specification

A test specification should identify:

• objective and requirement under examination;
• equipment, calibration status and connection arrangement;
• input conditions and environmental conditions;
• procedure and sequence of operations;
• measured variables and units;
• number of repetitions;
• pass/fail criterion;
• method of recording raw data;
• safety controls and stop conditions.

A test is repeatable when the same operator and equipment can obtain similar results under the same conditions. It is reproducible when different competent operators, equipment or laboratories can obtain comparable results. Both properties improve confidence in conclusions.

2.2 Measurement accuracy and uncertainty

Every measurement contains uncertainty. If a measured voltage is reported as 12.04 V using an instrument specified as ±0.05 V, reporting 12.04000 V falsely implies much greater certainty. The resolution, calibration, repeatability and environmental sensitivity of the instrument should be considered.

For repeated observations x₁, x₂, …, xₙ, the arithmetic mean is:

where x̄ is the sample mean, xᵢ is observation i, and n is the number of observations. The sample standard deviation is:

The denominator n − 1 is used when estimating population variability from a sample. These formulas assume numerical observations measured on a meaningful interval or ratio scale. Standard deviation describes spread; it does not by itself prove that a result is accurate.

For a quantity calculated from independent measured variables, uncertainty propagation may be approximated by:

where y = f(x₁, x₂, …, xₘ), uᵧ is the standard uncertainty in y, uⱼ is the standard uncertainty in xⱼ, and the partial derivatives are evaluated at the measured values. This first-order expression assumes small uncertainties and negligible correlation between input variables. If inputs are correlated, covariance terms are required.

For example, electrical power calculated as P = VI, with independent voltage uncertainty uᵥ and current uncertainty uᵢ, has approximate uncertainty:

The units are consistent: Iuᵥ and Vuᵢ each have units of watts.

2.3 Accuracy metrics

For a measured value xₘ and reference value xᵣ, the absolute error is:

The absolute error has the same unit as the measured quantity. Percentage error, when xᵣ ≠ 0, is:

For a time series, the mean absolute error is:

where yᵢ is the reference or actual value and ŷᵢ is the system estimate. MAE has the same units as the output. The root mean square error is:

RMSE gives greater weight to large errors. Neither metric should be used without explaining the reference measurement, operating range and sampling procedure.

3. Performance Metrics for Electrical Engineering Projects

The metric must correspond to the engineering purpose. Common categories include the following.

3.1 Functional performance

Functional metrics determine whether required operations occur. Examples are detection rate, switching accuracy, settling time, communication packet success and control error. For a step response, settling time is the time required for the output to remain within a specified band around its final value. The band must be stated, commonly ±2% or ±5%, because different definitions produce different results.

For a feedback system with reference r(t) and output y(t), the tracking error is:

The definition applies when r and y represent comparable quantities with the same units. A small average error does not guarantee safe transient behaviour; overshoot, delay and oscillation must also be examined.

3.2 Efficiency and energy

For a converter or power-processing system, efficiency is:

where Pᵢₙ and Pₒᵤₜ are input and useful output power, measured under the same steady-state condition. Losses, measurement timing and power-factor treatment must be made explicit. For AC systems, apparent power must not be substituted for real power unless the intended metric is apparent-power efficiency.

Energy consumed over an interval is:

where p(t) is instantaneous power in watts and E is energy in joules. If power is constant, E = P∆t. A watt-hour result requires time expressed in hours or appropriate conversion from joules.

3.3 Reliability and availability

Reliability is the probability that a system performs its required function for a stated time under stated conditions. It is not equivalent to durability, maintainability or availability.

If failures follow a constant-rate exponential model, reliability may be estimated as:

where λ is the constant failure rate and t is operating time. This model assumes independent failures and a constant hazard rate; it is not automatically valid during early-life defects or wear-out periods.

Mean time between failures (MTBF) is often estimated for repairable systems as total operating time divided by the number of observed failures. It should not be presented as a guaranteed lifetime, particularly when the sample is small or the test is censored because no failure occurred.

Availability for a simple repairable system is often approximated by:

where MTTR is mean time to repair. The approximation assumes stable long-run averages and clearly defined operating and repair states.

3.4 Robustness and stress testing

Robustness is the ability to maintain acceptable performance when conditions vary. Relevant factors may include supply-voltage variation, temperature, humidity, vibration, electromagnetic interference, component tolerance, sensor noise, communication delay and operator error.

A robustness test should vary one factor at a time when isolating a mechanism, or use a designed experiment when interactions are important. Testing only at nominal conditions is insufficient. However, stress levels must remain safe and technically justified; destructive over-stressing without a defined purpose is not evidence of normal reliability.

A useful test matrix may include nominal, minimum, maximum and selected combined worst-case conditions. Each condition should have a rationale and an acceptance criterion.

4. Benchmarking and Comparative Evaluation

A new design should be compared with a meaningful baseline: a previous project, commercial product, conventional algorithm, simulation prediction or accepted standard. Comparisons are valid only when boundary conditions are equivalent. For example, comparing converter efficiencies at different output powers or temperatures can produce a misleading conclusion.

Absolute improvement is:

For a positive-is-better metric, relative improvement is:

For a lower-is-better metric such as error, the interpretation should be stated explicitly. A 20% reduction in error means the new error is 80% of the baseline error, not 120%.

Benchmark reports should include hardware versions, software versions, parameter settings, input data, environmental conditions, number of trials and statistical treatment. If the baseline is reproduced from published work, differences in implementation must be disclosed. A result that cannot be reproduced is weak evidence even if its numerical value appears impressive.

5. Failure Diagnosis and Root-Cause Analysis

Failure is not merely an undesirable outcome; it is information about system behaviour. Diagnosis should preserve evidence and proceed systematically.

First, make the system safe. Isolate hazardous energy, protect personnel and prevent a fault from causing secondary damage. Next, record the symptom precisely: what failed, under what condition, when it occurred and whether it is repeatable. Avoid changing several variables simultaneously, because this destroys causal information.

A practical diagnostic sequence is:

1. Confirm the reported failure using a controlled reproduction.
2. Check instruments, power supplies, grounding and test connections.
3. Divide the system into functional blocks.
4. Measure signals at block boundaries and compare them with expected values.
5. Form competing hypotheses.
6. Change one relevant condition or substitute one known-good component.
7. Identify the root cause and verify the corrective action with regression tests.

The “five whys” method can expose process causes, but it should not replace technical measurement. Fault-tree analysis begins with an undesired top event and decomposes it into contributing events using logical AND and OR relationships. Failure mode and effects analysis (FMEA) examines possible failure modes, their effects, causes and controls. Risk-priority scoring may assist prioritisation, but numerical scores are not objective probabilities unless their scales are carefully defined.

6. Worked Example: Evaluation of a Buck Converter

Consider an illustrative 12 V-to-5 V buck converter. The requirement is at least 85% efficiency at a 5 V, 1.0 A output, with output voltage between 4.90 V and 5.10 V at an ambient temperature of 25 °C.

Measured steady-state values are Vᵢₙ = 12.1 V, Iᵢₙ = 0.48 A, Vₒᵤₜ = 4.98 V and Iₒᵤₜ = 1.00 A. Assume the instruments measure average real DC quantities and that input and output readings are sufficiently synchronized.

Input power is:

Output power is:

Therefore:

The efficiency requirement is met. The voltage is also within the permitted interval, so the test passes at this operating point. This conclusion is limited to the stated input, load and temperature. It does not establish performance across the complete operating envelope.

Suppose the same test is repeated five times, producing efficiencies of 85.6%, 85.9%, 85.8%, 85.7% and 85.9%. The mean is 85.78%. The sample standard deviation is approximately 0.13 percentage points. This indicates good short-term repeatability, but it does not measure long-term reliability or calibration accuracy.

A stronger evaluation would repeat the test at minimum and maximum input voltage, light and full load, elevated temperature and after an endurance period. Thermal images or component temperature measurements would help determine whether efficiency loss is associated with semiconductor conduction, switching loss or magnetic-component heating.

7. Illustrative Case Study: Solar-Powered Environmental Monitor

Background

The following is a fictional illustrative case. A student team develops a solar-powered environmental monitor for a small greenhouse. It measures temperature and humidity, transmits data wirelessly and activates a ventilation fan when temperature exceeds a threshold.

Problem

The initial demonstration worked indoors, but stakeholders required unattended operation for four weeks. The system had to operate from a photovoltaic panel and battery, maintain temperature measurement within ±0.5 °C, transmit at least 95% of scheduled records, and activate ventilation within 30 s of a sustained high-temperature condition.

Analysis

The team created a traceability matrix and identified three risks: insufficient battery energy during cloudy periods, sensor drift at high humidity and communication loss caused by the metal greenhouse frame. The original demonstration did not test any of these conditions.

The evaluation plan included calibrated reference sensors, a battery-voltage logger, packet sequence numbers, controlled threshold tests and a four-week field trial. The fan threshold was tested using a temperature chamber and a reference thermometer. Communication performance was measured as:

Missing packets were counted using sequence numbers rather than inferred from display gaps.

Approach and outcome

The threshold test showed activation after 18–24 s for five trials, satisfying the 30 s requirement. Temperature error had a mean absolute value of 0.31 °C, also satisfying the requirement. However, the initial field trial achieved only 91% packet delivery and the battery voltage fell below the controller’s minimum operating voltage after three cloudy days.

The team repositioned the antenna, added a communication retry policy and revised the energy budget. The revised trial achieved 97% packet delivery and remained operational for the four-week period. The result did not prove indefinite operation; it demonstrated compliance under the recorded weather and installation conditions.

Learning points

The case illustrates that laboratory functionality is not equivalent to field validation. It also shows why failures should be reported rather than concealed. The failed first trial identified design weaknesses and guided corrective action. Stakeholder acceptance depended on response time, data continuity and autonomy, not merely on whether individual sensors produced plausible readings.

8. Limitations, Ethics and Governance

Evaluation conclusions must match the evidence. A small number of trials cannot justify broad claims about population-level reliability. A simulation cannot establish hardware safety unless model assumptions and physical validation are addressed. A successful short demonstration cannot justify a claim of long service life.

Students must preserve raw data, document exclusions and report failed tests. Removing inconvenient observations without a predeclared technical reason is research misconduct. Calibration records, software versions and test scripts should be retained. Data involving people, locations or operational infrastructure may require consent, anonymisation, access control or institutional approval.

Electrical safety is non-negotiable. Appropriate isolation, fusing, protective earth, current limiting, enclosure design and emergency procedures must be used. Relevant national and international standards should be identified rather than claimed generically. If a project can affect health, public safety, grid operation or industrial control, supervision and formal risk assessment are essential.

Artificial intelligence may assist with anomaly detection, classification or test-log analysis, but an AI-generated conclusion is not evidence by itself. The dataset, labels, performance metrics, false-alarm costs and failure cases must be examined. Engineers remain responsible for verification, validation and safe deployment.

9. Best Practices for the Dissertation Report

A strong evaluation chapter should:

• restate each requirement and its measurable acceptance criterion;
• describe equipment, calibration and test conditions;
• distinguish measured results from calculated or simulated results;
• show representative raw data as well as summaries;
• report units, uncertainty and relevant sample size;
• compare results with a baseline under equivalent conditions;
• explain failures and corrective actions;
• discuss limitations and threats to validity;
• avoid unsupported words such as “perfect,” “fully reliable” or “proven”;
• conclude each test with a transparent pass, fail or inconclusive decision.

Graphs should have labelled axes, units, legends and captions. Error bars must be defined: they may represent standard deviation, confidence intervals, instrument uncertainty or another quantity. These are not interchangeable. Tables should not hide important variation behind excessive rounding.

Chapter Summary

Verification establishes whether an implementation conforms to specifications, whereas validation establishes whether it satisfies stakeholder needs in its intended context. Performance evaluation supplies measured evidence through planned, repeatable tests. Requirements should be made measurable and linked to design elements and test records through traceability.

Reliable evaluation requires suitable metrics, controlled variables, calibrated instruments, uncertainty awareness and clearly stated assumptions. Important measures include error, MAE, RMSE, efficiency, energy, reliability, availability and packet delivery ratio. Benchmarking is meaningful only when the baseline and new system are tested under equivalent conditions. Robustness testing explores variation in environmental, electrical and operational factors.

Failures should be reproduced safely, diagnosed by functional decomposition and verified after corrective action. Ethical evaluation requires honest reporting of negative results, preservation of raw data, respect for safety and privacy, and claims that do not exceed the evidence. A professional dissertation therefore presents not only successful demonstrations but also limitations, uncertainty and the reasoning behind acceptance decisions.

Key Terms

Verification: Evidence that an implementation conforms to specified requirements.

Validation: Evidence that a system is fit for its intended stakeholder purpose.

Performance evaluation: Systematic measurement and interpretation of system behaviour.

Acceptance criterion: A defined condition used to decide whether a requirement is satisfied.

Traceability matrix: A record linking requirements to design elements, tests and results.

Repeatability: Agreement of results under the same conditions and operator.

Reproducibility: Agreement of results across changed operators, equipment or laboratories.

Measurement uncertainty: Quantified doubt associated with a measurement result.

Baseline: A reference system or method used for comparison.

Robustness: Ability to maintain acceptable performance under variation or disturbance.

Reliability: Probability of performing a required function for a stated time and condition.

Availability: Proportion of time that a repairable system is operational.

MTBF: Mean time between failures for a defined repairable-system context.

MTTR: Mean time required to restore a system after failure.

Root cause: Fundamental condition that produces an observed failure.

FMEA: Structured analysis of failure modes, effects, causes and controls.

Regression test: Repeated test confirming that a change has not damaged existing functions.

Benchmarking: Fair comparison with a reference design, method or standard.

Threat to validity: A factor that may weaken the credibility or generality of a conclusion.

Review Questions

Short Answer Questions

1. Distinguish verification from validation and give one electrical engineering example of each.
2. Why is a traceability matrix useful in a major project?
3. Define repeatability and reproducibility.
4. State two limitations of using the constant-failure-rate reliability model.
5. Why should failed tests be reported in a dissertation?

Long Answer Questions

1. Explain how to develop a test specification for an embedded electrical system, including equipment, conditions, procedure and acceptance criteria.
2. Discuss measurement uncertainty and explain why numerical precision should match measurement capability.
3. Compare functional testing, robustness testing and endurance testing, giving suitable examples.
4. Explain how benchmarking can become misleading and identify controls needed for a fair comparison.
5. Describe a structured root-cause analysis process for an intermittent electronic-system failure.

Analytical / Application Questions

1. A converter has Vᵢₙ = 24.0 V, Iᵢₙ = 0.60 A, Vₒᵤₜ = 12.0 V and Iₒᵤₜ = 1.10 A. Calculate input power, output power and efficiency, and determine whether it meets an 80% efficiency requirement.
2. A wireless monitor transmits 4,800 records and receives 4,656. Calculate the packet delivery ratio and explain one limitation of this metric.
3. A prototype passes all nominal tests but fails at the minimum supply voltage and maximum temperature. Design a follow-up evaluation that distinguishes inadequate design margin from measurement or test-setup error.

Multiple Choice Questions with Answers

Question 1

Which question is primarily addressed by verification?

A. Does the system satisfy stakeholder needs in actual use?

B. Was the system built according to its specified requirements?

C. Will users prefer the product to competing products?

D. How profitable will the product be?

Correct Answer: B. Was the system built according to its specified requirements?

Question 2

A converter has an input power of 10 W and useful output power of 8.5 W. What is its efficiency?

A. 1.5%

B. 15%

C. 85%

D. 117.6%

Correct Answer: C. 85%

Question 3

Which metric gives greater weight to large individual errors?

A. Mean absolute error

B. Root mean square error

C. Packet delivery ratio

D. Availability

Correct Answer: B. Root mean square error

Question 4

What is the main purpose of a requirement traceability matrix?

A. To replace calibration certificates

B. To link requirements with implementation and evidence

C. To increase the supply voltage during testing

D. To estimate manufacturing profit

Correct Answer: B. To link requirements with implementation and evidence

Question 5

Under the constant-failure-rate exponential model, which expression represents reliability?

A. R(t) = e^(−λt)

B. R(t) = λt

C. R(t) = t/λ

D. R(t) = λ + t

Correct Answer: A. R(t) = e^(−λt)

Question 6

A system transmits 1,000 packets and receives 950. What is the packet delivery ratio?

A. 5%

B. 50%

C. 95%

D. 105%

Correct Answer: C. 95%

Question 7

Which action is most appropriate immediately after reproducing an unsafe electrical failure?

A. Increase the stress until the component burns out

B. Make the system safe and isolate hazardous energy

C. Delete the failed data point

D. Change several components simultaneously

Correct Answer: B. Make the system safe and isolate hazardous energy

Question 8

Which statement about validation is correct?

A. It concerns only schematic drawing quality

B. It determines whether the system is fit for intended use

C. It is identical to calculating component ratings

D. It can be established by simulation alone in every case

Correct Answer: B. It determines whether the system is fit for intended use

Question 9

For independent voltage and current measurements, the approximate uncertainty in P = VI is based on which expression?

A. u_P = u_V + u_I

B. u_P = √[(Iu_V)² + (Vu_I)²]

C. u_P = V/I

D. u_P = VIu_Vu_I

Correct Answer: B. u_P = √[(Iu_V)² + (Vu_I)²]

Question 10

Which practice most strengthens a comparative benchmark?

A. Testing the new system at favourable conditions and the baseline at nominal conditions

B. Reporting only the best trial

C. Using equivalent operating conditions and documenting parameters

D. Changing the metric after seeing the results

Correct Answer: C. Using equivalent operating conditions and documenting parameters