Verification, Validation and Debugging

Introduction

Embedded systems in mechatronics combine software, processors, sensors, actuators, communication networks, and mechanical components. A defect may therefore appear as a software failure while its cause lies in timing, electrical noise, calibration, mechanical loading, or an incorrect requirement. Verification, validation, and debugging provide complementary ways to control this complexity.

Verification asks whether the product has been built correctly against specified requirements, designs, and standards. Validation asks whether the resulting system satisfies its intended use and stakeholder needs. Debugging is the systematic process of locating, explaining, and correcting the causes of failures. Testing supplies evidence, but testing alone cannot prove that a system contains no defects.

For a mechatronic system, confidence should be developed progressively: clarify requirements, analyze designs, test individual software units, integrate subsystems, simulate physical behavior, execute on target processors, test with real hardware, and continuously repeat these activities as the design changes.

Learning objectives

By the end of this chapter, a student should be able to:

1. Distinguish verification, validation, testing, and debugging.
2. Construct and use a requirements traceability matrix.
3. Design unit, integration, system, and acceptance tests.
4. Apply static analysis, simulation, fault injection, and assertions.
5. Explain software-in-the-loop, processor-in-the-loop, and hardware-in-the-loop testing.
6. Use oscilloscopes, logic analyzers, debuggers, and logs to diagnose faults.
7. Interpret test coverage and defect evidence critically.
8. Design a practical continuous-integration workflow for embedded projects.

1. A systems view of quality

A useful quality argument links four elements:

• Requirements: what the system must do and under what constraints.
• Implementation: hardware, software, mechanics, control algorithms, and interfaces.
• Evidence: inspections, analyses, simulations, measurements, and test results.
• Decision: whether the evidence is sufficient for release or further engineering.

Verification and validation are related but not interchangeable. A motor controller may pass every software unit test and still fail validation because its emergency stop is too slow for a real operator. Conversely, it may meet a user expectation in a demonstration while violating a timing, safety, or coding requirement.

Important distinctions are shown below.

Quality activities should begin before code is written. Ambiguous requirements generate ambiguous tests. A requirement such as “the robot shall respond quickly” is not directly verifiable. A stronger form might be: “After a valid emergency-stop command is detected, motor torque shall be disabled within 50 ms under operating conditions from 0 °C to 45 °C.” This statement identifies an event, a measurable response, a limit, and environmental conditions.

2. Requirements traceability

Requirements traceability records relationships among stakeholder needs, system requirements, architecture, implementation elements, tests, and defects. A traceability matrix is a simple but powerful implementation. Each requirement receives a stable identifier, such as REQ-MOT-014.

A useful requirement should be:

• correct and necessary;
• unambiguous;
• feasible;
• consistent with other requirements;
• measurable or objectively assessable;
• uniquely identifiable;
• traceable to verification evidence.

Traceability is usually bidirectional. Forward traceability maps a requirement to design elements and tests. Backward traceability maps a test, code module, or design decision back to the requirement that justifies it. Unlinked requirements may be unimplemented; unlinked tests may be unnecessary or may reveal missing requirements.

Traceability does not guarantee correctness. A complete matrix can still trace an incorrect requirement through a perfectly executed implementation. Therefore requirements must also be reviewed with domain experts and tested against realistic operational scenarios.

3. Test levels and test design

Unit testing

A unit is the smallest separately testable software element, such as a function, class, driver abstraction, or control-law component. Unit tests isolate the unit using stubs, mocks, or simulated inputs. They should include normal, boundary, invalid, and exceptional cases.

For example, an angle-normalization function may be required to return an angle in the interval [−π, π). Tests should include zero, values just below and above the limits, multiple revolutions, and invalid numeric values. A test that checks only ordinary positive angles provides weak evidence.

Integration testing

Integration tests examine interactions: sensor drivers with middleware, a controller with a timer, or a communication parser with a command handler. Many embedded failures occur at interfaces because assumptions about units, byte order, sampling rate, ownership, or timing differ between modules.

An interface contract should define data type, units, valid range, update rate, initialization state, error behavior, and timing constraints. For example, a temperature interface should specify whether values are degrees Celsius or raw ADC counts, whether stale data is flagged, and what happens after sensor disconnection.

System and acceptance testing

System tests exercise the complete product in an environment representative of intended operation. Acceptance tests are derived from stakeholder acceptance criteria and may include usability, maintainability, safety, and performance. A system test should specify preconditions, stimulus, expected result, measurement method, tolerances, and pass/fail rule.

Regression testing

A regression test detects the reappearance of a previously corrected defect or an unintended effect of a change. Every significant defect should result in a reproducible test whenever practical. Regression suites should be organized by speed and risk: fast unit tests on every commit, integration tests on merges, and lengthy hardware or endurance tests on scheduled builds.

Equivalence classes and boundary analysis

Inputs can be partitioned into equivalence classes expected to behave similarly. A speed command might have valid, negative, excessive, missing, and nonnumeric classes. Boundary-value analysis then tests values at, just below, and just above each limit. These methods provide better coverage than selecting arbitrary values.

For a requirement that permits a supply voltage from 10.8 V to 13.2 V, useful tests include 10.8 V, 13.2 V, values slightly below and above both boundaries, and representative nominal voltage. The exact tolerance of the measurement apparatus must be considered; a displayed value of 10.800 V does not necessarily prove the physical voltage is exactly 10.800 V.

4. Static analysis and formal reasoning

Static analysis examines source code, binaries, configurations, or models without executing the program. Typical checks include syntax, type consistency, unreachable code, uninitialized variables, suspicious conversions, buffer bounds, concurrency risks, cyclomatic complexity, and compliance with a coding standard.

Compiler warnings should be treated as engineering information rather than cosmetic messages. A warning about signed-to-unsigned conversion may indicate that a negative sensor value will become a very large positive value. Static analyzers can identify paths that ordinary tests do not execute, but they may produce false positives. Results should be classified, justified, and reviewed rather than blindly suppressed.

For safety- or security-relevant code, coding standards and peer review are especially valuable. Examples include avoiding hidden state, documenting units, limiting dynamic allocation, checking return values, and making error handling explicit. Static analysis cannot establish that the control algorithm is physically appropriate, nor can it detect every wiring or timing defect.

Assertions express conditions that must hold at a particular point. For example, an assertion may require that a state-machine index is within its table range or that a command is accepted only when the system is in a permitted mode. In production, an assertion failure must have a defined safe response; simply stopping in an unsafe state is not acceptable.

5. Simulation and model-based testing

Simulation permits inexpensive exploration of situations that may be dangerous, expensive, or difficult to reproduce physically. A plant model may represent motor dynamics, friction, inertia, sensor noise, actuator saturation, and disturbances. A software model may represent the controller, scheduler, or communication network.

Simulation is useful for:

• checking control algorithms before hardware exists;
• exploring operating envelopes;
• generating repeatable test inputs;
• testing rare faults;
• evaluating numerical limits and timing assumptions;
• comparing alternative designs.

A model is an abstraction, not reality. Validation of the model requires comparison with measured plant behavior. A model that omits backlash may falsely suggest that a positioning controller is stable. A simulation with ideal communication may hide packet loss and scheduling jitter.

Software-in-the-loop, processor-in-the-loop, and hardware-in-the-loop

In software-in-the-loop (SIL), the algorithm and often the plant model execute on a host computer. SIL is fast and convenient for large test sets, but host arithmetic, operating-system timing, and compiler behavior may differ from the target.

In processor-in-the-loop (PIL), production or near-production code executes on the target processor while inputs and outputs are exchanged with a host model. PIL exposes processor arithmetic, compiler effects, memory limitations, and execution time more realistically than SIL.

In hardware-in-the-loop (HIL), real embedded hardware interacts with a real-time simulator representing the plant or external environment. HIL can test I/O, scheduling, communication, fault responses, and safety behavior without exposing a complete machine to damage. HIL quality depends on model fidelity, simulator timing, interface calibration, and correct fault representation.

Figure: Increasing realism from software-in-the-loop to processor-in-the-loop and hardware-in-the-loop.

A sound workflow uses SIL for broad algorithmic exploration, PIL for target-code effects, and HIL for interface and fault-response evidence. None replaces final validation on the actual product in its intended environment.

6. Debugging as an evidence-based process

Debugging should not begin with random edits. A disciplined sequence is:

1. Observe: record the exact symptom, operating conditions, firmware version, inputs, and timing.
2. Reproduce: create the smallest reliable reproduction, preferably automated.
3. Localize: determine whether the fault is in requirements, hardware, software, timing, data, or interaction.
4. Form a hypothesis: state a specific causal explanation.
5. Instrument and test: add measurements, logs, assertions, or controlled experiments.
6. Correct: make the smallest justified change.
7. Regression-test: demonstrate that the original failure is gone and no related behavior is damaged.

The scientific method is particularly important in intermittent faults. Changing several variables at once destroys causal information. A useful defect report includes a unique identifier, severity, frequency, reproduction steps, expected and actual behavior, logs, traces, environmental conditions, and suspected impact.

Time and concurrency faults

Embedded failures often arise from timing rather than incorrect sequential logic. Relevant quantities include task periods, interrupt latency, execution time, communication deadlines, and watchdog windows. If a periodic task has period T and its worst-case execution time is C, a necessary basic condition for that task not to overrun its own period is C ≤ T. For multiple tasks, processor utilization is commonly expressed as:

Here Cᵢ is the execution time of task i, Tᵢ is its period, and U is dimensionless. This expression estimates processor demand under stated scheduling assumptions; U ≤ 1 is necessary for a single processor in a simple independent-task model, but it is not by itself a proof of schedulability. Priorities, blocking, release jitter, deadlines, interrupts, and scheduling policy must also be analyzed.

A logic analyzer or oscilloscope can reveal that a control task starts late, a PWM signal changes too soon, or a chip-select line violates a peripheral timing requirement. Software timestamps may be misleading when clocks are unsynchronized or logging alters timing. Instrumentation must therefore be designed so that it does not substantially change the fault.

Debugging tools

A source-level debugger supports breakpoints, watchpoints, register inspection, and single stepping. It is powerful for deterministic faults but may perturb real-time behavior. Logging provides historical evidence but consumes memory, bandwidth, and CPU time. Circular buffers with event codes and timestamps are often more suitable than unrestricted text logging.

Oscilloscopes show analog waveforms and voltage levels over time. Logic analyzers decode digital transitions and protocols. A current probe can reveal actuator stalls, unexpected sleep-current consumption, or power sequencing faults. A protocol analyzer can expose malformed frames, incorrect acknowledgements, and bus arbitration errors. These tools should be connected with attention to grounding, probe loading, bandwidth, and signal integrity.

7. Fault injection and negative testing

A robust system must be tested not only under nominal inputs but also under abnormal conditions. Fault injection deliberately introduces faults such as:

• disconnected or implausible sensors;
• stuck-at, noisy, delayed, or biased measurements;
• corrupted, duplicated, delayed, or lost packets;
• actuator saturation and overcurrent;
• memory allocation failure;
• watchdog reset;
• low supply voltage;
• invalid commands and unauthorized access;
• clock drift or missed interrupts.

Each injected fault needs an expected safe response. “The system handles the fault” is not a testable criterion. A precise criterion might require detection within a specified interval, transition to a defined state, removal of actuator energy, storage of a diagnostic code, and controlled recovery after the fault is removed.

Fault injection must be performed under controlled conditions. Injecting a fault into a high-energy machine without suitable safeguards can create unacceptable risk. Test plans should specify isolation, emergency shutdown, authorization, data integrity, and restoration procedures.

8. Continuous integration and test evidence

Continuous integration (CI) automatically builds and tests changes whenever code is committed or merged. A practical embedded pipeline may contain:

1. formatting and configuration checks;
2. compilation for every supported target;
3. compiler warnings treated according to project policy;
4. static analysis;
5. unit tests on the host;
6. SIL regression tests;
7. target builds and binary checksums;
8. integration tests on development hardware;
9. scheduled HIL, endurance, and fault-injection tests;
10. publication of traceability and test reports.

Builds should be reproducible: source revisions, compiler versions, configuration, dependencies, calibration data, and test equipment should be identified. Test results should be retained with logs and artifacts. A green pipeline is not equivalent to a safe product if critical tests are absent, flaky tests are ignored, or the test environment differs materially from deployment.

Useful engineering metrics include requirement coverage, test pass rate, defect escape rate, code coverage, mean time to reproduce, and time from defect discovery to correction. Code coverage is informative but limited. Statement coverage shows which statements executed; branch coverage examines decision outcomes; modified-condition or higher assurance measures may be required in critical systems. High coverage can still miss incorrect expected values, inadequate requirements, hardware defects, and untested combinations.

9. Illustrative case: a servo positioning module

Background

Consider an illustrative laboratory servo axis used in a pick-and-place mechanism. An encoder supplies position, a microcontroller runs a position controller at 1 kHz, and a network command specifies the target position. The axis includes an overcurrent input and an emergency-stop circuit.

Problem

During endurance testing, the axis occasionally moves several millimeters after an emergency-stop command. The event occurs approximately once every few hundred cycles and is not reproduced by ordinary unit tests.

Analysis

The team first records firmware revision, command timing, encoder value, PWM output, overcurrent input, emergency-stop input, and motor current. A logic analyzer shows that the emergency-stop input changes during a high-priority interrupt, but the PWM output remains active for one additional control cycle. Code inspection reveals that the emergency-stop flag is set in an interrupt routine, while the PWM compare register is updated by a lower-priority control task. A second issue is found: the test fixture’s emergency-stop signal has contact bounce.

The requirement is refined: once the safety input is electrically detected, the hardware gate shall disable the power stage independently of the ordinary control task; software shall additionally record the event and enter a latched fault state.

Approach and solution

The engineers first create a regression test using a debounced simulated input and verify the software state transition. They then inject short pulses and delayed signals in HIL. A hardware modification routes the safety signal to a timer or gate capable of disabling PWM asynchronously. Firmware tests verify that a latched fault cannot be cleared by an ordinary network command. Oscilloscope measurements confirm the power-stage disable time under minimum and maximum supply conditions.

The team also adds tests for lost encoder data, invalid network packets, overcurrent, processor reset, and recovery after power cycling. The traceability matrix links the emergency-stop requirement to the schematic, safety firmware, HIL scenario, and measured hardware test.

Outcome

The original intermittent movement is eliminated in the tested conditions. More importantly, the design no longer relies on a non-safety control task to provide the first protective action. The software remains responsible for diagnosis and controlled state management, while the independent hardware path limits the hazard.

Learning points

The case demonstrates that a software unit can be logically correct while the integrated system is unsafe. It also illustrates the value of precise requirements, timing instrumentation, fault injection, hardware protection, regression tests, and independent safety mechanisms.

10. Limitations, governance, and professional practice

Verification evidence is always bounded by assumptions. Test equipment has accuracy limits; models have scope limits; test cases sample an enormous input space; and environmental conditions may differ from the laboratory. Engineers should document assumptions and uncertainty rather than present pass/fail results as absolute truth.

Safety-critical and regulated products may require documented lifecycle processes, independence of review, configuration control, hazard analysis, cybersecurity controls, and retention of objective evidence. The applicable standards depend on the domain, such as industrial machinery, automotive, medical devices, aerospace, or consumer products. A standard should be interpreted with competent domain guidance rather than copied mechanically.

Cybersecurity is part of validation because compromised inputs can produce physically dangerous behavior. Tests should include authentication failures, malformed messages, replay, denial of service, unsafe update attempts, and protection of debug interfaces. AI-assisted coding and test generation can accelerate development, but generated code and tests require human review, licensing and provenance checks, security analysis, and verification against requirements. An AI-generated test that merely repeats implementation assumptions may provide false confidence.

Best practices include:

• write measurable requirements before designing tests;
• use independent review for critical functions;
• test interfaces explicitly, including units and timing;
• automate repeatable tests and preserve artifacts;
• make every important defect reproducible where possible;
• test abnormal and environmental conditions;
• use multiple evidence types rather than relying on coverage alone;
• separate safe shutdown from ordinary functional control where hazards justify it;
• control firmware, hardware revisions, tools, calibration, and configurations;
• treat failed, skipped, and flaky tests as engineering issues;
• review whether the test environment represents actual use.

Chapter Summary

Verification determines whether an implementation conforms to specified requirements, whereas validation determines whether the complete system satisfies its intended purpose. Traceability connects requirements to architecture, implementation, tests, and defects. Unit, integration, system, acceptance, and regression tests provide evidence at increasing levels of realism. Static analysis finds many defects without execution, while simulation enables repeatable exploration of plant and software behavior.

SIL, PIL, and HIL progressively introduce target-code and physical-interface realism. Debugging is most effective when based on reproducible evidence, explicit hypotheses, controlled instrumentation, and regression tests. Logic analyzers, oscilloscopes, debuggers, logs, and current measurements reveal different classes of faults. Fault injection tests the system’s response to abnormal conditions, and CI makes quality activities repeatable throughout development. No single metric or test proves correctness; dependable mechatronic systems require layered evidence, controlled configuration, safety-aware engineering, and professional judgment.

Key Terms

Verification: Evaluation of whether a product conforms to specified requirements.

Validation: Evaluation of whether a product meets intended user and operational needs.

Debugging: Systematic localization and correction of defect causes.

Requirements traceability: Recorded linkage among requirements, design, implementation, tests, and defects.

Unit test: Test of an isolated software component.

Integration test: Test of interactions among components or subsystems.

System test: Test of the complete integrated system.

Acceptance test: Test against stakeholder or contractual acceptance criteria.

Regression test: Test intended to detect reintroduced or newly caused defects.

Static analysis: Examination of code or models without executing them.

Assertion: An explicitly checked condition expected to hold during execution.

Fault injection: Deliberate introduction of abnormal conditions for testing.

Software-in-the-loop: Testing an algorithm and model on a host computer.

Processor-in-the-loop: Execution of target code on the target processor with a host model.

Hardware-in-the-loop: Interaction of real embedded hardware with a real-time simulated environment.

Logic analyzer: Instrument for observing and decoding digital transitions.

Test coverage: Measure of exercised code, branches, requirements, or scenarios.

Continuous integration: Automated building and testing of changes in a shared development workflow.

Requirements-based testing: Test design derived explicitly from measurable requirements.

Safety case: Structured argument supported by evidence that a system is acceptably safe.

Review Questions

Short Answer Questions

1. Distinguish verification from validation using a mechatronic example.
2. What is the purpose of bidirectional requirements traceability?
3. State two differences between SIL and HIL.
4. Why are boundary-value tests useful?
5. Name three instruments or tools that can assist embedded debugging and state what each reveals.

Long Answer Questions

1. Explain how unit, integration, system, acceptance, and regression testing form a layered test strategy.
2. Describe a disciplined debugging process for an intermittent timing fault.
3. Explain the benefits and limitations of static analysis and code coverage.
4. Compare SIL, PIL, and HIL in terms of realism, cost, speed, and defects exposed.
5. Design a CI pipeline for a microcontroller-based motor controller, including artifacts and quality gates.

Analytical / Application Questions

1. A task has execution time C = 0.4 ms and period T = 2 ms. A second task has C = 1.2 ms and T = 5 ms. Calculate U using U = ∑(Cᵢ/Tᵢ). Interpret the result and explain why it is not a complete schedulability proof.
2. An emergency-stop requirement specifies that motor torque must be disabled within 20 ms. Propose a traceability entry, a test procedure, instrumentation, boundary conditions, and a pass/fail rule.
3. A robot intermittently accepts a corrupted network command and changes speed. Develop a fault-injection and debugging plan covering reproduction, packet analysis, software tests, HIL testing, safe response, and regression evidence.

Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes verification from validation?

A. Verification checks intended user value, while validation checks compiler warnings.

B. Verification checks conformance to requirements, while validation checks fitness for intended use.

C. Verification is performed only on hardware, while validation is performed only in simulation.

D. Verification corrects defects, while validation writes source code.

Correct Answer: B. Verification checks conformance to requirements, while validation checks fitness for intended use.

Question 2

What is the primary purpose of a requirements traceability matrix?

A. To replace all system testing with documentation

B. To record processor register values during debugging

C. To link requirements to design elements and objective verification evidence

D. To calculate processor utilization

Correct Answer: C. To link requirements to design elements and objective verification evidence

Question 3

Which activity is characteristic of static analysis?

A. Observing motor current with a probe during operation

B. Checking for unreachable code without executing the program

C. Running a complete machine acceptance test

D. Injecting packet loss into a live network

Correct Answer: B. Checking for unreachable code without executing the program

Question 4

In processor-in-the-loop testing, what normally executes on the target processor?

A. The production or near-production controller code

B. Only the mechanical plant model

C. Only the test report generator

D. The user acceptance procedure

Correct Answer: A. The production or near-production controller code

Question 5

Which test is most directly intended to detect the return of a previously corrected defect?

A. Regression test

B. Exploratory design review

C. Calibration test

D. Type inspection

Correct Answer: A. Regression test

Question 6

A task has C = 0.5 ms and T = 2 ms. Its contribution to U = C/T is:

A. 0.25

B. 0.40

C. 1.50

D. 2.50

Correct Answer: A. 0.25

Question 7

Which instrument is generally most suitable for examining digital bus transitions and protocol timing?

A. Logic analyzer

B. Thermocouple only

C. Mechanical caliper

D. Hydraulic pressure regulator

Correct Answer: A. Logic analyzer

Question 8

Why is fault injection used in embedded-system testing?

A. To eliminate the need for requirements

B. To test responses to abnormal conditions that nominal tests may not cover

C. To guarantee that the physical plant model is exact

D. To increase source-code size

Correct Answer: B. To test responses to abnormal conditions that nominal tests may not cover

Question 9

Which statement about code coverage is correct?

A. 100% statement coverage proves that requirements are correct.

B. Code coverage is irrelevant to testing decisions.

C. Coverage is useful evidence but does not by itself prove system correctness.

D. Branch coverage measures only electrical noise.

Correct Answer: C. Coverage is useful evidence but does not by itself prove system correctness.

Question 10

What is a principal benefit of continuous integration?

A. It prevents all hardware failures automatically.

B. It repeatedly builds and tests changes, providing early feedback on defects.

C. It removes the need for configuration control.

D. It guarantees that a simulation model represents reality.

Correct Answer: B. It repeatedly builds and tests changes, providing early feedback on defects.