Verification, Diagnostics, Reliability and Safety

Introduction

A microcontroller system is not complete when its firmware compiles or when a prototype produces the expected output once. In mechatronics, the controller interacts with sensors, actuators, power electronics, mechanical structures, communication networks, and human operators. Failures may therefore arise from software defects, electrical noise, timing errors, component degradation, incorrect assumptions, or unsafe system responses. Verification and validation provide evidence that the system has been built correctly and that it is appropriate for its intended use.

Verification asks, “Have we implemented the specified requirements correctly?” Validation asks, “Does the resulting system satisfy the real operational need?” Diagnostics determines the condition and likely cause of a fault. Reliability assesses the probability that required functions will continue to operate. Safety engineering ensures that failures do not create unacceptable risk.

This chapter presents a practical evidence-based workflow covering unit, integration, hardware-in-the-loop (HIL), and regression testing; instrumentation; fault analysis; failure mode and effects analysis (FMEA); reliability; diagnostics; and safety documentation.

Learning objectives

By the end of this chapter, a student should be able to:

• distinguish verification, validation, testing, diagnosis, reliability, and safety;
• design unit, integration, HIL, and regression tests for embedded systems;
• select and correctly use oscilloscopes, logic analyzers, debuggers, and trace tools;
• analyse faults using structured methods including FMEA and fault trees;
• calculate basic reliability measures and interpret their assumptions;
• design diagnostic mechanisms and fail-safe responses;
• produce traceable evidence for quality and functional-safety requirements.

1. A model of evidence-based assurance

A useful assurance process links four elements:

1. Requirement — an observable statement of what the system must do.
2. Implementation — hardware, firmware, configuration, and interfaces that realize the requirement.
3. Test or analysis method — a repeatable way to assess compliance.
4. Evidence — recorded results, conditions, tools, versions, and conclusions.

A requirement such as “the motor shall stop promptly when the emergency switch is pressed” is incomplete unless “promptly” is defined. A testable version might specify that, for a stated load and supply voltage, motor torque command shall become zero within 50 ms, and hazardous motion shall cease within a separately specified time.

Requirements should be atomic, unambiguous, measurable, and traceable. A requirements traceability matrix maps each requirement to design elements, tests, defects, and approval records. Traceability is particularly important when a change affects timing, interrupt handling, safety mechanisms, or calibration.

Verification and validation terminology

Testing demonstrates behaviour under selected conditions; it cannot prove that every possible input and physical condition is safe. Confidence must therefore combine testing with inspection, static analysis, design review, fault analysis, and operational controls.

2. Levels of testing

2.1 Unit testing

A unit is a small, independently testable item, such as a sensor conversion function, PID calculation, packet parser, or fault-state transition. Unit tests should isolate the unit from hardware where practical by using stubs, mocks, or recorded data.

Good unit tests include normal, boundary, invalid, and abnormal cases. For a temperature conversion function, tests might cover the minimum and maximum ADC codes, sensor disconnection, saturation, numerical overflow, and values just inside and outside an alarm threshold. Assertions should test externally relevant behaviour rather than private implementation details.

A unit test should be deterministic. Time, random numbers, interrupts, and hardware registers should be controlled or simulated. Coverage measures such as statement, branch, and modified-condition/decision coverage can reveal untested code, but coverage is not equivalent to correctness. Executing every statement does not guarantee that requirements or combinations of conditions have been adequately tested.

2.2 Integration testing

Integration testing examines interactions among units and interfaces. Typical targets include:

• ADC sampling and filtering;
• timer interrupts and control-loop execution;
• driver, middleware, and application layers;
• sensor scaling and actuator limiting;
• communication framing, timeouts, and retries;
• bootloader, application, watchdog, and reset behaviour.

Integration failures often result from assumptions about units, byte order, timing, ownership of peripherals, or initialization order. Interface contracts should specify data ranges, units, update rates, validity flags, timeout behaviour, and concurrency rules.

2.3 Hardware-in-the-loop testing

In HIL testing, the real controller runs its production firmware while a real-time model supplies simulated plant signals and receives controller outputs. HIL is valuable when physical testing is expensive, slow, hazardous, or difficult to reproduce. It can test sensor faults, actuator saturation, communication loss, startup sequences, and control behaviour over many scenarios.

HIL accuracy depends on model fidelity, real-time execution, I/O timing, and correct scaling. A plant model that omits backlash, quantization, thermal limits, or transport delay may produce false confidence. HIL should be correlated with bench and physical tests rather than treated as a replacement for them.

2.4 Regression testing

Regression testing repeats selected tests after a change to detect unintended effects. The regression suite should contain high-risk requirements, previously failed tests, interface tests, safety tests, and representative normal-operation tests. Tests should be versioned with firmware, configuration, test scripts, and calibration data.

A useful continuous-integration sequence is:

commit → static checks → build → unit tests → integration tests
       → target smoke test → HIL scenarios → report and archive

A failed test should not simply be deleted or weakened. It should be classified as a product defect, test defect, environment problem, or intentional requirement change. Any change to expected results requires review and traceability.

3. Instrumentation and observation

Oscilloscopes

An oscilloscope displays voltage as a function of time. It is appropriate for analogue signals, power-rail disturbances, PWM waveforms, reset behaviour, ringing, overshoot, and timing relationships. Probe grounding is critical: a long ground lead can introduce inductance and display ringing that is not present at the circuit node. Differential or isolated measurement may be required for high-side or floating signals, but the probe’s voltage and common-mode ratings must be respected.

The sampling rate must be sufficient for the fastest relevant event, and the analogue bandwidth must not unintentionally remove the feature being measured. A common practical rule is to select bandwidth and sample rate with substantial margin over the highest frequency of interest; the exact requirement depends on waveform rise time, probe response, and measurement objectives.

Logic analyzers

Logic analyzers capture digital transitions and decode protocols such as UART, SPI, I²C, CAN, and USB-related signals. They are excellent for checking bit order, framing, chip-select timing, missing acknowledgements, and bus contention. They generally provide less information about analogue voltage quality than an oscilloscope. A signal may decode correctly while having poor noise margin or excessive ringing.

Debuggers and trace tools

A source-level debugger permits breakpoints, single stepping, register inspection, watchpoints, and memory examination. Halting a real-time controller can change the fault: timers may stop, watchdogs may expire, and communication buffers may overflow. Non-intrusive trace, instruction trace, event trace, and timestamped instrumentation are preferable for timing-sensitive problems.

Trace records should include a monotonic timestamp, event identifier, relevant state, and sequence number. Logging must be bounded; excessive logging can alter scheduling and exhaust memory. The design should define behaviour when the log buffer is full.

Measurement discipline

Every measurement should record the test article identity, firmware version, supply conditions, probe or analyzer configuration, sampling settings, trigger condition, environmental conditions, and interpretation. “It looked correct on the scope” is not reproducible evidence without these details.

4. Fault analysis and diagnostics

A diagnostic architecture detects abnormal conditions, identifies their likely source, and places the system in a controlled state. It commonly includes:

• range, rate-of-change, and plausibility checks for sensors;
• communication CRCs, sequence counters, and timeouts;
• watchdog timers and independent clock supervision;
• memory tests, stack monitoring, and brownout detection;
• actuator feedback and command-versus-response comparison;
• redundant or diverse measurements for high-consequence functions;
• latched fault records containing code, timestamp, operating state, and reset cause.

Diagnostics must distinguish a transient anomaly from a persistent fault. A debounce or consecutive-sample rule can prevent nuisance trips, but excessive filtering delays detection. The delay must be compatible with the hazard response time.

A useful diagnostic state machine contains states such as healthy, suspect, degraded, shutdown, and recovery inhibited. Transitions should be explicit. Automatic recovery is appropriate only when it is safe, bounded, and observable; repeated restart attempts can create a hazardous oscillation.

Fault injection

Fault injection deliberately introduces faults such as disconnected sensors, stuck-at values, corrupted packets, delayed messages, excessive CPU load, low supply voltage, or blocked actuator motion. The test should define the injected fault, expected detection time, diagnostic indication, system response, and restoration procedure. Injection into a safety-related circuit requires controlled energy, authorization, and an independent means of returning to a safe state.

5. FMEA and fault-tree reasoning

FMEA proceeds from component or function to possible failure modes and their effects. A compact FMEA record includes the function, failure mode, local effect, end effect, cause, existing controls, detection method, and recommended action. Some organizations assign severity, occurrence, and detection rankings. If a risk-priority number is used, its ranking scales and limitations must be explicitly defined; multiplying ordinal scores does not produce a physically meaningful probability.

Fault-tree analysis proceeds in the opposite direction. It begins with a top event, such as “unexpected motor movement,” and decomposes contributing events using AND and OR logic. An OR gate means any input can produce the output; an AND gate means all required inputs must occur. The method exposes common-cause failures and missing independence assumptions.

For example, safe shutdown may fail if either the software shutdown path fails or the power-interruption path fails. If both paths are required to fail before the top event occurs, an AND relationship may apply—but only if the paths are genuinely independent. Sharing a power supply, clock, connector, or software component can invalidate that assumption.

6. Reliability assessment

Reliability is the probability that an item performs its required function for a stated time under stated conditions. Let R(t) denote reliability at time t and F(t) the probability of failure by time t. For a continuous lifetime distribution:

The failure rate or hazard rate is:

where f(t) is the lifetime probability density and R(t) is nonzero. For a constant failure rate λ, the exponential model gives:

Here λ has units of inverse time and the model assumes a constant hazard rate, which is often a reasonable approximation for random-useful-life failures but not for infant mortality or wear-out.

For n statistically independent, identical components connected in series, the system works only if every component works:

For independent parallel redundancy in which the system works if at least one of n components works:

These equations assume independence and correct definition of the system success criterion. Common-cause failures, imperfect switching, shared power, and diagnostic coverage reduce the benefit of redundancy.

The mean time to failure for an exponential model is:

MTTF is not automatically the same as service life, warranty duration, or mean time between failures. Repairable systems require availability analysis, including repair time and maintenance logistics.

Worked reliability example

Two independent sensors are used in parallel, and the function is available if either sensor operates. Suppose each sensor has reliability 0.98 over the mission interval. Then:

The result, 0.9996, depends on independence and on the controller detecting and rejecting a failed sensor. If both sensors share the same connector or environmental exposure, the calculated value may be optimistic.

7. Safety engineering and risk control

Safety begins with hazard identification. A hazard is a source of potential harm; risk combines the likelihood of a hazardous event with its consequence. A qualitative risk matrix can prioritize action, but risk categories must be defined by the organization and application. Risk reduction should follow a hierarchy:

1. eliminate the hazard where practicable;
2. use inherently safer design and passive limitations;
3. add engineered protective measures;
4. add information, procedures, training, and personal protective equipment.

Software warnings alone are weaker than an independent energy-limiting mechanism. A microcontroller emergency stop should be analysed for single-point failures, stuck outputs, reset behaviour, power loss, and recovery after the emergency condition.

Functional safety focuses on ensuring that safety-related functions respond correctly to faults. The applicable standard depends on the sector—for example, machinery, automotive, process control, or household equipment. Standards differ in terminology and required evidence, so a project must identify the governing standard early rather than copying isolated practices.

Safety integrity claims require a safety lifecycle: hazard analysis, safety requirements, architecture, implementation controls, verification, validation, configuration management, competence, change control, and field feedback. A safety case is a structured argument supported by evidence. It should state assumptions and residual risks rather than claiming that the system is “fail-safe” without qualification.

8. Illustrative case: servo-driven access gate

Background

Consider an illustrative university laboratory access gate driven by a brushless motor and controlled by a microcontroller. The gate includes position sensors, a current sensor, a wireless command interface, an emergency-stop circuit, and a mechanical obstruction sensor.

Problem

During testing, the gate occasionally moved after a communication dropout. A second issue was observed: a position sensor intermittently reported a valid but incorrect angle because its data line was disturbed by motor switching.

Analysis

The team created a fault tree for “unexpected gate movement.” Contributors included stale wireless commands, a firmware timeout defect, a corrupted sensor value, a reset during active drive, and welded power-switch contacts. Oscilloscope measurements showed that the sensor supply dipped during motor-current transients. A logic analyzer demonstrated that the communication task could be delayed by a high-priority diagnostic transfer.

The FMEA identified high severity for unexpected movement, moderate occurrence, and inadequate detection for a plausible-but-wrong position value. The analysis also revealed that the emergency-stop input and motor-enable output shared a software path, creating a common failure mode.

Approach and solution

The design was changed so that:

• commands contained sequence counters and expiration times;
• communication timeout forced a zero torque request;
• sensor values were checked for range, rate, and agreement with an independent limit switch;
• the motor-enable path required a healthy heartbeat and was interrupted by a separate hardware safety circuit;
• supply decoupling and PCB routing were improved;
• HIL tests injected stale commands, corrupted positions, resets, and sensor disconnections;
• regression tests were run for every firmware release;
• an event log recorded reset cause, command age, sensor validity, and safety-state transitions.

Outcome

The revised tests showed that stale commands were rejected, implausible sensor data caused controlled stopping, and loss of the communication task removed motor enable. Physical validation then confirmed the response under representative load, supply variation, and obstruction conditions. The evidence package included requirements, test procedures, captured waveforms, HIL reports, FMEA revisions, code revision identifiers, and independent review records.

Learning points

The case demonstrates that a communication timeout is not sufficient if a corrupted sensor value remains plausible. It also shows why software diagnostics, electrical design, timing measurement, fault injection, and independent safety mechanisms must be considered together.

9. Quality of evidence and professional practice

A defensible test record identifies the requirement, test objective, preconditions, equipment, configuration, inputs, expected result, observed result, pass/fail decision, date, operator, and artifact location. Requirements and test results should be under configuration control. Calibration status and measurement uncertainty matter when acceptance limits are narrow.

Defects should be reported with steps to reproduce, actual and expected behaviour, severity, affected versions, logs, and safety implications. A change-impact analysis should determine which tests must be repeated. Independent review is especially valuable for safety requirements, fault reactions, boundary conditions, and claims of diagnostic coverage.

Artificial intelligence can assist with test generation, log clustering, anomaly detection, and code review, but generated tests and interpretations require human verification. Training data may omit rare faults, and an AI system may recommend a plausible but unsafe conclusion. Safety evidence must identify tool limitations, approval responsibility, and the authoritative source of each decision.

Challenges and limitations

Testing is constrained by the state-space explosion of concurrent firmware, the difficulty of reproducing electromagnetic interference, model inaccuracies in HIL, probe loading, uncertain field environments, and changing requirements. Reliability predictions can be misleading when failure-rate data are sparse or when laboratory conditions do not represent use. Redundancy can introduce common-cause failures and diagnostic complexity. Safety mechanisms may themselves create hazards if they stop a machine abruptly or prevent controlled braking.

The correct response is not to claim certainty, but to combine complementary methods, state assumptions explicitly, test realistic boundaries, monitor field behaviour, and update the risk and reliability assessment when evidence changes.

Chapter Summary

Verification establishes that implementation satisfies specified requirements, while validation establishes suitability for intended operation. Unit, integration, HIL, and regression testing provide progressively broader evidence. Oscilloscopes reveal analogue and timing behaviour; logic analyzers decode digital transactions; debuggers and trace tools expose software state while requiring awareness of measurement intrusion.

Fault analysis links faults to errors and failures. Diagnostics use plausibility checks, timeouts, watchdogs, feedback, memory checks, and controlled state machines. FMEA examines failure modes from the component upward, whereas fault trees decompose a hazardous top event downward. Reliability equations require explicit assumptions about distributions and independence. Safety engineering combines hazard analysis, protective architecture, verification, validation, configuration control, and documented safety arguments. High-quality evidence is reproducible, traceable, reviewed, and honest about limitations.

Key Terms

Verification: Demonstrating that implementation satisfies specified requirements.

Validation: Demonstrating that the system meets its intended operational need.

Unit test: Test of an isolated software or hardware unit.

Integration test: Test of interactions among integrated units or interfaces.

Hardware-in-the-loop: Testing a real controller against a real-time simulated plant.

Regression test: Repeated test used to detect unintended effects of change.

Fault: Abnormal condition or defect.

Error: Incorrect internal state caused by a fault.

Failure: Loss of a required system function.

Diagnostic coverage: Fraction of relevant faults detected by diagnostic measures.

FMEA: Structured analysis of failure modes, causes, and effects.

Fault tree: Deductive logic model of contributors to a top event.

Reliability: Probability of performing a required function for a stated time and condition.

Failure rate: Rate or hazard of failure conditional on survival.

Redundancy: Multiple elements providing a function or protection.

Common-cause failure: A shared cause that defeats supposedly independent elements.

Functional safety: Safety achieved through correct response to dangerous faults.

Traceability: Ability to link requirements, implementation, tests, and evidence.

Fault injection: Deliberate introduction of a fault to evaluate detection and response.

Safety case: Structured safety argument supported by documented evidence.

Review Questions

Short Answer Questions

1. Distinguish verification from validation.
2. Why can debugger breakpoints alter a real-time fault?
3. State two advantages and one limitation of HIL testing.
4. What information should a reproducible measurement record contain?
5. Why may redundancy fail to provide its expected reliability improvement?

Long Answer Questions

1. Explain how unit, integration, HIL, and regression testing can be organized into one embedded-systems assurance process.
2. Compare an oscilloscope, logic analyzer, debugger, and trace tool for diagnosing a communication-related fault.
3. Describe how FMEA and fault-tree analysis complement each other in safety engineering.
4. Explain the assumptions and limitations of the exponential reliability model.
5. Develop a diagnostic architecture for a microcontroller-controlled actuator, including detection, fault states, logging, and recovery.

Analytical / Application Questions

1. A controller occasionally commands an actuator after communication is lost. Propose a test and fault-injection plan that distinguishes stale data, scheduler delay, reset behaviour, and output-driver failure.
2. Two independent sensors each have mission reliability 0.97, and the function is available if either is valid. Calculate the ideal parallel reliability and discuss at least three reasons why the field reliability may be lower.
3. Construct a concise FMEA for a motorized laboratory gate containing a position sensor, wireless interface, watchdog, and emergency-stop circuit. Identify the highest-priority design improvements and justify them.

Multiple Choice Questions with Answers

Question 1

Which activity most directly tests interactions between a peripheral driver and an application task?

A. Unit testing

B. Integration testing

C. Visual inspection

D. Reliability prediction

Correct Answer: B. Integration testing

Question 2

Which instrument is most suitable for observing ringing on a microcontroller power rail?

A. Oscilloscope

B. Logic analyzer

C. Compiler

D. Source-code formatter

Correct Answer: A. Oscilloscope

Question 3

In HIL testing, what normally remains real while the plant is simulated in real time?

A. Only the user manual

B. The production controller and its firmware

C. Only the mechanical enclosure

D. The laboratory power socket

Correct Answer: B. The production controller and its firmware

Question 4

For a constant failure rate λ, which expression represents exponential reliability?

A. R(t) = 1 − λt

B. R(t) = λt

C. R(t) = e^(−λt)

D. R(t) = e^(λt)

Correct Answer: C. R(t) = e^(−λt)

Question 5

A fault tree begins its analysis with which item?

A. A top event

B. A source-code comment

C. A calibration certificate

D. A unit-test assertion

Correct Answer: A. A top event

Question 6

Which feature helps detect a stale communication command?

A. A sequence counter or expiration time

B. A larger enclosure

C. A decorative status LED only

D. A slower oscillator without monitoring

Correct Answer: A. A sequence counter or expiration time

Question 7

If two independent components each have reliability 0.98 and either may provide the function, the ideal parallel reliability is:

A. 0.9604

B. 0.9800

C. 0.9996

D. 1.9600

Correct Answer: C. 0.9996

Question 8

Why is code coverage not proof of software correctness?

A. It measures only hardware temperature

B. Executed code may still implement the wrong requirement

C. It cannot be recorded

D. It always excludes branch execution

Correct Answer: B. Executed code may still implement the wrong requirement

Question 9

What is a principal limitation of multiplying ordinal FMEA rankings into a risk-priority number?

A. The result is always a probability

B. Ordinal scores do not necessarily have meaningful arithmetic spacing

C. It prevents all testing

D. It measures supply voltage directly

Correct Answer: B. Ordinal scores do not necessarily have meaningful arithmetic spacing

Question 10

Which response is generally strongest in a safety architecture?

A. A warning message alone for a severe hazard

B. An independent engineered protective measure

C. Deleting the fault record

D. Disabling the watchdog permanently

Correct Answer: B. An independent engineered protective measure