Debugging, Testing and Quality Assurance

Introduction

Embedded systems for the Internet of Things (IoT) combine software, electronic hardware, sensors, actuators, communication networks, cloud services and physical processes. A defect in any layer can produce an incorrect measurement, unsafe actuator operation, lost message, battery drain or security breach. Because an IoT device often operates remotely and continuously, quality cannot be added at the end of development. It must be engineered throughout the product life cycle.

Debugging seeks to locate and correct the cause of an observed failure. Testing provides controlled evidence that specified behaviour is present and unwanted behaviour is absent. Quality assurance (QA) is broader: it establishes processes, standards, reviews and measurements that reduce the probability of defects and provide confidence that the system is fit for use.

This chapter presents practical methods for debugging embedded IoT systems, including debuggers, logic analysers, oscilloscopes and network tools. It also explains unit, integration, hardware-in-the-loop (HIL), performance and security testing; fault handling and logging; requirements traceability; the V-model; and acceptance criteria.

Learning Objectives

After studying this chapter, you should be able to:

• distinguish debugging, testing, verification, validation and quality assurance;
• use a source-level debugger, logic analyser, oscilloscope and network analyser systematically;
• design unit, integration, HIL, performance and security tests;
• implement suitable fault handling, watchdog behaviour and diagnostic logging;
• construct a requirements-to-test traceability matrix;
• apply the V-model to embedded IoT development;
• write measurable acceptance criteria and evaluate test evidence;
• analyse limitations, safety, privacy and ethical issues in IoT quality assurance.

1. Quality in an Embedded IoT System

Quality has several dimensions. Functional correctness means that the device performs its specified functions. Reliability concerns continued operation over time. Availability concerns whether the service is usable when required. Performance includes timing, throughput, energy consumption and responsiveness. Safety concerns preventing unacceptable physical harm. Security protects confidentiality, integrity and availability. Maintainability concerns diagnosis, repair, updating and future modification.

These properties interact. Encrypting every message may improve security but increase CPU time and energy consumption. Aggressive power saving may reduce availability. A retry mechanism may improve delivery but create duplicate commands unless messages are designed to be idempotent. Testing must therefore evaluate the complete system rather than isolated features only.

Important distinctions are:

• Verification: “Are we building the product right?” It checks conformance to specifications, designs and coding rules.
• Validation: “Are we building the right product?” It checks whether the system satisfies user, operational and environmental needs.
• Testing: Executing or otherwise evaluating a system to reveal defects and provide evidence.
• Debugging: Investigating failures and changing the system to remove their causes.
• Quality control: Inspecting and measuring a product or artefact.
• Quality assurance: Planned processes that prevent defects and improve confidence in development outcomes.

A test can show that a defect exists, but a passing test does not prove that no defect exists. Confidence depends on requirements coverage, test quality, environmental realism, independence of review and the limits of the test oracle—the rule used to decide whether an outcome is correct.

2. The V-Model and Traceability

The V-model relates development activities on the left side to corresponding test activities on the right side. User and system requirements are developed first; architecture and detailed design then decompose them into implementable components. Coding occurs at the bottom. Integration, system and acceptance tests proceed upward and are linked to earlier specifications.

A typical correspondence is:

The V-model is not necessarily a rigid one-pass sequence. IoT projects commonly use iterative development, but the principle of planned verification remains valuable. Each requirement should have an owner, a verification method, a test or inspection procedure, and an objective pass condition.

Requirements Traceability Matrix

A traceability matrix connects requirements to design elements, source code, tests, defects and release evidence. For example:

Traceability is not merely documentation. It reveals untested requirements, duplicated tests and changes whose impact has not been assessed. A requirement that cannot be tested objectively should be clarified before implementation.

3. A Systematic Debugging Method

Unstructured experimentation often changes several variables at once and makes diagnosis difficult. A disciplined debugging process is:

1. Reproduce the failure. Record firmware version, hardware revision, configuration, environmental conditions and exact steps.
2. Characterise the symptom. Determine frequency, timing, affected devices, boundary conditions and whether the failure is deterministic.
3. Form a hypothesis. Examples include buffer overrun, race condition, clock mismatch, electrical noise, incorrect endian conversion or expired credential.
4. Collect discriminating evidence. Choose an observation that distinguishes competing hypotheses.
5. Localise the fault. Divide the system into hardware, firmware, driver, protocol, network and cloud boundaries.
6. Apply the smallest controlled change. Change one relevant factor and repeat the test.
7. Correct the root cause. Avoid merely hiding the symptom.
8. Add a regression test. The failure must not return unnoticed.
9. Document the result. Record cause, correction, evidence and affected versions.

A useful defect report contains an identifier, title, severity, priority, environment, reproduction steps, expected result, actual result, logs, waveform or packet capture, suspected component and resolution. Severity describes impact; priority describes how urgently the organisation should address it. These are not identical: a rare safety defect may have high severity and immediate priority, while a frequent cosmetic defect may have high priority for a scheduled release but low safety impact.

Source-Level Debuggers

A hardware debugger connects through interfaces such as JTAG or SWD and permits breakpoints, single-stepping, register inspection, memory examination and watchpoints. A debugger is especially useful for examining program flow, interrupt state, stack contents and peripheral registers.

Important precautions include:

• Breakpoints can alter timing and conceal race conditions.
• Halting a processor may leave communication peripherals or watchdogs in a different state.
• Reading a status register may clear a flag on some microcontrollers.
• Optimisation can make source-level variables appear missing or reordered.
• A debugger should not be the only evidence for real-time faults; use trace, GPIO markers or external instruments as well.

When investigating an interrupt or real-time defect, inspect interrupt enable and pending bits, priority configuration, shared data protection, stack usage and whether an interrupt service routine performs excessive work. Long operations should normally be deferred to a task or main-loop context.

Logic Analysers

A logic analyser samples digital signals and displays transitions over time. It is appropriate for checking SPI, I²C, UART, CAN and GPIO activity. It can reveal incorrect chip-select timing, missing acknowledgements, wrong baud rate, bus contention and unexpected reset sequences.

The sampling rate must be sufficiently high to distinguish transitions. A practical engineering rule is to use several samples per shortest pulse; the exact requirement depends on edge speed, protocol decoder quality and measurement uncertainty. A logic analyser records logic thresholds, not analogue voltage quality. It may therefore miss slow edges, ringing, overshoot or marginal voltage levels.

Oscilloscopes

An oscilloscope displays voltage as a function of time and is essential for analogue sensors, power integrity, clock signals, reset lines and signal quality. Probe grounding, bandwidth, attenuation and loading affect the measurement. A long ground lead can introduce apparent ringing. Differential or spring-ground probing may be required for fast signals.

Use trigger conditions to capture intermittent faults. Measure supply droop during radio transmission, reset timing, oscillator stability and actuator transients. If a sensor value is wrong, an oscilloscope can determine whether the problem originates in the sensor signal, reference voltage, ADC timing or software conversion.

Network Tools

IoT debugging often requires packet capture and protocol analysis. Tools may inspect Ethernet, Wi-Fi, Bluetooth Low Energy, IPv6, TCP, UDP, DNS, TLS, MQTT, CoAP or HTTP traffic. Useful observations include connection establishment, retransmissions, packet loss, latency, message ordering, keep-alive behaviour, certificate validation and payload encoding.

Capture data must be interpreted carefully. A missing packet in one capture point does not prove that the sender failed; it may have been filtered, captured on another interface or lost after capture. Timestamps from different devices may not be synchronised. Packet payloads can contain personal or confidential information and must be protected.

4. Test Levels and Test Design

Unit Testing

A unit is a small testable element, such as a CRC function, sensor conversion routine, queue or state machine. Unit tests should isolate dependencies using stubs, mocks or simulated interfaces. Test normal values, boundary values, invalid inputs, overflow conditions, timeouts and error paths.

For an ADC conversion, for example, test zero-scale input, full-scale input, values near thresholds, calibration coefficients and out-of-range readings. A unit test should not depend on network availability or physical sensor timing unless those dependencies are specifically being tested.

Code coverage metrics can identify unexecuted statements, branches or conditions. Coverage is evidence of execution, not proof of correctness. A test suite can achieve high statement coverage while failing to test meaningful combinations or incorrect expected results.

Integration Testing

Integration testing checks interfaces between modules: driver and sensor, scheduler and actuator, device and gateway, or gateway and cloud service. Interface defects commonly involve units, byte order, timing, ownership of buffers, error codes, authentication state and assumptions about retry behaviour.

A useful integration test specifies the stimulus, interface state, expected messages or signals, timing limits, environmental conditions and cleanup procedure. Fault injection should include disconnected sensors, malformed packets, delayed acknowledgements, duplicate messages and exhausted storage.

System and End-to-End Testing

System testing evaluates the assembled device in realistic configurations. End-to-end tests follow a complete path, such as sensor sampling → local validation → encrypted publication → cloud rule → mobile notification. These tests are slower and more difficult to diagnose than unit tests, so they should complement rather than replace lower-level tests.

Hardware-in-the-Loop Testing

HIL testing connects the real controller to a simulator that produces electrical signals or protocol responses representing the physical environment. The controller executes real firmware, while the plant, sensor or network conditions are generated reproducibly.

Examples include simulating temperature ramps, motor loads, battery voltage decline, GPS movement, packet loss and sensor disconnection. HIL is valuable for rare, dangerous or expensive conditions. Its limitations include model inaccuracy, simulator latency and the possibility that the model omits physical effects. HIL results should be complemented by tests on representative real hardware.

Performance and Resource Testing

Performance testing measures response time, throughput, jitter, memory use, CPU utilisation, energy consumption, boot time and communication reliability. Define workload and measurement conditions before collecting results.

For a periodic task with intended period T, timing error at an observation i can be represented as:

Here, t_i is the actual time of observation i, t_0 is the reference start time, T is the intended period, and e_i is the timing error. All times must use the same unit and clock basis. This expression is useful for periodic scheduling; it does not by itself measure long-term clock drift or execution-time variation.

Mean response time for n observations is:

where t_i is an individual response time and n is the number of observations. The mean should be accompanied by percentiles or a maximum when occasional long delays matter. A system with a low mean but unacceptable 99th-percentile latency may fail a real-time requirement.

For independent stages with success probabilities p₁, p₂, …, pₙ, the probability that all stages succeed is:

This multiplication applies only when the stage events are appropriately independent. In real systems, failures may be correlated, so the result is an estimate rather than a universal reliability law.

Energy testing requires a defined operating cycle. If current is approximately constant over intervals, charge is estimated by:

where Iₖ is current in amperes and ∆tₖ is time in seconds, producing coulombs. Average current over a cycle of duration T is I_avg = Q/T. Battery-life estimates additionally require battery capacity, discharge conditions, conversion losses, temperature and ageing; nominal capacity alone is insufficient.

Security Testing

Security testing should begin with threat modelling. Identify assets, trust boundaries, actors, attack surfaces and consequences. Test authentication, authorisation, secure boot, firmware update, key storage, certificate validation, random-number generation, replay resistance, input validation, debug-port control and failure behaviour.

Typical tests include malformed packet fuzzing, invalid credentials, expired certificates, replayed commands, privilege escalation attempts, downgrade attempts and interruption during firmware update. Tools may include static analysis, dependency scanning, fuzzers, protocol analysers and controlled penetration testing.

Security testing must be authorised and isolated from production systems. Test credentials and captured data must be protected. A vulnerability should be assessed by impact and exploitability, remediated or mitigated, and retested. Security is not achieved by hiding an interface or relying on an undocumented protocol.

5. Fault Handling, Recovery and Logging

IoT devices operate in noisy and incomplete environments. Fault handling should classify failures and define safe, observable responses. Common classes are transient faults, persistent faults, configuration faults, communication faults, sensor plausibility faults and security faults.

A robust pattern is:

1. detect the fault;
2. record enough diagnostic context;
3. isolate the affected function;
4. attempt a bounded recovery if safe;
5. retry with backoff where appropriate;
6. enter a defined degraded or safe state if recovery fails;
7. notify an operator or service;
8. preserve evidence for later analysis.

Retries require care. Unbounded retries can consume energy and prevent other tasks from running. A command that controls a physical actuator should include a unique identifier, expiry time and safe duplicate-handling policy. Watchdogs can recover from software hangs, but a watchdog does not correct the underlying defect and may complicate diagnosis if reset causes are not logged.

A useful log record may contain timestamp, device identifier, firmware version, severity, event code, state, error code, relevant measurement and sequence number. Logs should be structured rather than free-form where automated analysis is required. Rate limiting prevents a repeated fault from exhausting storage or bandwidth. Sensitive values, keys and personal data should not be logged unnecessarily; masking and access control are essential.

Time presents a special challenge. A device may have an unsynchronised real-time clock, reboot-relative time and network time. Logs should identify the clock source and synchronisation status. For event ordering, a monotonic sequence or monotonic timer may be more reliable than wall-clock time.

6. Acceptance Criteria and Test Evidence

Acceptance criteria convert expectations into observable conditions. Good criteria are specific, measurable, achievable, relevant and time-bounded where appropriate. They define input conditions, operating environment, pass threshold, measurement method and handling of exceptions.

Weak criterion: “The device communicates reliably.”

Stronger criterion: “Under the stated radio test environment, the device publishes at least 99 of 100 valid telemetry messages within 5 seconds, without duplicate sequence numbers, during a 30-minute test.”

Acceptance criteria should distinguish typical performance from guaranteed limits. If a requirement says “within 500 ms,” specify whether this means every observation, a maximum, a percentile or a confidence bound. Test reports should preserve raw data, instrument configuration, calibration status, software and hardware versions, test operator, date, deviations and result.

A release decision should consider failed tests, unresolved defects, known limitations, risk acceptance and change impact—not simply the percentage of tests passed. Independent review is particularly important for safety and security claims.

7. Illustrative Case Study: Smart Cold-Storage Monitor

Background

A fictional company develops a battery-powered IoT monitor for a medicine refrigerator. It measures temperature every minute, stores readings locally, publishes them securely through Wi-Fi and activates a local alarm if temperature remains outside the permitted range.

Problem

During pilot use, some devices reported delayed alarms. Others rebooted after Wi-Fi reconnection. Initial logs showed only “communication error,” so engineers could not determine whether the cause was firmware, power supply or network behaviour.

Analysis

The team created a traceability matrix linking alarm, sampling, reconnection and safety requirements to tests. A logic analyser was attached to the sensor bus, an oscilloscope monitored the 3.3 V rail, and a packet capture recorded reconnection traffic. GPIO markers identified sampling, message transmission and alarm activation times.

The oscilloscope showed a supply dip when the radio transmitted. The debugger then revealed that the brownout reset occurred before the reset cause was saved. Network traces also showed that the client retransmitted a message after reconnecting, but the cloud service treated the duplicate as a new event.

Approach and Solution

The team changed the power decoupling and regulator configuration, then verified the supply under a worst-case radio workload. Firmware stored reset cause and a short crash record in retained memory before normal startup. The telemetry protocol gained persistent sequence numbers and duplicate detection. Alarm evaluation was separated from network publication so that a network outage could not delay a local safety response.

Unit tests covered threshold comparison and duplicate handling. Integration tests covered sensor failure and reconnect sequences. HIL testing simulated temperature excursions, Wi-Fi loss, power dips and repeated resets. Acceptance testing required alarm activation within the specified interval and no duplicate cloud events in a defined test set.

Outcome

The revised system passed the acceptance tests and provided actionable logs. The most important improvement was architectural: local safety behaviour no longer depended on successful cloud communication. The team also discovered that a passing end-to-end communication test would not have exposed the power-integrity problem without electrical measurement.

Learning Points

• Combine software, electrical and network evidence.
• Preserve reset and fault context before recovery.
• Make safety-critical local functions independent of optional cloud services.
• Define duplicate, timeout and reconnect behaviour explicitly.
• Link every corrective change to regression tests and requirements.

8. Challenges, Limitations and Professional Practice

Testing cannot reproduce every combination of temperature, interference, component ageing, user action, network condition and adversarial behaviour. Test environments may be cleaner than deployment environments. Simulators may omit analogue or mechanical effects. Instruments introduce loading, quantisation and timing uncertainty. Coverage metrics can create a false sense of completion when tests are poorly designed.

Quality practices should therefore use risk-based prioritisation. Safety functions, security boundaries, external interfaces, complex state transitions and recently changed code deserve particular attention. Reviews, static analysis, automated regression testing, reproducible builds and controlled configuration reduce avoidable defects.

Ethical and governance responsibilities are significant. Test data may expose location, health, occupancy or operational information. Engineers must obtain permission for security tests, protect credentials, minimise retained data and report vulnerabilities responsibly. Devices deployed in homes, workplaces or public infrastructure should fail safely and provide understandable status information. Regulatory obligations may apply to radio equipment, electrical safety, medical applications, privacy and cybersecurity. Compliance certification is valuable evidence but does not replace engineering judgement or product validation.

Contemporary development increasingly uses continuous integration, automated hardware test farms, remote device telemetry, fuzzing, software bills of materials and machine-assisted log analysis. AI-based anomaly detection can identify unusual patterns, but it may produce false positives, inherit biased training data or conceal the reasoning behind an alert. It should support, not replace, validated test oracles, human review and secure data governance.

Best-Practice Checklist

• Write testable requirements before implementation.
• Maintain bidirectional traceability from requirement to evidence and back.
• Reproduce failures with controlled versions and documented conditions.
• Use the appropriate instrument: debugger for state, analyser for digital protocol timing, oscilloscope for analogue and power behaviour, and network tools for communication evidence.
• Test normal, boundary, invalid, delayed, duplicated and missing inputs.
• Include recovery, reset, power-loss and update-interruption scenarios.
• Measure worst-case or percentile performance when averages are inadequate.
• Protect logs, credentials, personal data and test environments.
• Automate repeatable regression tests, while retaining targeted exploratory testing.
• Require explicit acceptance criteria and independent review for high-risk claims.

Chapter Summary

Debugging, testing and quality assurance are complementary disciplines in embedded IoT engineering. Debugging identifies and removes causes of observed failures; testing produces evidence about specified behaviour; and QA establishes processes that make reliable outcomes more likely. The V-model connects requirements and design with unit, integration, system and acceptance testing, while traceability ensures that each requirement has corresponding evidence.

Source-level debuggers reveal program and register state, logic analysers reveal digital transitions and protocol timing, oscilloscopes reveal analogue, clock and power behaviour, and network tools reveal communication operation. Effective test programmes combine unit, integration, HIL, performance, end-to-end and security tests. Fault handling must provide bounded recovery, safe states and useful logs. Performance measurements require clearly defined workloads and valid metrics rather than reliance on averages alone. Acceptance criteria should be observable and measurable, and ethical practice requires authorised security testing, privacy protection and responsible governance.

Key Terms

Debugging: Systematic investigation and correction of the causes of failures.

Testing: Controlled evaluation intended to reveal defects and provide evidence of behaviour.

Quality assurance: Planned processes and practices that prevent defects and increase confidence in outcomes.

Verification: Checking conformity with specified requirements and design.

Validation: Checking fitness for user and operational needs.

V-model: A development model linking specification and design activities to corresponding test levels.

Traceability: The ability to relate requirements to design, implementation, tests and evidence.

Unit test: A test of a small, isolated software element.

Integration test: A test of interactions between modules or subsystems.

Hardware-in-the-loop: Testing real controller hardware with a simulated physical environment.

Logic analyser: An instrument that samples and displays digital signals.

Oscilloscope: An instrument that displays electrical signal amplitude against time.

Watchdog: A timer that can reset a system that fails to service it within a defined interval.

Fault injection: Deliberately introducing failures or abnormal conditions to evaluate response.

Regression test: A repeated test used to detect defects reintroduced by changes.

Acceptance criterion: An objective condition that must be satisfied for approval.

Test oracle: A rule or reference used to determine whether an observed result is correct.

Fuzzing: Automated testing using numerous unexpected, malformed or varied inputs.

Mean response time: The arithmetic average of measured response times.

Safe state: A defined condition that minimises harm when normal operation cannot continue.

Review Questions

Short Answer Questions

1. Distinguish verification from validation in an IoT project.
2. State two limitations of using a source-level debugger for real-time faults.
3. What information should a useful embedded-system fault log contain?
4. Explain the purpose of a requirements traceability matrix.
5. Why should security testing be authorised and isolated from production systems?

Long Answer Questions

1. Explain how a logic analyser, oscilloscope and network tool provide different forms of evidence during diagnosis of an IoT device.
2. Describe a complete V-model-based testing strategy for a sensor node, from requirements through acceptance testing.
3. Discuss unit, integration, HIL and end-to-end testing, including the strengths and limitations of each.
4. Explain how retry policies, watchdogs, safe states and logging should be designed for fault-tolerant IoT operation.
5. Discuss why high code coverage does not guarantee software quality, and identify additional evidence required for release.

Analytical / Application Questions

1. A device resets whenever its radio transmits, but software logs contain no reset event. Design a step-by-step investigation using a debugger and oscilloscope, and identify likely corrective actions.
2. An IoT alarm system has a low average response time but occasional delays that exceed its safety limit. Which measurements and tests should be performed, and how should acceptance criteria be rewritten?
3. Design a test plan for a secure actuator command received over MQTT. Include authentication, authorisation, replay, duplicate delivery, network loss, malformed input, logging and safe-state behaviour.

Multiple Choice Questions with Answers

Question 1

Which activity primarily seeks to identify and remove the cause of an observed failure?

A. Validation

B. Debugging

C. Acceptance testing

D. Requirements elicitation

Correct Answer: B. Debugging

Question 2

Which instrument is most suitable for observing voltage droop during a radio transmission?

A. Oscilloscope

B. Logic analyser

C. Source-code editor

D. Packet decoder

Correct Answer: A. Oscilloscope

Question 3

In the V-model, system requirements are most directly associated with which test level?

A. Unit testing

B. System testing

C. Compiler testing

D. Code formatting

Correct Answer: B. System testing

Question 4

What is a principal purpose of a requirements traceability matrix?

A. To increase processor clock speed

B. To replace all integration tests

C. To link requirements to implementation and test evidence

D. To encrypt sensor readings

Correct Answer: C. To link requirements to implementation and test evidence

Question 5

Which statement about code coverage is correct?

A. High coverage proves that the software is defect-free

B. Coverage measures whether executed tests had correct expected results

C. Coverage can identify unexecuted areas but does not prove correctness

D. Coverage is relevant only to hardware testing

Correct Answer: C. Coverage can identify unexecuted areas but does not prove correctness

Question 6

A logic analyser is particularly appropriate for investigating which problem?

A. Digital protocol timing and missing acknowledgements

B. Battery chemistry ageing only

C. Cloud user-interface colour contrast

D. Mechanical enclosure strength

Correct Answer: A. Digital protocol timing and missing acknowledgements

Question 7

Which design response best prevents an unbounded retry loop from consuming device resources?

A. Retry forever without delay

B. Use bounded retries and an appropriate backoff policy

C. Delete all diagnostic logs

D. Disable authentication after the first failure

Correct Answer: B. Use bounded retries and an appropriate backoff policy

Question 8

What is the main purpose of hardware-in-the-loop testing?

A. To test only source-code formatting

B. To connect real controller hardware to a simulated physical environment

C. To eliminate the need for acceptance criteria

D. To measure cloud advertising effectiveness

Correct Answer: B. To connect real controller hardware to a simulated physical environment

Question 9

If response times are occasionally very long, which measure is often more informative than the mean alone?

A. A relevant percentile or maximum

B. The number of source files

C. The display resolution

D. The device colour

Correct Answer: A. A relevant percentile or maximum

Question 10

Why should an actuator command include replay or duplicate-delivery protection?

A. To make packets physically larger

B. To prevent repeated delivery from causing unintended repeated actions

C. To disable all network diagnostics

D. To avoid writing acceptance criteria

Correct Answer: B. To prevent repeated delivery from causing unintended repeated actions