Development, Testing and Troubleshooting

Introduction

Firmware is the software that controls a microprocessor- or microcontroller-based system. Unlike ordinary desktop software, firmware interacts directly with electrical signals, memory devices, timers, communication buses, sensors and actuators. A program may be logically correct yet fail because of an incorrect clock source, unsuitable voltage levels, a floating input, timing violation, reset problem or defective connection. Consequently, professional development requires more than writing source code: it requires a controlled process for building, programming, observing, testing, diagnosing and documenting the complete hardware–software system.

This chapter presents a practical development and troubleshooting workflow using assemblers, compilers, simulators, debuggers, oscilloscopes, logic analyzers and in-circuit programmers. The emphasis is on diploma-level professional practice: forming testable hypotheses, collecting evidence, isolating faults and recording solutions so that they can be reproduced.

Learning Objectives

By the end of this chapter, a learner should be able to:

1. Explain the roles of assemblers, compilers, linkers, simulators and debuggers.
2. Construct a reproducible firmware build and programming process.
3. Design tests for reset, I/O, timing, memory and communication functions.
4. Use an oscilloscope and logic analyzer appropriately.
5. Distinguish hardware, firmware and integration faults.
6. Apply a systematic troubleshooting method and document the result.
7. Observe safe, ethical and configuration-control practices when working with embedded systems.

1. The Firmware Development Cycle

A typical cycle is iterative rather than strictly linear:

Figure: An iterative firmware development and troubleshooting cycle.

The cycle begins with requirements that can be measured. For example, “the motor shall stop within 100 ms of an over-temperature signal” is testable, whereas “the system shall respond quickly” is not. Source files are then assembled or compiled, linked with libraries, converted into a programming image and loaded into the target. Tests produce observations, and observations lead to a corrected design or a justified conclusion that the system meets its requirements.

A useful distinction is between a build artifact and a source file. The source is human-readable code; the artifact may be an ELF file containing symbols and debugging information, a HEX file containing address–data records, or a binary image containing raw bytes. The exact image, compiler version, configuration and source revision should be retained for every released unit.

2. Development Tools and Their Roles

2.1 Assemblers

An assembler translates assembly-language mnemonics into machine instructions. It resolves labels, interprets directives and creates relocation or listing information. Assembly gives precise control over registers, instruction timing and memory layout, but it requires detailed knowledge of the instruction set and is more difficult to maintain than a well-structured high-level program.

An assembler listing is valuable during diagnosis. It can show the address of each instruction, the generated opcode and the effect of directives. Common assembly errors include invalid addressing modes, incorrect register banks, missing symbols and branch distances that exceed the instruction’s range.

2.2 Compilers, linkers and libraries

A compiler translates a language such as C or C++ into object code. It performs lexical and syntactic analysis, type checking, optimization and code generation. The linker combines object files and libraries, assigns addresses and resolves references. A linker script or device configuration commonly defines program memory, data memory, interrupt-vector locations and reserved regions.

Compiler options affect behaviour. Optimization can remove apparently unnecessary variables, reorder instructions and make source-level single-stepping appear non-sequential. A variable shared with an interrupt service routine or hardware register may require a volatile declaration, although volatile does not by itself make a multi-byte operation atomic. For diagnosis, a low-optimization build with debugging symbols is often useful; the final release must be tested with the intended optimization settings.

Warnings should be treated as evidence, not decoration. A narrowing conversion, uninitialised variable, unreachable statement or conflicting declaration may reveal a defect before hardware testing begins.

2.3 Simulators and emulators

A simulator models processor instructions and, sometimes, peripherals without requiring the physical target. It is useful for checking arithmetic, branching, stack use, interrupt logic and memory access. A simulator may not accurately model analogue noise, oscillator start-up, electrical loading, bus contention or the exact behaviour of a peripheral.

An emulator or hardware debugger executes code on the real target while providing breakpoints, watch windows, register inspection and memory access. A halt may change timing and may prevent watchdog servicing, so a program that works under a debugger is not automatically correct in free-running operation.

2.4 Debuggers

A debugger supports controlled observation and modification of execution. Important facilities include:

• Breakpoints: halt at a program address or source line.
• Watchpoints: halt when a memory location is read or written.
• Stepping: execute one instruction, source statement or function.
• Register and memory views: inspect processor state.
• Call stack: identify the active sequence of function calls.
• Trace or real-time logging: observe execution with less intrusion than halting.

Breakpoints should be placed at decision boundaries: after initialisation, when a state changes, when an error flag is set, or immediately before a peripheral transaction. Excessive breakpoints can conceal race conditions and alter real-time behaviour.

2.5 In-circuit programmers and debug probes

An in-circuit programmer transfers firmware to the target through an interface such as SWD, JTAG, ICSP or a bootloader UART. The programmer may also erase memory, verify contents, set configuration fuses and protect the device. Before programming, confirm the target voltage, device identity, reset connection, interface pin assignment and power arrangement.

A successful “program complete” message does not prove that the application will run. Verification normally confirms that memory contents match the image; it does not confirm clock configuration, peripheral wiring or correct application logic. Never connect two active drivers to the same signal without confirming that the electrical design permits it.

3. Building a Reproducible Firmware Image

A professional build should be repeatable by another person on another workstation. Record:

• source revision or commit identifier;
• compiler, assembler and linker versions;
• compiler options and linker script;
• device variant and clock configuration;
• external libraries and their versions;
• generated image checksum or cryptographic hash;
• warnings and errors;
• programming and verification results.

A clean build is important because stale object files may hide dependencies. A typical sequence is: preprocess source, compile or assemble, link, generate a map and listing, convert to the programming format, then verify the final image. The map file can reveal excessive RAM use, an incorrectly placed interrupt vector or code overlapping a reserved memory area.

Configuration data must be separated from application logic where practical. Pin assignments, clock frequencies, communication rates and feature options should be visible, reviewed and version-controlled rather than hidden in ad hoc source edits.

4. Designing a Test Strategy

Testing should proceed from simple, observable functions toward integrated behaviour.

4.1 Test levels

Unit tests examine a small function, such as a checksum or state transition, often on a host computer or simulator. Module tests exercise a driver with a peripheral model or test fixture. Integration tests combine drivers, interrupts and hardware. System tests evaluate the complete product against its requirements. Acceptance tests demonstrate that the system is suitable for its intended use.

A test case should state the initial condition, stimulus, expected result, measurement method and pass/fail criterion. For example:

4.2 Boundary, fault and regression tests

Boundary tests use minimum, nominal and maximum permitted values. Fault tests deliberately remove a sensor, corrupt a message, hold a line active or interrupt power where safe. Regression tests repeat previously passed tests after a change. A defect is not considered fully resolved until a test reproduces the original failure, demonstrates the correction and is added to the regression suite.

Timing requirements should be stated quantitatively. If a periodic task has period T, its frequency is:

Here, f is frequency in hertz and T is period in seconds. This relation applies to a stable periodic signal. For example, a 10 ms task period gives f = 1/0.010 = 100 Hz. If an interrupt consumes 0.8 ms every 10 ms, its ideal duty ratio is:

where t_on is active time and T is the period, both in the same time units. The result is D = 0.8/10 × 100% = 8%. This calculation does not prove that deadlines are met; interrupt blocking, jitter and competing tasks must also be examined.

For asynchronous serial communication, the nominal bit time is approximately:

where R is the baud rate in bits per second. At 9,600 bit/s, t_bit ≈ 104.17 μs. The approximation assumes the configured baud rate represents the transmitted bit rate and ignores start-up and sampling details.

5. Measurement Instruments

5.1 Oscilloscope

An oscilloscope displays voltage as a function of time. It is appropriate for observing analogue and fast digital phenomena: reset rise time, clock waveform, ringing, overshoot, power-rail dips, PWM duty cycle and interrupt timing. Essential settings include volts per division, time per division, trigger source and trigger level.

Probe grounding is critical. A long ground lead can introduce inductance and display ringing that is not present at the circuit node. Use a short ground spring or differential probe when appropriate. Confirm probe attenuation in both the probe and oscilloscope settings. Exceeding the probe’s voltage or category rating is dangerous.

For a repeating waveform, frequency can be measured from its period using the relation above. Duty cycle is the proportion of a period for which a signal is in its active state. An oscilloscope can also reveal a reset line that briefly falls below the input-low threshold, even when a multimeter shows a normal average voltage.

5.2 Logic analyzer

A logic analyzer samples one or more digital channels and displays decoded transitions. It is particularly useful for SPI, I²C, UART, parallel buses and state-machine signals. It can show whether a byte was transmitted, whether an acknowledgement occurred and whether the order of events is correct.

Sampling rate and threshold matter. A sample rate that is too low can miss narrow pulses or misrepresent their timing. A protocol decoder is only as reliable as its configuration: clock polarity, clock phase, bit order, word length, baud rate and address convention must be set correctly. A decoded message should be checked against the raw waveform rather than accepted blindly.

An oscilloscope measures voltage continuously within its bandwidth; a logic analyzer generally records threshold crossings. Therefore, the logic analyzer may report a clean digital signal while an oscilloscope reveals slow edges, ringing or inadequate noise margin.

6. A Systematic Troubleshooting Method

Troubleshooting is an evidence-based process, not random component replacement.

1. Make the failure safe. Disconnect loads if overheating or overcurrent is possible.
2. Define the symptom. Record what happens, when it happens and what should happen.
3. Reproduce it. Establish the conditions and frequency of failure.
4. Check the simplest causes. Inspect power, ground, connectors, jumpers, reset and programming connections.
5. Divide the system. Test power, clock, reset, processor execution, I/O and communications separately.
6. Form a hypothesis. Predict an observation that would support or reject it.
7. Measure one variable at a time. Change one condition and record the result.
8. Localise the fault. Compare a failing signal with a known-good reference or boundary.
9. Correct and retest. Use the original reproducer and regression tests.
10. Document the result. Record cause, evidence, correction and prevention.

The fault domain can often be narrowed by a basic decision sequence. If there is no power rail, investigate the supply and protection. If power exists but reset is continuously active, investigate reset circuitry and configuration. If reset releases but no clock or instruction activity is visible, investigate clock selection, programming and device identity. If code executes but an output is wrong, inspect pin multiplexing, direction registers, timing and electrical loading. If local functions work but communication fails, inspect wiring, voltage levels, protocol settings and grounding.

A useful distinction is symptom versus root cause. “The display is blank” is a symptom. The root cause may be a disabled peripheral clock, incorrect chip-select polarity, an open ground connection or an uninitialised display controller. Replacing the display without evidence may remove the symptom temporarily while leaving the process defective.

7. Common Hardware–Software Faults

Digital inputs should not be left floating unless the design explicitly supports it. A floating node can respond unpredictably to noise and proximity. Open-drain buses such as I²C require suitable pull-up resistors and compatible voltage levels; a push-pull driver connected incorrectly to such a bus can cause contention.

When a peripheral is silent, check the entire path: application request, driver state, peripheral enable bit, pin multiplexing, physical wire, remote device power and protocol timing. Checking only the application code is insufficient.

8. Applied Case: Intermittent Sensor-Node Reset

The following is an illustrative case based on a common embedded-system investigation.

Background. A battery-powered environmental node reads a temperature sensor every second and transmits a packet over UART. The node passes a ten-minute bench test but resets intermittently when a radio transmitter is enabled.

Problem. Field logs show missing packets and a reset counter increasing. The firmware team initially suspects a buffer overflow.

Analysis. A debugger confirms that the reset handler is reached, but halting the processor prevents reproduction. A logic analyzer shows that the UART packet is correctly formed before the failure. An oscilloscope connected across the controller supply, using a short ground connection, captures a brief voltage dip coincident with radio transmission. The reset line falls shortly afterward. The reset-status register indicates a brownout rather than a software watchdog reset.

Approach and solution. Engineers first reproduce the failure with a controlled radio duty cycle. They separate the radio supply from the controller supply temporarily; the resets disappear. Inspection finds inadequate decoupling near the radio and excessive resistance in the battery path. The design is corrected with suitable local decoupling, improved power routing and a verified regulator current capability. Firmware is also changed to record reset cause in retained memory and to delay sensor initialisation until the supply is stable.

Outcome. The node completes an extended stress test, including low battery voltage and repeated radio transmissions. The test report includes supply captures, reset flags, hardware revision, firmware hash and acceptance criteria.

Learning points. The initial software hypothesis was reasonable but unsupported. The decisive evidence came from correlating the reset event with the supply waveform. A logic analyzer alone could not show the analogue voltage dip, while a debugger altered the timing sufficiently to hide it. Hardware and firmware evidence must therefore be considered together.

9. Documentation, Safety and Professional Practice

A troubleshooting record should include an identifier, date, operator, equipment, board and firmware revisions, environmental conditions, symptom, reproduction steps, measurements, hypothesis, corrective action, verification result and unresolved risks. Screenshots and waveform captures should include channel names, scales, trigger settings and probe points.

Configuration control prevents an old binary from being tested on a new board without recognition. Serial numbers, calibration status and instrument settings should be traceable. Where a product affects safety, medical equipment, transport or critical infrastructure, testing must follow applicable standards, risk controls, review procedures and change-approval requirements. Never disable a protective function merely to obtain a convenient test result without recording the risk and restoring it afterward.

Firmware can influence privacy and security. Debug ports should be controlled in production, secret keys must not be placed in publicly accessible logs, and update images should be authenticated where the threat model requires it. Test data containing personal or operationally sensitive information should be handled according to organisational policy.

Artificial-intelligence tools may assist with code explanation, test-case generation or log classification, but generated code and diagnoses require human review. An AI suggestion is not measurement evidence. Proprietary source code, credentials and confidential hardware details should not be uploaded to an unapproved service.

10. Best Practices and Limitations

The following practices substantially improve reliability:

• define measurable requirements before implementation;
• keep builds and tools version-controlled;
• enable meaningful compiler warnings;
• use assertions and diagnostic counters during development;
• expose safe test points for power, reset, clock and communication signals;
• label probe points and connector pinouts;
• test both nominal and boundary conditions;
• distinguish intrusive debugging from real-time observation;
• use the oscilloscope for analogue integrity and the logic analyzer for digital sequence analysis;
• change one variable at a time during diagnosis;
• preserve the failing image and original evidence;
• add every confirmed defect to regression testing.

No tool is complete. A simulator omits many electrical effects; a debugger changes execution timing; a logic analyzer depends on sampling and threshold settings; an oscilloscope can show a waveform without explaining its protocol meaning; and an in-circuit programmer verifies memory rather than system correctness. Reliable conclusions come from combining tools with a clear test hypothesis.

Chapter Summary

Firmware development is a controlled cycle of specification, implementation, building, programming, measurement, diagnosis and documentation. Assemblers translate assembly language, compilers generate object code, linkers create a memory image, simulators model execution, debuggers inspect live state and in-circuit programmers transfer and verify firmware. Unit, integration, system, boundary, fault and regression tests provide progressively stronger evidence.

Oscilloscopes are essential for voltage, timing, reset, clock, power and signal-integrity observations. Logic analyzers are effective for digital transactions and protocol sequences. Neither instrument replaces the other. Systematic troubleshooting begins by defining and reproducing the symptom, checking simple causes, dividing the system, forming hypotheses and measuring predicted evidence. Professional practice also requires reproducible builds, configuration control, safe instrumentation, security awareness and complete records of corrective action.

Key Terms

Assembler: Software that translates assembly-language instructions into machine code.

Compiler: Software that translates a high-level programming language into lower-level code.

Linker: Tool that combines object files, resolves symbols and assigns memory addresses.

Firmware: Software stored in or closely associated with an embedded device.

Build artifact: Generated file such as an ELF, HEX or binary image.

Simulator: Software model that executes instructions without the physical target.

Debugger: Tool that controls or observes program execution.

Breakpoint: A condition that suspends execution at a selected point.

Watchpoint: A condition that suspends execution when a memory location is accessed.

In-circuit programmer: Hardware that loads firmware into a target device while it is on a development board.

Unit test: Test of a small software function or module.

Regression test: Repeated test used to detect reintroduction of a defect.

Oscilloscope: Instrument that displays voltage versus time.

Logic analyzer: Instrument that records and decodes digital transitions.

Trigger: Measurement condition that determines when an instrument begins displaying or recording.

Brownout: Reset or malfunction caused by a supply voltage falling below an operating limit.

Pin multiplexing: Assignment of an integrated pin to a selected peripheral or GPIO function.

Root cause: Fundamental condition that produces an observed failure.

Configuration control: Management of versions and approved changes to hardware, software and tools.

Review Questions

Short Answer Questions

1. What are the principal roles of a compiler and a linker?
2. Why can a debugger hide a real-time firmware fault?
3. State one situation in which an oscilloscope is preferable to a logic analyzer.
4. What information should be retained for a reproducible firmware build?
5. Why should regression tests be repeated after a defect is corrected?

Long Answer Questions

1. Explain the complete process from source code to programmed and verified firmware.
2. Compare the capabilities and limitations of an oscilloscope and a logic analyzer.
3. Describe a systematic method for separating hardware faults from firmware faults.
4. Explain how boundary, fault-injection and integration testing complement one another.
5. Discuss the professional and safety responsibilities associated with in-circuit debugging and programming.

Analytical / Application Questions

1. A UART signal configured for 9,600 bit/s appears to have a bit time of approximately 100 μs. Identify possible causes and describe the measurements you would make.
2. A controller operates correctly under a debugger but resets during unattended operation. Develop a test plan that considers watchdog timing, power integrity, uninitialised state and debugger intrusion.
3. A sensor node reports valid packets but occasionally loses communication when an actuator starts. Explain how you would combine power-rail oscilloscope measurements, logic-analyzer captures and firmware reset diagnostics to locate the cause.

Multiple Choice Questions with Answers

Question 1

Which tool primarily translates assembly-language mnemonics into machine instructions?

A. Assembler

B. Linker

C. Logic analyzer

D. In-circuit programmer

Correct Answer: A. Assembler

Question 2

What is the principal purpose of a linker?

A. Measure reset voltage

B. Combine object files and resolve symbols

C. Decode SPI waveforms

D. Erase a circuit board

Correct Answer: B. Combine object files and resolve symbols

Question 3

Which instrument is most appropriate for detecting a brief analogue supply-voltage dip?

A. Compiler

B. Logic analyzer only

C. Oscilloscope

D. Assembler listing

Correct Answer: C. Oscilloscope

Question 4

A periodic task has a period of 10 ms. What is its frequency?

A. 10 Hz

B. 50 Hz

C. 100 Hz

D. 1,000 Hz

Correct Answer: C. 100 Hz

Question 5

At 9,600 bit/s, what is the approximate nominal bit time?

A. 10.4 μs

B. 104.2 μs

C. 960 μs

D. 9,600 μs

Correct Answer: B. 104.2 μs

Question 6

Why can a logic-analyzer protocol decoder produce a misleading result?

A. It always measures analogue voltage inaccurately

B. Its configuration may not match the bus settings

C. It cannot record digital transitions

D. It automatically changes the firmware

Correct Answer: B. Its configuration may not match the bus settings

Question 7

Which action is most appropriate immediately after defining a reproducible symptom?

A. Replace every component

B. Disable all protection circuits

C. Form a measurement-based hypothesis after basic checks

D. Delete the failing firmware image

Correct Answer: C. Form a measurement-based hypothesis after basic checks

Question 8

A reset-status register reports brownout after a radio transmission. Which evidence most directly supports this diagnosis?

A. A supply waveform showing a coincident voltage dip

B. A compiler warning about an unused variable

C. A longer source-code listing

D. A successful memory programming message

Correct Answer: A. A supply waveform showing a coincident voltage dip

Question 9

What does a regression test primarily determine?

A. Whether a previous defect has reappeared

B. Whether a probe is calibrated

C. Whether a linker can resolve symbols

D. Whether a circuit has a clock signal

Correct Answer: A. Whether a previous defect has reappeared

Question 10

Which statement about an in-circuit programmer is correct?

A. Successful programming alone proves the application is correct

B. It can transfer and often verify firmware memory contents

C. It replaces all requirements testing

D. It measures protocol noise margins automatically

Correct Answer: B. It can transfer and often verify firmware memory contents