Barnsley fern fractal

Thoughts on software architecture and development, and methods and techniques for improving the quality thereof.

David B. Robins (home)


Code Visions: Improving software quality
Over the air debugging

By David B. Robins tags: C++, Python, Tools, Debugging Saturday, April 11, 2015 15:57 EST (link)

I had the idea a few months ago to debug our embedded device application over a Bluetooth (Low Energy, BLE) link. Three weeks ago I finished over the air firmware updates, and this last week I got a chance to try my over the air debugging plan; and as it happened it worked fairly well.

We use SEGGER's J-Link GDB server to debug our application over a USB tag cable. I figured that I could write a GDB server that could instead use Bluetooth to communicate. It would be able to leverage GDB for symbol lookup and examine and write to memory although not (at least at first) run and step (stopping execution would terminate the Bluetooth connection). I looked up the GDB remote serial protocol, but as it happened an open source Python GDB server project already existed: pyOCD. I made some fixes to make it work with Python 3 (and stop obnoxiously logging to console), and wrote BLE transport and target classes.

When the pyOCD GDB server is launched, it runs in its own thread. My implementations of the read and write memory functions (readBlock32, readMem, writeBlock32, and writeMem) leveraged my existing ayncio-based tools framework that communicates to our embedded device over BLE using Bluegiga USB dongles. The functions use loop.call_soon_threadsafe to invoke the BLE communication to read or write on the main thread, and wait on an event until it completes.

This is of course a dangerous facility to leave enabled on shipping code, which is why it is (until we have better general security) enabled only under #ifdef DEBUGGER; our builds are automatically tagged with these feature defines so it will be clear when it is enabled.

Having this over the air memory debugging support is more convenient than having to attach a cable, and allows doing some useful debugging of state at remote locations too. More importantly, usually the case of attaching the normal debugger disrupts the device state (losing any information that could be obtained), or after attaching gets disrupted by static, and this will not. One of the current issues we're tracking has been very hard to reproduce, and we don't have enough physical debuggers to attach to every lab device. This new facility will be a great aid in resolving down such issues.

Content on this site is licensed under a Creative Commons Attribution 3.0 License and is copyrighted by the authors.