Repository navigation
EPIC-17: faster hardware iteration - #11
Merged
diegoparrilla merged 10 commits intoSep 15, 2026
Merged
Conversation
tools/dev/console.py watch reads the Debug Probe's UART at 921,600 baud, timestamps each line, appends it to tools/dev/logs/console.log and shows it in the terminal. It finds the probe by its USB name, and waits for it and reopens it after an unplug. since-boot, tail, grep and wait read the log, so a boot log no longer has to be pasted by hand. Standard library only. Verified on hardware: a probe reset produced a complete boot log, and wait matched the GEMDRIVE ready line; unplugging and replugging the probe resumed the capture.
…-17 STORY-02) Every build now carries the git commit it came from: <sha7>, or <sha7>-dirty.<diff7> with uncommitted changes, where <diff7> hashes the diff. rp/src/build_id.cmake regenerates build_id.h on every build and rewrites it only when the ID changes, so incremental builds never report a stale ID and only health.c and http_server.c recompile. The same tree gives the same ID and a byte-identical binary. The ID is printed on the console at boot and returned by GET /api/v1/system/health with a new "debug" flag; sidecart.py health prints both. The health reply buffer grows by 32 bytes for them. tools/dev/flash.sh builds out of tree and incrementally, keeps the ELF by build ID, flashes with picotool or through the Debug Probe, and waits until the health report shows the new ID and build type, printing the console since boot if it does not. Verified on hardware: debug with picotool and with the probe, release with the probe, and the timeout path; about 18 s from command to a confirmed boot for an unchanged build.
…C-17 STORY-02) The developer tools must work with any microfirmware from the template and with a hung RP, so they reach it only through picotool, the Debug Probe and the console UART. tools/dev/swd.py wraps OpenOCD and reads memory while the CPU runs: running waits until VTOR holds the ELF's RAM vector table, verify compares the whole flash with the ELF, build-id reads the new release_build_id string from flash, read dumps a range, and program flashes through the probe. Runs that fail on a momentary debug-port drop are retried. build_id.cmake now also generates build_id.c with release_build_id, kept through --gc-sections. flash.sh checks running, verify and build-id over SWD instead of polling GET /api/v1/system/health, which keeps reporting the build ID for API users. Verified on hardware: debug with picotool and release with the probe, each running, verified byte for byte and carrying its build ID; a release ELF against a running debug build fails verify.
swd.py screen reads the 320x200 framebuffer at the top of the cartridge window while the RP runs, or while it is halted, and writes it as a PNG, so the setup menu can be checked without looking at the ST. swd.py shared prints the command sentinel, the random token and the shared variables, named after the *_SVAR_* indexes in rp/src/include. Neither hard-codes an address: the window comes from the ELF's __rom_in_ram_start__ and the offsets from chandler.h, whose #defines are evaluated by a small parser. Without --elf the tools pick the cached ELF whose build ID the RP carries. swd.py running now also requires core 0 not to be halted, and swd.py resume releases both cores through DHCSR: a halted core 1 pauses the RP2040's timer and watchdog and leaves core 0 asleep, and OpenOCD's own resume fails in a new OpenOCD run. Verified on hardware: the PNG matched the ST's screen before HELLO, after an ST reset (Phystop, Screenmem and the countdown line), and after a forced panic (Recovered: panic @10005194, the PC on the console). After HELLO the shared variables showed reloc and memtop 0xF4000, drive C and hook vector $70.
…Y-04) swd.py select presses the SELECT button without firmware code: it forces the pin's input high through the RP2040's GPIO input override for the hold time. A long press needs --force, because md-devops then erases the global settings. For what the firmware must do itself, devhooks.h adds a debug-only mailbox in RAM, self-contained like debug.h. Tools write a request over SWD and devhooks_poll(), called from the main loop, runs it and acknowledges it. A protocol request is queued through the new chandler_injectProtocol() as if the ST had sent it, so swd.py key and swd.py inject use the firmware's normal command path. An app request goes to a handler the app registers; emul.c handles countdown stop and restart, named by DEVHOOKS_APP_* defines that swd.py app reads. Release builds contain none of it. Verified on hardware with the menu: countdown stop and restart and a key 'z' each showed the expected bottom line in swd.py screen, the console logged the injected keystroke, and a SELECT short press was detected after 304 ms and rebooted the RP with reason reset.
swd.py crash reads the watchdog reason and scratch registers, and the breadcrumb the firmware decoded at boot when the ELF has it, and resolves the crash PC and LR to source lines with addr2line. The RP keeps running. swd.py postmortem halts the RP and prints both cores' backtraces, the registers, the watchdog registers and the interesting variables through GDB, then releases both cores. --leave-halted keeps it stopped. It found a real bug in the crash breadcrumb: the once-a-second stall timer cleared watchdog scratch[1] whenever the watchdog had been fed, and that is where the panic and HardFault handlers store the crash PC just before the reboot. A crash whose reboot delay overlapped a tick lost its PC. The timer now only replaces its own values there, never a crash PC. Five test panics in a row now all report the PC. Verified on hardware: panic resolves to health.c:404 and HardFault to health.c:409 with the faulting LR, and a postmortem during a forced hang showed health.c:415 in health_tick from emul_start, before the watchdog rebooted.
…EPIC-17 STORY-06) smoke.py, measure_builds.sh and stackdepth.py move from the gitignored docs/epics/tools into tools/dev, so EPIC-16 and anyone else can run them. smoke.py now uses the Debug Probe: it stops the countdown through the debug mailbox, presses SELECT through the pin override, and checks the menu text and the firmware's own haltCountdown flag instead of asking. Only the cold boot, the [G] desktop file copy and switching the ST to [U] still need a person (--manual), and --no-swd turns the probe off. The JSON report now carries the build ID, the build type, the ELF used and the console log since the last boot. swd.py text prints the terminal's character buffer, which is the menu as text; the bottom status line is drawn straight to the framebuffer, so swd.py screen stays the way to read it. Verified on hardware: a probe-driven run with no prompts passed 8 steps and skipped the 3 that need a person or the Runner, in 33 s.
Two faults from moving smoke.py into tools/dev, both found in the first full manual run: The default --hellodbg path still went three directories up, which was right under docs/epics/tools, so the Runner step crashed with a FileNotFoundError. It now goes two up, and an unreadable test program fails the step instead of the run. The countdown step read the RP's screen as soon as the tester confirmed the cold boot and reported "menu text NOT found". After a reset the RP shows its boot screen for about 11 s before the menu, and the countdown starts with the menu; the step now waits for the menu text first. The SELECT step waits the same way.
flash.sh read the generated build_id.h unconditionally, so building an older commit or another microfirmware with --src failed. The ID is now optional: such a build reports no-build-id and is still verified against its ELF over SWD. Found while building release/v1.1.0 before this epic to compare sizes.
The countdown step used the debug mailbox whenever the probe was available, so a release run failed it with "has no devhooksMailbox" instead of falling back to the tester, as it does with --no-swd. It now checks for the mailbox symbol first; without it the step asks, and the probe still confirms the menu text. Without --manual it skips, saying the build has no mailbox. Found in the release smoke run of 39f69dc, where it was the only failure.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tooling epic, run before EPIC-11 (D-10): cut the round trips between the developer at the Atari ST and the agent at the Mac. The tools reach the RP only through picotool, the Debug Probe (SWD) and the console UART, never through the firmware's own HTTP server (D-11), so they work with any microfirmware built from this template and on a hung RP.
Host tools, in
tools/dev/console.pycaptures the debug console (921,600 baud) to a timestamped log and shows it live.since-boot,tail,grepandwaitread that log, so a boot log no longer has to be pasted by hand. It finds the probe by its USB name and reopens it after an unplug.flash.sh <release|debug>builds out of tree and incrementally (35 s cold, 4 s warm;rp/buildand the submodules untouched), flashes with picotool or the probe, and then checks over SWD that the RP booted the ELF, that its flash matches byte for byte and that it carries the new build ID. About 18 s from command to a confirmed boot.swd.pywraps OpenOCD:running,verify,build-id,read,program,resume,screen(the setup menu as a PNG),text(the menu as text),shared(the shared variables by name),select,key,app,inject,crashandpostmortem. Memory is read while the CPU runs, so a hung RP can be inspected too.smoke.py,measure_builds.sh,stackdepth.pymove in from a gitignored folder. The smoke script now stops the countdown, presses SELECT and checks the menu text and the firmware's own state itself; only the cold boot, the[G]desktop file copy and switching the ST to[U]still need a person.Nothing hard-codes an address: offsets come from
chandler.hand the ELF's symbols (C-04).Firmware changes
rp/src/build_id.cmakeregeneratesbuild_id.handbuild_id.con every build (not at configure time, so an incremental build is never stale) with<sha7>, or<sha7>-dirty.<diff7>when the tree has uncommitted changes. The same tree gives the same ID and a byte-identical binary. It is reported on the console at boot, inGET /api/v1/system/healthnext to a newdebugflag, and bysidecart.py health.rp/src/include/devhooks.h, self-contained likedebug.h): host tools write a request over SWD,devhooks_poll()in the main loop runs it. A protocol request goes through the newchandler_injectProtocol()as if the ST had sent it, so an injected keystroke takes the firmware's normal path; an app request goes to a handler the app registers (emul.c: countdown stop and restart).releasecontains none of it.watchdog_hw->scratch[1] = 0whenever the watchdog had been fed, and that is where the panic and HardFault handlers store the crash PC in the 100 ms before the reboot. Crashes whose reboot window overlapped a tick reportedpc=0x00000000. The timer now only replaces its own values there. This is an EPIC-09 bug the new tools found.swd.py selectforces the pin's input high throughIO_BANK0 GPIOn_CTRL.INOVER. A long press needs--force, because it erases the global settings.releasegrew 64 bytes of code and 32 bytes of RAM againstrelease/v1.1.0, all of it the build ID and the two new health fields. It exposes no debug-only route or symbol.Verified on hardware
debug(--manual)release(--manual)Recovered: panic @10005194)0xF4000, driveC, first file descriptor 64, hook vector$70released after 304 ms,Short press detected, next bootresethealth_runPendingTest health.c:415←health_tick←emul_start, RP resumedhealth.c:404andhealth.c:409; five panics in a row all kept the PC after the fixdebugwith picotool,releasethrough the probe; a wrong ELF failsverifyKnown limitations
rebootrather thanhang; the backtrace is the better evidence.debugbuild; onreleasethose steps fall back to prompts.