Bench boards: retire the three failing boot units; provisioning enforces it by dorkmo · Pull Request #144 · OpenSourceEcology/LifeTrac

dorkmo · 2026-10-13T00:37:47Z

What this is

Three LifeTrac systemd units were enabled on the two bench Portenta X8 boards and failed at every boot. On 2026-10-12 the operator approved retiring them. This PR records what was done on the boards and makes the repo match.

They were retired, not repaired. Each one failed for an incidental reason, and that failure was the only thing keeping a radio daemon from starting at boot without a GO.

Unit	Board	What it is	Why it failed	If it had succeeded
`lifetrac-base.service`	base	The planned production compose stack	Rebuilds the images at boot; the build dies with `layer does not exist`	Replaces the flown image, starts `web_ui` (which pushes a retained radio profile into the harness broker) and `lora_bridge`
`lifetrac-base-compose.service`	base	The May video-test demo	The bench broker already holds `127.0.0.1:1883`	`image_rx` opens the radio UART and resets the parked L072
`lifetrac-tractor-compose.service`	tractor	The same May demo	Its container names were held by leftover containers	Image TX at every boot

Done on the boards (radios parked throughout, no reboot)

Each unit: systemctl disable without --now, reset-failed, unit files moved to /etc/lifetrac-retired/20261012T212515Z/, daemon-reload.
Base: the four never-started lifetrac-vtest-* containers and their network removed.
Tractor: /opt/lifetrac/video-test copied to the private archive (sha256-verified), its docker-compose.yml renamed so the old scripts cannot start anything from it, then the three leftover containers and two networks removed.
Nothing held /dev/ttymxc3 on either board, before or after. The bench broker was not touched.

Evidence (scripts that ran, their output, the failure journals): bench-evidence/board_state_2026-10-12_unit_retirement/.

Repo changes

provision_bench_board.sh: the three units are now enforced as retired instead of only reported. It refuses, and changes nothing, while a unit is active, starting or has a job queued. It never stops a unit, never runs docker compose, never removes a container.
provision_bench_board.sh (bug found by the first real-board run): an OpenSSH client warning was read as pids holding the radio UART. Fixed.
deploy_base.sh: the unit guard is now default-deny and its messages follow the real ExecStop behaviour.
Bench docs (BENCH_BOARDS.md, DEPLOY.md, BENCH_SETUP.md, RS13_2_DESK_CHECK.md): the decision, the rules that follow from it, the ordered procedure for another board pair, and the undo. The desk check now handles the bench broker holding port 1883.
Labels, comment-only: OBSOLETE headers on the two deploy/*.service files, a warning header on base_station/systemd/lifetrac-base.service and on docker-compose.video-test.yml, and DO NOT RUN headers on 25 May one-off scripts that install these units, start a radio daemon, remove the bench broker, or rebuild over the flown image.
SETTINGS_REFERENCE.md, TODO.md: the two rows re-statused; follow-ups RS-5.10 to RS-5.19. Line-number citations in other docs that shifted were updated.

No behaviour change outside the two bench scripts. No firmware change.

Verification

pytest tests -q in base_station: 1598 passed, 2 skipped.
Fake-board harness for the provisioning script: 14 + 65 scenario runs, all expectations met, no forbidden call.
provision_bench_board.sh <role> --check against both real boards (read-only): the retire item reads ok on each.
Three independent reviewers (script logic, fact check, operator walk-through), then a fixer and a verifier. No blocker or major finding remains.

Not done, needs the operator

Next boot not yet observed. Expect systemctl --failed to list only psplash-start.service, and the base broker back up by itself.
RS-5.18: the base still has docker-compose.video-test.yml in its deployed tree, and deploy_base.sh still ships it. A manual up of it would start image_rx whenever port 1883 is free. Proposed: stop shipping it and rename the copy on the base, as on the tractor.
RS-5.13: the tractor's udev-started lifetrac-camera.service also maps the radio UART at boot. Separate decision.
RS-5.11 / RS-5.12: the production base unit needs a redesign before it is installed anywhere.
Not yet run on a real board: the provisioning script's apply path and deploy_base.sh itself. Both fail closed.

🤖 Generated with Claude Code

The three enabled-but-failing units (lifetrac-base.service, lifetrac-base-compose.service, lifetrac-tractor-compose.service) were disabled and moved to /etc/lifetrac-retired/20261012T212515Z/ on the operator's word; the dead video-test containers were removed after that. Radios stayed parked, nothing opened /dev/ttymxc3, no reboot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ssages follow the ExecStop rule provision_bench_board.sh: the three units retired on the bench on 2026-10-12 (lifetrac-base.service and lifetrac-base-compose.service on the base, lifetrac-tractor-compose.service on the tractor) were only reported. They are now an enforced step, so a re-provisioned board ends in the same state. - Check: each unit is neither enabled nor active. A unit that is not installed passes; a reply that cannot be read counts as TODO. - Apply, only while every listed unit is failed or inactive: systemctl disable (never --now), reset-failed, the unit file and its <unit>.d moved to a new /etc/lifetrac-retired/<PC UTC time>/ (plain mkdir, with a -2..-5 suffix when that name exists, so an earlier retirement is never reused), a README.txt, daemon-reload. - A unit that is active or still starting is not touched. The step fails and prints what to do: activating -> wait for it to fail; active -> stop the containers by name, then the unit (stopping lifetrac-base.service runs a bare 'docker compose down' that also removes the broker container; deploy_base.sh recreates it). - The step never starts, stops or masks a unit, never runs docker compose and never stops or removes a container. Leftover lifetrac-vtest-* / tractor-*-v2 containers and the tractor's /opt/lifetrac/video-test/docker-compose.yml are reported as info only. While the tractor unit is still enabled the report says not to remove the containers: their names are what makes the unit fail at boot. - On the tractor the step runs before the camera step, so lifetrac-camera.service and tractor-camera are checked, and stopped, after the daemon-reload. deploy_base.sh: the refuse-while-active guard behaves as before. Its messages no longer say "Stop and disable it first", which was wrong for a failed unit and removed the broker for an active one. Activating -> wait for it to fail. Active -> stop the containers, then the unit; for lifetrac-base.service that removes the broker container, which the deploy recreates. The "enabled" warning points at the retirement procedure in BENCH_BOARDS.md. Tested without a board: bash -n on both scripts, deploy_base.sh --dry-run, and a throwaway fake-board harness (not committed) that ran the unmodified provisioning script through 14 unit-state scenarios and the generated board-side deploy script through 7. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

lifetrac-base.service, lifetrac-base-compose.service (base) and lifetrac-tractor-compose.service (tractor) were enabled and failed at every boot. They were retired on 2026-10-12 (evidence: bench-evidence/board_state_2026-10-12_unit_retirement). The bench docs still described them as enabled with a decision pending. BENCH_BOARDS.md - close the two "Decide on ..." open items, with a link to the record - mark what changed on 2026-10-12 in the Base and Tractor sections: units retired to /etc/lifetrac-retired/20261012T212515Z/, the lifetrac-vtest-* and tractor-*-v2 containers removed, the tractor's video-test compose file renamed, a copy of the tree in the private archive - fix the provenance: the tractor compose unit came from tmp_install_tractor_unit.sh (2026-05-27), not from provision_x8.sh - new section "Retired boot units (2026-10-12)": what each unit was, why it failed, what it would have started; the rules (no repair, mask is impossible, the stop rule, no bare compose down, the tractor order); the ordered procedure for another board pair; the undo; the next boot is not yet observed - new open items: confirm at the next boot, lifetrac-camera.service, 335 anonymous volumes on the tractor, the base broker's missing image, later deletion of the video-test tree and volumes - base clock: note the ~239 s step early in each boot DEPLOY.md - replace the "disable both" recommendation with the decision taken; add the tractor unit, the stop rule and the tractor ordering rule - lora_bridge: as deployed it would probably crash-loop (compose fills in the single-$ variables on the host); with $$ it would open the L072 with KISS at 115200. Fix the $ only with a Method-G daemon. - traps: a bare `docker compose up -d` also starts web_ui; a bare `docker compose down` removes the bench broker BENCH_SETUP.md - rule 6: only lifetrac-base.service is a production unit, the other two are May-2026 video-test demo units; all three are retired on the original pair and provision_bench_board.sh enforces that state RS13_2_DESK_CHECK.md - rule 2: lifetrac-base.service = production stack with lora_bridge, lifetrac-base-compose.service = video-test stack with image_rx - step 3: check that the units are retired, then a recorded `docker stop design-controller-mosquitto-1` (it holds 127.0.0.1:1883 after every boot); step 8 starts it again and checks it is Up - rule 3: do not reuse the bench broker - step 1: deploy_base.sh (re)starts the bench broker, it does not "start nothing" - steps 3 and 8: the listener check falls back to netstat. `ss` does not work on the base (10-04 capture), so `ss | grep || echo free` printed "free" whatever held the port Docs only. No board was contacted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…2026-10-12) Follows the bench retirement of lifetrac-base.service, lifetrac-base-compose.service and lifetrac-tractor-compose.service (bench-evidence/board_state_2026-10-12_unit_retirement). - deploy/*.service: "OBSOLETE - DO NOT INSTALL" header (May-2026 video-test demo, what each starts at boot, retired on the bench 2026-10-12). - base_station/systemd/lifetrac-base.service: header saying it is the planned production unit, not for bench boards, with its known problems. - SETTINGS_REFERENCE.md: the two unit rows are no longer `live`; their file:line citations are re-counted after the headers (:32 and :33). - 19 repo-root one-offs and tractor_restart_video_test.sh: a "DO NOT RUN on the bench without an explicit radio GO" comment with each script's effect. - DESIGN-CONTROLLER/TODO.md: RS-5.10 records the retirement, RS-5.11 to RS-5.17 are the follow-ups. LifeTrac-v25/TODO.md: dated note on the Phase 5 systemd-units entry. Comments and docs only; no behaviour change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Scripts (fake-board harness only; no board was contacted): - provision_bench_board.sh, retire step: - "retired" now also means no unit file or <unit>.d left in /etc/systemd/system (a symlink passes), so a re-run finishes a retirement that stopped after the disable, and a unit that is already disabled is moved too. - a unit with a systemd job queued (early boot) is not touched; the board re-checks is-active and the job right before each disable. - an empty reply to a state or path query no longer reads as "not installed": the step fails and moves nothing. - is-enabled after the disable is tested positively. - a README.txt that cannot be written makes the step FAILED. - the camera item reads is-active as text, so "activating" is not "stopped". - report lines: no "--check" hint in apply mode, the tractor WARNING is keyed on is-enabled, leftover files are named. - deploy_base.sh, board-side unit guard: only failed or inactive passes (deactivating and unknown replies are refused), and a queued job is refused. Comment on lora_bridge aligned with DEPLOY.md. Docs: - BENCH_BOARDS.md: the script's behaviour stated exactly; one stop route for an active unit; rules 7 (the base's video-test compose file is still armed) and 8 (build only with deploy_base.sh); the manual procedure now uses the script for disable-and-move and keeps guarded container cleanup only; next-boot checks; "failed at every captured boot". - BENCH_SETUP.md, DEPLOY.md, RS13_2_DESK_CHECK.md: same facts; the desk check gets marker-form UART and listener checks and a recovery path for a base reboot between Step 3 and Step 8. - Evidence README: unit text is the same but not byte-identical (CRLF on the boards); the tractor undo needs the compose file renamed back; the "listeners" line in base/12 is a false negative (ss). Logs untouched. - docker-compose.video-test.yml header: do not bring up on a bench board. - SETTINGS_REFERENCE.md: index rows in 10.1 and 10.2 for the two re-statused rows; its one cited line below them moves 1247 -> 1249. - TODO.md: RS-5.15 and RS-5.17 updated, RS-5.18 and RS-5.19 added. - VECTOR_SCENE.md, DECISIONS.md: TODO.md line citations past the RS-5.10 block shifted by 95 (each range checked against cf71a58), and the SETTINGS_REFERENCE.md citation. Labels (comment-only, line endings unchanged): - five build one-offs get a DO-NOT-RUN header (tmp_base_prune_build.sh, base_build_bg.sh, tmp_base_build.sh, _bench_build3.ps1, _bench_build4.ps1). - the 20 scripts labelled earlier now say "not even with a radio GO" and name their rule; the three unit-file headers say what was observed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… over ssh Check-only run of provision_bench_board.sh against both retired boards (read-only): the "old boot units retired" item reads ok on each, and `systemctl show -p Job --value` answers as the new guards expect. That run showed the script reading OpenSSH's three-line post-quantum warning as pids holding /dev/ttymxc3 (stderr is merged into the command output). on() now passes LogLevel=ERROR, and uart_report uses a marker so a query that never ran no longer reads as "held" or "free". Docs: the desk broker holds 127.0.0.1:1883 while the desk stack is up, so the video-test compose file is unblocked only in the gaps around it. TODO RS-5.19 updated with what is now verified (line count unchanged). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot

🟡 Changes recommended

The provisioning success check can incorrectly accept queued jobs and non-mask unit symlinks as safely retired.

2 open findings

What changed in this PR

Retires unsafe bench boot units, enforces retirement during provisioning/deployment, and documents the board changes and recovery procedures.

Changes:

Adds fail-closed systemd guards to bench provisioning and deployment.
Documents retired units, cleanup, verification, and follow-up work.
Labels hazardous legacy scripts and service definitions.

File	Description
`tmp_tractor_vtest_up.sh`	Adds radio-transmission warning.
`tmp_short_keyframe.sh`	Warns against recreating the camera stack.
`tmp_restart_image_rx.sh`	Warns about UART reset behavior.
`tmp_recreate_image_tx.sh`	Adds transmission warning.
`tmp_recreate_camera.sh`	Adds camera-stack warning.
`tmp_recreate_both.sh`	Warns against recreating both services.
`tmp_install_tractor_unit.sh`	Marks retired tractor unit installer unsafe.
`tmp_install_base_unit.sh`	Marks retired base unit installer unsafe.
`tmp_hot_patch.sh`	Warns about retained radio-profile publication.
`tmp_base_up.sh`	Warns against starting the video-test stack.
`tmp_base_prune_build.sh`	Warns about destructive image/cache changes.
`tmp_base_build.sh`	Directs builds through the supported deploy tool.
`start_base.sh`	Marks production-unit starter unsafe for bench use.
`start_base_long.sh`	Marks timeout/unit recreation unsafe.
`LifeTrac-v25/TODO.md`	Records the production-unit redesign requirement.
`LifeTrac-v25/DESIGN-CONTROLLER/VECTOR_SCENE.md`	Updates shifted documentation citations.
`LifeTrac-v25/DESIGN-CONTROLLER/TODO.md`	Records retirement and follow-up tasks.
`LifeTrac-v25/DESIGN-CONTROLLER/SETTINGS_REFERENCE.md`	Reclassifies retired units and updates references.
`firmware/x8_lora_bootloader_helper/tractor_restart_video_test.sh`	Adds radio-transmission warning.
`bench_tools/RS13_2_DESK_CHECK.md`	Updates desk-check broker and retirement procedure.
`bench_tools/provision_bench_board.sh`	Enforces retirement and improves UART reporting.
`bench_tools/DEPLOY.md`	Documents deployment guards and retired-unit rules.
`bench_tools/deploy_base.sh`	Adds default-deny unit/job checks.
`bench_tools/BENCH_SETUP.md`	Integrates retirement into board setup.
`bench_tools/BENCH_BOARDS.md`	Documents board state, procedures, and undo steps.
`docker-compose.video-test.yml`	Adds prominent bench-use warning.
`deploy/lifetrac-tractor-compose.service`	Marks tractor demo unit obsolete.
`deploy/lifetrac-base-compose.service`	Marks base demo unit obsolete.
`DECISIONS.md`	Updates shifted settings and TODO citations.
`tractor/20_provision_check.txt`	Records post-retirement tractor check.
`tractor/13_disarm_and_remove_containers.txt`	Records tractor compose disarm and cleanup.
`tractor/12_video-test_tree.sha256`	Records archived tree checksums.
`tractor/11_disable_and_move.txt`	Records tractor unit retirement.
`tractor/10_preflight.txt`	Records tractor preflight state.
`tractor/00_failure_journal_2026-10-11.txt`	Captures tractor failure evidence.
`scripts/tractor_4_disarm.sh`	Preserves executed tractor disarm procedure.
`scripts/tractor_1_retire.sh`	Preserves executed tractor retirement procedure.
`scripts/tractor_0_preflight.sh`	Preserves tractor preflight script.
`scripts/diag_tractor.sh`	Preserves tractor diagnostic collection.
`scripts/diag_base.sh`	Preserves base diagnostic collection.
`scripts/base_3_containers.sh`	Preserves base container cleanup.
`scripts/base_1_retire.sh`	Preserves base retirement procedure.
`scripts/base_0_preflight.sh`	Preserves base preflight script.
`bench-evidence/.../README.md`	Summarizes retirement evidence and outcomes.
`base/20_provision_check.txt`	Records post-retirement base check.
`base/12_remove_dead_containers.txt`	Records base container cleanup.
`base/11_disable_and_move.txt`	Records base unit retirement.
`base/10_preflight.txt`	Records base preflight state.
`base/00_failure_journal_2026-10-11.txt`	Captures base failure evidence.
`base_station/systemd/lifetrac-base.service`	Documents why the unit is not installable.
`bringup_retry.sh`	Warns against transient production bring-up.
`base_videotest_up.sh`	Warns about broker removal and UART reset.
`base_build_bg.sh`	Warns against unmanaged on-board builds.
`_tractor_flip2.ps1`	Warns against remote FHSS stack recreation.
`_tractor_flip.ps1`	Warns against direct tractor stack changes.
`_tractor_flip_remote.sh`	Warns against image transmitter recreation.
`_bench_up2.ps1`	Warns about broker removal and image receiver start.
`_bench_up.ps1`	Warns against starting image reception.
`_bench_build4.ps1`	Warns about cache deletion and retagging.
`_bench_build3.ps1`	Warns about unmanaged image retagging.

🧠 Review effort: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

dorkmo and others added 6 commits October 12, 2026 16:28

Copilot AI balanced review requested due to automatic review settings October 13, 2026 00:37

Copilot started reviewing on behalf of dorkmo October 13, 2026 00:38

View session

Copilot AI reviewed Oct 13, 2026

View reviewed changes

Comment thread ...25/DESIGN-CONTROLLER/firmware/x8_lora_bootloader_helper/bench_tools/provision_bench_board.sh

Comment thread ...25/DESIGN-CONTROLLER/firmware/x8_lora_bootloader_helper/bench_tools/provision_bench_board.sh

dorkmo merged commit cfb93d8 into main Oct 14, 2026
23 checks passed

dorkmo mentioned this pull request Oct 14, 2026

RS-13: close the no-radio items before the desk check and range-edge session #150

Merged

Navigation Menu

Repository navigation

Bench boards: retire the three failing boot units; provisioning enforces it - #144

Bench boards: retire the three failing boot units; provisioning enforces it#144
dorkmo merged 6 commits into
mainfrom
bench-units-retired-2026-10-12

dorkmo commented Oct 13, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

dorkmo commented Oct 13, 2026

What this is

Done on the boards (radios parked throughout, no reboot)

Repo changes

Verification

Not done, needs the operator

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants