Validate POSIM images and fix Docker runtime shutdown failures

[yeseorizi]

Current status — both validation tags published; registry re-pull passed

PR #5 remains OPEN and unmerged; auto-merge is disabled. main is still e05a9fb7. General main/release publication stays disabled (POSIM_PUBLISH_IMAGES=false); no latest, main or release tag was written. The already-tested abad9d70 images were published without rebuilding.

Architecture	Public validation tag	Fresh headless Quickstarts after anonymous pull	Extra offline camera	Installed transport churn
ARM64	validation-pr5-abad9d70-arm64-rdp	14/14	1/1	5/5
AMD64	validation-pr5-abad9d70-amd64	14/14	1/1	5/5

Repository: ioeslab/posim. Both installed inventories passed. Independent audits of actual observations and raw child-process logs found no startup/shutdown failure in these post-publication checks. These 14+1 trials per architecture are additional to, not replacements for, the earlier native 77+5 acceptance trials.

Immutable published references:

• ARM64: ioeslab/posim@sha256:72179187c96a801184a15ab9cdaf3ca9714d3717ce04e97cd2ec60f34d83edb8.
• AMD64: ioeslab/posim@sha256:9783525a2a18ecc2e275e9af43e82ccab6202bb99b63790ffd165d8885fa9784.

Publication history is retained: ARM64's first upload timed out. The second upload completed, but the original post-push checker incorrectly compared Docker containerd's index ID with a configuration digest. Tooling commit 584953b5 distinguishes these identities and passed 12 guard tests. The final ARM64 run verified the exact index/platform/config chain, reused the identical published tag without overwriting it, pulled it anonymously and completed the fresh-container checks. No application code or acceptance thresholds changed, and no failed runtime trial was discarded. AMD64's successful attempt-1 evidence is retained separately from that workflow's historical ARM64 failures.

docker pull may reuse cached layers; this is digest-verified public retrieval followed by fresh-container execution, not a cold-download claim. GUI/RDP, joystick, CUDA/WGPU sonar and all-world coverage remain outside the scope. Earlier non-publication statements below describe their historical checkpoints.

---

Historical initial status: publication blocked — debugging runtime shutdown

POSIM_PUBLISH_IMAGES=false. This PR is not merged and no image from it has been published. A green diagnostic workflow means evidence was collected, not release acceptance.

Native image evidence

Source	Architecture	Initial runtime paths	Additional repeats	Inventory	Outcome
c3a4498d baseline	ARM64	11/14	—	PASS	Failed (raw-log audit)
c3a4498d baseline	AMD64	7/14	—	PASS	Failed (raw-log audit)
a74aa354	ARM64	12/14	45/45	PASS	Model-readiness observations failed twice
a74aa354	AMD64	13/14	43/45	PASS	Two readiness failures; one MAVROS shutdown SIGSEGV
bd5ec83c	ARM64	12/14	60/63	PASS	Three MAVROS SIGSEGVs; two Gazebo SIGTERM escalations
bd5ec83c	AMD64	12/14	Incomplete	PASS	Cancelled by a later PR update; not a full-matrix result

Original artifacts and all failed trials are retained. The earlier baseline summaries (13/14 and 10/14) missed child aborts hidden by a successful wrapper exit; they are not acceptance results.

Native runs: ARM64 bd5, AMD64 bd5, incomplete, ARM64 a74, AMD64 a74.

Implemented build/runtime fixes

• Preserve Docker Desktop's daemon endpoint when isolating Docker credentials. Fetch the pinned ArduPilot commit/submodules shallowly with bounded network retries.
• Persist AMD64 ArduSub and plugin/resource paths for noninteractive execution.
• Leave process signals to Gazebo in embedded ROS plugins. Guard context invalidation without hiding live-context errors.
• Drain underwater-camera callbacks before buffer destruction, initialize publisher before subscription, and validate image/depth/stride. Five C++ regressions cover these cases; the attenuation formula is unchanged.
• Build pinned Lyrical ros_gz_bridge with the upstream handle-ownership fix plus an explicit local callback-lifetime patch. A three-direction weak-reference probe checks actual node release.
• Rebuild MAVROS and libmavconn 2.15.1 with the upstream I/O self-close fix. A before/after sanitizer reproducer shows the self-close use-after-free and its correction. This fix alone did not eliminate the native MAVROS SIGSEGV.
• Wait for the exact entity within the existing startup budget. Retain every observation; do not restart a failed scene to obtain a pass.

Historical candidate: 6253abad — ARM64 passed; AMD64 release was blocked

GDB run captured MAVROS SIGSEGV in a DDS event thread while the main thread unloaded RMW libraries at exit. Router I/O and DDS threads were still alive. Code and a native-image reproducer identified the Router → endpoint → Router strong-reference cycle.

Controlled before/after diagnostic: the empty Router control released; ROS-only and ROS+UDP endpoints leaked before the local weak-parent patch. All three released after the patch. BlueROV and BlueROV Heavy then each passed five execution/payload/connection/shutdown trials (10/10) in the disposable patched container. This is diagnostic evidence, not final-image acceptance. The patch is now integrated into the Docker builds with its checksum and ownership probe.

Fresh-container Gazebo backtrace showed shutdown joining a background Fuel download. Jetty had already advertised the world and advanced its clock before creating the scene entities. The four launch entries now default to Jetty's --wait-for-assets (explicit wait_for_assets:=false opts back into asynchronous startup). The bimanual check requires the real grabbapole entity and spherical-coordinate service output. This mitigates ready-scene shutdown racing cold downloads; it does not fix interruption during a download or guarantee network availability.

Earlier clock-only world observations did not establish asset readiness. The new checks are stricter, so old and new pass counts should not be treated as identical acceptance criteria.

Final raw-log audit of the native candidate:

• ARM64: inventory PASS; initial 14/14 and additional 63/63, 77/77 runtime trials.
• AMD64: inventory PASS; initial 14/14 and additional 62/63, 76/77 runtime trials. rexrov_waves-r3 crashed on shutdown in Gazebo Transport's RunReceptionTask / zmq_poll / libzmq. The parent exit of zero and wrapper SIGINT do not override the logged segmentation fault. The later controlled transport investigation is documented below.
• Both BlueROV variants and the bimanual world passed 10/10 on both architectures. The remaining failure is distinct from the earlier MAVROS cycle and cold-asset download issue.
• Asset-readiness diagnostic: 10/10 fresh containers passed; diagnostic evidence only.
• First AMD64 GDB follow-up was invalid: the wrapper corrupted the Ruby CLI version probe, so none of 20 attempts reached world readiness. Its green job status is script completion only, not absence-of-crash evidence. All failed records are preserved.
• Diagnostic-only commit 38ada73c adds transparent version probing, bounded library-specific symbol retrieval, a preflight version comparison, fail-fast handling for startup failure, and three wrapper regression tests. Corrected diagnostic completed 20/20 instrumented trials with normal inferior exit, but did not reproduce SIGSEGV. It used one diagnostic container and does not override the fresh-container production failure; native evidence remains 77/77 and 76/77. Redundant image builds triggered by diagnostic pushes were cancelled.

Post-diagnostic symbolization of the original failure used the exact captured ZeroMQ library (build ID ac4eff7609d3a2693528173b6ad0df03a09bdb09, Ubuntu zeromq3 4.3.5-1build3) and matching Ubuntu DDEB symbols. The frame chain is zmq_poll -> socket_base_t::getsockopt -> xsub_t::xhas_in -> xsub_t::match; the top address maps to inlined trie_t::check, trie.cpp:219. Load bias was inferred from the unique page-aligned mapping compatible with the zmq_poll symbol range; this is not a newly captured full GDB stack. At that stage, concurrent subscription-trie access was a code-inspection hypothesis; the controlled follow-up below now reproduces it. The original final-image failure remains valid evidence. The publication opt-in is still disabled.

Image IDs: ARM64 sha256:b3452540588705bd021cf6dd89c80a7fd6f8557da249ea20226f1db0724e9011; AMD64 sha256:528f01785189660ae7c011980bed2fc2d1ac3732401de186dc6c718406cbc369. Native source head is 6253abad; PR merge-test checkout and image revision label are 1727fa79c1356f01135edf945370f04fe857b865.

Local classifier/readiness, diagnostic-wrapper, churn and linkage unit tests: 22/22. Full GitHub lint for the new candidate passed. All historical failures and diagnostic artifacts remain available; nothing has been relabeled as a pass.

Transport poll-race follow-up (2026-09-30)

A two-process payload probe repeatedly removes the last subscriber on active topics. Both variants use the same pinned gz-transport 15.1.0 source, compiler settings and probe; only the poll-serialization patch differs. The driver records real anchor/churn deliveries, child exit codes, timeouts and the loaded library hash. No failed trial is retried or discarded.

• Local ARM64 matched-source comparison: unpatched 1/5 SIGSEGV, patched 5/5 passed (20 seconds per trial). An additional installed-vendor run also reproduced SIGSEGV. Its valid core, symbolized using the matching Ubuntu libzmq build ID, follows trie_t::check -> xsub_t::match -> xsub_t::xhas_in -> socket_base_t::getsockopt -> zmq_poll -> RunReceptionTask, the same function chain implicated in the original AMD64 failure.
• Native AMD64 comparison, diagnostic commit 174bab1e: unpatched 5/5 SIGSEGV, patched 5/5 passed. Every passing patched trial delivered actual messages and terminated normally. These are diagnostic probe trials, not Gazebo Quickstart trials.
• Local ARM64 patched-source CTest: 106/106 entries passed, including result-checker entries. The initial run accidentally loaded the old vendor library and is invalid; the reported run explicitly selected the matching build library. This source-default build and the deployment's Zenoh-disabled ROS vendor configuration are distinct configurations.

The patch serializes zero-timeout socket polling with subscription updates, while blocking only on OS notification descriptors outside the mutex. It rechecks socket events on each loop for ZeroMQ's edge-triggered ZMQ_FD semantics. Candidate Docker builds use a pinned separate ROS vendor overlay, not replacement files in /opt/ros. The overlay preserves the installed public build configuration, including Zenoh disabled, and records source/patch/library hashes. Image inventory checks the actual Gazebo/probe linkage and repeats five payload-churn trials using only the installed setup chain.

The ROS-vendor-configured local ARM64 overlay also passed its five build-probe trials and five installed-inventory probe trials. Its public configuration matched the installed ROS vendor byte-for-byte, and both Gazebo and the probe resolved the patched library. These are additional local checks, not native final-image acceptance.

Historical final-image audit of 0fbb707b: camera startup blocked acceptance

Fresh ARM64 and AMD64 image builds completed. Downloaded artifacts were independently checked against all launch commands, result records, actual observations and raw child-process logs.

• ARM64: installed inventory PASS, transport payload/churn 5/5, 77/77 runtime trials PASS. Image sha256:2ad24f7e264b2bdfdefd62a2cbbf4d7dfacbab0c0ab46c68f0352a0ae9731584.
• AMD64: installed inventory PASS, transport payload/churn 5/5, 76/77 runtime trials PASS. Image sha256:b058652ea2ac64e4e36ce5a5466670ad3f02e181587956d638a195fbbda7431f.
• No child shutdown crash was logged in either 77-trial matrix. This does not establish zero failure probability. The new AMD64 failure is camera-r5: the world control service appeared, but the camera entity, advancing simulation and ROS image payload were not observed within their original budgets. /world/camera_world/create remained unavailable until shutdown began. Parent exit zero and normal shutdown do not override this startup failure. The original acceptance record does not contain a server stack; the separate same-image diagnostic below captured the cold-start wait.
• Both images identify synthetic PR-test checkout 55d11e37ce0a8ee041e58e8943ba9df013ab49f1, whose parents are main e05a9fb7 and candidate 0fbb707b. This is GitHub's test checkout, not a PR merge; main has not been changed.

Diagnostic-only commit c60bef43 adds a fresh-container camera readiness probe. It preserves the original startup budget and captures server stacks after a readiness failure, with verbose startup logs and Fuel cache inventory. Native AMD64 diagnostic completed and stopped on the first fresh-container readiness failure. At the unchanged entity-readiness deadline, GDB captured the server waiting on FuelClient::DownloadModel -> Rest::Request -> libcurl/poll while loading an SDF material. The server main thread was waiting for construction to finish. Verbose logs show a nested Sunken Vase resource download before scene initialization resumed about 155 seconds after launch. Real camera images arrived later, and shutdown was normal. This establishes an external Fuel-download wait in the reproduced failure, not a transport shutdown crash. It does not identify the remote server/network cause or prove the uninstrumented acceptance failure had an identical stack.

The diagnostic preserved the failed readiness verdict despite the later image payload. No acceptance timeout was raised, no failed run was replaced, and no application-level cold-asset fix has yet been applied. A deterministic asset-preparation/cache solution must be validated on rebuilt images before publication. These instrumented observations cannot replace the failed acceptance trial.

PR #5 remains open/unmerged. POSIM_PUBLISH_IMAGES=false; no candidate image was published. All earlier failures and the new camera failure are retained.

Validated build-time Fuel assets candidate abad9d70 (2026-09-30)

The reproduced camera wait involves Sunken Vase Distorted's version-3 Sunken Vase material dependency, in addition to the version-4 top-level model. The candidate now prepares the Fuel dependency closure used by the 14 Quickstarts during image build: 12 version-pinned model assets, all 105 prepared files hashed, nested SDF HTTP references checked, and original author/license metadata retained. No geometry, texture or simulation parameter is edited. Native installations are unchanged.

The image exports a shared Fuel cache and its locked provenance under /opt/posim_fuel. Inventory verifies the installed lock against the validation source and checks the files without downloading. CI retains the original 77 connected trials and adds five separate fresh-container camera trials with --network none, requiring actual ROS image payloads and normal shutdown. Startup budgets are unchanged; offline runs do not replace connected failures.

Local preflight: 33 Python unit tests passed; the production downloader fetched the critical version-3 dependency into isolated staging and verified the full lock; an older local ARM64 development image with the prepared cache produced a 320×240 BGR image and terminated normally with networking disabled. This is development evidence, not acceptance of the new native images. Both native image matrices have now completed and their downloaded artifacts passed an independent raw-observation/child-log audit:

Architecture	Installed inventory	Connected runtime trials	Additional offline camera trials	Installed transport churn
ARM64	PASS	77/77	5/5	5/5
AMD64	PASS	77/77	5/5	5/5

All offline records retain Docker's NetworkMode=none, an exited container with code zero, actual camera payload dimensions/byte count/hash, advancing simulation, entity readiness, and raw shutdown logs. No child crash was logged in either connected matrix or the additional offline trials. Both inventories verified all 105 cached files against lock SHA-256 9776a35839f1797769f7ba3a80dfe046366493889838d8e78cd40c16c9ab31fa. These results apply to the declared test scope, not every feature/world or zero failure probability.

• ARM64 image ID: sha256:72179187c96a801184a15ab9cdaf3ca9714d3717ce04e97cd2ec60f34d83edb8.
• AMD64 image ID: sha256:72d00bb8e32725bd8e4fe91b8f21de770d4e4f4753f93d596bce4cc97d018cc5.
• Image revision label: 32eedbacf781a0fd1e517481571e76bfaab73c51, GitHub's synthetic test checkout with parents main e05a9fb7 and candidate abad9d70. This is not an actual PR merge.
• ARM64 attempt 1 failed before runtime testing because the ArduPilot Git fetch repeatedly ended with a GnuTLS receive error/early EOF. The build failure log is retained. Same-source attempt 2 built successfully and then ran the full matrix. No failed runtime trial was rerun into a pass.
• Full GitHub lint passed, including 33 Python unit tests.

No PR merge or Docker Hub publication was performed. PR #5 is OPEN with auto-merge disabled; main remains e05a9fb7d25ec21399f897e9bddfcabc4493b703, and POSIM_PUBLISH_IMAGES=false. Publication and a registry re-pull/Quickstart check remain pending.

Acceptance and scope

The full native test is installed inventory + 14 distinct headless paths + 63 additional repetitions (77 runtime trials). It requires actual payloads, advancing simulation, model readiness, MAVROS connected state for both BlueROVs, and clean child-process shutdown. A parent launch exit of zero never overrides a child crash. PRs never publish; main/version-tag publication also requires the opt-in variable and success on the exact image ID being pushed.

GUI/RDP, joystick, CUDA/WGPU sonar execution, physical fidelity, and all 18 world files are outside this matrix. Do not infer zero failure probability from any finite passing sample.