feat(fleet): durable movement foundations and bounded reconciler

[forhappy]

Scope

Draft implementation of the complete fleet operations plan. Full W1–W10 completion remains active.

• Durable controller leases, maintenance/movement state machines, enrollment and action journals with unknown-outcome retention.
• Receiver preparation with real resource reservations, canonical fenced release/recovery and retained native work across cancellation.
• Request-bound fresh inspections and native snapshots through the existing bounded action owner.
• Monotonic cordon, managed boot admission, live intent refresh, reader/follower enrollment and lifetime observations.
• Public caller-driven reconciler and durable SQLite reference adapter, with real overload, count-balancing and controller-restart scenarios.
• Qualification root-capture ordering and bounded timing evidence, preserving required rates, deadlines and coverage gates.

Latest Cron maintenance checkpoint 1bd98d8

Adds a public native regression for accepted Cron dispatch across quiescence and exact-root handoff. The accepted source Tick calls the public scheduler inside the canonical SQLite command transaction with pinned logical time and an explicit worker gate; new registered Ticks and Effect claims are refused at the barrier. The successor restores the exact Idle root, resolves the original Tick outcome, retains occurrence/generation/due-time identity and uses the registered maintenance command to process due work. Stale Ticks generate nothing, replay returns the original Tick receipt, and duplicate signed Effect delivery returns the same destination Inbox receipt with exactly one application row per occurrence.

All three independently disk-budgeted runtime ledgers return to zero through native shutdown. Fixture compile, protected-table and shared-default-budget failures remain archived separately from the final source; resource assertions and production scheduling rules are unchanged.

Isolated all-feature native verification passes the complete primitive suite (50), complete protocol suite (30, two existing manual provider diagnostics ignored), and selected public runtime maintenance suite (2, 202 other runtime cases unselected): 82 distinct passed cases, excluding focused repeats. Workspace all-target/all-feature locked check and Clippy pass with warnings denied. Format/diff, boundaries/layout, 110 Rust snippets, 1174 local Markdown links and 28 SQL/peer assertions pass. All 643 active/snapshot/staged Rust/Cargo/lock paths match; source manifest SHA256 32bc2b69c178f9e8295b3fa187e5d976995eb52654761e4e3a475d0934c2057f. Sources, logs and three final isolated executables are archived. No new process/provider/mixed-version qualification is claimed.

PR merge conflicts are resolved. Parent 9ed469c workspace CI still fails the original overload scenario: 111 example cases pass, one planned movement succeeds and another is safely cancelled. This checkpoint does not repair required two-movement convergence; fresh CI is required for this head. Full W1–W10 remains active, including the remaining primitive/fault matrix, Blob external owners, role settlement/finalization, executable maintenance/failure scenarios and qualification.

Prior exact-source refusal checkpoint 9ed469c

The native pressure race is reproduced: partition 4 was locally evicted two milliseconds before fleet Release acceptance, which then returned Unknown with CellNotActive. The canonical actor now identifies refusal before this exact request began deactivation through Error::CellReleaseRefused. The host records Rejected with the original cause, so existing refusal/cancellation transitions can join unused receiver credit. Close/publication/response-loss errors and prior accepted actions without original results remain unknown; an independently released Idle root supplies no movement proof.

Two public regressions cover actual receiver preparation followed by source eviction, durable refusal/replay, immediate zero disk credit before shutdown, and the complementary unknown-result case with credit retained. The original refusal regression fails before the fix; original and diagnostic sources, logs and executables are archived.

Final isolated native source: 6 idle-release integration cases, 515 runtime library cases (3 existing ignores), 29 host library, all 104 public host node cases, all 112 fleet example cases (71.41 seconds), and 39 application integration cases (same 16 manual ignores): 805 distinct passed cases. Other 198 runtime integration cases were unselected. Workspace all-target/all-feature locked check and Clippy pass; runtime/host API docs pass with warnings denied. Format/diff, boundaries/layout, 110 Rust snippets, 1173 Markdown links and 28 SQL/peer assertions pass. Complete source-set/byte checks cover all 641 Rust/Cargo/lock paths before and after each final command; manifest SHA256 4e84a44ffa02ed0b7f55e21cedecffa015664b3a9e2d42a46ac38de4de47395e. Final sources and six executables are archived. No new Linux, provider/process or mixed-binary qualification is claimed.

This fixes refusal classification. Reliable two-movement overload convergence remains open: preparation can also refuse a source invalidated by native shedding while the scenario disables subsequent scheduling. Required movement counts, deadline/resource bounds, native protection and qualification profiles remain unchanged. The positive native run does not erase the reproduced race or establish reliability.

Parent 5d350d1 Rust CI passed 111 example cases and failed overload (one required movement). Its object-proof qualification failed repeat 3 with skewed: no fully served capacity point; the driver completed 58/60 at that point. Complete artifact and logs are retained; that capacity failure remains undiagnosed. Fresh CI is required for this head.

Full W1–W10 remains active. SettleRoles/Finalize remain refused; complete role observation and settlement, terminal handoff, Cron/Blob owners, remaining runnable scenarios, fault/mixed-version qualification and rollout remain required.

Prior closing-owner checkpoint 5d350d1

The complete canonical host drain is now retained in one fixed task slot. Shutdown and terminal scale-down transfer their existing shared lane guard to that task. Cancelling the sole caller cannot abandon an accepted facility callback or release the lane early. Original return/join results and first/latest source failures remain inspectable; native runtime shutdown still occurs once. Genuine task failures remain fatal and incomplete phase deadlines keep lease maintenance retained. Three public regressions cover autonomous completion, queued scale-down and original failure history. The native-retirement fixture preserves its original assertions while observing its weak receipt before automatic host completion clears it.

Two synthetic deadline models retain the original 100/300 ms budgets and all original assertions, injecting the first timeout after observed acceptance. This prevents SQLite setup and journal rereads from introducing a second, unintended model fault. Native/process qualification clocks and profiles are unchanged.

Current exact-source native results: host library 29 passed, public host 102 passed, application integration 39 passed (same 16 manual ignores), fleet example 111 passed and overload failed. These are 281 passed cases and one failure, excluding focused repeats. Host Clippy/API docs pass with warnings denied; format, boundaries/layout and documentation gates pass.

The pinned Debian 12/aarch64 Rust 1.97.1 run passes the complete application/host default-feature command, including all 102 public host cases and five application doctests. Both corrected model cases pass in the full example; that example still has 111 passes and an overload failure. Complete source checks match all 641 Rust/Cargo/lock paths before/after commands: manifest SHA256 4377077c0762a8422f73642fdbdced6b6d105c0ce70d50656ff53c9dcedb33a6. Logs, sources, original/rejected evidence and final native/Linux executables are archived; owned runners are terminal and removed.

The parent merge's Rust workflow also fails overload with 111 passes: one movement completed while two attempts retired. Current native/Linux failures remain recorded and rejected as fleet qualification. The local pressure/observed-generation race remains under investigation; no required movement count or qualification gate was weakened. Fresh CI is pending for this checkpoint. SettleRoles/Finalize remain refused; complete role settlement and terminal handoff, cross-session recovery, Cron/Blob ownership, remaining runnable scenarios, fault/mixed-binary qualification and rollout remain required. Full W1–W10 stays active.

Prior merge conflict resolution 4cc8ed3

Merged main 0f4ca09 and resolved all four conflicts. Admitted owner fences remain attached to transfer admission; expiry-driven log rotation uses the existing bounded supervisor claim and retains strict accepted maintenance. Both branches' shutdown regressions and publication-hint diagnostics remain included. The upstream rotation fixture now uses the canonical retirement observation and matching host/log session; original deadlines and assertions are unchanged.

The exact merged source passes workspace all-target/all-feature locked check, 515 runtime unit cases (three existing ignores), three admitted-owner-fence integration cases, 29 host library cases, 99 public host cases, all 112 fleet example cases, and 39 application integration cases (same 16 manual ignores): 797 distinct passed cases. Host all-target/all-feature Clippy and API docs pass with warnings denied. Format, boundaries/layout, 110 Rust snippets, 1173 Markdown links and SQL/peer validation pass.

Complete checks before/after each isolated command match all 638 Rust/Cargo/lock paths: manifest SHA256 c1393c29486a7bc1bffcdf61e88db8cc3a5f6110abe4d9c4fa3d46d4fe62f784. Rejected compile/fixture evidence is retained separately. Fresh CI for this merge is pending; prior Linux results below qualify their original checkpoint only. Full W1–W10 remains active.

Prior checkpoint 6726aae

The bounded node task group now retains original joins/results. Previously it removed handles before joining; a failed first shutdown could be followed by a successful retry that withdrew lease maintenance and reported Stopped. A public regression reproduces that false success before the fix.

Concurrent drains now share the same join. Caller cancellation leaves accepted task ownership retained. Original task and panic failures remain the same source object on later drains; siblings still join, runtime cleanup runs, and a required task failure keeps the node Draining with lease maintenance retained.

Ordinary deadline aborts target the work task while retaining its supervisor until work cancellation and its destructor join. The existing node-log facility watcher uses private retained-work registration in the same 256-task supervisor, so a deadline cannot replace its native cleanup result with watcher cancellation. Both existing resumable native-cleanup deadline cases pass unchanged. No second scheduler, authority, public configuration or alternate cleanup path was added.

Five new public cases cover repeated shutdown, cancelled waiters, concurrent drains, deadline cancellation and original panic retention. Lifecycle documentation records the ownership and deadline contracts.

The earlier reader lease-boundary fix and model/snapshot fixture repairs remain included. Their original intervals, resource bounds, exact assertions and qualification profiles remain unchanged.

Verification

Complete isolated native scope	Passed	Exceptions
Host library	29	None
Public host node suite	98	None
Fleet operations example	112	None; 67.98 seconds
Application integration	37	Same 16 documented manual cases ignored

276 distinct native cases pass with all features and locked dependencies; focused repetitions are excluded. Host all-target/all-feature Clippy and API docs pass with warnings denied. Format/diff, boundaries/layout, 110 Rust snippets and 1173 local Markdown links pass.

The isolated Linux ARM default-feature commands also pass:

• cargo test -p cellule-app -p cellule-host --locked: application library 10, contracts 3, integration 37 (same 16 manual ignores), host library 29, public node 98, and five application doctests.
• cargo test -p cellule-host --example fleet_operations --locked: all 112 pass, none ignored or filtered, 81.35 seconds.

Linux environment: pinned Rust image, Debian 12/aarch64, Rust/Cargo 1.97.1, two CPUs, four GiB and 256 PIDs. Only the inherited descriptor soft limit is raised from 1024 to its existing 524288 hard limit, following the earlier frozen-binary descriptor diagnosis.

Complete source-set and byte checks run before/after each final command. All 637 active/snapshot/staged Rust/Cargo/lock paths match manifest SHA256 6e921d09fca19a8e91438e1bc078e315ff42847252691cb21dffc13a49184db4. Original failures, rejected intermediate results, final logs, sources, manifests and frozen native/Linux executables are archived. The owned Linux runner is terminal and removed.

CI and unfinished work

The parent 0132b60 qualification contract, coordination model, website, fuzz and capacity workflows passed. Rust MSRV and workspace tests passed, but its full fleet example step failed: 111 cases passed and the overload scenario settled one required movement instead of two. Current local example passes do not establish that failure's cause or claim it repaired. The historical x86 publication-hint failure also has an unestablished cause. Fresh CI for this head remains required.

An intermediate task-retention run failed two existing native-cleanup deadline cases after storing forced watcher cancellation. That source is archived and rejected; retaining the original facility watcher fixes those cases without weakening their assertions. A Linux launcher attempt failed before Cargo because its copied script lacked an interpreter line; the corrected launcher supplied the complete results above.

Full W1–W10 remains active. Complete role/current-authority observation, replacement and failed-owner evidence, evacuation/finalization without self-join, cross-session recovery, Cron/Blob owners, runnable maintenance/receiver-loss scenarios, fault/mixed-binary qualification and rollout remain required. SettleRoles and Finalize remain refused.

Execution evidence and historical failures retain every checkpoint and its exact scope.