Bound authentication, stabilize residency, and restore retained catalog identities

[forhappy]

Summary

Follow-up to merged #17: bound authentication work, stabilize cold-residency admission, restore verified retained catalog identities, preserve HTTP listener reservations through startup, and add reproducible three-node qualification tools and documentation.

Cellule is now pinned to 0f4ca0919b0dfe20a3dcd964d21da03135e42eed, the upstream main revision checked on October 2. Only five direct pins and six lockfile source entries changed from 1914096; unrelated dependencies, features and runtime budgets are unchanged. The dependency checkpoint distinguishes the new pin from historical results.

The benchmark retains bounded, redacted Git failure details and independently validates them without changing deadlines, retrying failed arrivals or rewriting historical evidence.

Verification

Scope	Result and limitation
Current PR head bbd3c54	Documentation-only native and retained-corpus checkpoint; all 91 local Python harness tests and git diff --check passed. Production, Cargo lockfile, tests and workflows are unchanged from e8eef77. All five PR/push checks passed, including release correctness. The new release artifact has not yet been independently audited
Previous PR head e8eef77	All five checks passed, including release correctness. This documentation-only update preserves the production, dependency and test inputs audited at 7497fc7; the newer release artifact has not yet been independently audited
Latest pin at 7497fc7	All five PR-head checks passed, including release correctness: 244 top-level Rust tests, nine ignored, 91 Python tests and all eight fresh RustFS gates. Release artifact, all 268 source/workflow inputs, six locked Cellule packages and ELF digest independently audited. This is not native retained-store recovery or performance qualification
Previous PR head cb78793, Cellule 1914096	All five checks passed, including release correctness: 244 top-level Rust tests, nine ignored, 91 Python tests and eight fresh RustFS gates. Release artifact and 268 source/workflow bindings independently audited; these results do not qualify 0f4ca09
Earlier release failure	Run at 5581d5c failed the disconnected-admission regression with HTTP 503. Subsequent passes and unchanged-source diagnostics do not establish its cause or resolution
Separate SSH test candidate, excluded from this PR	Post-fix diagnostic passed three reservation checks, ten stock-Git SSH checks and one full-target run; the second full-target run failed. Its debug suite also failed the new reservation regression. Not presented as a fully qualified correction
Separate active-owner recovery candidate d1f9250, excluded from this PR	All 14 declared diagnostic cases passed; exact source, dependencies, test executables and logs were independently audited. Debug verification and release/RustFS verification passed. The release artifact was independently audited: 247 top-level Rust tests, nine ignored, 91 Python tests, eight fresh RustFS gates, all 270 bound source/workflow inputs and six locked Cellule packages; ELF digest and local evidence copies verified. Depends on draft Cellule #44, not upstream main; retained-store recovery, historical timing failures and performance remain unqualified. Temporary admission diagnostics at 9a5ff42 are also excluded
Frozen native baseline and terminal fleet	Baseline remains 0dc04a6. On revalidation, all three previously recorded owners were absent; terminal records show node-lease-bounds errors and unconfirmed drains at 17:40 UTC on October 2. No restart or upgrade was performed. All closed runtime files and 658 bound inputs were preserved; see the checkpoint below

See the dependency checkpoint and listener qualification.

Complete baseline and failed recovery

The original RustFS corpus passed controlled activation and pre-load remote verification: 10,000 identities, 100 complete LFS bodies, 200 stock-Git v0/v2 clones and both critical fixtures. This is not post-load recovery evidence.

• All 108 windows, 114,960 arrivals and 8,640 scheduled seconds completed and were audited: 59,554 OK / 55,406 failed arrivals. The separate critical schedule completed 19/20 OK, retaining its busy drop.
• Positive ACK inventory: 1,464 creations, 1,023 ref-only pushes, 467 fresh-object pushes, 242 LFS uploads and 19 critical workflows covering 38 repositories. Inventory integrity is not remote recovery proof.
• The 1-MiB push phase recorded 150/1,200 OK, 855 busy drops, 186 Git errors and nine timeouts. Two 16-client, 4-push/s windows delivered 0.117 / 0.050 successful in-window pushes/s, with successful-only scheduled p95 70.764 / 100.998 seconds. Completion counts include drain; in-window throughput excludes it.
• The exact original owners were removed after closed-ledger and every-ACK preflight; confirmed absence was 32.008191 seconds. RustFS identity and resource envelope remained unchanged. The first fresh launch failed a helper readiness ordering check and drained; a separate launch reached readiness within the unchanged deadline.
• Full recovery verification failed at 08:13:33 UTC on October 2: a stock-Git v2 clone exited 128 after 0.570 seconds. Its ledger contains 3,632 identity checks, 41 LFS downloads and 325 Git commands, with one failed command. No complete stage was recorded; remaining corpus and every-ACK stages did not complete. The original verifier retained stderr's digest, not its text; root cause remains unresolved.
• All 3,684 closed attempt files and remaining cloned data were copied and hash-checked on a different local filesystem; all 651 bound inputs were checked before and after copying. This is local preservation, not an off-machine backup.
• A separate full attempt with supplemental Git capture failed at 15:57:27 UTC: 768 identity requests included nine 30-second timeouts and one HTTP 503; nine LFS downloads included one HTTP 500. All 64 Git commands passed and reconciled with the supplemental ledger. No complete stage or every-ACK verification completed. All 1,454 closed attempt/input files and 653 bound inputs were preserved on another local filesystem. A fixed four-route metadata probe retained three 503s, including direct-node failures; this does not resolve the earlier clone failure. See the detailed checkpoint.

The full campaign audit is 23836f6a437826a58e5157b534a17346b939e9c4e7213168b1c872c19f387f59; failed recovery receipt is 54c3e058871c7c57ef65f9b8f32f3f94cdd93151915d757af49cfcce333a3b54; preservation manifest is c9b314798f1071dcb625f95e9cd4da8b9d64618051d8f7d61b100dfe85a601fc.

See complete outcomes and recovery evidence and the performance plan.

Native and retained-store checkpoint

The excluded active-owner candidate d1f9250 also passed native Mac release lints, 247 top-level Rust tests (nine ignored), 91 Python tests and eight fresh RustFS gates. Independent audits verified all 270 source/workflow inputs and six Cellule package revisions. The fresh-provider gates ran the retained test executable, not the standalone CLI. Two post-test collector failures remain preserved; separate collectors and audits verified the results without rerunning tests or builds.

A write-refusing inspector completed two identical full scans: 11,509 Cells, including all 11,505 required Cells and four extras. It observed 10,993 Idle / 516 Serving, six retired owners, no advertised writers and zero provider write attempts. Catalog visibility is not remote Git/LFS recovery proof.

The inspector's current-code-only predicate rejected Directory's retained schema-1 code. The fetched stored release descriptor was independently BLAKE3-verified and explicitly supports that retained code. Source inspection found the same restriction in the frozen maintenance worker's Registry::is_current_cell check. This is a source-level compatibility defect, not a failed maintenance execution: no maintenance, activation, original-provider restart or upgrade was performed. All 938 negative-attempt/input files were preserved and hash-checked on another local filesystem.

The candidate remains excluded: retained recovery is unqualified, and draft Cellule #44's follower-proof capacity check failed. The PR still pins latest upstream main 0f4ca09, rechecked at publication.

See the native qualification checkpoint and retained-corpus inspection.

Follow-up recovery prerequisites

The separate maintenance correction 5ab8638 reproduced two exact retained-code predicate failures twice before the fix. Changing only the catalog predicate to supports_cell passed all three regressions, all 13 deployment tests and release all-target lints. Persisted-Control validation, quotas, leases and deadlines are unchanged. Published-root and stored-byte preservation passed; unknown code/schema/role/namespace, live-owner, wrong-operation and missing-root refusals remain intact. The owned-fixture audit and 556-file cross-filesystem copy are verified. This candidate remains excluded from PR #18.

The corrected write-disabled inspector passed six tests and completed two matching full scans of 11,509 Cells, including all 11,505 required Cells and four extras. Independent reconciliation verified that catalog, Control, roots, owners, ETags, release and shard inventory exactly match the preserved negative snapshot; only its two predicate errors disappeared. The descriptor explicitly supports Directory's retained code. 516 Cells remain unsettled, and frozen-maintenance-worker qualification remains false. The original provider was not written, restarted or activated. The 944-file inspection/input copy is hash-verified.

The separate image-transition fixture 049f5d1 passed four regressions, all 14 deployment tests and release all-target lints. It uses canonical release preparation to select a new image with the same descriptor, retires the owned old session, refuses the old-image worker and preserves the retained Directory root/data. Its independent audit and 280-file local copy are verified. Synthetic fixture image digests do not qualify a physical maintenance bridge. That executable, supported original-store transition, complete remote recovery and performance remain open.

Remaining gates

Resolve retained correctness failures; verify complete original-corpus and every-ACK recovery; qualify the new pin natively and against retained stores; run concurrent faults, higher admission profiles, uniform/skewed active sets, independent operation rates/concurrency, matched comparisons, non-sparse five-GiB transfers and isolated Linux capacity.

No matched speedup, latest-pin recovery or isolated Linux capacity is claimed. Future comparisons must use the same instrumented driver on both binaries; new diagnostics cannot reconstruct missing historical stderr.

[forhappy]

Latest Cellule candidate Linux CI is now closed and passed: https://github.com/crabbuild/canopy/actions/runs/36965371936 . Exact head 63c93b0 pins Cellule 191409685b001a82bd02780def45102b4fc2f164. Log reconciliation confirms 244 top-level Rust tests passed, zero failed, nine ignored; two nested child results are not double-counted. All 84 Python harness tests and all eight exact fresh RustFS compatibility gates passed, as did formatting, all-target Clippy and the debug server build. Native/release qualification, retained-store upgrade/recovery and performance do not follow from these debug CI results. The live fleet and PR pin remain on the separately qualified 0dc04a6 revision.

Both PR and push CI at documentation-only head ec81650 also passed all four Rust/harness jobs, including RustFS compatibility. The PR description now records those closed scopes.

Closed candidate CI/source/log evidence has nine copied and independently reread files; manifest SHA-256 880d2100bf28801b80aeaa0cd6b38bfd50abe0b263b19189b273cf03531de1c1. The independent log-accounting copy has manifest 7683fcea17f284d593cbfdea6f5975aa5e5a51c653e151c16b41a832e0f475f1. These are local cross-filesystem evidence copies, not off-machine/provider-data backups.

The original full campaign has now closed 64 windows and moved to incremental pull. The eight incremental-fetch windows finished at 771 OK / 1,200 scheduled, 418 busy drops and 11 Git errors; all 32 finalized source/copy file pairs were reread. At concurrency 16 / 4/s, the first repetition returned 211/240 OK (18 busy, 11 Git errors), delivering 3.333 successful operations/s with completed-attempt p95/p99 7,245.835/8,568.187 ms; the second returned 240/240 OK, delivering 3.883/s with p95/p99 3,847.302/4,845.643 ms. Variability and failed arrivals remain recorded, with no matched speedup claimed.

A separate complete-recovery verifier is prepared outside the repository. It gates on the full terminal audit/ACK inventory, the exact three old owners being absent for at least 32 recorded monotonic seconds, unchanged provider/deployment/budgets, distinct new owners, the same executable and fresh local-state provenance. It invokes full original 10K/100 content checks plus every creation, ref/fresh-object Git push, LFS and complete critical-workflow ACK verifier, retaining a request ledger. Eleven pure guard/accounting tests passed; ledger reconciliation also matched the actual earlier complete pre-load record (10,002 identities, 100 full LFS downloads, 824 Git commands). The real live-campaign invocation refused before reading nonexistent owner/fresh-launch inputs or creating any verifier output. No owner signal or provider request was sent. The real post-terminal/fresh-fleet path and every-ACK remote recovery have not run; this is preparation, not a correctness pass. Its four-file verified qualification copy has manifest febe3e79bc3a3be77274084b79b0d0dd5bce07224ecc6abea13a2e7092eca231.

The existing large-transfer workflow requires a dedicated self-hosted Linux runner. The repository runner inventory currently returns zero registered runners, so no unserviceable job was queued. Full schedule/audit and owner-loss recovery continue to be required before reference capacity or any improvement claim; all 638 frozen live input bindings remain unchanged.

[forhappy]

Closed verification update (no changes to the live benchmark):

• Latest Cellule 1914096 candidate Linux release CI passed. Independent artifact accounting confirms 244 top-level Rust tests / 0 failed / 9 ignored, 84 Python tests, all eight exact release RustFS gates, formatting, release lints and the release executable build. Two filtered child results were not double-counted.
• Verified the GitHub ZIP digest, all 267 bound source/workflow hashes, six locked Cellule package revisions, provider image identity, the executable TAR and binary checksum. Current PR production/test/script/Cargo inputs match the candidate. Linux executable SHA-256: 5ff3a4daf2c2012357b4643b896e4f89b8402f7cad9c29679ea20c12becc880d. Seventeen artifact/audit files have a reread-verified second-filesystem copy. This is Linux fresh-fixture correctness, not native Mac, retained-store recovery, performance or reference capacity. Current PR-head CI remains separately in progress.
• Re-audited all 8 incremental-pull windows: 743 OK / 1,200 scheduled; 457 busy drops, zero Git errors. The two concurrency-16 / 4-RPS windows each returned 240/240 OK; delivered in-window RPS was 3.900 / 3.833, with successful p95/p99 2,228.861/2,604.536 ms and 4,364.562/5,076.078 ms.
• Re-audited all 4 ref-only-push windows: 1,023 OK / 1,200 scheduled; 85 busy drops, 92 Git errors. Both 1-RPS windows returned 120/120 OK; the 4-RPS windows returned 427/480 and 356/480. Preserved all 1,023 distinct positive ref ACKs for post-owner-loss verification. Forty-eight finalized source/copy pairs, deterministic selections, arrival/timing counters and resource boundaries reconcile. Failed pushes are not assumed to have rolled back.
• The frozen driver discards Git stderr, so git_error alone cannot establish a cause. No error detail, root-cause finding or performance improvement is inferred. Instrumentation must be a separate subsequent diagnostic, not a change to the live baseline.
• Prepared the full-campaign owner-loss controller outside the repository. Six guard tests, real process/config identity checks and an actual live-load refusal passed, with six qualification files reread on a second filesystem. No gateway was signalled and no recovery performed. Execution requires all 108 windows, 114,960 arrivals, preserved terminal ledgers and every positive ACK; it will then require exact three-owner loss, unchanged provider and at least 32 seconds of confirmed absence. Concurrent-load faults remain a separate gate.

Immutable receipts: release artifact audit 1c524ecb218e2fafe2f27f98fac96673e0de2a9bbc047773d569bc0d12868c3b; pull/ref-push audit 439e4688e0043adc1918fd9333e34206b1617b74b4213c561a71352b9c38b882; owner-loss guard qualification 89ff5d1c25d971771df7f952e174e4d71af05959fa8a7fa98f1aed56a95e3ba2. All 638 frozen campaign input bindings remain unchanged. The full campaign is live and measuring fresh-object pushes; original-corpus/every-ACK recovery, matched comparisons, five-GiB transfers and isolated Linux reference capacity remain open.

[forhappy]

New negative correctness evidence and closed push accounting:

• At current PR head 5581d5c, release CI failed residency::faults::disconnected_admission_finishes_release_and_allows_a_later_restore, returning HTTP 503 from repository creation. Observed top-level results before Cargo stopped: 240 passed / 1 failed / 9 ignored; this is not a complete workspace pass. The later harness, RustFS and release-binary steps were skipped. All failed logs and the exact artifact ZIP (digest verified against GitHub) have a reread-verified second-filesystem copy. Both debug PR and debug push workflows passed: each Rust job has 244 top-level tests, nine ignored, and eight fresh RustFS gates; both 84-test harness jobs passed. These successes and the earlier candidate release success do not erase the release failure.
• Launched an unchanged-source release diagnostic, not retry-until-green: 100 isolated repetitions of the exact failing test, then two original full-target runs at four test threads. All repetitions and the exact release test executable are retained. The first diagnostic stopped before tests because the shallow checkout lacked its baseline commit; that failed setup is preserved. The corrected run passed setup and source equivalence and is executing the declared repetitions. The diagnostic branch changes only its workflow; no Canopy, test, dependency or benchmark input is changed. No root cause or repair is claimed.
• Re-audited the entire 256-KiB fresh-object-push subphase / eight windows / 1,200 arrivals: 317 OK, 736 busy drops, 147 Git errors. Preserved all 317 distinct positive ref/commit ACKs, their original parents and payload declarations: 83,099,648 acknowledged new Git payload bytes. This excludes protocol overhead and is not a remote recovery proof. Both concurrency-16 / 1-RPS windows returned 60/60 OK, but concurrency-16 / 4-RPS windows returned only 23/240 and 48/240; successful p95/p99 was 71,213.366/73,063.333 ms and 58,241.442/65,166.040 ms, with delivered in-window throughput 0.183/0.600 RPS. Thirty-two finalized source/copy pairs, deterministic selections, timing/resource and positive-payload accounting reconcile. The frozen driver still has no retained Git stderr; failed writes are not assumed rolled back.
• Prepared the fresh-fleet supervisor and immutable handoff outside the repository. Six guard tests and a real live-load refusal passed. It requires closed full-campaign/every-ACK evidence, exact three-owner loss, >=32 seconds confirmed absence, unchanged original provider/budgets, the same native executable and absent local scratch. No owner was signalled, no server started and no recovery performed. The actual post-terminal launch/recovery path remains unverified.

Immutable receipts: CI contradiction audit 1501ebf40273c369412e3c253c4a9f4acd3112acf85615d51e34f4f9606907b1; 256-KiB push audit 054b21bea943e2e6db806f9fc23fd51ba321aa36bef5568d3f855f2d98cdd934; fresh-launch guard qualification d920974678e8a4bd7defc4c62859ff8a15cb3937435fab259664abc7f91cc699. All 638 frozen campaign input bindings remain unchanged. The baseline is live on one-MiB pushes. Full 108-window closure, original-corpus/every-ACK recovery, concurrent faults, matched comparisons, five-GiB transfers and isolated Linux reference capacity remain open; no performance improvement is claimed.

[forhappy]

Updated this PR to 0ba7775 with the latest audited qualification status. All 84 local Python tests passed; documentation links/fences and diff whitespace passed. All 638 live-campaign inputs remain unchanged.

The unchanged-source release diagnostic passed 100 isolated runs and both full-target runs (104 passed, zero failed, nine ignored each). The GitHub archive, all 264 source inputs, retained ELF and every result/log binding were independently verified; 115 evidence files were copied and reread. This does not clear the original release failure, establish a fix, or qualify recovery/performance.

The update is documentation-only. No production code, assertions, dependencies, workflows, workload budgets or provider state changed. Fresh PR-head CI is separate.

[forhappy]

The complete unchanged baseline is now closed and independently audited: 108 windows / 114,960 arrivals / 59,554 OK / 55,406 failed arrivals. Every positive ACK is preserved: 1,464 creations, 1,490 Git writes, 242 LFS uploads and 19 critical workflows. This is not a performance or capacity pass.

After the closed-evidence preflight passed, only the exact three owned gateways were removed. 32.008191 seconds of confirmed owner absence was recorded, with RustFS unchanged. A first fresh-start observer race was preserved and all its nodes drained normally; a separately qualified, bounded same-child metrics wait allowed a new attempt to reach readiness. Full original-corpus and every-ACK read-only verification is running; recovery is not yet proven.

Fresh PR-head checks are green, but the earlier residency 503 remains unresolved. The seven-case stage-context diagnostic did not reproduce it: the target passed all seven cases. A different SSH SHA-256 bind test failed AddrInUse in one serial full-target run; that failure and all cases were independently audited and preserved. No production change, assertion relaxation, workload adjustment or provider restart was made.

[forhappy]

Published cb78793: bounded redacted Git failure evidence, independent schema checks and complete baseline/recovery documentation. All 91 local Python tests, Rust formatting, whitespace checks and 27 local documentation links passed. Fresh PR/push and release RustFS CI are running. Correction to the previous progress update: full post-owner-loss verification terminated at 08:13:33 UTC with a Git v2 clone exit 128; no complete stage or every-ACK recovery pass was recorded. All 3,684 failed-attempt files were copied and hash-checked, with all 651 bound inputs unchanged. The PR description and documentation now reflect this failure. No runtime, provider, frozen checkout, workload, deadline or admission budget changed for this publication.

[forhappy]

Separate recovery diagnosis: one isolated v2 clone passed exact commits, fixture digests and fsck. A fixed diagnostic covering the two original 16-entry batches also passed once through the proxy and each direct node: 32 identities per route, the original LFS body, v0/v2 clones and fsck; all 20 traced Git HTTP responses were 200. These fresh-client, already-serving-fleet probes did not reproduce or clear the original failed full-corpus/every-ACK gate and are not performance samples. Closed diagnostics and the SSH pre-fix CI evidence were audited/copied: 384 files, manifest SHA-256 619320c999c82e01c734df7be2a486152534ebd5c581ab866b83cf2458173c56. The SSH injected HTTP-reservation regression failed AddrInUse 3/3 before the fix; ten affected stock-Git runs passed and both full-target runs retained the injected failure. A listener-retention correction is isolated on 9a0ed8e, with the same 3+10+2 release diagnostic queued at https://github.com/crabbuild/canopy/actions/runs/37024939150 and verification at https://github.com/crabbuild/canopy/actions/runs/37024938646. It has not been folded into this PR pending qualification. Neither this correction nor passing probes establish a cause for the original clone failure or residency 503. Full recovery, current-pin performance and reference capacity remain open.

[forhappy]

All five checks at cb78793 have now passed. The exact release run 37023329269 was independently audited: 244 top-level Rust passes, zero failed, nine ignored (nested children counted once); 91 Python passes; all eight exact release RustFS gates; all 268 retained source/workflow inputs, six Cellule package pins and the retained ELF digest verified. Linux binary SHA-256 remains 5ff3a4daf2c2012357b4643b896e4f89b8402f7cad9c29679ea20c12becc880d. Nineteen closed files were copied and reread on a second local filesystem. Audit SHA-256: 28a209445c9ac6774eed1fd65bf0e209e3050d1cbae10632a5d81431a3be45a5; preservation manifest: 2f495842920d209684dee026da237e66d1d8b47e3c02470b216afee930bdfd2c. This is current-head Linux correctness, not original/every-ACK recovery, native latest-pin qualification or a speedup; earlier 503 and clone failures remain unresolved. A separate wrapper around the frozen full verifier now retains bounded redacted Git failure evidence, with nine passing checks and preserved before/after evidence. Its actual full-run preflight refused the unchanged 20-GiB floor with zero Git traffic and no verification workspace created. SSH candidate 9a0ed8e is now compiling under the unchanged fixed diagnostic schedule, still outside this PR pending its results.

[forhappy]

Full-scope recovery and diagnostic checkpoint (October 2):

• With the unchanged 20-GiB scratch floor satisfied, one new full original-corpus/every-ACK verification ran against the existing recovered three-node/proxy fleet. It failed at 15:57:27 UTC during the original-corpus stage: 768 identity requests included nine 30-second timeouts and one HTTP 503; nine LFS downloads included one HTTP 500. All 64 Git commands passed. No complete stage or every-ACK verification completed. This is a separate failed attempt, not replacement evidence for the earlier v2-clone failure or a cold-recovery claim.
• The supplemental Git ledger reconciles all 64 commands with the original partial ledger. All 1,454 closed attempt/input files, including all 653 bound external inputs, plus four live log/metrics snapshots were copied and reread on another local filesystem. Preservation manifest: 145eae59926887277bd18d7a3d1271dda746aa09e67cb9ed5aea35d85f6ccde1.
• A fixed metadata-only probe issued one 16-repository batch through each of four routes, without retries or deadline increases. Proxy: 15/16 matching identities; node 0: 15/16; node 1: 16/16; node 2: 15/16. All three failed requests returned HTTP 503 / Repository is unavailable. Direct-node failures show this is not confined to the proxy, but the original timeouts were not reproduced. Warm, sequential route probes are not matched performance or recovery proof. Probe receipt: ec0ae2e0d7edf17fb7ed8559ab8302bdccb0098406ad449fb777b4463cac256d; verified copy manifest: 6d541cd099b03b647533b456c1dbd3ee0e7c7df53025f89707c94ed5c088292f.
• Node logs during the failed full attempt show SQL query deadline expirations and slow lease refreshes. This warrants testing shared-provider latency versus directory/actor contention; no Cellule root cause or speedup is established.
• Clarification from the independently audited SSH candidate release artifact: all three focused reservation checks, ten stock-Git SSH checks and both full-target SSH cases passed. The second full target failed the existing residency HTTP-503 regression, not SSH. The separate debug suite failed the new reservation regression. All 264 source hashes and 15 case/log bindings were verified; audit digest d428451dd3736d331818c76e29b40141be9dfa257ad9b647f268bb9e6e6c8e87. A diagnostic-only branch run labels startup, shutdown and HTTP/SSH rebind boundaries; it is excluded from this PR and is not a claimed fix.

The live fleet and RustFS provider remain unchanged on the frozen 0dc04a6 baseline. The latest 0f4ca09 pin has passed push verification; release qualification must be evaluated separately. Full recovery, every acknowledged write, matched performance, higher admission profiles, concurrent faults, large non-sparse transfers and isolated Linux capacity remain open. Local evidence copies are not off-machine backups.

[forhappy]

All five checks for 7497fc720e68a440e27614a83a1442576511cfd2 / Cellule 0f4ca0919b0dfe20a3dcd964d21da03135e42eed have passed.

The closed Linux release run was independently audited: 244 top-level Rust tests passed, zero failed, nine ignored; two nested child results were counted once; 91 Python tests and all eight exact fresh RustFS compatibility gates passed. All 268 source/workflow input hashes, the six locked Cellule package sources, release output and ELF digest match the tested commit. Cargo changes normalize exactly to the prior pin: no unrelated dependency, feature, implementation or runtime-budget changes.

• GitHub artifact archive SHA-256: d2cb4060a1f6dc68dc122a53a833f2be04e0e3f812df21b88147cde48008329b.
• Retained Linux executable SHA-256: 381ec918026d7d6421a86b0ed916ff40b847c83dccdc881f82624c104c88b33b.
• Independent audit SHA-256: 7c28852384d53d5265721af4daa58b37b898b08e5525ff49a61dbce29642abc6.
• Verified 19-file local copy manifest SHA-256: 16c7775c6525aa3fa7cd42b177a9ff7dba98d4f4118ffafb68f58d2994ac5276.

The executable was inspected, not run on the Mac. The live baseline fleet and RustFS provider remain unchanged. This establishes Linux release correctness on a fresh fixture, not native qualification, retained-store upgrade, full-corpus/every-ACK recovery, matched performance or isolated capacity. Both failed full recovery attempts and earlier CI failures remain retained and unresolved. The dependency document's pending status records publication time; this comment records the now-closed qualification for the exact published head.

[forhappy]

Updated this PR with commit e8eef77: the performance plan now names the current Cellule pin, separates current verification from historical checkpoints, and records both failed full recovery attempts. The dependency checkpoint includes the independently audited Linux release results at 7497fc7: 244 top-level Rust tests, 91 Python tests and all eight fresh RustFS gates.

This update changes only two documentation files. All 91 local Python harness tests passed and git diff --check passed. New PR-head CI is pending; previous green checks are explicitly bound to 7497fc7.

A separate test-only diagnostic, excluded from this PR, now reproduces a recovery defect on Cellule 0f4ca09: after the SQL deadline, a fenced handle and confirmed idle/no-owner Control with unchanged root, the next metadata request returns HTTP 503. Three isolated release cases and one full-library run failed at that assertion; the debug suite failed it too. I independently checked all 270 diagnostic source/workflow hashes, six locked Cellule packages, the retained ELF digest and all four result/log bindings. Diagnostic: https://github.com/crabbuild/canopy/actions/runs/37034626141

No recovery fix, end-to-end recovery pass or performance gain is claimed. The frozen native fleet and RustFS provider were not restarted or upgraded for this publication.

[forhappy]

Publication checkpoint: all five checks for the current PR head e8eef7702f09f1fb582ad2517ab3996e367ff71b now report success, including release correctness against fresh RustFS. The PR is open and mergeable. The independently audited artifact counts in the documentation remain bound to 7497fc7; the new-head artifact has not yet received that independent audit.

The recovery-fix experiment remains excluded from this PR: its focused deadline regression passed, but broader debug and release suites failed the existing cross-account cold-admission regression. A separate safety diagnostic is still running; no experimental runtime fix has been promoted. Complete original-corpus/every-ACK recovery and matched performance remain unverified. The frozen native fleet and RustFS provider are unchanged.

[forhappy]

Recovery investigation checkpoint: the independently audited safety diagnostic at 025c03a passed three unchanged deadline cases, three pinned-file/acknowledged-data cases, three cross-account admission cases and the 122-test library. Its full multi-server case failed the existing disconnected-admission regression (HTTP 503); release workspace failed that same regression, while debug CI passed. The diagnostic metadata fallback is excluded from this PR and is not a production correction.

The audit observed 254 resident-only misses that still had valid active local owners. Draft Cellule PR #44 adds metadata-free active-owner lookup, keeping resident-only lookup and fencing semantics intact. Rust workspace/MSRV and fresh RustFS three-process smoke passed; a separate follower-capacity window failed (59/60 completed, one scheduler-late arrival, matching expected/actual readback for all 12 entities), so the framework candidate remains draft.

Separate Canopy candidate d1f9250026bd517576793af44bdb4fc849cedce7 pins that API commit and removes the diagnostic metadata fallback and debug logs. Fixed 14-case diagnostic, debug verification, and release/RustFS qualification are running. No candidate code was added to this PR or deployed to the frozen three-node fleet. Full original-corpus/every-ACK recovery and matched performance remain open.

[forhappy]

Updated the PR description at unchanged head e8eef77: all five current-head checks are green and GitHub reports MERGEABLE.

The separate metadata-free recovery candidate d1f9250 now has independently audited positive evidence:

• Diagnostic run https://github.com/crabbuild/canopy/actions/runs/37042619416: all 14 declared cases passed. Exact source, dependencies, both release test executables and case logs were audited; audit e13f51a4a18af22f16c77502fc4e53654c591168ae9180b0459de4bfe97206a3.
• Release run https://github.com/crabbuild/canopy/actions/runs/37042781552: 247 top-level Rust tests passed, nine ignored; two nested subprocess results were not double-counted; 91 Python tests and all eight fresh RustFS gates passed. All 270 source/workflow inputs and six locked Cellule packages were checked against the exact revision. ELF SHA-256 32489617877c9a4d1ec2e0514cccbae040350a3a571c6862fe6e5663d886050c; audit e9f01be96193095af1329151c50f25fcf321f777226de652945a2c7840571bc0.
• Release evidence copied and reread on another local filesystem (20 files), manifest aceac946a87319c11f46660011750762966defcf2b528e526624737a752f8a89. Diagnostic evidence likewise preserved (31 files), manifest 83d822b5b5a2dcbadf27baff6ea0550b1d8560197d6df88ae0d1a2facb8fc545. These are local preservation, not off-machine backups.

Candidate code remains excluded from this PR: it depends on draft Cellule #44, whose follower-capacity proof failed 59/60 arrivals (one scheduler-late arrival; all 12 readbacks matched), and has not passed native retained-store or complete original/every-ACK recovery. Temporary admission snapshots at 9a5ff42 are also excluded; their fixed diagnostic workflow passed but artifact inspection is still pending. Passing scheduled runs do not establish that historical timing failures are resolved.

The live three-node fleet, RustFS provider and UI were not rebuilt, restarted or changed. No matched speedup or isolated Linux capacity is claimed.

[forhappy]

Authoritative fleet revalidation corrects the earlier live-fleet status: supervisor session closed with exit 1, and PIDs 10143/10617/10794/12878/13313 are absent. The final supervisor record is timestamped 2026-10-02T17:40:07Z. All three node logs report node lease bounds are invalid, unconfirmed Cell drain, and Server(Runtime(Fenced)); outcome records all three exit 1, forced=false. This is not a successful graceful drain. No new fault or restart was injected. RustFS remains running with the same container/image/start/resource envelope and no OOM or restart.

Preserved and reread 1,644 closed fleet/input files (328,057,447 bytes), including all local runtime database/checksum files and 658 historical input bindings, on another local filesystem. Manifest SHA-256 727ba4ef7b9d5870da117cc1e1c0e79a8cc2e0d0968821fba3d7d4dbd559ec6d. No data cleanup, provider writes or reseeding occurred. These are local copies, not off-machine backup.

The separate diagnostic-only 9a5ff42 run https://github.com/crabbuild/canopy/actions/runs/37044047072 passed its fixed 21-case schedule, including ten disconnected-admission cases. Debug verification https://github.com/crabbuild/canopy/actions/runs/37044047338 also passed. The diagnostic schedule did not reproduce the historical failure; no production timing fix is inferred. The new terminal baseline failure is not evidence against the separate candidate, which was not deployed into that fleet. Root cause remains unproven.

Next: native qualification of the instrumentation-free candidate, using external build/scratch storage because the primary volume has less than the existing 20-GiB qualification floor. Retained-corpus recovery and every-ACK verification remain mandatory before qualification claims.

[forhappy]

Verification checkpoint — October 2, 18:55 UTC

The published PR head remains e8eef7702f09f1fb582ad2517ab3996e367ff71b: mergeable, all five checks successful. No unqualified candidate was promoted into this branch.

New retained-corpus evidence:

• Reconciliation of the complete closed campaign and positive-ACK ledgers passed: 11,504 distinct original/ACK repository identities, preserving all 1,464 creations, 1,023 ref-only pushes, 467 fresh-object pushes, 242 LFS uploads and 19 critical workflows. This checks ledger integrity, not remote recovery. Required-identities receipt SHA-256: 9f095648e0bd2258f78c97e1013401045416acd0538dd87ff7f9081bee47efd5.
• The supported old native binary completed a read-only maintenance status check against the unchanged original RustFS provider, at 18:43:58 UTC: release/image unchanged, revision 9, zero advertised sessions, 516 unsettled Cells, and drained=false. No maintenance begin/recover/end, activation, owner restart or provider mutation was performed. Status log SHA-256: 8447acad0881e1322de45001d3ffee2085897caa675c32d7505a0711ac91e618. Both closed files were copied and hash-verified on another local filesystem; preservation manifest: 76f942cdcbe6b4646e28caf5a12b83a9cf3e8a5eca114bfa5ed596e2b7b459b4.
• Native Mac qualification of the separate instrumentation-free d1f9250 candidate, pinned to Cellule 4992118, is still running on external build/cache/scratch storage. Formatting and release workspace/all-target Clippy passed; release tests are now executing. There is no final native verdict yet. Its existing independently audited Linux release result remains 247 top-level Rust tests, nine ignored, 91 Python tests and eight fresh RustFS gates.
• A full-catalog, write-disabled inspector has been prepared to classify all retained catalog entries, including failed-request leftovers separately from acknowledged repositories. It has not yet been compiled, tested or executed, and supplies no upgrade admission evidence.

The earlier full-recovery failures and terminal fleet failure remain unresolved. Complete original/every-ACK recovery, native fresh-RustFS qualification, deployment upgrade and matched performance/capacity comparisons remain required. Cellule #44 remains a draft; its follower capacity check failed and no speedup is claimed.

[forhappy]

Native candidate checkpoint — October 2, 19:28 UTC

The separate instrumentation-free d1f9250 candidate, pinned to draft Cellule #44 revision 4992118, now has independently audited native Mac evidence:

• Formatting and release workspace/all-target Clippy passed; 247 top-level Rust tests passed, nine ignored, and 91 Python tests passed. Six exact locked Cellule packages and all 270 source/workflow bindings were verified. The retained standalone Mach-O executable is SHA-256 a6e827989d850e34101276d7ddbf3698b1a2171e73183c298f5d21b8620111ef.
• All eight fresh RustFS test-artifact gates passed: SHA-256 round trip and merge candidates, signed push, signed SHA-256 SSH, stock SSH, bulk mirror, filtered clones and SSH/LFS. The actual multi_server test executable is SHA-256 99640caa91b0eeb198f6498d0dc8ea26ba680b5a6a467465caa6399b63aa67be. Exact gate accounting, provider identity/envelope and source bindings were independently audited. Native gate audit: 178064b07b05c31d4082499ccca8231834119ad7515703d3cb3343d103a7dd73; local preservation manifest: 633f58ce2c827f06458e00a67ba5a826526162c166e2f2234509ee73e1161590.
• Two evidence-collector failures remain preserved separately: metadata stderr contaminated JSON, and Cargo reselected the previously cached test-dependency CLI in the mutable target path. Supplemental receipts use separated streams and actual artifact identities; three regression tests cover each collection seam. No application tests or builds were rerun to obtain these passing results. The retained standalone launch executable stayed unchanged. Fresh-provider gates execute the test artifact, not that standalone CLI; deployment/CLI recovery proof remains required.
• The temporary fresh RustFS container and volume were removed by the unmodified fixture helper; provider logs and lifecycle receipts were retained and copied. The original retained provider identity/resource envelope remained unchanged. No original-corpus maintenance, activation, restart, reseed or cleanup was performed.
• A write-disabled full-catalog inspector is being compiled on a separate external target. Fresh offline resolution failed on an already-locked yanked dependency; that failure was preserved. Seeding from the exact qualified lock allowed root registration without dependency-version/source drift. No runtime inspection verdict exists yet.

These results are not a retained-store upgrade, original/every-ACK recovery, historical-failure resolution, five-GiB qualification, matched speedup or isolated Linux capacity. The candidate remains excluded from this PR. Upstream Cellule main was rechecked and remains 0f4ca09; Cellule #44 stays a draft with its follower capacity failure unresolved. The original retained status still has 516 unsettled Cells, and both earlier full recovery attempts remain failures.