feat(protocol): publish Git and Xet state through capsules

[forhappy]

Status

Protocol v2 includes per-ref publication, Xet xorbs/shards, stable layered packs, and exact-tip fetch. This PR is still under qualification; v1 should remain available until correctness, performance, and product-parity gates are met. Current head: 13ff72520faef3fc3468b65ff5f5a05e945344c9.

Incremental-fetch object-store request count is diagnostic, not a pass/fail threshold. The latest 500-commit fetch completed correctly in 5.55 s and installed one pack with 27 object-store requests; that request count is acceptable for this workload. Correctness, fetch latency, push scaling, and clone behavior remain relevant qualification measures.

RustFS qualification evidence

A Kubernetes-derived 5,000-commit replay passed exact-tip checks, strict Git and Crab fsck, cold/warm clones, and sampled blob-digest verification. Push mean/p95/p99 was about 335/579/932 ms with about 7.062 mean origin requests. Ten fetch/repack intervals completed in about 3.1–6.3 s and installed one pack each. Full evidence: crab/docs/benchmarks/capsule-v2-kubernetes-5000-rustfs-ga.md.

Before the current copy-on-write change, the cold clone took 68.149 s and warm clone took 25.703 s. The current head adds copy-on-write materialization for cached Git packs, with verified-copy fallback and no hard links. This should avoid copying cached pack contents on supported filesystems, but a full post-change clone benchmark has not yet measured the effect; no speedup is claimed yet.

A separate 100 GiB Xet workload passed 3,850 file-byte comparisons, but the metered seed push saw three 60-second proxy timeouts on duplicate xorb PUTs. The cause was not established. Cold 100 GiB hydration took 37.9 minutes, so Xet performance is not qualified. Evidence: crab/docs/benchmarks/capsule-v2-xet-100g-rustfs-ga.md.

Remaining gates

• Run the full clone and Xet qualification on the current head; retain correctness and latency evidence.
• Resolve the observed Xet proxy timeouts and high cold-hydration cost.
• Complete matched-v1 performance and hosted-provider/platform, replica, tiering, mount, browser, backup/restore, restart, and reclamation parity before considering v1 retirement.
• Current-head CI is still running. No green CI claim is made.

Design and release gates: crab/docs/design/capsule-layered-packs.md and crab/docs/design/capsule-xorbs-shards.md.

[forhappy]

Follow-up qualification and hardening (commit 9393caf):

• Terminal upload-pack now retains delta bases external only when the request is unfiltered, non-shallow, non-deepen, advertises thin-pack, and every client have is authenticated in the pinned v2 visibility plan. OFS deltas are rewritten to REF_DELTA; the full dependency sort/materialization path remains for filtered, shallow, incomplete, and legacy requests.
• Catalog-selected objects retain authenticated physical pack order; legacy materialized selection keeps canonical OID ordering.
• Fresh release binary v2-tiny-external-final-20260917 on RustFS: two incremental pushes 304–327 ms at 8 requests each; two fetches 225–247 ms at 9 requests; final clone 501 ms at 18 requests; matching tip and strict fsck passed.
• Focused tests: crab-remote-git 133, crab-read 188, upload-pack wire 35; cargo check -p crab, release build, architecture gates, and format/diff checks pass.

The Kubernetes 5,000-commit stress artifact remains explicitly negative at the 500-commit fetch/repack boundary because the interrupted pre-fix repository state still requires a large historical pack; it is not being reported as parity proof. Hosted-provider, multipart, replica/tiering, mount/browser, S3 gateway, migration/recovery, and backup/restore rows remain release gates.

[forhappy]

Parity follow-up (commit d6431f8): removed the stale client rejection for protected pushes carrying a v2 mirror-plan ID. The plan ID is already authenticated in CapsuleTransaction::for_plan; the auth-server capsule publisher commits the same transaction-scoped capsule plan receipt. The protected capsule receive test now exercises that planned transaction, verifies the receipt, and retries successfully. cargo test -p crab-auth-server --lib (101), capsule-push tests (9), and the release build pass.

[forhappy]

Added external_thin_subset_pack_keeps_the_proven_base_outside_the_pack (commit 36f2ea6). It generates the public external-base thin-pack API from a real REF_DELTA fixture, verifies the one-object thin pack with strict git index-pack --fix-thin, and passes.

[forhappy]

Documentation follow-up (commit 0233c25): the main capsule publication design now records the authenticated external thin-base rule and protected mirror-plan receipt path alongside the parity inventory and RustFS evidence.

[forhappy]

V1 parity pass

Implemented and pushed in 6ad3b0b0e9a (rebased onto current main):

• Legacy read replicas now use the v1 manifest/index/object readiness contract only when v2/root is absent; a present or corrupt v2 root never downgrades. The resolver accepts verified legacy manifests, and tests cover both acceptance and corrupt-root rejection.
• migrate import, migrate export, and adopt --rewrite-history now use the built-in verified fast-export/fast-import engine with v2 Xet staging/hydration. Shared Git blobs are converted inline only for selected paths; ref rollback is attempted on post-import failures, checkout failures are surfaced, and staging is closed before success.
• Remote snapshot download/export, mount, hydrator, browser/HTTP, protected publication, and checkpoint readers carry one authenticated v2 view and immutable pointer catalog through reconstruction.
• Documentation now contains a complete v1 product-parity inventory, cross-surface contracts, closure order, and Level-3 acceptance gates.

Proof after rebase:

• cargo check -p crab --locked passed.
• cargo test -p crab --locked --lib: 4,250 passed, 0 failed, 3 ignored.
• cargo fmt --all -- --check, git diff --check, and python3 crab/scripts/check-architecture-gates.py passed.

Remaining release blockers are intentionally explicit: hosted-provider checksum/multipart and 5,000-commit current-format replay; managed replica failover/repair; tier/archive restore; mount range/cancellation/unmount; browser/HTTP load and fault matrix; S3 gateway operation/concurrency/restart matrix; lifecycle/workflow/admin inventory; backup/restore export inventory; and migration fault/resume/provider plus older-Git/interrupted/adversarial qualification. Full v1 production parity is not claimed until those Level-3 gates pass.

[forhappy]

Parity closure update (c2d87d8):

• Hydrate and remote mount now share one restore-availability adapter. It is built from the resolved physical store identity, so managed and replica read views target the bucket that owns the authenticated v2 catalog instead of reconstructing a provider from the logical crab:// URL.
• --no-restore does not construct a cloud restore client, but archived shard/xorb reads still fail closed with the typed archive-class admission error.
• A standalone crab:// mount now refuses to start if its authenticated v2 read context cannot be built; it no longer starts with stub readers and defers the failure until first pointer access. Local Git-native mount fallback is unchanged.
• Design matrix and closure notes are updated in crab/docs/design/capsule-xorbs-shards.md.

Local proof after this change:

• cargo test -p crab --locked --lib: 4,254 passed, 0 failed, 3 ignored.
• cargo test -p crab --locked --lib cmd::mount: 120 passed before the final guard, plus the two new fail-closed/fallback tests passed individually.
• cargo check -p crab --locked, cargo fmt --all -- --check, git diff --check, and python3 crab/scripts/check-architecture-gates.py all pass.

This closes the local wiring gap, but is not a claim of complete v1 production parity. Release gates remain: live S3/GCS/Azure lifecycle and restore behavior; replica readiness/failover/repair; restored-content verification; the full FUSE/NFS range/cache/cancellation matrix; browser and smart-HTTP load/fault coverage; S3 gateway restart/concurrency/request-count coverage; and migration, backup inventory, and delete/restore qualification on populated v1/v2 repositories.

[forhappy]

Follow-up test hardening: the focused mount module now passes 122/122. I also serialized the unmount test's HOME override through the existing test guard; this removes a process-global HOME race that could make local_pipeline_config_rejects_active_cache flaky when mount tests ran in parallel. This is test-only and does not alter local Git-native fallback behavior.

[forhappy]

Final local rerun after the test-only race fix: cargo test -p crab --locked --lib --quiet passed 4,254 tests (3 ignored) in 99.51s; cmd::mount passed 122/122. No working-tree changes are pending other than pre-existing generated Python __pycache__ directories, which were not added.

[forhappy]

Layered-pack qualification update (commit 1640af9):

• Fixed the ref-only layered-run regression: physical pack source validation now runs only for runs that carry Git members; ref-only runs still retain authenticated visibility and transition checks.
• Exact release build verified on local RustFS/Kubernetes fixture: 1.1 GiB cold git clone --no-checkout completed in 3.75 s; default checkout clone completed in 8.04 s; exact tip 71f0fc6e72d53d5caf50b1314ca4d754463117f0; connectivity fsck passed; 26,892 files checked out.
• Test proof: crab-read 200/200, remote-helper 139/139, metadata 210/210, release build, formatting, and diff checks pass.
• The earlier ~20 s shared-volume result is destination write contention (RustFS and destination sharing the qualification volume), not object-store request or connectivity amplification.

The 5,000-commit replay with 500-commit fetch/repack checkpoints and hosted-provider matrix remain explicit release gates; v1 is not being retired until those pass.

[forhappy]

Follow-up pushed in ee72375b4b09b3df74d90b15ffcecfcd5edd8868.

The first fresh PR-208 5,000-commit run found a correctness blocker before replay could continue: an append-only in-memory visibility dictionary could be non-canonical when serialized into the layered capsule (CRAB-E0020: layered visibility dictionary is not in canonical order). The fix canonicalizes the OID dictionary at the capsule boundary and remaps ref, transition, and history ordinals; it preserves the append-only runtime representation and keeps the decoder fail-closed.

Proof for the fix:

• cargo test -p crab-metadata --lib --locked: 211 passed
• cargo test -p crab-read --lib --locked: 200 passed
• cargo test -p crab --lib git::remote_helper --locked: 139 passed
• cargo fmt --all -- --check and git diff --check: passed

I am rebuilding the exact release binary from this commit and rerunning the fresh local-RustFS qualification. The earlier run remains recorded as a failed negative qualification at seed + replay 1; no 5,000-commit success is claimed until all replay, checkpoint fetch/repack, clone, and fsck gates pass.

[forhappy]

Layered-pack follow-up pushed in 4ed9664ab70 (on top of 042c129cd55):

• The writer now canonicalizes append-only visibility ordinals at the wire boundary and remaps refs, transitions, history closures, and authenticated member admission through the same old-to-new map. This closes the two fresh-replay correctness failures found after the ref-only fix.
• crab-metadata passes 212/212 and crab-read passes 200/200 on the patched tree.
• The local-RustFS smoke (seed + 10 replay pushes, fetch/repack checkpoints at 5 and 10) now passes every push, fetch, repack, tip, and fsck gate; incremental pushes remain about 1.04 s after the seed.
• The design doc records the remaining cold-clone blocker: a multi-member checkpoint correctly declines the one-source/one-member direct installer, then the normal path performed 357,313 uncoalesced range reads in 163 s before the run was intentionally stopped (1.89 GB read, 15.2 GB inflated). This is not being reported as a cold-clone or 5,000-commit qualification pass.

The PR is updated with the fixes and evidence, but v1 is not retired and the 5,000-commit/multi-pack cold-clone gates remain open until the authenticated multi-member install or equivalent whole-member union path is implemented and requalified.

[forhappy]

Post-push test completion for 4ed9664ab70:

• cargo test -p crab-metadata --lib --locked: 212 passed
• cargo test -p crab-read --lib --locked: 200 passed
• cargo test -p crab --lib git::remote_helper --locked: 139 passed
• cargo fmt --all -- --check and git diff --check: passed

The branch remains clean apart from the pre-existing untracked local crab.toml RustFS config, which is not part of the PR.

[forhappy]

Updated in commit 1cfdd63 (perf(fetch): preload layered locators for cold clones).\n\nWhat changed:\n- Complete layered views now coalesce and authenticate index, reverse-index, and kind-bearing locator sidecars, then expose inline locators to the planner.\n- Ordinary incremental haves stay on the footer/tip-bound path; cold, filtered, shallow, and tag requests promote only when complete visibility is required.\n- Multi-member cold clones no longer fall back to per-object visibility reads; direct one-pack installation remains fail-closed.\n- Design history and qualification evidence are recorded in crab/docs/design/capsule-layered-packs.md.\n\nVerification:\n- crab-read: 200 passed.\n- remote-helper: 139 passed.\n- upload-pack wire: 39 passed.\n- Local RustFS, Kubernetes-derived fixture: blob:none clone 14.50 s with 173 ms planning; cache-miss shallow blob:none clone 6.12 s with 1,095 planning reads and 372 terminal response-pack reads; unfiltered multi-member clone 85.94 s, exact source tip, native git fsck --full clean.\n- Full, filtered, and shallow clone tips all match the source.\n\nThe fresh 5,000-push/fetch/repack matrix, hosted-provider latency, and v1-retirement gates remain open; this update does not claim those are complete.

[forhappy]

CI follow-up: GitHub did not emit a synchronize run for the new head, so I manually dispatched the current commit (1cfdd63) against the repository workflows:\n- CI: https://github.com/crabbuild/crab/actions/runs/35675208869\n- Git protocol v2 partial-clone qualification: https://github.com/crabbuild/crab/actions/runs/35675210814\n- Large repository RustFS qualification: https://github.com/crabbuild/crab/actions/runs/35675212409\n\nAt this update they are queued, not yet green; the prior completed run was for an older head.

[forhappy]

Pushed dba2cb4 (fix(protocol-v2): retain negotiated fetch haves).

What changed:

• Accumulate and de-duplicate protocol-v2 haves across negotiation rounds before any tip-bound → complete-view promotion.
• Copy the complete negotiated have set into the terminal done request, so incremental planning remains wants-minus-haves instead of regenerating the repository.
• Added design evidence in crab/docs/design/capsule-layered-packs.md.

Proof:

• cargo test -p crab --lib upload_pack_wire --locked: 39 passed
• cargo test -p crab-read --lib --locked: 200 passed
• cargo test -p crab --lib remote_helper --locked: 140 passed
• Full-history K8s + local RustFS: incremental fetch after pushes 1 and 5 passed with exact tips and no missing objects; response packs were 49.9 MiB and 55.7 MiB instead of the prior 1.09 GiB complete response.

Qualification status remains honest: the replay later reached an existing 503,980,520-byte Crab/Xet pointer commit and stopped with CRAB-E0086 because the replay harness had not staged its local chunks. The 5,000-push/xorb qualification gate is still open; this is a staging-contract failure, not evidence that the haves fix is incorrect.

[forhappy]

Pushed follow-up commit 1696f53 to PR 208.

The fresh full-history GitHub-origin Kubernetes RustFS qualification (pr208-v2-fresh-github-smoke2-20260921, binary dba2cb4) passed seed publication, protocol-v2 incremental fetches at pushes 1/5/10, exact tip checks, cold and warm full clones, and native fsck. Measured incremental fetches were 33.457s / 15,494 storage range reads / 50.1MB response at push 1 and 24.227s / 16,999 reads / 55.5MB response at push 5. Cold full clone was 217.5s with 10 store requests; warm clone was 99.8s with zero store requests.

The run stopped only at the blobless-clone qualification assertion blob-none-ordinal-metadata-lookup: the exact ordinal-metadata lookup branch emitted no locator_lookup_mode event, so the harness observed zero metadata events even though the request completed through the catalog-filter plan. The new commit adds that trace at the crab-metadata reader boundary; no data-path or authorization behavior changes. Focused crab-metadata tests pass (212/212).

Fresh workflows have been dispatched at this new head:

• CI: https://github.com/crabbuild/crab/actions/runs/35684015947
• protocol-v2 partial clone: https://github.com/crabbuild/crab/actions/runs/35684017903
• large repository RustFS: https://github.com/crabbuild/crab/actions/runs/35684019821

I am not marking the PR green until those runs complete.

[forhappy]

Pushed f0db75ac994 to PR 208.

This fixes the v2 incremental-fetch regression caused by generation-owner checkpoint compaction: the control-only reader previously saw no live capsule transitions after compaction and fell back to a visibility traversal. Layered checkpoints now carry a bounded, authenticated recent per-ref transition suffix in the footer. Control-only fetches consume that suffix; older/incomplete have chains still fail closed to the existing catalog/traversal planner. The complete ordinal visibility body remains authoritative for cold/strict paths.

Proof:

• cargo fmt --all --check
• cargo test -p crab-metadata --lib --locked: 213 passed
• cargo test -p crab-read --lib --locked: 200 passed
• local RustFS tiny fixture, committed binary SHA matched source SHA; seed + 10 replay pushes, incremental fetches at 1/5/10, clones, and fsck reached exact tips. Incremental logs report strategy=tip_bound_transition, visibility_plan_ms=0, and exact-pack-member responses.
• Qualification report is intentionally not called fully green: the tiny fixture's blob-none-ordinal-metadata-lookup check is false because that small clone selected the valid catalog_filter path; the incremental correctness gates all passed.

Fresh manual workflows for this commit:

• CI: https://github.com/crabbuild/crab/actions/runs/35687962739
• protocol-v2 qualification: https://github.com/crabbuild/crab/actions/runs/35687958669
• large-repository RustFS qualification: https://github.com/crabbuild/crab/actions/runs/35687960517

[forhappy]

Qualification update for f0db75ac994:

• The local RustFS K8s replay is still running with the refreshed GitHub-origin source and 5,000 replay commits. It has completed 302 pushes so far with no push failure, missing-object error, or tip mismatch.
• Seed publication, cold clone, and incremental fetch gates at pushes 1, 10, and 100 have passed with exact tips. At push 10, the v2 transition planner took 0 ms, used 3 object-store reads for 149 objects, and generated the response pack in about 30 ms.
• This is not a green qualification result yet: the full replay, 500-push fetch/repack gate, final clone, and fsck are still pending.
• The three fresh workflows for this head remain pending/queued: CI, protocol-v2 qualification, and large-repository RustFS qualification.

[forhappy]

Update: pushed a1d745f41abc9234c18bdebfd4d092318fa913a8 (perf(metadata): bound visibility updates and fetch history).

• crab-metadata lib tests: 214 passed; targeted visibility tests: 17 passed.
• Release binary built from this commit.
• Local RustFS/Kubernetes-shaped K8s qualification is running with the exact binary, seed + 5,000 first-parent pushes, fetch/repack every 500, and end-to-end verification.
• At 500 pushes: 501/501 pushes succeeded; visibility planning used tip_bound_transition in 1 ms; incremental fetch completed in 6.95 s with 51 measured object-store requests (53 in the operation log). The old pre-change 500-fetch control was 69.7 s and 74,135 requests.
• The remaining fetch latency is Git pack/index materialization; the v2 server still generated a response pack for this protocol-v2 request. The 5,000-commit run is intentionally still in progress, so this is not being claimed as final green qualification yet.

Known local gate: workspace cargo clippy -p crab-metadata --lib -- -D warnings still reports six pre-existing/unrelated lint failures in adjacent capsule/visibility code; no lint suppression was added.

[forhappy]

Qualification update from the exact a1d745f binary: the 1,000-push checkpoint completed with 1,001/1,001 pushes successful. Owner/checkpoint maintenance was 608.9 s (two active packs, peak child RSS ~0.86 GiB). The 1,000-commit incremental fetch completed in 59.7 s, with 100 storage requests and 6.53 s pack generation; visibility planning remained 24 ms. This exposes the remaining v2 bottleneck clearly: the protocol-v2 server is still materializing response packs as the requested delta grows, so PR 208 is not yet a final sub-second/under-10-request qualification. The local 5,000 replay remains in progress.

[forhappy]

Focused regression gate from a1d745f: cargo test -p crab-read --lib capsule_protocol --locked -j2 passed 21 tests (179 filtered), covering layered member admission, authenticated pack identity, coalesced/split range windows, control-only checkpoint reads, and incremental transition retention.

[forhappy]

Qualification update: the 1,500-boundary checkpoint completed with 1,501/1,501 pushes successful. Owner maintenance was 511.8 s (peak child RSS ~1.69 GiB, two active packs). The incremental fetch then completed in 15.4 s with 146 storage requests and 17,445 logical objects; visibility planning was 1 ms. This reinforces the outstanding protocol-v2 response-pack scaling gap; correctness remains intact and the 5,000 replay is continuing.

[forhappy]

Qualification update: the 2,000-boundary checkpoint completed with 2,001/2,001 pushes successful. Owner maintenance was 659.3 s (peak child RSS ~0.79 GiB, two active packs). Incremental fetch completed in 5.88 s with 126 storage requests and 20,279 logical objects; visibility planning was 3 ms. Fetch wall time varied versus the 1,500 sample, but the request count remains dominated by protocol-v2 response-pack reads. The 5,000 replay continues with no correctness failure.