WIP Shadow test ai - #1803
Draft
lispc wants to merge 119 commits into
Draft
WIP Shadow test ai#1803lispc wants to merge 119 commits into
lispc wants to merge 119 commits into
Conversation
…d in current code base)
… blob-base-fee); accept Fusaka-era fee floor
Simulate a production zk upgrade with continuous finalization on a single fork/DB: Phase 1 runs the production stack against the fork's untouched production MVRV verifier, then 20-upgrade.sh cuts over to the new stack (coordinator assets swap + restart, prover restart, new wrapper registered via the genuine updateVerifier path so the old wrapper moves to legacyVerifiers). Bundles proven pre-boundary keep the old verifier (mixed mode); everything at/after the boundary batch is reset and re-proven. - follow/scripts/20-upgrade.sh: cutover orchestrator (boundary computation with hard-cutover fallback, in-flight task reset, genuine registration, MVRV routing verification, boundary recorded in follow-run.env) - 10-follow-up.sh: --skip-verifier for Phase 1 (use the fork's production verifier; assert routing, code presence, protocolVersion) - lib/03-deploy-verifier.sh: --start-batch and --genuine-register (real updateVerifier instead of forcing the latestVerifier storage slot) - lib/04-prover-up.sh: PROVER_BIN and ASSETS_DIR env overrides so phases can point at binaries/assets from different checkouts (e.g. a develop worktree for Phase 1) - lib/01-setup-anvil.sh: --skip-verifier flag; fix the fallback that registered a codeless wrapper because cast code returns 0x for empty accounts - monitor-catchup.py: verifier drift check follows MVRV routing (getVerifier(lastFinalized+1)), works across the two-wrapper boundary - generate-catchup-report.py: split metrics pre/post upgrade - Makefile: follow-old / follow-upgrade targets - docs: mid-run upgrade section in follow/GUIDE.md incl. how to determine the production zk stack; new docs/bundle-digest-encoding.md (v0.8.0 S3 digests are Montgomery, v0.9.0+ canonical); Traps 30-34 in the TROUBLESHOOTING files; AGENTS.md updates
…eration - Pin scroll-zkvm-* to zkvm-prover master bf887150 (halo2-gpu support): the SNARK (bundle) aggregation prover can now run on GPU via the new `halo2-gpu` cargo feature (24 GB-class GPUs), exposed as the `prover_halo2gpu` target in zkvm-prover/Makefile. - Prover now downloads agg_vk.bin alongside app.vmexe/openvm.toml: without the pre-built aggregation VK the prover derives it from the SDK, which is slow and allocates GPU memory that is never reclaimed (the issue the upstream commit fixed). agg_vk.bin must exist on S3 next to each circuit. - Cargo.lock: revert revm-family resolution drift introduced by `cargo update -p scroll-zkvm-*` (alloy-evm back to revm 30.1.1, revm-primitives edges back to the scroll-v91 fork) so the [patch] fork resolution keeps compiling. See follow/TROUBLESHOOTING.md Trap 32. - Docs: new-stack checklist in follow/GUIDE.md mid-run upgrade section, Trap 32 (revm drift) + Trap 33 (agg_vk.bin / batch_root_verifier_vk mandatory assets) in follow/TROUBLESHOOTING.md, AGENTS.md build command and S3 asset notes.
…_vk.bin
libzkp/coordinator:
- Batch-proof verification now reads the batch circuit's agg_vk.bin from
the assets dir instead of a separate batch_root_verifier_vk file (same
content, one less artifact to track).
Upgrade-run fixes learned from the first live 20-upgrade.sh run:
- 20-upgrade.sh reset SQL: also reset rows already at proving_status=1 —
their ready flags (batch_proofs_status/chunk_proofs_status) and proof
blobs can still reference old-circuit proofs — and DELETE stale assigned
prover_task rows while provers are stopped. Previously the new
coordinator rebuilt poisoned tasks from old-format proofs on every poll
and the priority dispatcher starved all task types (Trap 35).
- 03-deploy-verifier.sh: accept foundry >= 1.6 'cast receipt' status
format ('1 (success)') so a successful updateVerifier is not misreported
as reverted (Trap 36).
Docs:
- TROUBLESHOOTING: Trap 34 (stale ProverProofInvalid rows silently starve
batch assignment; sweeper does not clear prover_task), Trap 35
(post-upgrade leftover state), Trap 36 (cast status format); Trap 33
rewritten for the agg_vk.bin-only asset story.
- GUIDE: upgrade notes — aggressive reset rationale, slow first chunk
task generation (per-block witness fetching) is normal.
- AGENTS.md: coordinator reads batch circuit's agg_vk.bin.
…_hook_commit Two crashes hit the first halo2-gpu bundle proving on the shadow fork (both provers died at the SNARK stage, peak 22.9 GiB): 1. quotient.cu cudaErrorInvalidConfiguration: UniversalHandler::agg_vk() built the child's full GPU aggregation prover (~5.7 GiB/circuit, never reclaimed by the VPMM pool) just to read the VK, starving the SNARK phase until quotient chunking computed batch_size = 0. Now uses Prover::load_agg_vk() which reads the pre-built agg_vk.bin asset (matches zkvm-prover bf887150's VRAM-budgeting guidance). 2. 'def_hook_commit must be defined': the bundle verify circuit's hook commit comes from the batch child SDK's deferral prover, which used to be initialized as a side effect of sdk.agg_vk(). do_prove now runs batch.enable_deferral(chunk) before bundle.enable_deferral(batch) for bundle tasks, mirroring the zkvm integration tester. Also: generate-catchup-report.py accepts +0000 timezone offsets. Verified end-to-end on the mid-run upgrade shadow fork: bundles 18246/18247 (end_batch 518861/518862 >= upgrade boundary) were re-proven with the halo2-gpu stack and finalized through the newly-registered wrapper; older bundles finalized through the legacy wrapper. Documented as Trap 38; also adds Trap 37 (stale embedded libzkp.so after coordinator rebuild).
… VRAM starvation Fix 1 (load_agg_vk from file) removed the 5.7 GiB/circuit agg-VK derivation, but a prover process that served chunk/batch tasks before a bundle task still crashed at the halo2_outer SNARK stage with quotient.cu cudaErrorInvalidConfiguration (peak 22.8 GiB on 24 GB cards): every circuit's GPU SDK stays resident once lazily built, and the VPMM pool never returns pages, so the quotient chunking heuristic (raw cudaMemGetInfo) computed batch_size = 0. do_prove now calls Prover::reset() on the child/grandchild handlers right after enable_deferral (+ file-based agg_vk), mirroring the zkvm integration tester: the parent captures agg_vk / cached commit / def_hook_commit by value, so the child SDK is no longer needed and its GPU proving keys are released before the parent's STARK/SNARK phase. Verified live on the mainnet shadow fork (mid-run upgrade test): bundle 18394 was proven by a mixed-type process (same process had proved chunks minutes earlier) and finalized via the new wrapper — halo2_outer create_proof live peak 8.6 GiB vs 22.8 GiB pre-fix.
…ssons - 03-deploy-verifier.sh: cast 1.7 prints 'true'/'false' for receipt status — the genuine-register path misread a successful updateVerifier as reverted and aborted 20-upgrade.sh mid-cutover. Also document the manual recovery (finish steps i/j/k, do NOT re-run the script). - Trap 38: Symptom 3/Cause 3 (mixed task types re-create VRAM starvation post-Fix-1), per-GPU supported_proof_types isolation workaround, Fix 3 (child SDK reset after deferral, verified on bundle 18394), upstream pool-free-bytes budgeting suggestion, and crash-recovery corollaries (reset active_attempts, wipe the local SDK db before restart, cleanup ordering). - Trap 36: note the cast 1.7 output drift.
…9.0) with halo2-gpu SNARK acceleration - Bump scroll-zkvm-prover/verifier/types pins ed3b964 -> bf887150 (OpenVM v1.6 -> v2.0.0); rust-toolchain nightly-2025-08-18 -> nightly-2025-11-20. - prover-bin: OpenVM v2 deferred STARK verification for batch/bundle aggregation — new deferral module computes input_commits / DeferralInputs / DeferralStates from child proofs; handlers enable_deferral against child circuits (bundle also initializes batch-over-chunk) and release child GPU SDKs after setup to avoid VRAM starvation of the halo2-gpu SNARK phase. - prover-bin: download pre-built agg_vk.bin circuit asset and new child_circuit_vks config; new halo2-gpu cargo feature + prover_halo2gpu make target for GPU SNARK (bundle) proving. - libzkp: read batch circuit agg_vk.bin for root-proof verification; tasks carry input_commits; drop pre-v0.9.0 universal task compatibility shim. - common/coordinator: adopt v0.9.0 StarkProof wire format (proof / user_pvs_proof / baseline / deferral_merkle_proofs) in message types, proof receiver, mock verifier and tests. - Point prover/e2e circuit base_url at scroll-zkvm/releases/v0.9.0/.
…lows The workspace rust-toolchain moved to nightly-2025-11-20 for the OpenVM v2.0 upgrade (openvm/halo2 crates require rustc >= 1.91.1); the workflows' hardcoded nightly-2025-08-18 override (rustc 1.91.0-nightly) fails the coordinator lint job while resolving the new dependencies.
- types.rs: derive Default with #[default] variant instead of manual impls (clippy::derivable_impls; pre-existing code newly flagged by the bumped toolchain) - prover.rs: drop explicit &*guard reborrows (clippy::explicit_auto_deref) - deferral.rs: factor compute_deferral_data return into a DeferralData type alias (clippy::type_complexity)
Extends (does not replace) the 20-upgrade.sh hard-switch mode: instead of proving with the old stack locally, production bundle proofs are imported from the remote DB into a quarantine table (remote_bundle_proof) and applied to shadow bundles below a boundary; at the boundary the new wrapper is registered via the genuine updateVerifier path, so old (imported proofs, legacyVerifiers routing) and new (local proofs, latestVerifier) bundles genuinely finalize in parallel. 31-canary-rollback.sh restores the old wrapper at the boundary and applies quarantined proofs >= N. - sync-mainnet-db.py: optional --import-proofs / SYNC_PROOFS=1 proof importer; proved-only cursor advance + 200-row lookback (late-attaching mainnet proofs), PROOF_IMPORT_MAX_END_BATCH boundary from canary.env - 10-follow-up.sh: --import-proofs flag (no local coordinator/prover); --reset also drops canary state - 30-canary-upgrade.sh: boundary N = max(first unproven unfinalized, max proven unfinalized + 1), register new wrapper, mark finalized history proved, start new coordinator + provers - 31-canary-rollback.sh: stop new stack, swap wrapper back, apply proofs - stop_component in 30/31 always falls through to the pattern sweep (setsid zombie-pidfile trap) - Makefile targets follow-canary / canary-upgrade / canary-rollback; GUIDE.md canary chapter; TROUBLESHOOTING Trap 39/40; AGENTS.md notes - AGENTS.md: remote RDS query discipline (partial deleted_at indexes, no unbounded count(*) on l2_block) - Test report: docs/testing_reports/canary-parallel-upgrade-2026-08-25.md (all acceptance criteria passed, incl. interleaved finalize, rollback drill, and t2 new-guest proofs finalized via the new wrapper)
Mainnet indexes on l2_block/chunk/l1_message are partial (WHERE deleted_at IS NULL), so sync queries without the predicate could not use them: the per-minute MAX(index) watermark on chunk scanned an entire 6.6M-row index, and the per-cycle l2_block range copy ran two parallel seq scans of the 62 GB heap whenever new chunks arrived — TB-scale RDS reads per day from an idle follow stack. Add a NOT_DELETED constant and apply it to every src-side query (get_watermarks, copy_range, baseline copy_table_pipe clauses, proof import). EXPLAIN-verified on the remote: watermark MAX is now a 1-page index-only backward scan, range copies are plain index scans. Also document Trap 41 (root cause of the Trap 40 Cause 3 slowness) and the "stop the stack when the test ends" cost discipline.
04-prover-up.sh --docker now actually works (the flag was parsed but ignored): each prover runs as a shadow-prover-<gpu> container of PROVER_IMAGE with same-path work-dir mounts, host networking, and --user $(id -u) so the containerized prover is killable from the host; its host PID goes to the usual pidfile, keeping 11-follow-stop.sh / 31-canary-rollback.sh semantics unchanged (plus a docker rm -f shadow-prover-* backstop in both). The SRS dir is mounted at the container HOME so ~/.openvm/params resolves (Trap 12). The per-GPU work_dir is now canonicalized (lib/.. resolved): dockerd path-cleans bind-mount targets but the prover opens its config by the literal path, which ENOENTs inside the container when it still contains ".." — this made the first docker-mode t1 prover start fail. 30-canary-upgrade.sh gains --docker-provers (image check instead of bare-metal binary check, flag + PROVER_IMAGE passed through to 04-prover-up.sh); follow/Makefile wires it as `make canary-upgrade DOCKER_PROVERS=1 PROVER_IMAGE=<tag>`. Image for the current run was built locally from this checkout via build/dockerfiles/prover.Dockerfile (the same production-style packaging as the devops prover-image-build.yml workflow): scrolltech/prover:v4.7.14-c500b7d6-bf88715.
Second canary parallel-upgrade run with provers as GPU docker containers (image built via build/dockerfiles/prover.Dockerfile). All acceptance criteria passed: Phase A (5 bundles finalized from imported proofs), t1 (N=519539), rollback drill with clean docker stop/start, t2 (bundle 18791 docker-proven and finalized via the new wrapper), and a halo2-gpu image swap that cut the bundle SNARK from ~33 min (CPU) to 6.05 s. Trap 42 documents the docker-mode pitfalls found: ".." in bind-mount paths (fixed in 04-prover-up.sh), cuda-only images running the bundle SNARK on CPU, stale coordinator task assignments after prover restarts, and the MAINNET_DSN localhost-tunnel default.
…bled set -e 11-follow-stop.sh intentionally starts without -e so one dead pidfile cannot abort the cleanup, but sourcing lib/anvil-utils.sh (which begins with set -euo pipefail) silently re-enabled it. stop_pid's stale-pidfile branch (return 2) inside the bare prover loop then killed the script midway, leaving docker provers, coordinators, and Anvil running. Latent until docker-mode provers made the stale branch fire: the expect pattern target/release/prover never matches a containerized prover's cmdline. - set +e right after the source - match prover pidfiles on "prover.json" (works for bare-metal and docker cmdlines alike) - trailing exit 0 (the && idiom made the script always exit 1 without --keep-anvil, so make reported Error 1 even on full success) Documented as Trap 43.
… facts sheet and RDS query rules; archive early dry-run postmortems
…hive + thematic groups; slim AGENTS.md and snapshot GUIDE; make doc-check
…hardcoded path — derive from __file__; follow-status showed (no metrics yet)
…route wording (tunnel vs LAN), anvil port, pipeline latency note
…te, fail-fast verifier, dead-path removal - prover config (blocker 1): add child_circuit_vks + asset_detours carrying the v0.9.0 release circuit VKs to zkvm-prover/config.json.template and tests/prover-e2e/docker-e2e prover.json; document both fields and make prover_halo2gpu in zkvm-prover/README.md. Without child_circuit_vks every batch/bundle task failed with "missing child circuit vk". - version gate (blocker 3): the StarkProof wire-format change is breaking, so bump workspace version 4.7.12 -> 4.8.0, version tag v4.7.16 -> v4.8.0, and min_prover_version v4.4.x -> v4.8.0 in coordinator + e2e configs. Pre-v4.8.0 provers are now rejected at login instead of failing verification. In-flight v0.8.0 proofs no longer deserialize and must be drained or reset before deploy (see the new testing report). - coordinator verifier (blocker 4): Verifier::new now loads agg_vk.bin eagerly and panics at startup with an actionable message when it is missing, instead of panicking at verify time where panic_catch would blame the prover. setup_releases.sh downloads agg_vk.bin into the verifier assets dir. NOTE: releases/v0.9.0/verifier/agg_vk.bin (a copy of the batch circuit agg_vk.bin) still needs to be uploaded to S3 by ops. - dead paths (blocker 5): delete univ_task_compatibility_fix (Rust fn, CGO symbol, Go wrapper incl. mock, and both dispatch call sites), the unused OpenVMProof message type, and the feynman/galileo verifier entries (incl. the openvm_13 feature) from coordinator/conf/config.json. - docs: add testing report for OpenVM v2.0 / guest v0.9.0.
Those forks are finalized history on mainnet and the coordinator no longer dispatches tasks for them (the openvm_13/feynman path was removed in the previous commit). Their release dirs are vk-keyed archives of pre-v0.9.0 circuits for which no agg_vk.bin was ever built, so a prover configured from the template would fail asset loading for no benefit (blocker 2).
…, phony halo2gpu - SCROLL_ZKVM_VERSION is a bare version again (v0.9.0, not releases/v0.9.0) in the e2e .make.env files — every other consumer (download-release.sh, permissionless-batches, release-verifier-stuff.sh) already treats it as bare and appends releases/ itself. setup_releases.sh now probes the S3 layout: v0.9.0+ publishes verifier assets under releases/<ver>/verifier/, older releases under <ver>/verifier/. - prover: bail with a clear error when child_circuit_vks maps a circuit to the same vk as its parent (or chunk == batch for the bundle branch). Handlers are cached by vk, so such a misconfiguration made the second .lock().await wait on a mutex the task already held — a silent hang. - zkvm-prover/Makefile: add prover_halo2gpu to .PHONY.
Runbook distilled from the v0.9.0 upgrade work (shadow canary validation, scroll#1816 review rounds, helm/devops ops PRs): component/PR map across scroll, zkvm-prover, sre-helm-charts and devops; S3 circuit-release layout facts (flat v0.9.0 dirs, asset_detours mandatory, verifier asset paths, IAM gotcha); prover config/params/GPU requirements; prover image build caveats (pinned SCROLL_COMMIT, cuda vs halo2-gpu, solc 0.8.24); coordinator version gate + agg_vk.bin fail-fast; deploy sequencing, rollback, and open items. Linked from AGENTS.md documentation index.
OpenVM v2 / guest v0.9.0 upgrade (PR #1816): workspace v4.8.0, child_circuit_vks config, agg_vk.bin mandatory, dead feynman/galileo paths removed. # Conflicts: # AGENTS.md # common/types/message/message.go # coordinator/internal/logic/provertask/batch_prover_task.go # coordinator/internal/logic/provertask/bundle_prover_task.go # crates/libzkp/src/lib.rs # crates/libzkp/src/verifier/universal.rs # crates/prover-bin/src/deferral.rs # crates/prover-bin/src/prover.rs # crates/prover-bin/src/zk_circuits_handler/universal.rs # tests/prover-e2e/cloak-galileoV2/.make.env # tests/prover-e2e/docker-e2e/conf/prover.json # tests/prover-e2e/mainnet-galileoV2/.make.env # zkvm-prover/config.json.template
…ployed verifier wrapper t1 can now skip the plonk/wrapper deployment and register the wrapper named by .contracts.deployed_verifier in the next config (e.g. the production- prepared contract on mainnet, present on the anvil fork by construction). The genuine updateVerifier() path is unchanged; the reused wrapper's on-chain verifierDigest1 is checked against the S3 bundle digest so a misconfigured address fails at t1 instead of at the first >= N finalize. Also fixes a Makefile flag-concatenation bug (--docker-provers--reuse-wrapper).
…production-deployed verifier, happy path)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.