Repository navigation
ci: restrict cache writes to trusted pushes and trim cache usage - #688
Merged
Merged
Conversation
The non-release CI already cached Cargo artifacts and bench fixtures, but the repo-wide 10 GB cache budget was exhausted: one Cargo.lock-changing PR (#679) had saved three full per-OS generations under its merge ref (7.5 GiB of the 11.06 GiB total) -- entries no other run can read. Over-budget usage forces evictions, and evictions force the cold dependency rebuilds this cache exists to avoid. Split every actions/cache step into restore + save: - actions/cache/restore always runs, so PRs still build against their base branch's caches. - actions/cache/save is gated on github.event_name == 'push' and an exact-key miss, so only pushes to master/main/v3.0 can create cache entries; PR and merge-queue runs become restore-only and per-PR copies no longer evict the durable entries. - The bench-fixture dir is deleted after its own save and before the Cargo save, so its ~330 MB per OS aren't archived twice. - release.yml gains a SECURITY note that it must never restore caches: released binaries are built from checked-out source only, so a poisoned cache cannot reach signed artifacts. Also cancel superseded runs via a concurrency group (same pattern as deploy-demos-website.yml), and fix AGENTS.md's stale build notes (the dev opt-level=3 dependency override and release lto=fat were removed earlier; release uses lto=thin).
The lint job and the ubuntu test job shared one cache key (ubuntu-latest-cargo-<lock hash>). Clippy runs cargo check under the hood, so its dependency artifacts are check-only .rmeta files with no codegen; the lint job finishes first and always wins the save, leaving the ubuntu test job an archive it cannot reuse. Result: the ubuntu test job rebuilt the entire ~430-crate dependency tree on every run (2m32s, verified against the v3.0 push run before this change), while macOS and Windows -- which only run the test job -- stayed warm. Move the test matrix to <os>-build-cargo-<lock hash> restore keys, with a fallback to the old <os>-cargo- prefix so the first run after this change warm-starts on macOS/Windows from the archives the previous builds saved. Ubuntu's build job will do one full dependency build to seed its own archive (it has never had one), then stays warm from the first trusted push onward.
The save steps are gated on push events, and pushes only run CI on master/main/v3.0, so no PR run can exercise them. Add this branch to the push triggers once to run the save path and seed the caches, then revert before merge. TEMPORARY - removed by the follow-up commit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Non-release CI already cached Cargo artifacts and bench fixtures, but three problems made the caching much less effective than it looked:
Cargo.lock-changing dependency update) saved under its merge ref — three OSes' Cargo + fixture caches from two lock generations, none of which any other run can ever read. An over-full budget forces evictions, and evictions force cold dependency rebuilds.lintandtest (ubuntu)shared one cache key, but clippy runs oncargo check-style dependency artifacts (check-only.rmetafiles, no codegen). The faster lint job always won the save, so the ubuntu test job could only ever restore an archive it cannot build from — it recompiled the entire ~430-crate dependency tree on every run (2m32s, reproduced on thev3.0push run before this change), while macOS/Windows stayed warm.This PR fixes all three and pins the policy down so future edits can't quietly undermine it.
1. The test matrix gets its own
build-cargocache namespaceThe test jobs now use
<os>-build-cargo-<lock hash>keys while the lint job keepsubuntu-latest-cargo-<lock hash>. The two jobs' archives are different kinds of artifacts and no longer overwrite each other. Ubuntu's build job will do one full dependency build after merge to seed its archive (it has never had one) and is warm from then on; macOS/Windows warm-start immediately via a fallback to the archives the previous builds saved.2. Cache writes are trusted-push-only
Every combined
actions/cachestep is split into:actions/cache/restore— always runs, so PRs still compile against the caches built from their base branch;actions/cache/save— gated on${{ github.event_name == 'push' && steps.<cache>.outputs.cache-hit != 'true' }}, i.e. only pushes tomaster/main/v3.0that missed the exact key can create an entry.Consequences:
3. Bench fixtures are no longer duplicated inside the Cargo archive
target/media-sort-fixtureslives inside thetarget/that the Cargo cache archives and also has its own cache entry (~330 MB × 3 OSes). From the next cache generation on, the fixture directory is deleted between the fixture save and the Cargo save, so the same bytes aren't stored twice within the 10 GB budget.4. Superseded runs are cancelled
A
concurrencygroup (${{ github.workflow }}-${{ github.ref }},cancel-in-progress: true— same pattern asdeploy-demos-website.yml) stops rapid successive pushes from running several full 3-OS matrices in parallel.5. Release pipeline guard rail
release.ymlperforms no cache restore (and never has); it now documents that as a security invariant to protect against future edits. See the analysis below.Validation (performed on this branch before merge)
Because saves are push-only, validation used a temporary push trigger on this branch (added and then reverted; both commits are in the branch history) together with a noop commit:
actions/cache/savesteps executed and saved successfully:ubuntu-latest-cargo-*,ubuntu-latest-build-cargo-*,macos-latest-build-cargo-*,windows-latest-build-cargo-*, and the fixtures.Finished dev profile ... in 6.96sFinished test profile ... in 31.49s(vs 2m32s when deps rebuilt)33.99s; Test (Windows):1m13sSave/Excludesteps:skipped(exact hits — no redundant uploads)Savestepsskipped; ubuntu test falls back to the legacy check-only archive until the first post-merge push seeds the new namespace.actionlintv1.7.12 clean on all workflows (only pre-existing shellcheck infos inrelease.yml's Velopack script).Cache-poisoning analysis
pushruns onmaster/main/v3.0(ci.yml)pull_requestruns (ci.yml)merge_groupruns (ci.yml)release.ymldeploy-demos-website.ymlis intentionally untouched: it runs onworkflow_run(low-trust — GitHub grants read-only cache access to it) orworkflow_dispatch, and its key namespace (Linux-…via${{ runner.os }}) is separate from this workflow's.Notes
gh cache delete --ref refs/pull/679/merge --all), bringing repository cache usage from 11.06 GiB back to 3.70 GiB.v3.0performs one full ubuntu dependency build to seedubuntu-latest-build-cargo-*; every run after that is warm on all three OSes.CARGO_PROFILE_DEV_DEBUG=0for CI (smaller archives, faster transfers) and/or an sccache-based layer.