Skip to content

ci: restrict cache writes to trusted pushes and trim cache usage - #688

Merged
Lolle2000la merged 7 commits into
v3.0from
ci/non-release-build-cache
Oct 11, 2026
Merged

Lolle2000la merged 7 commits into
v3.0from
ci/non-release-build-cache

Conversation

@Lolle2000la

@Lolle2000la Lolle2000la commented Oct 11, 2026 •

Copy link
Copy Markdown
Owner

What

Non-release CI already cached Cargo artifacts and bench fixtures, but three problems made the caching much less effective than it looked:

  1. The repo's shared 10 GB cache budget was exhausted — 11.06 GiB across 14 entries on 2026-10-11. 7.4 GiB of that was eight orphaned entries that PR chore: update Cargo, GitHub Actions, and website npm dependencies #679 (a Cargo.lock-changing dependency update) saved under its merge ref — three OSes' Cargo + fixture caches from two lock generations, none of which any other run can ever read. An over-full budget forces evictions, and evictions force cold dependency rebuilds.
  2. The ubuntu test job never actually used the cache for dependencies. lint and test (ubuntu) shared one cache key, but clippy runs on cargo check-style dependency artifacts (check-only .rmeta files, no codegen). The faster lint job always won the save, so the ubuntu test job could only ever restore an archive it cannot build from — it recompiled the entire ~430-crate dependency tree on every run (2m32s, reproduced on the v3.0 push run before this change), while macOS/Windows stayed warm.
  3. Cache writes were not restricted to trusted refs.

This PR fixes all three and pins the policy down so future edits can't quietly undermine it.

1. The test matrix gets its own build-cargo cache namespace

The test jobs now use <os>-build-cargo-<lock hash> keys while the lint job keeps ubuntu-latest-cargo-<lock hash>. The two jobs' archives are different kinds of artifacts and no longer overwrite each other. Ubuntu's build job will do one full dependency build after merge to seed its archive (it has never had one) and is warm from then on; macOS/Windows warm-start immediately via a fallback to the archives the previous builds saved.

2. Cache writes are trusted-push-only

Every combined actions/cache step is split into:

  • actions/cache/restore — always runs, so PRs still compile against the caches built from their base branch;
  • actions/cache/save — gated on ${{ github.event_name == 'push' && steps.<cache>.outputs.cache-hit != 'true' }}, i.e. only pushes to master/main/v3.0 that missed the exact key can create an entry.

Consequences:

  • Pull-request and merge-queue runs never write a cache entry, so no untrusted code can affect the caches other runs consume. GitHub already scopes PR-created caches to the PR merge ref (unreadable by the base branch or other PRs), but the explicit gate means we don't have to depend on that isolation — or on GitHub preserving it — for the guarantee.
  • Per-PR cache copies stop eating the repo-wide 10 GB budget, so the durable branch caches stop being evicted by one-off PR generations.
  • Trade-off: re-running CI on the same PR recompiles the PR's own crates on top of the restored dependency cache (only the workspace crates; dependencies stay cached).

3. Bench fixtures are no longer duplicated inside the Cargo archive

target/media-sort-fixtures lives inside the target/ that the Cargo cache archives and also has its own cache entry (~330 MB × 3 OSes). From the next cache generation on, the fixture directory is deleted between the fixture save and the Cargo save, so the same bytes aren't stored twice within the 10 GB budget.

4. Superseded runs are cancelled

A concurrency group (${{ github.workflow }}-${{ github.ref }}, cancel-in-progress: true — same pattern as deploy-demos-website.yml) stops rapid successive pushes from running several full 3-OS matrices in parallel.

5. Release pipeline guard rail

release.yml performs no cache restore (and never has); it now documents that as a security invariant to protect against future edits. See the analysis below.

Validation (performed on this branch before merge)

Because saves are push-only, validation used a temporary push trigger on this branch (added and then reverted; both commits are in the branch history) together with a noop commit:

  • Cold push run: every job built from source, then all five new actions/cache/save steps executed and saved successfully: ubuntu-latest-cargo-*, ubuntu-latest-build-cargo-*, macos-latest-build-cargo-*, windows-latest-build-cargo-*, and the fixtures.
  • Next push run (warm): every compile job restored its exact key and compiled only the 8 first-party workspace crates — no third-party dependency was built anywhere:
    • Format & Lint: Finished dev profile ... in 6.96s
    • Test (ubuntu): Finished test profile ... in 31.49s (vs 2m32s when deps rebuilt)
    • Test (macOS): 33.99s; Test (Windows): 1m13s
    • All Save/Exclude steps: skipped (exact hits — no redundant uploads)
  • PR run (restore-only): cache Save steps skipped; ubuntu test falls back to the legacy check-only archive until the first post-merge push seeds the new namespace.
  • actionlint v1.7.12 clean on all workflows (only pre-existing shellcheck infos in release.yml's Velopack script).
  • The validation-only branch caches were deleted after verification.

Cache-poisoning analysis

Consumer Can read Can write
push runs on master/main/v3.0 (ci.yml) base-branch + default-branch caches yes (the only writer)
pull_request runs (ci.yml) base-branch + own merge-ref caches no (gate)
merge_group runs (ci.yml) per GitHub's cache scoping (unchanged) no (gate)
release.yml nothing — no cache steps at all no
  • Every writable cache entry therefore traces back to merged code. A malicious PR cannot create, modify or evict anything in the scopes that later runs consume.
  • Release builds are always compiled from the checked-out source with no cache in the loop, so a poisoned CI cache cannot influence signed / published artifacts.
  • deploy-demos-website.yml is intentionally untouched: it runs on workflow_run (low-trust — GitHub grants read-only cache access to it) or workflow_dispatch, and its key namespace (Linux-… via ${{ runner.os }}) is separate from this workflow's.

Notes

  • The 7.4 GiB of orphaned PR chore: update Cargo, GitHub Actions, and website npm dependencies #679 entries were pruned on 2026-10-11 (gh cache delete --ref refs/pull/679/merge --all), bringing repository cache usage from 11.06 GiB back to 3.70 GiB.
  • After merge, the first push to v3.0 performs one full ubuntu dependency build to seed ubuntu-latest-build-cargo-*; every run after that is warm on all three OSes.
  • Possible follow-ups, intentionally out of scope: CARGO_PROFILE_DEV_DEBUG=0 for CI (smaller archives, faster transfers) and/or an sccache-based layer.

The non-release CI already cached Cargo artifacts and bench fixtures, but
the repo-wide 10 GB cache budget was exhausted: one Cargo.lock-changing PR
(#679) had saved three full per-OS generations under its merge ref (7.5 GiB
of the 11.06 GiB total) -- entries no other run can read. Over-budget usage
forces evictions, and evictions force the cold dependency rebuilds this
cache exists to avoid.

Split every actions/cache step into restore + save:

- actions/cache/restore always runs, so PRs still build against their base
  branch's caches.
- actions/cache/save is gated on github.event_name == 'push' and an
  exact-key miss, so only pushes to master/main/v3.0 can create cache
  entries; PR and merge-queue runs become restore-only and per-PR copies no
  longer evict the durable entries.
- The bench-fixture dir is deleted after its own save and before the Cargo
  save, so its ~330 MB per OS aren't archived twice.
- release.yml gains a SECURITY note that it must never restore caches:
  released binaries are built from checked-out source only, so a poisoned
  cache cannot reach signed artifacts.

Also cancel superseded runs via a concurrency group (same pattern as
deploy-demos-website.yml), and fix AGENTS.md's stale build notes (the dev
opt-level=3 dependency override and release lto=fat were removed earlier;
release uses lto=thin).
The lint job and the ubuntu test job shared one cache key
(ubuntu-latest-cargo-<lock hash>). Clippy runs cargo check under the hood,
so its dependency artifacts are check-only .rmeta files with no codegen;
the lint job finishes first and always wins the save, leaving the ubuntu
test job an archive it cannot reuse. Result: the ubuntu test job rebuilt
the entire ~430-crate dependency tree on every run (2m32s, verified
against the v3.0 push run before this change), while macOS and Windows --
which only run the test job -- stayed warm.

Move the test matrix to <os>-build-cargo-<lock hash> restore keys, with a
fallback to the old <os>-cargo- prefix so the first run after this change
warm-starts on macOS/Windows from the archives the previous builds saved.
Ubuntu's build job will do one full dependency build to seed its own
archive (it has never had one), then stays warm from the first trusted
push onward.
The save steps are gated on push events, and pushes only run CI on
master/main/v3.0, so no PR run can exercise them. Add this branch to the
push triggers once to run the save path and seed the caches, then revert
before merge. TEMPORARY - removed by the follow-up commit.
@Lolle2000la
Lolle2000la merged commit 0ade2e7 into v3.0 Oct 11, 2026
5 checks passed
@Lolle2000la
Lolle2000la deleted the ci/non-release-build-cache branch October 11, 2026 18:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant