Skip to content

feat(ci): Add revdep3, pair checks in isolated containers on a shared revdepx core - #2856

Open
krlmlr wants to merge 26 commits into
mainfrom
claude/revdep-check-collision-cfavj8
Open

feat(ci): Add revdep3, pair checks in isolated containers on a shared revdepx core#2856
krlmlr wants to merge 26 commits into
mainfrom
claude/revdep-check-collision-cfavj8

Conversation

@krlmlr

@krlmlr krlmlr commented Aug 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Proposes revdep3, the pair engine: the reverse-dependency check rebuilt on Docker containers. Its queue-engine sibling revdep4 (#2857) is merged; the two were designed as interoperable siblings on one shared core, and with that core now on main via #2857, this PR's diff is just the pair engine itself — revdep3.yaml + revdep3/ (the branch has main merged in; the shared .github/workflows/revdepx/ matched main byte-for-byte, and the committed revdep/ record is main's).

What revdep3 adds over the merged revdep4

Both engines isolate every check in its own container on the shared pipeline (plan → base → build → universe → shards → collect). They differ in where the concurrency sits:

  • revdep4 (merged) runs a package's old and new halves sequentially and wins the time back across packages — 4 packages in flight per shard, one container each.
  • revdep3 (this PR) keeps revdep2's pair: one package at a time per shard, its two halves running side by side in two containers. Per-package wall clock is halved; a package's verdict arrives in one step.

What the live runs measured about the trade: pair shards averaged 2.5 h for ~86 packages against queue shards' 4.05 h for ~172 — the pair engine buys per-package latency at moderately higher total runner cost. And because only two containers run at once, the pair engine is much gentler on the 4-core runners: the queue engine's measured per-half times run 1.5× the pair engine's wall clock at the median (p90 1.8×) from core contention, and the timeout analysis of the final revdep4 run showed all seven of its floor-timeouts were contention-driven — four of those packages finished in 421–715 s under this engine.

The old half always runs fresh; a stored old result is a second opinion only. Where the plan certifies an earlier run's old result as comparable (same revdep version, our CRAN version, container R, base-image tag, dependency fingerprint, within age), the fresh old check is compared against it: baseline_agrees on the manifest line, disagreements reported as drift. The stored result never substitutes for the check.

Interoperability with the merged workflow

One artifact family (revdepx-*), one manifest/baseline/timings schema, one comparison code path, one universe-image lineage on GHCR, one concurrency group per ref. Either workflow reads the other's baselines as second opinions, consumes the other's timings, retries the other's runs (retry-run accepts a run id of either), and starts delta from whichever universe image was refreshed last. The second-opinion machinery's first live exercise ran this engine against revdep4's baselines: 3288 agree, 134 disagree — and the disagreements decomposed exactly into the two then-known infrastructure effects plus 31 true flakes (0.9%).

Live-test results (this engine)

The validation run (32196879628, most, depth 2, R 4.6.1) completed green end to end: 3435 reverse dependencies, 40 shards in two waves, all 45 jobs green, ~8.9 h wall clock — 3336 ok · 28 newly broken · 9 fail to install · 58 with uninstallable dependencies · 2 timeouts · 2 deferred, and it was the first run to publish the universe image to GHCR (all 40 shards pulled it in minutes). It reproduced all 21 of revdep4's newly-broken and unmasked 4 more that revdep4's infrastructure failures had hidden. Its residual buckets (Bioconductor planner metadata, Rust and tcl toolchains, the memory cap) were fixed on both branches afterwards and validated by the reruns and the final revdep4 full run that produced the record now on main; this branch carries the identical fixes, so a fresh revdep3 run starts from the same platform.

Notes for review

  • The shared core arrived on main with feat(ci): Add revdep4, sequential halves driven by a two-ended bash work queue #2857 and is untouched here; this PR was kept byte-identical to it throughout, so merging adds no second copy and no conflicts.
  • Once merged, revdep3.yaml's workflow_dispatch becomes registered like revdep4's; the temporary push trigger that drove the live run is long reverted — the workflow is dispatch-only with strong/depth-1 defaults and the check platform pinned at R 4.6.1.
  • Whether to keep both engines or let revdep4 serve alone is a judgment call: the pair engine is the lower-latency, lower-contention option (better per-package timeouts, gentler on 4-core runners), the queue engine the cheaper-per-run one. They share everything but the shard driver, so the maintenance cost of keeping both is the ~600 lines under revdep3/.
  • By submitting this pull request, I assign the copyright of my contribution to The igraph development team.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UB7YutLzVWU7xCTYUvF3kF

@claude
feat(ci): Add revdep3, pair checks in isolated containers on a shared…
24c2491 
… revdepx core

revdep2 checks each reverse dependency's CRAN and dev halves
as two simultaneous `R CMD check` processes on one host.
Simultaneously checking the same package against two libraries
is not a supported mode of operation for the packages being checked:
the PSOCK port collision that needed the `R_PARALLEL_PORT` split
was one failure class of an open-ended family
(shared TMPDIR, caches, locks, every singleton a check assumes it owns).

revdep3 keeps the pair -- concurrency still halves a package's wall clock --
but runs each half in its own Docker container,
dissolving the interference class instead of patching mechanisms.
Checks run under a pinned R version (oldrel by default)
on a rocker/r-ver base,
and the dependency universe travels as a GHCR image,
built once and delta-updated across runs,
which replaces the preflight, the per-shard installs
and the prebuilt-library artifacts.

The engine-agnostic core lands in .github/workflows/revdepx/,
shared byte-for-byte with the sibling revdep4 proposal:
one artifact family (revdepx-*),
one manifest, baseline and timings schema,
one comparison code path, one universe-image lineage --
so either workflow reuses the other's baselines, timings,
images and reports, and `retry-run` accepts a run of either.

revdep2 stays untouched;
this is a proposal beside it, not a replacement of it yet.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UB7YutLzVWU7xCTYUvF3kF
@krlmlr krlmlr mentioned this pull request Aug 17, 2026
Merged
1 task
claude and others added 25 commits August 17, 2026 15:47
@claude
fix(ci): Always run the old half; a stored old result is a second opi…
76260ae 
…nion only

The queue engine briefly revived what revdep2 had retired:
substituting a stored old-version result for the old check
when the plan judged it comparable.
The container platform makes that far safer than it was,
but a fresh old check is the only result
whose provenance the run fully controls --
so the old half now always runs, in both engines,
and a comparable stored result rides along
purely as a second opinion:
`baseline_agrees` records whether the fresh check reproduced it,
and a disagreement is printed as drift.

Concretely: the queue engine's per-package price is always two halves
(no baseline discount in the plan),
the queue file loses its skip_old column,
and compare_halves() loses its baseline-substitution mode.
The baseline artifact, its validity conditions
and the cross-workflow lineage are unchanged --
only what a valid row is *for* has narrowed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UB7YutLzVWU7xCTYUvF3kF
@claude
docs(ci): Align the shared README with the second-opinion baseline po…
a350845 
…licy

Two sentences still described the retired skip-old design:
the timings bullet halved the queue plan's bill
where a baseline covered the old half,
and the baseline bullet spoke of reusing rows.
Both halves always run fresh now,
and a stored row is only ever a second opinion.

(This push also re-triggers the checks
that failed during the GitHub outage.)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UB7YutLzVWU7xCTYUvF3kF
@claude
fix(ci): Apply the clean-context review findings to the shared core
c759c0f 
An independent review of both revdepx branches confirmed four bugs
and a handful of gaps; this applies everything that lives in the
shared core or on this branch.

- The check-slice cut no longer dies on shards with fewer runnable
  packages than slices: `seq(index, n, by = of)` is an R error when
  index exceeds n, which crashed slices of 1-package retry shards and
  -- with an empty runnable set -- erased recorded depfail diagnoses
  into `missing`. The slice is also cut before the source downloads
  now, so each slice fetches only its own tarballs instead of the
  whole shard's three times over. The same seq() trap is fixed in
  plan.R's `part` split.
- The base image now installs callr: util.R's run_with_timeout()
  silently degrades to an unbounded inline call without it, and every
  "bounded" pak call of the in-container universe build ran with no
  clock -- the exact hang class revdep2 added callr for on the host.
  A denied base-image push is now a hard error too: unlike the
  universe image it has no artifact fallback, and everything
  downstream pulls it.
- timing.json's `script_seconds` accumulates across check slices;
  before, the final slice's overwrite dropped the earlier slices'
  driver time and the calibration charged it to per-shard setup,
  inflating every later plan.
- The `compared` commit gate counts only this run's own comparisons:
  carried-over retry results could pass it and let a run that learnt
  nothing overwrite the committed report.
- ensure_check_sysreqs() runs `apt-get update` before its direct
  install -- the base image deletes the apt lists, so the install
  otherwise failed quietly.
- The shared concurrency group is built from `github.ref_name`, so a
  dispatch with the `ref` input filled in serializes against one
  without it on the same branch.
- check-half.sh tells its outer safety-net timeout apart from a real
  check timeout (both exit 124) by when the axe fell, and reports the
  outer case as a runner failure, not a package one.
- The queue's claimed.log and queue-state.json are copied into the
  results artifact per slice; in the work directory alone they died
  with the runner, which is exactly when they are wanted.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UB7YutLzVWU7xCTYUvF3kF
@claude
fix(ci): Build the dev binary without vignettes
68a0c11 
The first live run failed in the build container:
"vignette builder 'knitr' not found".
The pak bootstrap installs the package's hard dependencies,
and building vignettes needs the Suggests tree on top --
which revdep2's host build inherited from setup-r-dependencies
without anyone deciding it.

The binary exists to be installed into the checks' new-half
library, and no check ever builds or reads the package-under-test's
own vignettes, so `--no-build-vignettes` removes the requirement
instead of importing it.
Also forward GITHUB_SHA into the container,
so meta.json records the real commit
when git cannot answer inside the mount.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UB7YutLzVWU7xCTYUvF3kF
@claude
fix(ci): Harden the universe commit and the dev binary against the on…
3e4c84e 
…e-disk runners

Mirror of the revdep4 branch's fixes after its first live run
(the shared `revdepx/` scripts stay byte-identical by construction):

- `docker commit` copies the container's entire rw layer onto the
  runner's ONE disk (there is no /mnt volume), and the copy filling
  the disk killed the runner agent mid-commit -- with its logs,
  artifacts and cache saves. The commit step now measures the delta
  and the free space first, skips the commit with a loud warning
  when the copy cannot fit (the shards' local-build fallback carries
  the run), drops the build container right after a successful
  commit so the peak subsides before the push, gives the artifact
  fallback its own space check, and gates the artifact upload and
  download on a new `saved` output.
- build.R moved the dev binary with file.rename() across two bind
  mounts; rename(2) fails there with EXDEV and file.rename() only
  says so in an unread return value, so every shard failed
  installing a binary the artifact never contained. Copy and check
  instead.
- The long steps (universe build, commit, shard image prep) run
  in-step resource samplers: a sampler backgrounded in its own step
  keeps writing its file, but its stdout stops reaching the job log
  when that step ends, and the streamed lines are all that survives
  a dying runner.
- chown the pak cache back to the runner user after root-in-container
  wrote it, so the cache post-step's tar stops failing on 600 lock
  files and the cache finally saves.
- Rename the "Move docker onto the big disk" steps: there is no big
  disk to move to, only room to make.
- Bump docker/login-action to v4 for the Node.js 24 runtime.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UB7YutLzVWU7xCTYUvF3kF
@claude
fix(ci): Put the universe commit under a watchdog and publish its evi…
33ecd8a 
…dence first

Mirror of the revdep4 branch's hardening after its second dead
runner in the guarded commit (~34 minutes in, both runs, and a dead
runner loses even the log lines it already streamed):

- measure the delta in bytes AND inodes and upload the numbers as an
  artifact BEFORE the commit runs, so the next post-mortem starts
  with data;
- the guard reads those numbers, refuses an unmeasured delta, and
  checks inodes as well as bytes;
- the commit runs under a watchdog that stops the docker daemon when
  disk, inodes or memory approach zero -- killing the client would
  not abort the daemon-side copy -- then restarts it so the rest of
  the job still works.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UB7YutLzVWU7xCTYUvF3kF
@claude
fix(ci): Keep temporary files out of the universe layer, and re-chunk…
625671d 
… before retrying singly

Mirror of the revdep4 branch's commit
(the shared `revdepx/` scripts stay byte-identical by construction):
bind-mount /tmp from the host in the universe build container
so killed installs' build residue never enters the committed layer,
sweep apt lists and /tmp leftovers at the end of image.R,
and re-try a failed chunked install in chunks of 50
before falling back to the one-at-a-time salvage pass,
so one broken package no longer drags hundreds of stranded
chunk-mates through per-package pak calls.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UB7YutLzVWU7xCTYUvF3kF
@claude
perf(ci): Isolate failing universe packages by trisection instead of …
f31d55d 
…a flat scan

Mirror of the revdep4 branch's commit
(the shared `revdepx/` scripts stay byte-identical by construction):
the salvage pass after a failed chunked install now divides and
conquers with fan-out 3 instead of scanning one package at a time --
d culprits hiding in n packages cost about 3 * d * log3(n) pak calls
instead of n, and single-package leaves still name each genuine
failure with its own log.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UB7YutLzVWU7xCTYUvF3kF
@claude
fix(ci): Give the bind-mounted /tmp real /tmp permissions
d44fcc4 
Mirror of the revdep4 branch's fix
(the shared `revdepx/` scripts stay byte-identical by construction):
a /tmp bind mount created by plain mkdir is 755,
apt-key cannot create its temporary config in it,
every repository then fails signature verification,
and every apt-get pak runs fails with it --
reproduced and fixed locally against rocker/r-ver:4.5.3.
chmod 1777 on every host directory
that becomes a container's /tmp.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UB7YutLzVWU7xCTYUvF3kF
@claude
feat(ci): Check under R 4.6.1 with a virtual display, and finish the …
cac319a 
…hardening

Mirror of the revdep4 branch's commit
(the shared `revdepx/` scripts stay byte-identical by construction):
R 4.6.1 hard-coded as the pinned default
(at least one reverse dependency needs R-release),
Xvfb in the base image and started for installs, load tests and
checks (Tk-based packages die headless while lazy-loading),
the universe delta measured via `docker ps -as` when the containerd
image store leaves GraphDriver empty,
the check-sysreqs survey chunked (one call over 3435 packages was
OOM-killed at 14 GB) with memory caps on the build containers,
load-test roots computed from Depends+Imports so LinkingTo-only
packages are load-tested too,
and in-step resource samplers on the check slices.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UB7YutLzVWU7xCTYUvF3kF
@claude
ci: TEMPORARY push trigger for the revdep3 live test run (most, depth 2)
6ae2a4a 
Same device as on the revdep4 branch: a workflow_dispatch workflow is
only registered once its file exists on the default branch, so the
live test fires from a push trigger with the inputs -- and the
run title -- hardcoded to the test parameters.
Revert this commit after the test run.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UB7YutLzVWU7xCTYUvF3kF
@claude
Revert "ci: TEMPORARY push trigger for the revdep3 live test run (mos…
c6541dd 
…t, depth 2)"

This reverts commit 6ae2a4a.

The run it started (32148370376) was cancelled within a minute:
the slice-truncation bug found in revdep4's completed run would have
destroyed two thirds of its results too. The trigger returns after
the fixed revdep4 run (most, depth 2, R 4.6.1) completes, so the two
workflows do not contend for the runner pool.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UB7YutLzVWU7xCTYUvF3kF
@claude
fix(ci): Never truncate the shared manifest between check slices
284536e 
Mirror of the revdep4 branch's fix
(the shared `revdepx/` scripts stay byte-identical by construction):
the check phase opened with file.create(manifest_path),
which truncates an existing file,
so under slicing every slice wiped its predecessors' results
and the sweep re-wrote them as `deferred` --
run 32114635495 lost 2293 finished checks to it.
Create the manifest only when it does not exist yet.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UB7YutLzVWU7xCTYUvF3kF
@claude
fix(ci): Publish the universe image the containerd store's way
5bf3c3f 
Mirror of the revdep4 branch's commit
(the shared `revdepx/` scripts stay byte-identical by construction):
push and save with `--platform linux/amd64` --
the containerd image store commits an OCI manifest index,
which a plain push refuses and a plain save reduces to a
1336-byte manifest shell (run 32148999976) --
refuse to upload a save under 100 MB,
let a bad pulled or loaded image fall through to the
shard-local build instead of failing the shard,
and survive single failing chunks of the sysreqs survey.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UB7YutLzVWU7xCTYUvF3kF
@claude
fix(ci): Run docker on the classic graphdriver store
d3c3033 
Mirror of the revdep4 branch's commit: the containerd image store's
commit is an OCI index that can be neither pushed nor usefully
saved (run 32158907637); the daemon.json this workflow already
writes now disables the containerd snapshotter, restoring the
classic store whose committed images push, save and du-measure
exactly as everything here assumes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UB7YutLzVWU7xCTYUvF3kF
@claude
ci: TEMPORARY push trigger for the revdep3 live test run (most, depth 2)
47cb586 
Same device as on the revdep4 branch: a workflow_dispatch workflow is
only registered once its file exists on the default branch, so the
live test fires from a push trigger with the inputs -- and the
run title -- hardcoded to the test parameters.
Revert this commit after the test run.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UB7YutLzVWU7xCTYUvF3kF
@github-actions
chore: Update revdep report from revdep3 run 32196879628
91ae7b1 
https://github.com/igraph/rigraph/actions/runs/32196879628
@claude
feat(ci): Resolve Bioconductor dependencies, diagnose compile OOMs, c…
fb9ee04 
…ap checks at 6g

The shared revdepx core mirrored from the revdep4 branch (see its
commit of the same name for the full story), plus the pair engine's
side of the memory knob: check-pair.sh honors
REVDEPX_MEMORY_PER_CHECK -- the name both engines share -- before
deriving (RAM - 2 GiB) / 2, and the yaml defaults it to 6g, keeping
the two engines' caps identical.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UB7YutLzVWU7xCTYUvF3kF
@claude
Revert "ci: TEMPORARY push trigger for the revdep3 live test run (mos…
04f1190 
…t, depth 2)"

This reverts commit 47cb586.
@claude
feat(ci): Compile checks with -g0 and -j1 inside the memory cap; docu…
5cafd96 
…ment the backlog

The shared revdepx core mirrored from the revdep4 branch (see its
commit of the same name for the full story), plus the same
REVDEPX_CHECK_FLAGS / REVDEPX_CHECK_MAKEFLAGS forwarding in the yaml.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UB7YutLzVWU7xCTYUvF3kF
@claude
fix(ci): Carry the committed record through subset runs; add Rust to …
5b2c0fc 
…the base image

The shared revdepx core mirrored from the revdep4 branch; see its
commit of the same name for the two findings from the broken-packages
retest (a subset run shrank the committed manifest to its own rows;
all six residual install failures were a missing Rust toolchain).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UB7YutLzVWU7xCTYUvF3kF
@claude
fix(ci): Ship the tcl/tk runtime in the base image
0114f7c 
The shared revdepx core mirrored from the revdep4 branch: tcltk is a
base R package, invisible to pak's sysreqs, so a universe built for a
small package set had no libtcl and every tcltk-using package failed
to install (run 32281237129).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UB7YutLzVWU7xCTYUvF3kF
@claude
fix(ci): Render carried rows truthfully in the report
8b6a406 
The shared revdepx core mirrored from the revdep4 branch: a carried
ok row without a check payload becomes a clean two-sided comparison
instead of an rcmdcheck error shim, and the section writer skips
payload-less carried rows -- run 32281237129's report had announced
"Failed to check (3407)" with a phantom failure section for every
carried ok package.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UB7YutLzVWU7xCTYUvF3kF
@claude
Merge main after #2857: one shared core, main's revdep record
945aea7 
The squash-merge of #2857 put the byte-identical
.github/workflows/revdepx/ core and the revdep4 engine on main,
so this branch now adds only the pair engine.
The committed revdep/ record diverged
(this branch carried revdep3's 2026-08-19 report,
main the newer 2026-08-22 full-run record);
main's record wins wholesale,
including the removal of section files only this branch had,
so a future subset run finds no stale committed sections.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UB7YutLzVWU7xCTYUvF3kF
@claude
chore: Drop the revdep3-era section files main's record does not carry
ef1a8d8 
The merge kept the per-package sections
only this branch had added
(the pair run's install failures and three since-cleared problems);
the record is main's 2026-08-22 full run,
and stale committed sections would be treated as evidence
by the next subset run's collector.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UB7YutLzVWU7xCTYUvF3kF
@krlmlr krlmlr mentioned this pull request Sep 3, 2026
Open
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@krlmlr @claude