fix(ci): the liveness probe measured latency, and its one right answer was unreachable (REQ-354)

[avrabe]

The defect

The workflow is named Runner Liveness. Its own header calls queued-run age "the authoritative signal" that "directly measures the symptom".

Queued age is latency. A saturated pool and a dead pool both show jobs stuck in queued, and they need opposite responses.

classify_stall already emitted label-saturated — but reaching it needs the runner list, which needs the administration scope, which is not grantable to GITHUB_TOKEN. So every scheduled probe 403s on that lookup and short-circuits at runners-unknown, which files an issue.

The right answer existed with no path to it. Measured today:

$ gh issue list --state all --search 'CI runner pool liveness alert in:title'
CLOSED  95
OPEN     2

Liveness needs no new scope

It is observable from the jobs API the probe already reads. A job requiring labels L that is in_progress proves a runner carrying at least L exists and is working. A queued job whose labels are a subset of L is therefore behind a live runner, not behind an outage.

Required of every queued job, not any — a mix where some labels have live siblings and others have none still alarms, so a real outage cannot hide behind one unrelated running job. When it cannot be shown for all, the answer is unchanged from before.

Why a distinct mode, not label-saturated

That mode means "capable runners exist and none is idle", established from the runner list, and its caller prints per-label idle counts read from that same list. Reached by this path the list is absent, so reusing it would print idle by label: none and imply an empty pool — the exact false alarm being removed. queue-behind-live-runners names what was actually observed.

Why a notice, not a problem

Same grounds as dependency-blocked. This workflow's charter, in its own words:

when the pool goes offline every gate queues forever with no fallback and no alarm

A pool with jobs actively running on the labels the queued jobs need is not that condition. This stays inside the charter rather than widening or narrowing it, and it is one line to reverse.

The shape that filed 96 alerts

Measured on #940 (closed today with this evidence): a 16-shard mutation matrix against 4 lean-mem runners serializes into four waves by construction, so shards queue past the 30-minute threshold every night.

org runners: total=12 online=12 busy=12
  self-hosted,linux,x64,hetzner,lean-mem   count=4  busy=4
run 34576277782 (scheduled, main): done=37 running=4 queued=5
  running: Mutation Testing (rivet-core 01,07,13,15 -of-16)

Nothing was down. Every runner was working.

Verification

Oracle first, red on exactly one case with all four controls already green:

classify_stall (liveness from running siblings):
  FAIL saturation is not a liveness failure: expected 'queue-behind-live-runners', got 'runners-unknown'
  ok   nothing running on those labels still alarms
  ok   a running sibling on other labels proves nothing
  ok   a readable offline pool still outranks the jobs signal
  ok   dependency waits still short-circuit first

Four mutations, each reddening only the controls that name the property:

mutation	reddens
every queued job → any	"nothing running still alarms" + "other labels proves nothing"
superset → any overlap	"a running sibling on other labels proves nothing"
count queued jobs as live	"nothing running still alarms" + "other labels proves nothing"
run the check when the runner list IS readable	7 cases, incl. "a readable offline pool still outranks the jobs signal"

fmt=0 · clippy --all-targets -D warnings (1.97.0)=0 · cargo test --workspace=0 (2390 passed) · cli_commands=0 (216) · rivet validate=0 · rivet docs check=0 · yamllint=0 · diagnose_test=0 · rivet coverage --tests shows REQ-354 2 test markers

Not resolved here

REQ-351 proposes scoping the probe to CI Gate's needs — that changes which jobs the alarm watches and is the maintainer's call. This changes nothing about which jobs are watched; it stops the probe calling a demonstrably live pool dead.

[github-actions]

📐 Rivet artifact delta

Change	Count
Added	1
Removed	0
Modified	0
Downstream impacted (depth ≤ 5)	0

Graph

graph LR
  REQ_354["REQ-354"]:::added
  classDef added fill:#d4edda,stroke:#28a745,color:#155724
  classDef removed fill:#f8d7da,stroke:#dc3545,color:#721c24
  classDef modified fill:#fff3cd,stroke:#ffc107,color:#856404
  classDef overflow fill:#e2e3e5,stroke:#6c757d,color:#495057,stroke-dasharray: 3 3

Loading

Added

• REQ-354

📎 Full HTML dashboard attached as workflow artifact rivet-delta-pr-941 — download from the workflow run.

_{Posted by rivet-delta workflow. The graph shows only changed artifacts; open the HTML dashboard (above) for full context.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!