retries_exhausted reports only the last attempt's failure and hides the first

[agent-relay-code]

retries_exhausted now reports every failed attempt, not only the last

The bug

A commit-and-push step was refused by a GitLab pre-receive hook for a
committer-identity reason. The retry re-ran a commit that had already landed,
died on nothing staged inside the declared scope, and reported that. The
terminal retries_exhausted carried the last attempt's evidence and nothing
else, so the real rejection was invisible without opening the journal by hand —
and "the push was rejected" and "nothing was staged" point at different bugs.

The evidence was never lost. The kernel appends a step.completed entry per
attempt (completion_actions, kernel/relayflowd-core/src/machine.rs); the
reader threw the earlier ones away.

What changed

The diagnostic reads every failed attempt. stepFailureDetails
(packages/sdk/src/cli/step-failure.ts) now accumulates a per-step history as
it walks the journal instead of overwriting one candidate. The history is keyed
by step, so interleaved steps never pool their attempts and a page boundary
never splits one step's record. A later success clears the history: what a
step that eventually succeeded printed on the way is not the diagnosis of a
later, different failure.

StepFailedDetails gained two additive fields (failure-kinds.ts):
attempts?: StepAttemptFailure[], present only when the step failed more than
once, and attemptEvidence?: 'differs' | 'unchanged' | 'unknown'. The existing
scalar fields still describe the terminal attempt; no diagnostic kind was added
or renamed, and a single-attempt failure renders exactly as before.

Divergent failures are named explicitly. compareAttempts
(packages/sdk/src/cli/step-evidence.ts, new) fingerprints each attempt from
its unbounded journal record — not the display excerpt — because two attempts
sharing a 256-byte prefix are not the same failure. Two label pairs are
normalised first: the kernel relabels an identical cause verification_failed
on a retry and retries_exhausted on the terminal attempt, which is a policy
decision, not a different failure. unknown is returned honestly when an
attempt journaled no account of itself, or when its producer had already
truncated one — equal evidence that was never complete is not evidence of
equality.

Agent and llm steps are covered too. The old deterministicFailureDetails
returned undefined for any run without a deterministic step, which is every
f.agent and f.llm step. Those steps' {exit_code, stdout_tail, stderr_tail}
is nulled out of output by preserve_failure_output and survives only as the
daemon's bounded verification.detail render (worker_failure_detail,
kernel/relayflowd/src/engine/remote.rs). selectEvidence reads both shapes.

The history crosses the IPC boundary. stepFailedFrame
(authored-node-runner.ts) validates attempts element by element on the same
terms as the frame that carries it. A malformed element is dropped rather than
voiding the whole history: a report naming three of four attempts still carries
the first attempt's error, which is the fact the terminal scalars cannot supply.

Output

A real run, real daemon, real YAML flow with maxIterations: 2:

FAILED [step_failed] Run "01M2ZXWXQTR2BHCM543ZC4NW2S" failed with completionReason: step_failed. Step "commit-and-push" (deterministic) completionReason: retries_exhausted attempt=2/2 exit=1.
Attempts: 2 failed; recorded failure evidence differs. An earlier attempt may have had side effects.
  attempt 1: verification_failed exit=1 — stderr: remote: GitLab: You cannot push commits for 'factory@example.com'
  attempt 2: retries_exhausted exit=1 — stderr: nothing staged inside the declared scope
Stderr (last 1,024 bytes):
nothing staged inside the declared scope
Inspect: flows replay 01M2ZXWXQTR2BHCM543ZC4NW2S --at commit-and-push --data-dir '<tmp>/data'
Journal: <tmp>/data/runs/01M2ZXWXQTR2BHCM543ZC4NW2S.sqlite3

attempt=2/2 is printed beside retries_exhausted because a deterministic
step's default budget is 1, so a single failed attempt with no retry at all
terminates under that same reason. The vocabulary does not change (AGENTS.md
rule 7); printing the facts beside it stops it being misread.

Acceptance

Criterion	Pinned by
retries_exhausted includes the first attempt's error	tests/step-attempt-history.test.ts — "reports the first attempt's error beside the last and says they differ"; tests/retried-step-failure.test.ts (live kernel, end to end)
Distinct reasons visible without reading the journal	compareAttempts tests: identical excerpts over divergent records still report differs; a changed gate verdict behind identical process output is seen; the exhaustion relabel is not called a different failure
flows logs <run-id> shows every attempt	tests/cloud-read.test.ts — "prints every attempt of a retried step the runner log captured", built from the real producer renderStepEvidence so the assertion cannot drift from the CLI's actual output

The cloud runner log is external to this repo: runCloudLogsCli prints it
verbatim, redacted, with no elision, and there is no journal-export endpoint.
Criterion 3 is therefore satisfied by the CLI diagnostic itself carrying every
attempt. docs/CLOUD.md records the one limit that remains — the log is a
recording, so a run executed by an older build carries only the terminal
attempt.

Out of scope, and left alone

Retry policy and counts are unchanged. Nothing here decides whether a step with
side effects should be retried at all; the diagnostic only reports that an
earlier attempt's evidence differs, which is the observation an operator needs
to make that call.

Tests

Kernel contract — three tests pinning the journal facts the reader depends on
(per-attempt output preserved beside an identical verdict; the
verification_failed → retries_exhausted relabel over byte-identical
evidence; a worker-reported reason not rewritten by the retry branch):

$ cd kernel && sh ../ops/cargo.sh test -p relayflowd-core
test machine::tests::a_worker_reported_reason_is_not_rewritten_by_the_retry_branch ... ok
test machine::tests::each_failed_attempt_journals_its_own_output_beside_an_identical_verdict ... ok
test machine::tests::the_exhaustion_label_replaces_verification_failed_over_identical_evidence ... ok
test result: ok. 68 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.69s

SDK:

$ cd packages/sdk && npm test
Test Files  7 failed | 149 passed | 3 skipped (159)
     Tests  40 failed | 2373 passed | 25 skipped (2438)

The 40 failures are pre-existing and environmental, not caused by this
change. Verified by taking a baseline: git stash push --include-untracked,
re-running the same seven files on a clean tree, and getting the identical 40
failures. Causes, all sandbox:

• spawn .../kernel/target/debug/relayflowd ENOENT — those tests hardcode
  kernel/target/debug, but ops/cargo.sh sets CARGO_TARGET_DIR to
  $HOME/.relayflows-toolchain/target/<key>, deliberately outside the worktree.
• bun --version 1.3.6 against an expected 1.4.0.
• expected an @relayflows/surface flow handle — module identity between the
  built dist and the source copy.

New SDK tests: tests/step-attempt-history.test.ts (18 tests — history
collection, comparison, redaction-before-bounding, page boundaries, IPC frame
round-trip) and tests/retried-step-failure.test.ts (one live-kernel run that
asserts the rendered JSON diagnostic and then reads the journal back to pin
the kernel contract it relies on).

Files

 docs/CLOUD.md                               |  11 ++
 docs/SURFACE.md                             |  43 ++++++-
 kernel/relayflowd-core/src/machine/tests.rs | 177 ++++++++++++++++++++++++++++
 packages/sdk/src/authored-node-runner.ts    |  76 ++++++++++--
 packages/sdk/src/authored-worker-step.ts    |   2 +
 packages/sdk/src/cli/step-evidence.ts       | 273 +++++++++++++++++++++++++++ (new)
 packages/sdk/src/cli/step-failure.ts        | 147 +++++++++--------------
 packages/sdk/src/failure-kinds.ts           |  61 ++++++++++
 packages/sdk/tests/cloud-read.test.ts       |  28 +++++
 packages/sdk/tests/retried-step-failure.test.ts  (new)
 packages/sdk/tests/step-attempt-history.test.ts  (new)

step-evidence.ts is a new sibling rather than more lines in
step-failure.ts: raw extraction, bounding, redaction and comparison are one
concern, and the reader plus the renderer are another. Same reason
step-attempt-history.test.ts is a new file rather than 300 more lines in the
existing 339-line step-failure-diagnostic.test.ts (AGENTS.md: files
approaching 500 lines are a design smell).

---

Note

Medium Risk
Changes failure diagnostics and JSON/--json shape for multi-attempt steps (additive fields) and redaction/bounding of attempt excerpts; behavior is well-tested but affects operator-facing error output on retried runs.

Overview
Retried step failures now surface every attempt, not only the terminal retries_exhausted record. The journal reader walks all failed step.completed entries per step, keeps terminal scalars unchanged, and adds optional attempts plus attemptEvidence (differs | unchanged | unknown) when there were multiple failures.

New step-evidence.ts centralizes journal extraction (deterministic output vs agent verification.detail), redacted 256-byte per-attempt excerpts, unbounded failureCause comparison (so shared tail bytes or verification_failed → retries_exhausted relabels do not hide real differences), and Attempts: rendering with a side-effect warning when evidence differs. stepFailedFrame and authored paths forward the new fields over IPC.

Docs (SURFACE.md, CLOUD.md) describe the expanded diagnostic and that hosted flows logs only shows what the runner printed (no journal export). Kernel tests pin per-attempt journaling without changing retry policy. SDK tests cover history, gates, redaction, cloud logs, and a live kernel retried-step scenario.

^{Reviewed by Cursor Bugbot for commit b9ae9ac. Bugbot is set up for automated code reviews on this repo. Configure here.}

---

Summary by cubic

Fixes retries_exhausted to report every failed attempt, so a retry that fails differently no longer hides the first attempt's error — exactly when the first attempt's failure is the diagnosis. Fixes #506.

• The reader accumulates a per-step history of failures from the journal (keyed by step, cleared on later success); scalar fields still describe the terminal attempt, and multi-attempt failures add attempts and attemptEvidence.
• Comparison runs on the unbounded journal record rather than the display excerpts and treats the verification_failed → retries_exhausted relabel as the same cause.
• Gate verdicts are withheld only when they restate bytes already printed, so a first attempt rejected by a gate on exit 0 with empty tails is still explained.
• Evidence truncated by its producer — the daemon's render cap or the transcript digest's truncation flag — yields unknown, and an attempt that journaled no evidence is listed but never compared, so neither can raise the side-effect warning without evidence.
• Agent and llm steps are covered by reading the daemon's bounded render where their output is nulled; single-attempt failures render exactly as before.

^{Written for commit b9ae9ac. Summary will update on new commits.}

Fixes #506

[coderabbitai]

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 723a675f-9487-46de-8069-bbe207efaf52

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[agent-relay-code]

Relayflow: the adversarial review did not pass. This branch is not approved: the flow stopped here and did not mark it ready to merge.

Review of PR #519

Head: c997e3189cac93b6ce07623c6a1282e7adb3d300
Diff base: e21caad1e4020d9e2675d2840301cb7eeb6f1aac
Decision: Changes requested. Do not create review.clean.

P2 — Honor transcript truncation before reporting unchanged failures

Location: packages/sdk/src/cli/step-evidence.ts:207–208.

failureCause compares trajectory_tail.transcript.failure.excerpt, but its producerTruncated flag only checks the daemon's truncation suffix in verification.detail. It ignores trajectory_tail.transcript.failure.truncated, which the SDK's own buildTranscriptDigest and boundTranscriptDigest produce. Thus two different tool errors reduced to the same excerpt are reported as unchanged, even though the journal explicitly says those excerpts are incomplete.

This affects an agent whose JSON output is promoted by worker.ts while its process fails: the daemon can record a short, identical verification detail such as {}, while the tool error is carried only in the transcript digest. A long shared prefix with different error suffixes then produces identical, truncated excerpts. The diagnostic asserts “recorded failure evidence is unchanged across them” instead of admitting that the causes cannot be compared. This defeats the new comparison's explicit protection against producer-truncated evidence and can misdirect the diagnosis of a retry.

Include the transcript's failure truncation flag when determining whether matching records can establish unchanged. Add a regression alongside the daemon-truncation case: matching excerpts with failure.truncated: true must produce unknown; genuinely different surviving evidence may still produce differs.

Reproduction: source. It uses the production transcript digest builder and journal reader/renderer, with journal-shaped entries. It is not a live worker or daemon test. Command from repository root (requires the built SDK):

node review-artifacts/pr519-transcript-truncation.mjs > review-artifacts/pr519-transcript-truncation.log 2>&1

Exit status: 1. Complete captured output:

Distinct original tool errors: true
Producer marked both excerpts truncated: true
Surviving excerpts equal: true
 Step "agent" (agent) completionReason: worker_error attempt=2.
Attempts: 2 failed; recorded failure evidence is unchanged across them.
  attempt 1: worker_error (excerpt truncated) — detail: ostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared dia…[566 bytes truncated]
  attempt 2: worker_error (excerpt truncated) — detail: ostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared dia…[566 bytes truncated]
Detail: t shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared diagnostic context shared dia…[566 bytes truncated]
node:internal/modules/run_main:107
    triggerUncaughtException(
    ^

AssertionError [ERR_ASSERTION]: truncated transcript evidence cannot establish unchanged
+ actual - expected

+ 'unchanged'
- 'unknown'
     ^

    at file:///home/daytona/.relayflow-v2-supervisor/durable/repository/review-artifacts/pr519-transcript-truncation.mjs:39:8 {
  generatedMessage: false,
  code: 'ERR_ASSERTION',
  actual: 'unchanged',
  expected: 'unknown',
  operator: 'strictEqual',
  diff: 'simple'
}

Node.js v25.6.0

Previous finding

The previous review's missing deterministic gate verdict and missing-account comparison case are resolved by this head. The original reproduction was rerun without modification:

node review-artifacts/pr519-repro.mjs > review-artifacts/pr519-current-repro.log 2>&1

Exit status: 0. Complete captured output:

CASE 1: first attempt only has a verification error
 Step "check" (deterministic) completionReason: retries_exhausted attempt=2 exit=1.
Attempts: 2 failed; recorded failure evidence differs. An earlier attempt may have had side effects.
  attempt 1: verification_failed exit=0 — detail: output did not contain "READY"
  attempt 2: retries_exhausted exit=1
    detail: exit code was 1; output did not contain "READY"
    stderr: nothing staged inside the declared scope
Detail: exit code was 1; output did not contain "READY"
Stderr (last 1,024 bytes):
nothing staged inside the declared scope
first verification error visible: true

CASE 2: one attempt has no account; the next has evidence
 Step "check" (deterministic) completionReason: worker_error attempt=2 exit=1.
Attempts: 2 failed; recorded failure evidence is incomplete, so whether the causes differ is unknown.
  attempt 1: worker_error — no failure evidence recorded
  attempt 2: worker_error exit=1
    detail: exit code was 1; output did not contain "READY"
    stderr: nothing staged inside the declared scope
Detail: exit code was 1; output did not contain "READY"
Stderr (last 1,024 bytes):
nothing staged inside the declared scope

Scope and PR comments

Read AGENTS.md and RFC-0001, reviewed all 11 changed files, and traced the reader, renderer, IPC propagation, transcript producer, and kernel completion shapes. Retry policy and counts are unchanged. No implementation, tests, gates, generated source, or docs/evidence files were edited by this review. Build outputs were produced only by the package test command.

Read the current PR body, conversation comments, submitted reviews, and inline comments with:

gh pr view 519 --json number,url,baseRefName,headRefOid,body,comments,reviews > review-artifacts/pr519-current-comments.json
gh api repos/AgentWorkforce/flows/pulls/519/comments --paginate > review-artifacts/pr519-current-inline.json

Captured responses: PR discussion, inline comments. Submitted reviews and inline comments are empty ([]); the sole conversation comment is CodeRabbit's review-skipped notice. Automated summaries in the PR body are not treated as independent verification.

The Cloud tests mock the runner-log endpoint. They establish rendering of captured diagnostics, not actual hosted capture. No live Cloud verification or mutation verification was performed.

Verification

Kernel command, run from kernel/:

sh ../ops/cargo.sh test -p relayflowd-core > ../review-artifacts/pr519-current-core-tests.log 2>&1

Exit status: 0. Literal result lines:

test result: ok. 68 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.77s
test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Complete captured output: kernel tests.

SDK command, run from packages/sdk/:

npm test > ../../review-artifacts/pr519-current-sdk-tests.log 2>&1

Exit status: 1. Literal captured lines for the changed tests, the existing diagnostic tests, and suite totals:

 ✓ tests/cloud-read.test.ts (41 tests) 46ms
 ✓ tests/step-attempt-history.test.ts (21 tests) 23ms
 ✓ tests/step-failure-diagnostic.test.ts (21 tests) 41ms
 ✓ tests/retried-step-failure.test.ts (1 test) 573ms
 Test Files  7 failed | 149 passed | 3 skipped (159)
      Tests  41 failed | 2376 passed | 25 skipped (2442)
     Errors  1 error
   Duration  204.57s (transform 2.28s, setup 0ms, collect 37.41s, tests 528.93s, environment 20ms, prepare 6.28s)

Complete captured output, including all failure messages and stack traces: SDK tests.

The full SDK suite is not green. The changed test files pass, but do not cover the transcript truncation finding. This review did not rerun the clean base and therefore does not independently classify the 41 failures and unhandled error as pre-existing. The PR body’s baseline claim is not substituted for a captured baseline run.

[khaliqgant]

Marking ready for review.

This PR was drafted by the Software Garden when its adversarial review withheld signoff. The flow drafts on a failed review and never re-evaluates, so the "not approved" verdict above is a permanent record of one moment, not a current statement — three PRs merged today (#512, #521, #545) were in exactly this state with their findings long since fixed.

Re-reading this PR's verdict against the current head: it identifies no open production-code defect. CI is green (6 checks, 0 failures).

Whoever reviews this should still read the verdict for the caveats it records — they are real, they are simply not code defects.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-24T05:23:50.548689Z	562fcb8	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 562fcb884f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Treat crash completion reasons as recorded evidence

When an attempt is abandoned as crashed or lease_expired, abandonment_actions journals that typed completionReason but intentionally leaves output and verification empty, so this expression marks the attempt unrecorded. If a later attempt then reaches a normal terminal failure, compareAttempts filters out the abandoned attempt and reports unknown instead of differs, suppressing the side-effect warning precisely when RFC Appendix A says a crashed agent may have left a dirty workspace. Count non-policy completion reasons as evidence, while retaining the existing normalization of verification_failed and retries_exhausted.

AGENTS.md reference: AGENTS.md:L16-L18

Useful? React with 👍 / 👎.