Skip to content

fix(watch): stop wedge-escalating a crew whose run is gone - #10

Merged
NewAiCoder merged 2 commits into
mainfrom
fm/watcher-done-task-wedge-escalation
Sep 27, 2026
Merged

NewAiCoder merged 2 commits into
mainfrom
fm/watcher-done-task-wedge-escalation

Conversation

@NewAiCoder

@NewAiCoder NewAiCoder commented Sep 27, 2026 •

Copy link
Copy Markdown
Owner

Intent

Stop the firstmate watcher from wedge-escalating a finished crew: once a crew's run is gone (latest status done:, crew state unknown), the wedge timer must stop, with at most the ordinary one-time stale surfacing and then quiet, while a genuinely working-and-frozen run still escalates and declared pauses are unchanged. PR #10 already exists for this branch.

What Changed

  • wedge_timer_check in bin/fm-watch.sh takes a new reverify argument, passed by both stale-pane call sites. With it set, the watcher re-reads crew state before starting a wedge timer or escalating on one. If the crew is no longer provably working (for example a done: status with unknown crew state), no timer is started. An already-expired timer is dropped along with the hash suppressor, so the next poll classifies the hash afresh. The finished crew then surfaces at most once through the ordinary stale path and stays quiet. A crew that is still provably working and frozen escalates as before, and paused: handling is unchanged.
  • A new .stale-settled-<key> marker records the hash that was found no longer provably working. While it matches the current stale hash, later polls return early without re-reading crew state. Every place that clears stale or pause tracking (handle_paused_stale, clear_stale_hash_tracking, and the busy-pane resets) now also removes it.
  • tests/fm-watch-triage.test.sh adds two cases, one for a done: status and one for a non-terminal status. Each drives working absorb (timer starts), then run gone at the threshold, then the same hash staying quiet. docs/architecture.md gains a short note on the wedge timer's crew-state gate.

Risk Assessment

✅ Low: The change is bounded to the wedge-timer path in fm-watch.sh. The reverify flag is passed only from the two call sites that started a timer on run evidence, and declared-pause and busy-turn callers are unchanged. The added tests exercise both status shapes through working, run-gone and quiet phases by running the watcher.

Testing

I drove the real bin/fm-watch.sh as a subprocess through a filtered copy of tests/fm-watch-triage.test.sh (19 tests). It covers the two new working-then-gone cases, plus the genuine-wedge, declared-pause, busy-turn, timer-repair and write-deferral regressions, and all passed on the fix. Against the base watcher the new done-run-gone test fails with a "possible wedge, escalation 1" wake, so it reproduces the bug. I stopped there and did not run the quiet-run-gone case against base. The filtered copy is removed and the worktree is clean.

  • Live validation: ✅ go - 5 of 5 scenarios driven live against the product
Scenario Result Live Evidence
A done: crew whose run vanished stops wedge-escalating: one ordinary stale wake, wedge timer and escalation count gone, same hash quiet afterwards ✅ pass live test_terminal_done_crew_whose_run_is_gone_stops_wedge_escalating in head-run.log; the same test fails on base with 'possible wedge, escalation 1' in base-run.log
A quiet working: crew whose run vanished never restarts the wedge timer and never wakes again ✅ pass live test_nonterminal_crew_whose_run_is_gone_stops_wedge_escalating in head-run.log
A genuinely working-and-frozen run still wedge-escalates on schedule, including after write deferral and timer repair ✅ pass live provably_working_absorbed_then_escalated, wedge_escalation_deferred_while_worktree_is_written, write_deferral_resurfaces, timer_repair_drops tests in head-run.log
Declared pauses (including a busy pause and a live no-mistakes-run pause) are unchanged: rechecked on cadence, never wedge-escalated ✅ pass live paused_absorbed_then_resurfaced, busy_declared_pause_is_rechecked, nm_run_declared_pause, paused_authoritative_working tests in head-run.log
Busy-turn bound and not-provably-working immediate surfacing paths unchanged ✅ pass live busy_pane_stable_hash_escalates, busy_pane_below_turn_age_bound, nonterminal_stale_not_working_surfaced tests in head-run.log
Evidence: Fixed watcher: filtered triage tests, all ok

Source: Fixed watcher: filtered triage tests, all ok

ok - a stale pane sitting on a terminal status is surfaced (queue + exit)
ok - a stale terminal-looking status is overridden and absorbed while a run is actively working, then wedge-escalated
ok - provably-working non-terminal stale is absorbed on first sight, then wedge-escalated past the threshold
ok - a done: crew whose run is gone stops wedge-escalating after the ordinary surface
ok - a quiet crew whose run is gone stops wedge-escalating and never restarts the timer
ok - consecutive wedge escalations on the same pane accumulate and demand deep inspection at the threshold
ok - a pane becoming active again resets the consecutive wedge-escalation counter
ok - a busy worker below the turn-age bound remains working with no escalation
ok - a busy worker with a stable pane hash still escalates once its completed-turn age reaches the bound
ok - a busy pane under a declared pause is rechecked on the long cadence, and lifting the pause restores the wedge escalation
ok - a not-provably-working non-terminal stale is surfaced immediately (never left to wait out the timer)
ok - a declared pause is absorbed on first sight, then re-surfaced as a recheck past the threshold, never wedge-escalated
ok - a declared pause is periodically rechecked against authoritative active-run state
ok - a paused status overridden by authoritative working preserves its wedge timer and escalates
ok - a declared no-mistakes-run pause backed by a live run holds the pause cadence and never wedge-escalates
ok - matching non-terminal stale suppressors repair missing or corrupt stale-since timers
ok - a quiet pane writing its own worktree is deferred, while one writing nothing still wedge-escalates on the unchanged schedule
ok - a write deferral re-surfaces once on the bounded pause cadence, so a churning worktree cannot stay invisible
ok - an idle-window timer repair drops a finished write-deferral chain, so the next deferral gets a fresh re-surface window

real	1m20.926s
user	0m44.997s
sys	0m23.346s
Evidence: Base watcher: new done-run-gone test fails with a wedge wake

Source: Base watcher: new done-run-gone test fails with a wedge wake

not ok - done-run-gone: a crew whose run is gone was wedge-escalated: stale: test:fm-donegone (idle 500s, possible wedge, escalation 1)
1790476036	1	stale	test:fm-donegone	stale: test:fm-donegone (idle 500s, possible wedge, escalation 1)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

✅ **Review** - passed

✅ No issues found.

✅ **Test** - passed

✅ No issues found.

  • Live validation: ✅ go - 5 of 5 scenarios driven live against the product
Scenario Result Live Evidence
A done: crew whose run vanished stops wedge-escalating: one ordinary stale wake, wedge timer and escalation count gone, same hash quiet afterwards ✅ pass live test_terminal_done_crew_whose_run_is_gone_stops_wedge_escalating in head-run.log; the same test fails on base with 'possible wedge, escalation 1' in base-run.log
A quiet working: crew whose run vanished never restarts the wedge timer and never wakes again ✅ pass live test_nonterminal_crew_whose_run_is_gone_stops_wedge_escalating in head-run.log
A genuinely working-and-frozen run still wedge-escalates on schedule, including after write deferral and timer repair ✅ pass live provably_working_absorbed_then_escalated, wedge_escalation_deferred_while_worktree_is_written, write_deferral_resurfaces, timer_repair_drops tests in head-run.log
Declared pauses (including a busy pause and a live no-mistakes-run pause) are unchanged: rechecked on cadence, never wedge-escalated ✅ pass live paused_absorbed_then_resurfaced, busy_declared_pause_is_rechecked, nm_run_declared_pause, paused_authoritative_working tests in head-run.log
Busy-turn bound and not-provably-working immediate surfacing paths unchanged ✅ pass live busy_pane_stable_hash_escalates, busy_pane_below_turn_age_bound, nonterminal_stale_not_working_surfaced tests in head-run.log
  • bash tests/zz-filtered.test.sh (temporary filtered copy of tests/fm-watch-triage.test.sh, 19 tests, all ok, about 81s)
  • Base-watcher run of the two new tests with git show 3ce7f05:bin/fm-watch.sh swapped in: done-run-gone fails with a wedge escalation (fail-before-fix confirmed), then the fixed file was restored
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

NewAiCoder added 2 commits September 26, 2026 22:22
A stale pane absorbed as provably working starts a wedge timer. When the run
later vanished (crew state unknown, latest status done:), the timer kept
escalating 'possible wedge' every STALE_ESCALATE_SECS, and the repair path
restarted it after each escalation.

Re-confirm provable work where the idle-pane paths use the timer: at the
threshold, a crew no longer working drops the timer and hash suppressor so the
ordinary first-sight handling surfaces it once; when the timer is missing, a
crew no longer working gets no new timer and the hash is marked settled.
Busy-turn bound callers and genuinely working runs are unchanged.
@NewAiCoder NewAiCoder changed the title fix(watch): stop the wedge timer once a crew's run is gone fix(watch): stop wedge-escalating a crew whose run is gone Sep 27, 2026
@NewAiCoder
NewAiCoder merged commit 8e0e7e6 into main Sep 27, 2026
14 of 15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant