feat(watch): shadow graded wedge score alongside the fixed timer

[devin-ai-integration]

Intent

Add plumbing-only wedge settlement recording and shadow graded wedge scoring to FirstMate, lift ARGYLE gates.py verbatim with provenance, preserve the fixed 240-second escalation behavior, and validate and deliver the existing branch through the no-mistakes pipeline.

What Changed

• bin/fm-watch.sh gains wedge_shadow_settle/wedge_shadow_score/wedge_shadow_resumed helpers that record one settlement row per wedge escalation and per pane resume, and log a graded score line to the triage log. All three are gated on FM_WEDGE_SHADOW (default on) and on python3 being present, and they never feed back into escalation — the fixed FM_STALE_ESCALATE_SECS timer remains the sole decision path.
• New bin/fm-wedge-score.py owns the state/.wedge-settlements.jsonl schema, the task→lane rule (read from <task>.meta project=, else unknown), and the log's 256 KiB size bound. Its score subcommand fits a lognormal over resumed idle times per lane, counts repeated escalations of an unresolved window as a single incident, and reports graded_flag beside fixed_flag, returning insufficient data below --min-obs. bin/argyle_gates.py is lifted verbatim from nakatomi with a provenance header and supplies the pure-stdlib norm_cdf/norm_ppf used by the score.
• Adds tests/fm-wedge-score.test.sh (7 cases: lane recording, minimum-observation refusal, graded/fixed flags, caller-supplied threshold, repeat-escalation incident collapsing, missing-meta lane, log bounding) and registers it in bin/fm-test-run.sh. Docs record FM_WEDGE_SHADOW in docs/configuration.md and restate .watch-triage.log as the watcher's general bounded observational log rather than an absorbed-wake-only log.

Risk Assessment

⚠️ Medium: The change is strictly additive and shadow-only - escalation still depends solely on STALE_ESCALATE_SECS and both scorer calls are failure-tolerant, so it is safe to merge - but the fix round's new settlement bookkeeping emits two provably wrong values (a resumed row with idle_secs=0 for a busy non-resume, and an incident count that permanently under-reports repeat wedges), and the watcher/scorer seam where the previously fixed defects lived still has no test.

Testing

I drove the real bin/fm-watch.sh in a hermetic sandbox and captured the surfaces an operator actually sees: a resumed pane appends a settlement row with its true idle, a pane wedged past the threshold prints the same stale: ... possible wedge wake, queues the same drain entry, appends an escalated row, and logs one shadow wedge score: {...} line into state/.watch-triage.log. The decisive check is scenario 3 — on a lane where long healthy idles are normal the graded comparator returns graded_flag=false while fixed_flag=true, and the watcher escalates anyway, showing the score is observational only; with FM_WEDGE_SHADOW=0 nothing is recorded and the wake is identical, and with FM_STALE_ESCALATE_SECS unset the shipped 240s bound still absorbs at ~200s and escalates at ~260s. Alongside that, the new fm-wedge-score suite passes, is picked up by fm-test-run.sh, and proved non-vacuous: four seeded regressions in the scorer each made it fail (source restored, worktree clean). The 40-test fm-watch-triage regression suite is green, and the lifted argyle_gates primitives match reference normal values to ~1e-9 — though the upstream nakatomi repo is absent on this host, so verbatim-ness rests on the in-file provenance header rather than a diff. No screenshots: every surface here is a CLI wake line and a state-directory log file, so the transcripts and JSONL rows are the end-user artifact.

Evidence: End-to-end watcher transcript: settlements, shadow score, and preserved 240s escalation (5 scenarios)

========== SCENARIO 1  pane wedges, resumes -> a 'resumed' settlement is recorded ==========
wedge timer armed at: 1788623759  (epoch now 1788623763)
--- state/.wedge-settlements.jsonl ---
{"ts":"2026-09-05T15:56:04.498282Z","window":"test:fm-quiet","task":"quiet","lane":"demo","idle_secs":64,"outcome":"resumed"}
--- watcher stdout (no wake expected: resume is not actionable) ---

========== SCENARIO 2  pane stays wedged past 240s -> escalation + shadow score ==========
wedge timer armed, watcher absorbed the first sighting (stdout empty: yes)
--- watcher stdout: the actionable wake an operator sees ---
stale: test:fm-quiet (idle 500s, possible wedge, escalation 1)
--- state/.watch-triage.log ---
[2026-09-05T08:56:05-0700] absorbed non-terminal stale (provably working): test:fm-quiet
[2026-09-05T08:56:10-0700] shadow wedge score: {"lane":"demo","n":8,"n_total":9,"base_rate":0.1818,"z":9.2421,"p_tail":0.0,"score":8.3336,"graded_flag":true,"fixed_flag":true,"reason":"ok"}
--- new settlement row (escalated) ---
{"ts":"2026-09-05T15:56:10.667620Z","window":"test:fm-quiet","task":"quiet","lane":"demo","idle_secs":500,"outcome":"escalated"}
--- queued wake (fm-wake-drain.sh) ---
1788623770	1	stale	test:fm-quiet	stale: test:fm-quiet (idle 500s, possible wedge, escalation 1)
--- shadow score, pretty ---
{
    "lane": "demo",
    "n": 8,
    "n_total": 9,
    "base_rate": 0.1818,
    "z": 9.2421,
    "p_tail": 0.0,
    "score": 8.3336,
    "graded_flag": true,
    "fixed_flag": true,
    "reason": "ok"
}

========== SCENARIO 3  graded score DISAGREES (graded_flag=false) - fixed 240s timer still escalates ==========
--- watcher stdout: escalated anyway (fixed 240s behavior preserved) ---
stale: test:fm-quiet (idle 501s, possible wedge, escalation 1)
--- shadow score for that same event ---
{
    "lane": "demo",
    "n": 8,
    "n_total": 9,
    "base_rate": 0.1818,
    "z": -0.104,
    "p_tail": 0.5414,
    "score": -1.0125,
    "graded_flag": false,
    "fixed_flag": true,
    "reason": "ok"
}
VERDICT: graded_flag=False fixed_flag=True -> the graded comparator would NOT have escalated; the fixed timer did.

========== SCENARIO 4  FM_WEDGE_SHADOW=0 - no recording, identical escalation ==========
--- watcher stdout ---
stale: test:fm-quiet (idle 500s, possible wedge, escalation 1)
--- settlements file present? ---
absent (as expected)
--- 'shadow wedge score' lines in the triage log ---
0

========== SCENARIO 5  production default bound (FM_STALE_ESCALATE_SECS unset = 240s) on a fresh fleet ==========
idle ~200s: no wake, watcher still absorbing - under the 240s bound
shadow score lines so far: 0  (the score is computed only on escalation)
idle ~260s: stale: test:fm-quiet (idle 261s, possible wedge, escalation 1)
--- shadow score with no settlement history yet ---
{"lane":"demo","n":0,"score":null,"graded_flag":null,"fixed_flag":true,"reason":"insufficient data: 0<8"}
--- settlement log after that first escalation ---
{"ts":"2026-09-05T15:56:37.153027Z","window":"test:fm-quiet","task":"quiet","lane":"demo","idle_secs":261,"outcome":"escalated"}

========== DONE ==========

Evidence: E2E driver script used to produce the transcript (evidence dir, not committed)

#!/usr/bin/env bash
# Manual end-to-end verification of the wedge shadow settlement/score plumbing.
# Drives the REAL bin/fm-watch.sh against a hermetic fake tmux/crew-state
# sandbox (same fixture shape tests/fm-watch-triage.test.sh uses) and shows the
# operator-visible surfaces: state/.wedge-settlements.jsonl rows, the
# state/.watch-triage.log "shadow wedge score:" line, and the wake the watcher
# prints on escalation.
set -u
ROOT=$1
OUTDIR=$2
. "$ROOT/tests/lib.sh"
. "$ROOT/tests/wake-helpers.sh"
TMP_ROOT=$(fm_test_tmproot fm-wedge-e2e)
WATCH="$ROOT/bin/fm-watch.sh"
WINDOW="test:fm-quiet"
KEY=$(printf '%s' "$WINDOW" | tr ':/.' '___')

seen_sig() {
  if [ "$(uname)" = Darwin ]; then stat -f '%z:%Fm' "$1"; else stat -c '%s:%Y' "$1"; fi
}
reap() { kill "$1" 2>/dev/null || true; wait "$1" 2>/dev/null || true; }
wait_live() { local pid=$1 n=${2:-30} i=0; while [ $i -lt "$n" ]; do kill -0 "$pid" 2>/dev/null || return 1; sleep 0.1; i=$((i+1)); done; return 0; }
wait_exit() { local pid=$1 n=${2:-60} i=0; while [ $i -lt "$n" ]; do kill -0 "$pid" 2>/dev/null || { wait "$pid" 2>/dev/null || true; return 0; }; sleep 0.1; i=$((i+1)); done; return 1; }

setup_case() {  # <name> -> echoes dir
  local dir; dir=$(make_case "$1")
  printf 'idle building output' > "$dir/pane.txt"
  printf 'window=%s\nkind=ship\nproject=demo\n' "$WINDOW" > "$dir/state/quiet.meta"
  printf 'working: still compiling\n' > "$dir/state/quiet.status"
  printf '%s' "$(seen_sig "$dir/state/quiet.status")" > "$dir/state/.seen-quiet_status"
  printf '%s' "$(hash_text 'idle building output')" > "$dir/state/.hash-$KEY"
  printf '1\n' > "$dir/state/.count-$KEY"
  printf '%s\n' "$dir"
}

run_watch() {  # <dir> <escalate-secs> <out> [extra env...]
  local dir=$1 esc=$2 out=$3; shift 3
  env PATH="$dir/fakebin:$PATH" FM_FAKE_TMUX_WINDOW="$WINDOW" \
    FM_FAKE_TMUX_CAPTURE="$dir/pane.txt" FM_STATE_OVERRIDE="$dir/state" \
    FM_CREW_STATE_BIN="$dir/fakebin/fm-crew-state.sh" \
    FM_FAKE_CREW_STATE='state: working · source: run-step · ci running' \
    FM_STALE_ESCALATE_SECS="$esc" FM_POLL=1 FM_SIGNAL_GRACE=1 \
    FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 "$@" "$WATCH" > "$out" 2>&1 &
}

banner() { printf '\n========== %s ==========\n' "$1"; }

# --------------------------------------------------------------------------
banner "SCENARIO 1  pane wedges, resumes -> a 'resumed' settlement is recorded"
D=$(setup_case resumed); S="$D/state"
run_watch "$D" 999 "$D/w1.out"; PID=$!
wait_live "$PID" 40 >/dev/null || true
echo "wedge timer armed at: $(cat "$S/.stale-since-$KEY" 2>/dev/null)  (epoch now $(date +%s))"
# Backdate the armed timer by 63s, then let the pane change (crew resumed).
echo $(( $(date +%s) - 63 )) > "$S/.stale-since-$KEY"
printf 'compiling done, running tests' > "$D/pane.txt"
for _ in $(seq 60); do [ -e "$S/.wedge-settlements.jsonl" ] && break; sleep 0.2; done
reap "$PID"
echo "--- state/.wedge-settlements.jsonl ---"
cat "$S/.wedge-settlements.jsonl" 2>/dev/null || echo "(none)"
echo "--- watcher stdout (no wake expected: resume is not actionable) ---"
cat "$D/w1.out"

# --------------------------------------------------------------------------
banner "SCENARIO 2  pane stays wedged past 240s -> escalation + shadow score"
D=$(setup_case escalated); S="$D/state"
# Seed a healthy lane history (8 resumed settlements, typical short idles) so
# the shadow score has enough observations to grade.
python3 - "$S/.wedge-settlements.jsonl" <<'PY'
import json, sys
with open(sys.argv[1], "w") as fh:
    for i, idle in enumerate((38, 44, 51, 47, 60, 55, 42, 66)):
        fh.write(json.dumps({"ts": "2026-09-05T00:0%d:00Z" % i, "window": "test:fm-quiet",
                             "task": "quiet", "lane": "demo", "idle_secs": idle,
                             "outcome": "resumed"}) + "\n")
PY
run_watch "$D" 999 "$D/w2a.out"; PID=$!
wait_live "$PID" 40 >/dev/null || true
reap "$PID"
echo "wedge timer armed, watcher absorbed the first sighting (stdout empty: $([ -s "$D/w2a.out" ] && echo no || echo yes))"
# The pane never changes; backdate the timer past the fixed 240s threshold.
echo $(( $(date +%s) - 500 )) > "$S/.stale-since-$KEY"
: > "$D/w2b.out"
run_watch "$D" 240 "$D/w2b.out"; PID=$!
wait_exit "$PID" 100 || { reap "$PID"; echo "WATCHER DID NOT EXIT"; }
echo "--- watcher stdout: the actionable wake an operator sees ---"
cat "$D/w2b.out"
echo "--- state/.watch-triage.log ---"
cat "$S/.watch-triage.log" 2>/dev/null || echo "(none)"
echo "--- new settlement row (escalated) ---"
tail -1 "$S/.wedge-settlements.jsonl"
echo "--- queued wake (fm-wake-drain.sh) ---"
FM_STATE_OVERRIDE="$S" "$ROOT/bin/fm-wake-drain.sh" 2>/dev/null
echo "--- shadow score, pretty ---"
grep -o '{.*}' "$S/.watch-triage.log" | tail -1 | python3 -m json.tool

# --------------------------------------------------------------------------
banner "SCENARIO 3  graded score DISAGREES (graded_flag=false) - fixed 240s timer still escalates"
D=$(setup_case shadow-only); S="$D/state"
# A lane where long healthy idles are normal (a crew that legitimately waits on
# slow CI): the graded model considers a 500s idle unremarkable, so graded_flag
# comes back false while the fixed 240s timer escalates regardless.
python3 - "$S/.wedge-settlements.jsonl" <<'PY'
import json, sys
with open(sys.argv[1], "w") as fh:
    for i, idle in enumerate((120, 200, 350, 480, 700, 900, 1500, 2400)):
        fh.write(json.dumps({"ts": "2026-09-05T00:0%d:00Z" % i, "window": "test:fm-quiet",
                             "task": "quiet", "lane": "demo", "idle_secs": idle,
                             "outcome": "resumed"}) + "\n")
PY
run_watch "$D" 999 "$D/w3a.out"; PID=$!
wait_live "$PID" 40 >/dev/null || true
reap "$PID"
echo $(( $(date +%s) - 500 )) > "$S/.stale-since-$KEY"
: > "$D/w3b.out"
run_watch "$D" 240 "$D/w3b.out"; PID=$!
wait_exit "$PID" 100 || { reap "$PID"; echo "WATCHER DID NOT EXIT"; }
echo "--- watcher stdout: escalated anyway (fixed 240s behavior preserved) ---"
cat "$D/w3b.out"
echo "--- shadow score for that same event ---"
grep -o '{.*}' "$S/.watch-triage.log" | tail -1 | python3 -m json.tool
grep -o '{.*}' "$S/.watch-triage.log" | tail -1 | python3 -c '
import json,sys
r=json.loads(sys.stdin.read())
print("VERDICT: graded_flag=%s fixed_flag=%s -> the graded comparator would NOT have"
      " escalated; the fixed timer did." % (r["graded_flag"], r["fixed_flag"]))
'

# --------------------------------------------------------------------------
banner "SCENARIO 4  FM_WEDGE_SHADOW=0 - no recording, identical escalation"
D=$(setup_case shadow-off); S="$D/state"
run_watch "$D" 999 "$D/w4a.out" FM_WEDGE_SHADOW=0; PID=$!
wait_live "$PID" 40 >/dev/null || true
reap "$PID"
echo $(( $(date +%s) - 500 )) > "$S/.stale-since-$KEY"
: > "$D/w4b.out"
run_watch "$D" 240 "$D/w4b.out" FM_WEDGE_SHADOW=0; PID=$!
wait_exit "$PID" 100 || { reap "$PID"; echo "WATCHER DID NOT EXIT"; }
echo "--- watcher stdout ---"
cat "$D/w4b.out"
echo "--- settlements file present? ---"
[ -e "$S/.wedge-settlements.jsonl" ] && echo "PRESENT (unexpected)" || echo "absent (as expected)"
echo "--- 'shadow wedge score' lines in the triage log ---"
grep -c 'shadow wedge score' "$S/.watch-triage.log" 2>/dev/null || true

# --------------------------------------------------------------------------
banner "SCENARIO 5  production default bound (FM_STALE_ESCALATE_SECS unset = 240s) on a fresh fleet"
run_prod() {  # <dir> <out>   - no FM_STALE_ESCALATE_SECS: the shipped 240s default applies
  env PATH="$1/fakebin:$PATH" FM_FAKE_TMUX_WINDOW="$WINDOW" \
    FM_FAKE_TMUX_CAPTURE="$1/pane.txt" FM_STATE_OVERRIDE="$1/state" \
    FM_CREW_STATE_BIN="$1/fakebin/fm-crew-state.sh" \
    FM_FAKE_CREW_STATE='state: working · source: run-step · ci running' \
    FM_POLL=1 FM_SIGNAL_GRACE=1 FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 \
    "$WATCH" > "$2" 2>&1 &
}
prime() {  # <dir>  - arm the wedge timer via a real absorb, then stop the watcher
  local pid; run_prod "$1" "$1/prime.out"; pid=$!
  wait_live "$pid" 40 >/dev/null || true; reap "$pid"
}

# 5a: 200s idle - comfortably under the fixed bound (a few polls of slack).
D=$(setup_case default-under); prime "$D"
echo $(( $(date +%s) - 200 )) > "$D/state/.stale-since-$KEY"
run_prod "$D" "$D/under.out"; PID=$!
if wait_live "$PID" 50 >/dev/null; then
  echo "idle ~200s: no wake, watcher still absorbing - under the 240s bound"
else
  echo "idle ~200s: ESCALATED EARLY -> $(cat "$D/under.out")"
fi
reap "$PID"
echo "shadow score lines so far: $(grep -c 'shadow wedge score' "$D/state/.watch-triage.log" 2>/dev/null | head -1)  (the score is computed only on escalation)"

# 5b: 260s idle - over the fixed bound, on a fleet with no settlement history.
D=$(setup_case default-over); prime "$D"
echo $(( $(date +%s) - 260 )) > "$D/state/.stale-since-$KEY"
run_prod "$D" "$D/over.out"; PID=$!
wait_exit "$PID" 100 || { reap "$PID"; echo "WATCHER DID NOT EXIT"; }
echo "idle ~260s: $(cat "$D/over.out")"
echo "--- shadow score with no settlement history yet ---"
grep -o '{.*}' "$D/state/.watch-triage.log" | tail -1
echo "--- settlement log after that first escalation ---"
cat "$D/state/.wedge-settlements.jsonl"

banner "DONE"

Evidence: Mutation check proving the new subtests fail on a scorer regression

\### Mutation check: do the new tests actually fail when the scorer regresses?
(round-1 review flagged two subtests whose python assertions were unreachable;
 each mutation below is reverted with 'git checkout --' immediately after.)

--- mutation: min-obs-ignored ---
not ok - insufficient-data assertions failed for {"lane":"demo","n":3,"n_total":3,"base_rate":0.2,"z":2.827,"p_tail":0.0023,"score":1.9854,"graded_flag":true,"fixed_flag":false,"reason":"ok"}
not ok - insufficient-data assertions failed for {"lane":"demo","n":3,"n_total":3,"base_rate":0.2,"z":2.827,"p_tail":0.0023,"score":1.9854,"graded_flag":true,"fixed_flag":false,"reason":"ok"}
RESULT: suite FAILED as it should (mutation caught)
--- mutation: graded-flag-inverted ---
not ok - graded/fixed assertions failed for high={"lane":"demo","n":10,"n_total":11,"base_rate":0.1538,"z":7.5465,"p_tail":0.0,"score":6.5264,"graded_flag":false,"fixed_flag":true,"reason":"ok"} low={"lane":"demo","n":10,"n_total":11,"base_rate":0.1538,"z":-0.7693,"p_tail":0.7792,"score":-1.7894,"graded_flag":true,"fixed_flag":false,"reason":"ok"}
not ok - graded/fixed assertions failed for high={"lane":"demo","n":10,"n_total":11,"base_rate":0.1538,"z":7.5465,"p_tail":0.0,"score":6.5264,"graded_flag":false,"fixed_flag":true,"reason":"ok"} low={"lane":"demo","n":10,"n_total":11,"base_rate":0.1538,"z":-0.7693,"p_tail":0.7792,"score":-1.7894,"graded_flag":true,"fixed_flag":false,"reason":"ok"}
RESULT: suite FAILED as it should (mutation caught)
--- mutation: fixed-threshold-hardcoded ---
not ok - fixed_flag did not follow the supplied threshold: {"lane":"unknown","n":0,"score":null,"graded_flag":null,"fixed_flag":false,"reason":"insufficient data: 0<8"}
not ok - fixed_flag did not follow the supplied threshold: {"lane":"unknown","n":0,"score":null,"graded_flag":null,"fixed_flag":false,"reason":"insufficient data: 0<8"}
RESULT: suite FAILED as it should (mutation caught)
--- mutation: incident-dedup-removed ---
not ok - incident counting failed for repeated={"lane":"demo","n":8,"n_total":14,"base_rate":0.4375,"z":6.6581,"p_tail":0.0,"score":6.5007,"graded_flag":true,"fixed_flag":true,"reason":"ok"} distinct={"lane":"demo","n":8,"n_total":19,"base_rate":0.5714,"z":6.6581,"p_tail":0.0,"score":6.8381,"graded_flag":true,"fixed_flag":true,"reason":"ok"}
not ok - incident counting failed for repeated={"lane":"demo","n":8,"n_total":14,"base_rate":0.4375,"z":6.6581,"p_tail":0.0,"score":6.5007,"graded_flag":true,"fixed_flag":true,"reason":"ok"} distinct={"lane":"demo","n":8,"n_total":19,"base_rate":0.5714,"z":6.6581,"p_tail":0.0,"score":6.8381,"graded_flag":true,"fixed_flag":true,"reason":"ok"}
RESULT: suite FAILED as it should (mutation caught)

\### Baseline restored:
bin/fm-wedge-score.py clean
ok - settle appends a row with the project lane
ok - score reports insufficient data below the minimum observations
ok - score separates graded and fixed threshold flags
ok - fixed_flag follows the caller's escalation threshold
ok - repeated escalations of one window count as a single incident
ok - settle uses unknown lane when task metadata is absent
ok - the settlement log is trimmed once it passes its size cap

Evidence: Lifted ARGYLE gates: provenance header and numeric validation of the two live functions

\### Provenance header of the lifted file (bin/argyle_gates.py)
# Provenance: lifted verbatim from cipherholdingsllc/nakatomi
# branch t2-argyle-frontier, path crypto-crawler/crypto_crawler/argyle/gates.py
# (commit 02be13ca139e847089d3ce7c6503d9f5dae914ba).
# Pure stdlib. Do not edit here; upstream fixes land in nakatomi first.

\### The two functions FirstMate actually calls, checked against known normal-distribution values
norm_ppf(0.975 ) =  1.959963986   reference  1.959963985   |err| 1.1e-09
norm_ppf(0.5   ) =  0.000000000   reference  0.000000000   |err| 0.0e+00
norm_ppf(0.99  ) =  2.326347874   reference  2.326347874   |err| 3.9e-10
norm_ppf(0.01  ) = -2.326347874   reference -2.326347874   |err| 3.9e-10
norm_ppf(0.02  ) = -2.053748909   reference -2.053748911   |err| 2.0e-09
norm_cdf(1.959963985 ) =  0.975000000   reference  0.975000000   |err| 2.7e-11
norm_cdf(0.0         ) =  0.500000000   reference  0.500000000   |err| 0.0e+00
norm_cdf(-1.0        ) =  0.158655254   reference  0.158655254   |err| 6.9e-11
norm_ppf(0.0) -> ValueError: p must be in (0,1), got 0.0

\### The upstream repo (cipherholdingsllc/nakatomi) is not present on this host, so a
\### byte-for-byte diff against the recorded commit could not be performed here.

Evidence: Shadow-only proof: graded comparator disagrees, fixed timer escalates regardless

=== SCENARIO 3 graded score DISAGREES (graded_flag=false) - fixed 240s timer still escalates ===
--- watcher stdout: escalated anyway (fixed 240s behavior preserved) ---
stale: test:fm-quiet (idle 501s, possible wedge, escalation 1)
--- shadow score for that same event ---
{
"lane": "demo",
"n": 8,
"n_total": 9,
"base_rate": 0.1818,
"z": -0.104,
"p_tail": 0.5414,
"score": -1.0125,
"graded_flag": false,
"fixed_flag": true,
"reason": "ok"
}
VERDICT: graded_flag=False fixed_flag=True -> the graded comparator would NOT have escalated; the fixed timer did.

Evidence: Escalation surface: wake, triage-log score line, settlement row, queued wake

--- watcher stdout: the actionable wake an operator sees ---
stale: test:fm-quiet (idle 500s, possible wedge, escalation 1)
--- state/.watch-triage.log ---
[2026-09-05T08:56:05-0700] absorbed non-terminal stale (provably working): test:fm-quiet
[2026-09-05T08:56:10-0700] shadow wedge score: {"lane":"demo","n":8,"n_total":9,"base_rate":0.1818,"z":9.2421,"p_tail":0.0,"score":8.3336,"graded_flag":true,"fixed_flag":true,"reason":"ok"}
--- new settlement row (escalated) ---
{"ts":"2026-09-05T15:56:10.667620Z","window":"test:fm-quiet","task":"quiet","lane":"demo","idle_secs":500,"outcome":"escalated"}
--- queued wake (fm-wake-drain.sh) ---
1788623770 1 stale test:fm-quiet stale: test:fm-quiet (idle 500s, possible wedge, escalation 1)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 6 issues (3 warnings, 3 infos)

• ⚠️ bin/fm-wedge-score.py:25 - fm-wedge-score.py resolves state as os.path.join(home, "state") (line 25) and data as os.path.join(home, "data") (lines 39, 63), ignoring FM_STATE_OVERRIDE/FM_DATA_OVERRIDE, which every other bin/ script honors via ${FM_STATE_OVERRIDE:-$FM_HOME/state} / ${FM_DATA_OVERRIDE:-$FM_HOME/data}. Concrete reachable path: tests/fm-watch-triage.test.sh:508 starts the real fm-watch.sh with FM_STATE_OVERRIDE="$state" and FM_HOME unset, and forces an escalation (stale-since backdated 500s). fm-watch.sh:63 then resolves FM_HOME to the repo root (the live firstmate home), so wedge_shadow_settle appends a synthetic escalated row for a fixture window into the LIVE $FM_HOME/data/wedge-settlements.jsonl and looks the lane up in the live $FM_HOME/state, outside the test's temp sandbox. Simultaneously, wedge_shadow_score (fm-watch.sh:286) reads the overridden $STATE for the lane, so settle records lane=unknown while score queries the real lane and the shadow score degrades to permanent "insufficient data" under any override. Remedy: honor FM_STATE_OVERRIDE/FM_DATA_OVERRIDE in _lane_for_task/_settle/_read_rows (or pass the resolved dirs from the shell).
• ⚠️ bin/fm-watch.sh:291 - wedge_shadow_score invokes the scorer without --fixed-threshold, so fm-wedge-score.py:142 falls back to a hardcoded 240 while the shell's real escalation bound is STALE_ESCALATE_SECS=${FM_STALE_ESCALATE_SECS:-240} (fm-watch.sh:140). The score is only ever called from the escalation branch, i.e. exactly when the fixed timer just fired, so fixed_flag must be true there by definition. With FM_STALE_ESCALATE_SECS=120 (a documented knob, docs/configuration.md:572), escalation happens at age=120, fm-wedge-score.py:89 computes 120 >= 240 -> false, and the triage log records "fixed_flag":false for the very poll that escalated the pane - a wrong label emitted without any error. Remedy: pass --fixed-threshold "$STALE_ESCALATE_SECS" from fm-watch.sh:291 (or drop fixed_flag, which is constant-true at the only call site).
• ⚠️ tests/fm-wedge-score.test.sh:48 - test_score_requires_minimum_observations (line 48) and test_score_flags_graded_and_fixed_thresholds (line 74) run their assertions inside python3 - "$out" <<'PY' heredocs whose exit status is discarded: the test file sets only set -u, tests/lib.sh does not set -e, and bin/fm-test-run.sh:1532 judges a script solely by its exit code. So if the scoring logic regresses - e.g. graded_flag comes back True for idle 50, or score returns a number instead of None below min-obs - Python raises AssertionError to stderr and the very next statement still calls pass, the script exits 0, and the suite reports ok. Both subtests are currently incapable of failing. The sibling tests at lines 22-32 and 88-99 use the correct pattern (capture output, then [ "$out" = ok ] || fail). Remedy: append || fail "..." to those two heredoc invocations, or switch them to the capture-and-compare pattern used by the other two.
• ⚠️ bin/fm-wedge-score.py:81 - The healthy sample (lines 81-85) is right-censored by the same timer it is meant to shadow-grade, and the escalation rows are per-poll rather than per-incident. wedge_timer_check deletes the stale-since file at escalation (fm-watch.sh:336), so a "resumed" row can only ever record an idle up to STALE_ESCALATE_SECS + POLL (~255s at the defaults), while _score is only ever invoked from the escalation branch with idle >= 240. mu/sigma are therefore fit to a distribution truncated exactly at the boundary being evaluated, and the graded score can never observe the healthy long tail (a pane legitimately idle for 10 minutes) - the precise failure mode of the fixed 240s timer that a shadow experiment exists to detect. Separately, a single wedge that persists an hour re-escalates every 240s across watcher restarts and contributes ~15 "escalated" rows, so base_rate = (n_esc+1)/(n_total+2) (line 98) measures escalation polls, not distinct incidents, and drifts upward with one bad pane, lowering the graded bar via norm_ppf(1-base_rate). Numerically both directions are reachable: 8 resumes spread 30..255s with 3 escalations scores +0.54 (graded_flag true) at idle=240, while 8 resumes clustered at 240..255 score -1.36 (graded_flag false) for the same input. The remedy - recording per-incident settlements and/or a censor-aware fit - adds new state and schema beyond this change's plumbing-only intent, so it needs your authorization rather than an in-place patch; the alternative is to accept and document that the shadow number is a biased comparator.
• ⚠️ bin/fm-watch.sh:288 - Simplification: wedge_shadow_score re-derives the lane in shell (grep '^project=' "$meta" | cut -d= -f2-, line 288), a second definition of the rule that fm-wedge-score.py:_lane_for_task already owns, even though the header of that script declares it the single owner of the settlement schema. Nothing in the intent requires two implementations, and they already disagree in two ways: the shell reads $STATE (override-aware) while Python reads $home/state (not), and grep emits every matching line while Python returns the first, so a meta file with two project= lines yields a multi-line --lane argument that matches no stored row and forces permanent "insufficient data". Remedy: delete the shell-side derivation and let the scorer resolve the lane from --task, so the rule exists once.
• ⚠️ bin/fm-wedge-score.py:131 - Simplification: the required --key option (line 131) and the key field it writes (line 43) are a pure function of --window - wedge_shadow_settle computes it as printf '%s' "$win" | tr ':/.' '___' (fm-watch.sh:276), a third copy of the key derivation already inlined twice in fm-watch.sh. No stated requirement needs a settlement row to carry both the window and its own slug, and the scorer never reads the key field. Remedy: drop --key and the key field and derive the slug from window at read time if it is ever needed.
• ⚠️ bin/fm-wedge-score.py:126 - Simplification: the top-level --home (line 126) is a second definition of the same option the subparsers already declare (lines 129, 139), and it is silently discarded rather than merged. CPython's _SubParsersAction copies the subparser's namespace over the parent's, so fm-wedge-score.py --home /x settle ... yields home=None and _home() falls back to $FM_HOME or the CWD - I verified this on python3.12: --home /x settle -> Namespace(home=None), settle --home /x -> Namespace(home='/x'). The pre-subcommand form is accepted, produces no error, and writes the settlement to the wrong home. The intent needs one --home spelling, and the current callers all pass it after the subcommand, so the narrower form is enough. Remedy: delete the top-level --home.
• ℹ️ bin/fm-wedge-score.py:50 - wedge-settlements.jsonl is appended to on every resume and every escalation (line 50) with no rotation or retention bound, and _read_rows (line 63) re-parses the whole file on each escalation. This is the one new watcher-owned log that is unbounded: its siblings in the same path are bounded (triage_log trims at TRIAGE_LOG_MAX_BYTES, fm-push-transition-lib.sh:60 trims the delivery log). Growth is slow (~150 bytes/row) so this is not urgent, but it never self-heals. Flagging as ask-user because the smallest honest remedy adds retention/rotation machinery - new durable behavior beyond the change's plumbing-only intent - not because the defect is subtle.
• ℹ️ bin/fm-watch.sh:274 - FM_WEDGE_SHADOW (default 1) is a new operator-facing watcher knob with no entry in docs/configuration.md, where its immediate siblings FM_STALE_ESCALATE_SECS (line 572) and FM_WEDGE_DEMAND_INSPECT_COUNT (line 575) are both documented. Nothing enforces this, but an operator who needs to turn the shadow recording off in the field has no documented way to discover the switch. Remedy: add one line to the watcher block of docs/configuration.md.

🔧 Fix: fix wedge shadow state resolution, thresholds, and tests
6 issues (3 warnings, 3 infos) still open:

• ⚠️ bin/fm-watch.sh:398 - clear_pause_tracking is a tracking-reset helper, not a resume site, so hanging wedge_shadow_resumed off it labels a non-resume as "resumed". Concrete reachable sequence: window w has .paused-KEY from an earlier handle_paused_stale (which removed .stale-since-KEY); on a later poll the pane is busy (busy_now=0), its hash is unchanged, n>=2, and busy_turn_over_age is true. Line 1099-1100 then calls wedge_timer_check, which finds no since-file and writes date +%s > .stale-since-KEY. Line 1105 immediately sees [ -e "$pf" ] && [ "$n" -ge 2 ] and calls clear_pause_tracking, which at line 398 runs wedge_shadow_resumed against the since-file that was created microseconds earlier -> idle = 0 -> a row {"outcome":"resumed","idle_secs":0} is appended for a pane that is busy and has completed no turn in over BUSY_TURN_MAX_SECS. In _score that row becomes math.log(max(0,1.0)) = 0.0, an extreme low outlier against a healthy sample whose logs sit near 3.7-4.6, dragging mu down and inflating sigma for the whole lane, i.e. a wrong value in the emitted score with no error. Secondary cost: the $(window_to_task "$win" "$STATE") argument is evaluated unconditionally, re-running the full $STATE/*.meta scan (two greps per meta) on every clear_pause_tracking call even when FM_WEDGE_SHADOW=0, although $task is already in hand at every call site (fm-watch.sh:967). Remedy: delete the wedge_shadow_resumed call from clear_pause_tracking. It is a no-op at its other three call sites - lines 1102 and 1115 already record the resume and remove the since-file before lines 1105/1125 run, and the working) branch at line 1069 is only reached after the hash changed, which already cleared the since-file at line 1115 - so nothing genuine is lost.
• ⚠️ bin/fm-wedge-score.py:98 - _escalation_incidents assumes every escalated window is eventually discharged by a resumed row, but the watcher usually never writes that row, so the dedup permanently swallows every repeat incident for a window. At fm-watch.sh:332 the escalation deletes the since-file and then wake() exits the process. When firstmate acts on that wake and the crew resumes, the next watcher run takes the h != prev branch and calls wedge_shadow_resumed (fm-watch.sh:1115) with a since-file that no longer exists, so it returns early and no resumed row is written. unresolved therefore keeps that window forever and every later escalation of the same window hits if window not in unresolved and is not counted. handle_paused_stale (fm-watch.sh:369) has the same shape: it rm's .stale-since-KEY with no settlement row. Concrete wrong output: a lane with 8 resumed rows where window s:w wedges, escalates, is fixed, and wedges again a day later yields n_esc=1 -> base_rate=(1+1)/(9+2)=0.1818 -> norm_ppf(1-0.1818)=0.908, where the true two incidents give base_rate=(2+1)/(10+2)=0.25 -> norm_ppf(0.75)=0.674; any z in (0.674, 0.908) is reported graded_flag=false when correct counting says true. The bias is systematic and grows with fleet age. Related: line 329 settles the escalated row before line 330 scores it, so the very event being graded is already inside its own base rate. The smallest honest remedies both extend the change - either durably tracking which windows are unresolved so a resume can be recorded after the since-file is gone, or adding a time-gap heuristic to separate incidents - so this needs your authorization rather than an in-place patch; the alternative is to drop _escalation_incidents and accept per-poll escalation counting with the bias documented alongside the censoring note already in the header.
• ⚠️ tests/fm-watch-triage.test.sh:510 - All seven new tests drive bin/fm-wedge-score.py directly; nothing exercises the fm-watch.sh -> scorer seam, which is exactly where both fixed defects lived (the settlement log being written under the live FM_HOME instead of FM_STATE_OVERRIDE, and --fixed-threshold not being passed). The reported state-override failure has no reproduction: no test asserts that an escalation writes into the sandbox state dir. This is directly testable in the existing fixture - test_terminal_stale_overridden_by_run_step Phase B (line 505-512) already backdates .stale-since-KEY by 500s, runs the real watcher with FM_STATE_OVERRIDE="$state" and FM_STALE_ESCALATE_SECS=240, and waits for the escalation - so one added assertion that "$state/.wedge-settlements.jsonl" gained a row with outcome=escalated would fail before the fix and pass after it. Without it, a regression that silently drops --state or --fixed-threshold degrades to an empty score and no triage line, with every test still green.
• ℹ️ AGENTS.md:120 - AGENTS.md section 2 is the exhaustive state/ inventory - it enumerates watcher artifacts down to individual dotfiles (.hash-* .count-* .stale-* .stale-since-* .paused-* .wedge-escalations-* ... on line 120, .watch-triage.log on its own line 121) - and the new watcher-owned state/.wedge-settlements.jsonl (plus its transient .tmp sibling from _trim) is absent. docs/configuration.md:576 documents the FM_WEDGE_SHADOW knob but the layout inventory that tells firstmate which state files are watcher internals never to touch does not list the file, so a future cleanup or a reader of the inventory has no record of it. Remedy: add one line beside .watch-triage.log describing it as the watcher's size-capped wedge settlement sample, safe to delete (forces the score back to "insufficient data"). CLAUDE.md is a symlink to AGENTS.md, so one edit covers both.
• ℹ️ bin/fm-wedge-score.py:25 - The fix round relocated the settlement log from $FM_HOME/data/wedge-settlements.jsonl to $STATE/.wedge-settlements.jsonl. The original finding's remedy was only to honor the state/data overrides ("or pass the resolved dirs from the shell"); the directory change is an additional design call. AGENTS.md section 2 and docs/configuration.md:13-14 - the single owner of the home layout - split these deliberately: data/ holds durable private fleet records, state/ holds volatile runtime records, and AGENTS.md:121 tells firstmate the neighboring watcher log is "safe to delete". This log is not volatile in that sense: it is a multi-session accumulating sample that must reach min_obs=8 resumed rows per lane before the score reports anything, so clearing state/ silently resets every lane to "insufficient data". The header comment at lines 4-7 asserts the state/ placement as the rationale, but the layout owner was not updated to match. Either move it back under a FM_DATA_OVERRIDE-aware data/ path, or confirm the sample is intentionally disposable and say so where the layout is defined.
• ℹ️ bin/argyle_gates.py:88 - Informational, no action expected: only norm_cdf and norm_ppf (lines 43-71) are reachable from FirstMate. sharpe, _moments, GateResult, deflated_sharpe, block_permutation, oos_calibration, promotion_gate, EULER_GAMMA and the random/dataclasses imports - about 170 of 241 lines - are dead here. The User intent explicitly requires the lift be "verbatim with provenance", and the header (lines 1-4) records the upstream repo, branch, path and commit and directs fixes upstream first, so this is the required form, not a component to trim. Noting it so it is not re-flagged: the dead surface is intent-mandated. I confirmed the two live functions are safe at every call site - base_rate = (n_esc+1)/(n_total+2) is strictly inside (0,1), so norm_ppf's ValueError guard at line 49-50 is unreachable from _score.

✅ **Test** - passed

✅ No issues found.

• bash tests/fm-wedge-score.test.sh — 7 subtests covering settle rows/lane resolution, min-obs gating, graded vs fixed flags, caller-supplied threshold, escalation-incident dedup, and log trimming
• bin/fm-test-run.sh tests/fm-wedge-score.test.sh — confirms the new file is routed by the runner as family=pure-contract-unit
• bin/fm-test-run.sh --check-coverage — coverage guard satisfied (total=137) with the new script classified
• bash tests/fm-watch-triage.test.sh — 40-test watcher regression suite, including every wedge-timer absorb/escalate path the change touches
• Manual E2E: bash ~/.no-mistakes/evidence/01M1S2H988FP2CT6NK6N0T3FPP/e2e-wedge-shadow.sh <repo> <evidence-dir> — drives real bin/fm-watch.sh over 5 scenarios (resume settlement, escalation + shadow score + queued wake, graded/fixed disagreement, FM_WEDGE_SHADOW=0, shipped 240s default on a fresh fleet)
• Mutation check: 4 targeted mutations of bin/fm-wedge-score.py (min-obs bypass, inverted graded_flag, hardcoded 240s threshold, removed incident dedup), each re-running tests/fm-wedge-score.test.sh; every mutation was caught and the source restored via git checkout --
• Numeric validation of the lifted bin/argyle_gates.py primitives norm_ppf/norm_cdf against known normal-distribution reference values and the p∈(0,1) domain guard

🔧 **Document** - 1 issue found → auto-fixed ✅

• ℹ️ docs/architecture.md:63 - Two surfaces state that state/.watch-triage.log is exclusively the absorbed-wake debug log (docs/architecture.md:63 "state/.watch-triage.log remains exclusively the absorbed-wake debug log"; docs/watcher-continuity.md:56 "remains only the watcher's bounded absorbed-wake debug log and carries no lifecycle semantics"). This change adds a non-absorb line to that log: wedge_shadow_score writes "shadow wedge score: ..." via triage_log at escalation time (bin/fm-watch.sh:288), i.e. on the surfaced path, not an absorbed one. I did not edit either sentence because the drift is pre-existing rather than introduced here - bin/fm-watch.sh:885,887 already wrote non-absorb "merged PR poll retirement ..." lines before this branch - and because the load-bearing part of both claims (it is a bounded debug log with no lifecycle semantics, contrasted against the state/.watch-cycle-exits.log ledger; AGENTS.md:121 "never relied on, safe to delete") is still true. Fixing it properly means picking one owner for the triage-log role and reducing the other two mentions (architecture.md, watcher-continuity.md, AGENTS.md:121) to pointers, which is a three-surface consolidation beyond this change's scope. Proposed follow-up: reword the owner to "the watcher's bounded debug log, never a lifecycle ledger" and point the duplicates at it.

🔧 Fix: clarify triage log role for non-absorb watcher lines
✅ Re-checked - no issues remain.

⚠️ **Lint** - 1 warning

• ⚠️ linter found issues (exit code 127)

🔧 Fix: no lint fixes needed; ShellCheck absence was environmental
1 warning still open:

• ⚠️ linter found issues (exit code 127)

✅ **Push** - passed

✅ No issues found.

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[devin-ai-integration]

What landed - the watcher records wedge settlements (resumed / escalated with idle seconds, per lane) and logs a graded per-lane score to the triage log beside the fixed 240s timer. No escalation decision reads the score. gates.py is a verbatim lift from the nakatomi fork with a provenance header.

Residual risk - the resumed label is inferred from the three .stale-since-* resets that mean "pane progressed"; if one of those paths also fires for a non-progress reset, healthy idle durations get polluted and the base rate skews low. Worth a second pair of eyes on the two main-loop $ssf sites. Also the graded formula here is first-principles (the smelt report's exact probe spec was not available in this checkout).

Verified - bin/fm-lint.sh clean on ShellCheck 0.11.0; fm-test-run.sh tests/fm-wedge-score.test.sh and tests/fm-watch-triage.test.sh pass; --check-coverage ok; diff of gates.py body against origin/t2-argyle-frontier is empty. Not verified on a live fleet.

[devin-ai-integration]

Devin Review found 4 potential issues.

[devin-ai-integration]

🟡 Pause cleanup records false resumptions

When an over-age busy pane leaves pause tracking, clear_pause_tracking records a zero-second resumption despite continued inactivity. wedge_timer_check creates the timer immediately before this cleanup. The false healthy sample distorts every later lane score.

Learn more

Pause tracking can coexist with a busy pane whose completed-turn age exceeds BUSY_TURN_MAX_SECS. In that case, the busy-pane branch calls wedge_timer_check, which creates a missing timer, then immediately calls clear_pause_tracking. The new call records that freshly created timer as a resume even though no turn completed and the pane remains busy. The recorded idle time is usually zero seconds. The scorer clamps it to one second before taking its logarithm, making it an extreme low healthy observation for the lane.

Example: A paused pane becomes busy but completes no turn for over an hour. Its first qualifying poll creates .stale-since-*; pause cleanup immediately appends {"idle_secs":0,"outcome":"resumed"} although the pane is still inactive.

Recommended fix: Remove settlement recording from clear_pause_tracking. Record resumed settlements only in branches that observe a genuine pane/hash recovery while an existing wedge timer remains active.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

🟡 Later wedge incidents disappear

After an escalation resolves before monitoring restarts, _escalation_incidents suppresses every later wedge on that window. The escalation path deletes its timer before a resume can be recorded. Incident rates and graded flags remain understated.

Learn more

The scorer considers an incident unresolved until it reads a resumed row for the same window. However, the escalation path deletes the only timer used to generate that row and exits monitoring. If the pane recovers before the next monitoring cycle, wedge_shadow_resumed sees no timer and records nothing. The window therefore remains forever in _escalation_incidents' in-memory unresolved set whenever history is replayed. Every genuinely separate later escalation for that window is collapsed into the first one.

Example: Window session:fm-build escalates Monday, recovers before monitoring restarts, and wedges again Tuesday. The log contains two escalated rows but no intervening resumed row, so the scorer reports one incident instead of two.

Recommended fix: Persist incident resolution independently of the deleted timer, or record a recovery marker whenever a previously escalated window is observed active. Ensure repeated polls for one continuous wedge still collapse while later wedge episodes count separately.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

🔍 Watcher-to-scorer seam lacks coverage

Captain, direct scorer tests cannot catch broken watcher arguments or settlement timing. Add one real monitoring-cycle test covering escalation and recovery rows.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

🔍 Settlement state lacks an owner

The state inventory omits .wedge-settlements.jsonl and its trim file. Document whether this cross-session sample is disposable or belongs with durable fleet data.

---

Was this helpful? React with 👍 or 👎 to provide feedback.