F1d: startup recovery and the single-runner lock

[mchwang]

Lane F, step F1, slice F1d: startup recovery and the single-runner lock. Stacked on #57 (F1c) → #56 (F1b) → #53 (F1a). Related: #22, #51.

What this does

Single-runner lock (runner/recovery.ts, acquireRunnerLock; decision 1):

• Step 0. The database's canonical parent directory must be owned by the user and not writable by group or others.
• Step 1. Identify the database file with O_NOFOLLOW. If it doesn't exist, create it with O_CREAT|O_EXCL|O_NOFOLLOW (mode 0600), handling a creation race. Refuse symlinks, non-regular files, and files with more than one hard link.
• Step 2. Take an OS lock on ~/.codeboost/locks/<device>-<inode>.runner-lock. It is a small SQLite file held in locking_mode=EXCLUSIVE after a first write, and the OS releases it on exit or crash. There is no stale-lock takeover, and lock files are never deleted. A renamed database, or one moved to another directory on the same filesystem, meets the same lock; a copy gets its own.
• Step 4. verify() checks, after the Store opens, that the path still names the locked file.

The CLI now takes the lock before starting the server and releases it after shutdown. A second start prints "Another codeboost runner is using this database." and exits 1 (smoke-tested with npm run demo twice).

Startup recovery (recoverStartup), the contract's steps 2b–7, fail-closed at each step:

1. 2b. Kill and await every leftover preparation process group (matched by recorded start time) before D recovery or any storage work.
2. 2c. D recovery is injected: recoverLeftovers(runnerOwner), until D follow-ups required by the F1 runner lifecycle contract #51 delivers it. A rejection, or older-build resources without a runner label, stops startup before any row is finalized.
3. 3. Export phase, outside any transaction: only stopped writable attempts, fixed <attemptId>.diff names, a 60 s abortable deadline, and failures recorded as diagnostics. Then one finalization transaction (Store.recoverInterrupted):
   • the settlement precedence: first reason, then a changed context giving stale, then an expired budget giving time-limit, then an expired deadline giving "Timed out", then "Interrupted";
   • a D stop reason beats a shutdown reason;
   • a pending cancel task beats the time limit, and a closed task never changes;
   • the requeue claim (requeue_pending) is set for shutdown-cancelled or interrupted tasks.
4. 4. Abort recorded rebases (F3; none exist yet), then remove every matched storage handle. Storage whose attempt ID matches no row is kept and reported. Sweep <runnerRoot>/<token>/attempts: owned entries are removed; symlinks, other devices and unknown names are reported and left alone.
5. 5. Repair confirmed merges missing their closed status or task-closed event.
6. 7. An interrupted attempt that started preparation but never saved its process group blocks startup. releasePreparation removes its directory only if no process has files open or a working directory there (lsof +D on macOS, /proc on Linux), failing closed if that check can't run.

Store additions:

• runnerOwnerToken(file): a 32-hex token tied to the database file's device and inode. A copy gets a new token; a malformed stored value refuses startup.
• Preparation markers (markPreparationStarting, recordPreparationGroup, cancelPreparationStart) and recordAllocation, where F chooses the allocation ID.
• admitAttempt({ claimRequeue }): the requeue claim is cleared by compare-and-swap in the admitting transaction, and a second claim is refused.

Validation (head cfba95b)

• npm run typecheck: passes.
• CI's unit set: 518 passed, 0 failed. 17 of those are new, in test/runner-recovery.test.ts:
  • the lock in one process and across processes (a child killed with SIGKILL releases it);
  • rename and cross-directory move; copy; hard links, symlinks and an unsafe parent; path swap;
  • token stability, copy and malformed values;
  • every finalization mapping and the requeue rules;
  • cancel versus time limit after a crash; the requeue claim; merged-task repair;
  • step order; D rejection and unowned resources stopping before finalization;
  • an export timeout; a finalization failure removing nothing; the unmatched-storage and directory sweep;
  • unrecorded preparation, and release refused while a process uses it.
• npm run test:browser: 53 passed. The browser tests call startServer directly, so the CLI lock was smoke-tested by hand.
• Mutation check: eight guards were broken one at a time, and each was caught:
  • no exclusive locking mode;
  • a lock keyed by name instead of inode;
  • hard links allowed;
  • preparation stopped after D recovery;
  • no requeue claim set;
  • unowned legacy resources ignored;
  • unrecorded preparation not blocking;
  • orphan storage removed.

CI fix after opening (5d163f2)

The cross-process lock test failed once on CI (surfaced on #60). The cause is real: a caller that drops the returned RunnerLock lets its SQLite connection be garbage-collected, and that silently releases the OS lock. Held connections are now kept in a module-level set until release(). The new test "keeps holding the lock when the caller drops the returned object and garbage collection runs" fails without the fix and passes with it (3 of 3 runs).

Limits and follow-ups

• Network filesystems are refused on Linux (statfs magic numbers for NFS, SMB, CIFS and FUSE). On macOS the statfs type values are not stable enough to classify, so that check is skipped there.
• Recovery is not yet called at startup. It needs D's recoverLeftovers and the storage handles from D follow-ups required by the F1 runner lifecycle contract #51. Until then there is no runner in production, so no attempt can be left running.
• Preparation markers are not yet written in practice. The markers are Store methods; F2's preparation calls them once D's abortable clone and storage helpers (D follow-ups required by the F1 runner lifecycle contract #51) exist.
• --list-unowned-agent-resources also waits for D follow-ups required by the F1 runner lifecycle contract #51. It needs D's report.

🤖 Generated with Claude Code