DRU-380 - Add sandbox hold

[druks-operator-treadstone]

Linear ticket: DRU-380

Plan

Add sandbox hold (opt-in warm-VM retention across a park)

Goal. Let a warm-VM run parking on a gate keep its host (clipping the lease) instead of reaping it, so a resume reattaches warm. review() and terminal exit still reap. Default preserves today's behavior.

What the repo already gives us

• _park (backend/druks/workflows.py:333) unconditionally await workflow._reap_run() before suspending on DBOS.recv_async. _reap_run (:859) deletes the VM and nulls self._host.
• Gate.wait (:294) calls _park; review() (:815) calls _park directly (so the flag must live on _park).
• Warm reattach across steps is already keyed by the idempotency key f"{workflow_id}:sandbox" in _lease_host (:836), which rotates when remaining lease < SANDBOX_HOST_ROTATE_BEFORE_SECONDS (75 min; constants at sandbox/constants.py:6-8).
• The terminal finally: await instance._reap_run() (workflows.py:1084) already reaps on every exit.
• Clipping needs Client.set_expiry(host_id, expires_at) from DRU-379 (PR DRU-379 - Druks sandbox client: set_expiry / release unchanged #409, still In Review — not present in this checkout). See scope note below.

Change (one module + tests)

1. Gate.wait — add hold_sandbox: bool | timedelta | None = False; forward it to _park.
2. _park — add hold_sandbox param (default False). Replace the unconditional _reap_run() with a hold-aware branch: falsy → _reap_run() (today); truthy and self._host set → clip the lease via a new _hold_host(hold_sandbox) that calls sandbox_client.set_expiry(...) and retains self._host. No warm host held → nothing to hold (no set_expiry, no delete).
3. _hold_host (mirror _reap_run) computes the clipped expiry: True → min(current expiry, now + SANDBOX_HOST_ROTATE_BEFORE_SECONDS) (one worst-case call still fits); timedelta → min(current expiry, now + hold).
4. review() — unchanged; passes no hold, so it still reaps.
5. Resume — no code change: same-instance resume reuses the retained self._host; a restarted worker re-leases via the existing {workflow_id}:sandbox idempotency key (warm if the clipped host survives, cold if Drukbox reaped it).
6. Tests — extend backend/tests/test_warm_host_rotation.py's fake (_FakeSandboxClient + set_expiry/deleted tracking) to cover the hold decision and the follow-on _lease_host.

Scope / risk

• Depends on DRU-379's Client.set_expiry. This PR does not re-add it (that would collide with PR DRU-379 - Druks sandbox client: set_expiry / release unchanged #409). Production wiring calls sandbox_client.set_expiry; tests patch a fake client that exposes it. Assumes DRU-379 lands first.
• Out of scope: no durable_runs host-id column (reattach stays the idempotency key), no new reconciler/janitor, no Drukbox cap. No app currently passes hold_sandbox=True; this PR only adds the capability + the set_expiry caller path.

Acceptance Criteria

• AC1: Gate.wait accepts a keyword parameter hold_sandbox: bool | timedelta | None = False and forwards it to _park; _park accepts a hold_sandbox parameter defaulting to False.
  • Verification: Read the signatures of Gate.wait and _park in backend/druks/workflows.py and the call from wait to _park.
• AC2: When hold_sandbox is falsy (False/None), a park releases (deletes) any warm host exactly as today — _park reaps via _reap_run/sandbox_client.release and does not call set_expiry.
  • Verification: Unit test: park with no/False hold on a workflow holding a warm host records a delete/release and no set_expiry call.
• AC3: When hold_sandbox is truthy and a warm host is held, _park calls sandbox_client.set_expiry for that host and never deletes it, and retains workflow._host (not nulled) across the suspend.
  • Verification: Unit test asserts set_expiry called once, delete/release not called, and workflow._host still set after the hold path runs.
• AC4: The clipped expiry passed to set_expiry is: for hold_sandbox=True, no later than now + SANDBOX_HOST_ROTATE_BEFORE_SECONDS and no later than the host's current expiry; for a timedelta, min(current expiry, now + hold).
  • Verification: Unit test asserts the expires_at argument to the fake set_expiry matches the clip for both True and a timedelta input.
• AC5: review() continues to park without a hold and therefore still reaps the warm host (no set_expiry); parking with a hold when no warm host is held (e.g. steps_reuse_sandbox=False or _host is None) issues neither a delete nor a set_expiry.
  • Verification: Read review()'s _park call (no hold arg) and a unit test that a hold park with _host is None records no set_expiry and no delete.
• AC6: Resume uses the existing lease path unchanged: after a hold park that retained _host, a same-instance _lease_host returns the same host id without provisioning; with _host cleared and the clipped lease lapsed, _lease_host provisions exactly once under the {workflow_id}:sandbox key (no second host on a pre-expiry resume).
  • Verification: Unit tests over _lease_host mirroring test_warm_host_rotation.py: retained-host reuse, and re-provision-once after clip/expiry.

[druks-operator-treadstone]

Verdict: approve

Verification

All six acceptance criteria pass against 6b350ce591652c72b91a7254bd40985508bcca0c...3163b711ef5c2c5dcf5ad8390d7657ca0216306e, round 1 (implementation revision 1).

• AC1 — pass. Gate.wait (workflows.py:294-301) takes hold_sandbox: bool | timedelta | None = False, forwards it to _park (:341-343); _park (:342-347) accepts the same param, default False.
• AC2 — pass. _park's falsy branch calls workflow._reap_run() (:352-355). Verified by test_park_without_hold_releases_the_warm_host.
• AC3 — pass. Truthy hold_sandbox + a warm host routes to _hold_host: calls set_expiry, never release, retains self._host. Verified by test_park_with_hold_clips_the_lease_and_keeps_the_host.
• AC4 — pass. _hold_host (:879-896) clips via min(current expiry, now+span) for both True (rotate threshold) and a timedelta. Verified by test_hold_true_clips_to_one_more_worst_case_call, test_hold_never_outlasts_the_lease_drukbox_granted, test_hold_timedelta_clips_to_the_requested_span.
• AC5 — pass. review() (:829) still calls _park with no hold arg, so it reaps. No-warm-host hold case covered by test_hold_without_a_warm_host_touches_nothing.
• AC6 — pass. No change to _lease_host's idempotency-keyed re-lease path. Same-instance reuse and restarted-worker re-provision each covered by a dedicated test.

Checks: uv run ruff check backend — pass (clean locally; GitHub backend check green for head_sha). uv run pytest backend/ — pass (backend check green for head_sha; the 11 new/existing tests in test_warm_host_rotation.py pass locally). Frontend build/test — not_run, diff is backend-only.

Open findings (non-blocking)

1. [High, informational] sandbox_client.set_expiry doesn't yet exist on the real Client (backend/druks/sandbox/client.py) — only the test fake defines it. This is the explicitly plan-scoped DRU-379 (#409) dependency: this PR intentionally doesn't re-add set_expiry to avoid a merge collision, and no caller currently passes hold_sandbox=True, so nothing exercises this path in production yet. But the hold capability this PR ships won't actually work until #409 lands and wires the real method.
2. [High, informational] _hold_host clips the lease server-side but never updates the in-memory self._host.expires_at (read-only, derived from self.record), so _lease_host's rotation check can compare against a stale, pre-clip expiry after a hold — a same-worker resume in the gap between the real clip and the stale threshold could try to reuse an already-reaped host. Not exercised by the new tests (none advance simulated time between hold and the follow-up lease check).

Both items are inline on the diff and filed together as DRU-413, a child of DRU-380, for follow-up once DRU-379 lands.

Code review

An independent clean-room lens (no access to the plan/ticket) reviewed the diff for maintainability. It found the same set_expiry gap above as its one high finding, and confirmed: no reuse/duplication issues, idiomatic fit with the neighboring _reap_run/_lease_host methods, tests targeting behavior rather than implementation, no dead branches, no secret/log leakage, and no out-of-scope edits — both changed files are squarely scoped to the sandbox-hold feature. It noted one low-severity nit (the bool | timedelta | None signature never distinguishes None from False) but didn't treat it as a separate finding.

[druks-operator-treadstone]

Code-review lens (advisory, non-blocking): sandbox_client.set_expiry doesn't exist on the real Client in backend/druks/sandbox/client.py — only the test's _FakeSandboxClient defines it. This is the known, plan-scoped DRU-379 (#409) dependency, so it's not a defect in this diff, but it does mean the hold path will AttributeError in production until #409 lands and wires the real method. Filed as follow-up DRU-413.

[druks-operator-treadstone]

Verification lens open finding (non-blocking — ACs pass as literally specified and tested): _hold_host clips the lease via set_expiry but never updates self._host's local expires_at (it's a read-only property derived from self.record). _lease_host's rotate check then compares against the stale, pre-clip expiry, so a same-worker resume in the window between the real clipped expiry and the stale threshold could reuse a host_id the provider already reaped, surfacing as HostGone rather than a clean cold re-provision. Not exercised by the new tests since none let simulated time pass between the hold and the follow-up _lease_host call. Worth addressing alongside the set_expiry wiring — folded into DRU-413.