fix(server): bound the provisioning-deadline scan's lock wait

[EmilienM]

Summary

The provisioning-deadline scan waited with no deadline for each sandbox's local mutation keys, so a slow provider or profile write could stall lifecycle operations on an unrelated sandbox, or stall provider writes behind a busy sandbox. The scan now waits at most 500 ms per pass and picks a busy sandbox up on the next pass.

Related Issue

Closes #4547 (needs state:accepted for the issue acceptance check)
Part of #4307

Changes

• reconcile_provisioning_deadlines gives each pass a 500 ms lock-wait budget and skips a sandbox whose keys stay busy. Once the budget is spent, the rest of the pass only takes keys that are free, so a busy workspace or global key costs one wait per pass, not one per sandbox. The acquisition is cancel-safe, so a skip releases the keys it took.
• Skips log at debug and count in openshell_server_provisioning_deadline_skips_total{overdue}.
• Trade-off: a skipped sandbox gives up its place in the lock queue, so expiry can run a few passes late, against a 300-second repair window. The counter shows it.
• Doc updates: the new counter in the gateway metrics page and the debug-openshell-cluster skill, the lock table in mutation_lock.rs, and the lock_sandbox_local_in_workspace doc comment.
• Merge note: feat(kubernetes): add template-driven warm pooling #3508 moves the post-lock scan body into refresh_provisioning_deadline and conflicts textually with this PR. The lock acquisition stays in the scan loop there, so keep the per-pass budget and skip around it.

Testing

• mise run rust:lint, rust:format:check, cargo test -p openshell-server --features test-support and mise run test:rust:postgres pass.
• Four tests replace the one that pinned the old wait: one per key order, one that checks a skip doesn't end the pass, and one that checks sandboxes behind a busy workspace share one wait. All four fail on main.
• Checks appropriate to the affected code and behavior pass
• Unit tests added/updated (if applicable)
• E2E tests added/updated (if applicable)

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[mrunalp]

/ok to test 859406f

[github-actions]

Label test:e2e applied for 859406f. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[FrostGod]

if one workspace stays busy and has many sandboxes setting up, each one costs 500 ms before it's skipped. For example, 50 sandboxes make a pass take about 25 seconds. Should the scan skip the rest of that workspace after its first timeout?

[FrostGod]

with a steady stream of writers, an overdue sandbox could be skipped on every pass, and nothing bounds that. The only signal is a warning log.
and Log noise: in that same case, the warning fires once a second for each sandbox.

[EmilienM]

1. Agreed on the cost, but I'd bound the pass rather than the workspace. The timeout can come from the sandbox's own key, so one slow lifecycle op would skip its whole workspace on every pass. It can also come from the global key, where a global write costs 500 ms in every workspace. Next push gives each pass one 500 ms wait budget. Once it's spent, the rest of the pass only takes locks that are free right now, so a pass waits about 500 ms at most, whichever key is busy.

2. Fair on the noise: skips drop to debug and get a counter labeled overdue or not, so you can alert on it. I'd leave starvation unbounded, though. The lock is FIFO, so a skip needs more than 500 ms of writes already queued when the scan arrives. Starving a sandbox means that's true on every pass, i.e. the workspace lock is saturated, which mutation_lock_wait_seconds{scope="workspace"} already shows. A hard bound means waiting longer, which brings back the stall this PR removes.

Both are in 098b69c (openshell_server_provisioning_deadline_skips_total{overdue}).

[FrostGod]

/ok to test 098b69c