bug(vm): gateway restart can retain stale supervisor ownership and fail sandbox recovery #3546

User Story

As an operator running VM-backed sandboxes, I want a graceful gateway restart to restore running sandboxes and preserve explicitly stopped sandboxes, so that routine restarts do not leave workloads in Error.

Problem Statement

The gateway can exit before an asynchronous supervisor-session cleanup task deletes its persisted ownership record. After restart, the new supervisor cannot replace the still-fresh record belonging to the previous instance. ConnectSupervisor returns Unavailable; the supervisor's 10-second acceptance timeout expires before the 45-second ownership TTL, and the sandbox enters Error.

Observed in vm_gateway_restart_preserves_running_and_stopped_intent on both attempts of the VM CI job for #3386:

• First failure
• Rerun failure

The race mechanism was subsequently reproduced locally on main at 50230616d51f352954450c527f9dd64218be0b80 using a controlled delay in ownership cleanup. This establishes the failure mechanism, but does not establish the precise scheduling or database delay in CI.

Impact / Why This Matters

A graceful restart can leave a previously healthy, running-intent sandbox unavailable instead of restoring it. The VM restart E2E failed twice in CI, each time exhausting the readiness wait. Retrying CI did not resolve this occurrence, and repeated manual retries are not a reliable recovery strategy for workloads. No reliable operational workaround was validated in this investigation.

Acceptance Criteria

• A graceful gateway restart restores the running VM sandbox with its overlay state intact, while explicitly stopped sandboxes remain stopped.
• Shutdown waits for supervisor-session ownership cleanup to complete before exiting; delayed cleanup does not leave stale ownership blocking the replacement supervisor.
• Cleanup retains session, replica, and resource-version checks so it cannot delete ownership acquired by a replacement session.
• Shutdown remains bounded and reports incomplete cleanup if persistence is unavailable.
• A deterministic regression test covers shutdown while owner deletion is pending, including a session already removed from the in-memory registry.
• The focused VM restart E2E passes in the Linux x86-64 CI lane.

Reproduction Steps

Existing E2E, without instrumentation

On a machine with the VM driver's documented prerequisites (Linux KVM or supported macOS virtualization, repository build dependencies, and container/image tooling), check out 50230616d51f352954450c527f9dd64218be0b80 in a disposable checkout and run:

OPENSHELL_E2E_VM_TEST=vm_gateway_start mise run e2e:vm

The test creates a running VM sandbox, writes and flushes an overlay marker, creates and explicitly stops another sandbox, then stops and restarts the gateway. It expects the first sandbox to execute cat /sandbox/vm-gateway-start-state and return before-restart while the second remains stopped.

This failure is timing-sensitive. Three uninstrumented local runs with the failing PR's ARM64 binaries passed. A control using the gateway built from updated main also passed in 23.25 seconds.

Controlled reproduction used in the investigation

In the disposable checkout, edit crates/openshell-server/src/supervisor_session.rs. Inside the detached task in handle_connect_supervisor, immediately before owner_index.release_if_current(&owner_guard).await, insert:

// Diagnostic scheduling hook only; not a proposed production change.
if state_clone.gateway_shutting_down.load(Ordering::Acquire)
    && let Ok(delay) = std::env::var("DIAGNOSTIC_VM_OWNER_RELEASE_DELAY_MS")
    && let Ok(delay_ms) = delay.parse::<u64>()
{
    info!(sandbox_id = %sandbox_id_clone, delay_ms, "diagnostic: delaying shutdown owner release");
    tokio::time::sleep(Duration::from_millis(delay_ms)).await;
    info!(sandbox_id = %sandbox_id_clone, "diagnostic: shutdown owner release delay completed");
}

The following reproduces the tested Linux ARM64 binary combination: build the gateway from main and reuse the other executables from the failing CI run. Artifact downloads require GitHub access and are subject to retention; source builds via mise run e2e:vm are the alternative once artifacts expire.

VM_REPRO_ARTIFACTS="$(mktemp -d)"
gh run download 35723405169 --repo NVIDIA/OpenShell \
  --pattern openshell-driver-vm-aarch64-unknown-linux-gnu \
  --pattern openshell-supervisor-aarch64-unknown-linux-gnu \
  --pattern openshell-aarch64-unknown-linux-musl \
  --pattern openshell-conformance-aarch64-unknown-linux-musl \
  --dir "$VM_REPRO_ARTIFACTS"
chmod +x "$VM_REPRO_ARTIFACTS"/*/openshell*
mise exec -- cargo build -p openshell-gateway

export OPENSHELL_GATEWAY_BIN="$PWD/target/debug/openshell-gateway"
export OPENSHELL_BIN="$VM_REPRO_ARTIFACTS/openshell-aarch64-unknown-linux-musl/openshell"
export OPENSHELL_VM_DRIVER_BIN="$VM_REPRO_ARTIFACTS/openshell-driver-vm-aarch64-unknown-linux-gnu/openshell-driver-vm"
export OPENSHELL_VM_SUPERVISOR_BIN="$VM_REPRO_ARTIFACTS/openshell-supervisor-aarch64-unknown-linux-gnu/openshell-supervisor"
export OPENSHELL_CONFORMANCE_BIN="$VM_REPRO_ARTIFACTS/openshell-conformance-aarch64-unknown-linux-musl/openshell-conformance"

DIAGNOSTIC_VM_OWNER_RELEASE_DELAY_MS=2000 \
  OPENSHELL_E2E_VM_TEST=vm_gateway_start \
  mise run --no-deps --skip-deps e2e:vm

For the control, run the same command without DIAGNOSTIC_VM_OWNER_RELEASE_DELAY_MS. Remove the diagnostic hook and rebuild after investigation.

Observed: the delay-start message appeared, but the delay-completed message did not: the old gateway exited while cleanup was pending. A read-only query of the per-run SQLite objects table, filtered to object_type = 'supervisor_session_owner', confirmed that the previous session and replica still owned the sandbox after restart. New supervisor connections were rejected, and the full test failed at tests/vm_gateway_start.rs:161 after 260.04 seconds, exit code 101.

Environment

• CI: Linux x86-64, VM compute driver, run 35723405169, PR head c1bb9fdc0d9bedb30d2096880dfeabeb4b8f8a69.
• Local: Ubuntu 24.04.5 LTS, Linux aarch64, kernel 6.17.0-1018-nvidia, KVM available.
• Investigated gateway source: main 50230616d51f352954450c527f9dd64218be0b80, after refactor(runtime): retire Community image dependencies #3386 merged.
• Local CLI, VM driver, supervisor, and conformance executables: ARM64 artifacts from run 35723405169. This was not a full fresh-main runtime build or an x86-64 local reproduction.
• Gateway persistence: per-run on-disk SQLite; managed standalone gateway with mTLS.
• VM bootstrap image: nvcr.io/nvidia/base/ubuntu:24.04.
• Workload image: ghcr.io/astral-sh/uv:0.12.17-python3.12-trixie-slim@sha256:9a59bb7206905ccaae4f7dab222fbac47c125a21e5fc16f43f427cd6c940ade3.
• Latest release checked: v0.0.116; this report concerns the tested commits, and that release was not tested.

Logs

Condensed local diagnostic timeline; identifiers omitted:

12:50:35.127 diagnostic: delaying shutdown owner release (2000 ms)
12:50:35.163 VM driver shutdown begins
12:50:36.141 new gateway starts
12:50:38.190 SQLite still contains previous session/replica owner record
12:50:41–48 ConnectSupervisor returns gRPC Unavailable
12:50:51.987 sandbox transitions Provisioning -> Error
gateway did not accept supervisor session within 10 seconds
test vm_gateway_restart_preserves_running_and_stopped_intent ... FAILED
test result: FAILED; finished in 260.04s

Investigation Notes

• Gateway shutdown awaits listener and compute cleanup, then returns without joining the supervisor-session tasks.
• Session cleanup runs in a detached tokio::spawn task. It removes the in-memory session before awaiting persisted owner deletion. Merely waiting for the registry to become empty would not close this gap.
• Owner replacement rejects a different supervisor instance while the old ownership record is fresh (45-second TTL).
• Supervisor startup waits only 10 seconds for acceptance.

The fix direction is a bounded shutdown drain that tracks cleanup completion through the conditional database deletion, preserving the existing replacement-session protections. A shutdown sleep or longer startup timeout would mask the missing synchronization.

The workload image was identical between the passing control and failing diagnostic run. Image-dependent timing may expose the race, but a direct image incompatibility was not demonstrated. No production fix was applied during this investigation.