Conversation
… cancelled When a create request is cancelled or times out on the client side while the node-side create is still running, the node can finish creating the instance even though the gRPC create returns Canceled. The API classifies placement as timed out and returns the error without ever registering the sandbox, so the instance is left running on the node with no running-store record, no team index entry, and no catalog entry. The periodic reconcile reclaims it, but only after a full orphan grace period, and the create failure path itself does nothing. The compensation that removes a node instance only runs on the sandboxStore.Add failure branch, i.e. after a successful placement. The placement-failure branch never compensated. placeSandbox now remembers the node whose in-flight create was interrupted by the context being cancelled (a ResourceExhausted refusal never started a create, so it is skipped) and returns it as PlacementResult.InterruptedNode. On the placement-failure branch, CreateSandbox issues a best-effort kill of that exact (sandboxID, executionID) on that node, detached from the cancelled request context. It reuses the same node-side kill the orphan reconciler runs, just eagerly, so the leak window drops from an orphan grace period to ~0. killSandboxOnNode already treats NotFound as success, so if the node never actually completed the create the kill is a cheap no-op. Reconcile stays as the backstop. Fixes e2b-dev#3637
AdaAibaby
requested review from
ValentaTomas,
dobrac and
jakubno
as code owners
September 16, 2026 08:25
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
If a sandbox create request is cancelled or times out on the client side while the node-side create is still in progress, the node can finish creating the instance even though the gRPC create returns
Canceled. The API classifies placement as timed out and returns the error without ever registering the sandbox. The instance is left running on the node with no running-store record, no team index entry, and no catalog entry.An existing safety net reclaims it —
Store.Reconcile → killOrphanSandboxkills instances present on the node but absent from the store — but only afterorphanGracePeriod(1 minute). During that window a VM and its network slot run completely unregistered and invisible to the API, and the create failure path itself performs no compensation.Fixes #3637.
Root cause
The node-side create and the API-side registration are separate steps, and the only compensation (
removeSandboxFromNode) runs on thesandboxStore.Addfailure branch — i.e. after a successful placement. Whenplacement.PlaceSandboxreturns because the request context was cancelled,CreateSandboxreturnsplacementAPIError(err)immediately; the placement-failure branch never compensated. The gRPC create already dispatched to the node is not cancelled atomically with the API request, so the node completes it.Fix
Compensate on the placement-failure branch, targeting the exact node that leaked:
placeSandboxnow records the node whose in-flightSandboxCreatewas interrupted by the context being cancelled/timing out, and returns it asPlacementResult.InterruptedNode. AResourceExhaustedrefusal never started a create, so it is explicitly skipped — only a node that actually began a create can hold an instance.CreateSandboxissues a best-effort kill of that exact(sandboxID, executionID)onInterruptedNode, using a context detached from the cancelled request (the request context is already dead).Why this shape
firstTriedNode(which is tracked for retry-warming and can differ in a multi-node retry) or every node tried. A hard-failed node (context still live) cleaned up itself and is left alone.compensateInterruptedCreateruns the samekillSandboxOnNodethe orphan reconciler uses — just eagerly. An unregistered sandbox has no catalog entry, so the heavierremoveSandboxFromNode(which deletes a routing entry) is unnecessary.killSandboxOnNodealready treatsNotFoundas success, so if the node never actually completed the create, the kill is a cheap no-op.The alternative in the issue — making node-side create self-cleaning within a bounded window if the caller never confirms registration — is more robust to an API crash mid-create, but it is a node/protocol change with a much larger blast radius, and reconcile already covers the crash case. Not pursued here.
Testing
placement/interrupted_create_test.go— new: an interrupted create reports the node asInterruptedNode; aResourceExhaustedrefusal does not (would be a pointless kill); a hard failure with a live context does not (genuine node failure, self-cleaned).go build ./...,go vet, fullplacementpackage tests, andgolangci-lint(v2, pinned) all clean.Failed to create sandbox … "[Canceled] context canceled"/failed to place sandbox: request timed out, returns HTTP 499, and the sandbox is absent from the running store and team index (GET 404) — the exact conditions this fix now compensates.