rfc-0012: Isolation Backend interface by jganoff · Pull Request #2048 · NVIDIA/OpenShell

jganoff · 2026-06-29T15:22:24Z

Summary

Adds RFC 0012 for the Isolation Backend, a proposed pluggable component that establishes and enforces an agent's isolation boundary.

OpenShell runs untrusted agent code inside an isolation boundary: the network, filesystem, syscall, and identity constraints that decide what the agent can reach and what can reach it. Building that boundary takes privilege, and today that privilege lives inside the agent's own container, beside the code it is meant to confine. That blocks restricted and multi-tenant clusters, whose Pod Security Standards reject the capability set the in-pod setup needs.

This RFC proposes making the boundary a pluggable component, the Isolation Backend, that separates the privileged work that builds the boundary from the supervisor that operates it as the policy authority (proxy, policy, audit) and the agent's only egress. The supervisor drives any backend through one runtime contract, so the privileged setup can move out of the agent's container (a sidecar, a separate pod, a microVM, a node component, and eventually outside the agent's kernel) without changing how the supervisor operates it. Each placement is a new backend, not a new supervisor.

Feedback especially welcome on: the lifecycle state boundaries and what each one guarantees, the provenance-based identity and attestation model, and the resource-binding split between the backend and the compute driver.

Related to #1737.

Checklist

Follows Conventional Commits
Commits are signed off (DCO)
Architecture docs updated (if applicable)

copy-pr-bot · 2026-06-29T15:22:27Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

github-actions · 2026-06-29T15:22:35Z

All contributors have signed the DCO ✍️ ✅
_{Posted by the DCO Assistant Lite bot.}

jganoff · 2026-06-29T15:28:50Z

I have read the DCO document and I hereby sign the DCO.

Tighten RFC 0012 for review without changing its structure or tone: - Remove the undefined `BoundaryIntent` from `attach`, and state that `claim` is the only transition that binds sandbox identity, policy, agent, and resources; an attached boundary holds no claimed or untrusted workload. - Add the runtime interface definitions (`BoundaryExec`/`ExecSession`, `BoundaryPortForward`, `EventSource`) and their semantics: owned distinct stdio, PTY resize, placement-neutral exit and signals, stable repeated `wait`, validated loopback targets, and single-consumer events. - Make boundary confirmation mandatory (`MUST`) and lifetime-long, and state retry, termination, and cleanup semantics (only attach is auto-retryable; a lost `start_agent` returns the existing process; descendants are owned; `wait_terminated` and an idempotent `shutdown`). - Require trusted admission to supply the expected backend id, contract version, policy digest, and capabilities that a backend cannot lower; add the no-silent-weakening policy invariant; mark `VerifiedBoundaryDescriptor` privately constructed and distinguish envelope from contract version; make `ResourceBinding` driver-issued, bound, and non-wideable. - Make the rollout executable and forbid a silent default backend. topology-matrix: rename "sidecar proxy" to sidecar-assisted (the proxy stays with the supervisor), frame sidecar and node as composite backends, move the single-pod outer sandbox to the shared-kernel cell, and note a node enforcer alone is not `restricted`-compatible. codebase-grounding: re-pin anchors to the RFC's parent ba21bb3 (driver.rs and proxy.rs line numbers refreshed) and add a permalink base; correct the event wording so a denial carries `Evidence` plus request-specific L7 data. Set state: review and link PR NVIDIA#2048. Signed-off-by: Jordan Ganoff <jordan.ganoff@docker.com>

drew · 2026-07-02T00:42:45Z

I'd be interested to see if we can support MXC using this interface, #2071.

ericcurtin · 2026-07-02T16:37:37Z

@maxamillion @derekwaynecarr @mrunalp thoughts?

jganoff · 2026-07-05T21:55:18Z

@TaylorMutch thanks for the review and discussion last week. Based on our discussion, I've made the following changes:

Introduced topology nomenclature to align with how we've been talking about this in other conversations
Simplified the explanation in the proposal section
Clarified binary identity
Simplified the initial contract to require the minimal interface necessary to satisfy all known topologies
Confirmed this will work with the proposed mxc (RFC 0013) proposal: docs(rfc): add RFC 0013 native Windows support via MXC #2071 (cc @drew )
Confirmed this will work with warm pooling (though I haven't built a POC for it yet)

I've also updated my POC implementation for the current in-pod supervisor topology to align with these updates.

Please let me know what you all think!

github-actions · 2026-07-21T09:09:18Z

This pull request has had no activity for 14 days and is now marked stale. It may be closed in 7 days if there is no further activity.

Tighten RFC 0012 for review without changing its structure or tone: - Remove the undefined `BoundaryIntent` from `attach`, and state that `claim` is the only transition that binds sandbox identity, policy, agent, and resources; an attached boundary holds no claimed or untrusted workload. - Add the runtime interface definitions (`BoundaryExec`/`ExecSession`, `BoundaryPortForward`, `EventSource`) and their semantics: owned distinct stdio, PTY resize, placement-neutral exit and signals, stable repeated `wait`, validated loopback targets, and single-consumer events. - Make boundary confirmation mandatory (`MUST`) and lifetime-long, and state retry, termination, and cleanup semantics (only attach is auto-retryable; a lost `start_agent` returns the existing process; descendants are owned; `wait_terminated` and an idempotent `shutdown`). - Require trusted admission to supply the expected backend id, contract version, policy digest, and capabilities that a backend cannot lower; add the no-silent-weakening policy invariant; mark `VerifiedBoundaryDescriptor` privately constructed and distinguish envelope from contract version; make `ResourceBinding` driver-issued, bound, and non-wideable. - Make the rollout executable and forbid a silent default backend. topology-matrix: rename "sidecar proxy" to sidecar-assisted (the proxy stays with the supervisor), frame sidecar and node as composite backends, move the single-pod outer sandbox to the shared-kernel cell, and note a node enforcer alone is not `restricted`-compatible. codebase-grounding: re-pin anchors to the RFC's parent ba21bb3 (driver.rs and proxy.rs line numbers refreshed) and add a permalink base; correct the event wording so a denial carries `Evidence` plus request-specific L7 data. Set state: review and link PR NVIDIA#2048. Signed-off-by: Jordan Ganoff <jordan.ganoff@docker.com>

Introduces RFC 0012, the runtime-selectable Isolation Backend contract: a pluggable component that establishes and enforces an agent's isolation boundary across network, filesystem, syscall, and identity, while the supervisor stays the policy authority (proxy, policy, audit) and the agent's only egress. Includes the supporting topology matrix and codebase-grounding notes. Signed-off-by: Jordan Ganoff <jordan.ganoff@docker.com>

Tighten RFC 0012 for review without changing its structure or tone: - Remove the undefined `BoundaryIntent` from `attach`, and state that `claim` is the only transition that binds sandbox identity, policy, agent, and resources; an attached boundary holds no claimed or untrusted workload. - Add the runtime interface definitions (`BoundaryExec`/`ExecSession`, `BoundaryPortForward`, `EventSource`) and their semantics: owned distinct stdio, PTY resize, placement-neutral exit and signals, stable repeated `wait`, validated loopback targets, and single-consumer events. - Make boundary confirmation mandatory (`MUST`) and lifetime-long, and state retry, termination, and cleanup semantics (only attach is auto-retryable; a lost `start_agent` returns the existing process; descendants are owned; `wait_terminated` and an idempotent `shutdown`). - Require trusted admission to supply the expected backend id, contract version, policy digest, and capabilities that a backend cannot lower; add the no-silent-weakening policy invariant; mark `VerifiedBoundaryDescriptor` privately constructed and distinguish envelope from contract version; make `ResourceBinding` driver-issued, bound, and non-wideable. - Make the rollout executable and forbid a silent default backend. topology-matrix: rename "sidecar proxy" to sidecar-assisted (the proxy stays with the supervisor), frame sidecar and node as composite backends, move the single-pod outer sandbox to the shared-kernel cell, and note a node enforcer alone is not `restricted`-compatible. codebase-grounding: re-pin anchors to the RFC's parent ba21bb3 (driver.rs and proxy.rs line numbers refreshed) and add a permalink base; correct the event wording so a denial carries `Evidence` plus request-specific L7 data. Set state: review and link PR NVIDIA#2048. Signed-off-by: Jordan Ganoff <jordan.ganoff@docker.com>

Contract completion: add MediationIngress/MediatedConnection so the proxy receives connections placement-neutrally; complete the runtime types (BoundaryExitStatus/BoundarySignal, BoundaryTerminal::resize, IdentitySource Send+Sync) and a machine-readable BackendErrorKind; name the AdmittedBoundary- Requirements admission block retained by VerifiedBoundaryDescriptor; show claim_id/generation in ClaimContext; require a canonical bounded descriptor encoding; define capability comparison. Mark MediationIngress, the admission block, and claim generation as contract additions the in-pod POC does not yet implement. Security semantics: make confirmation mandatory and lifetime-long, but correct the enforcement model: Landlock/seccomp are monotonic while netns/nftables are mutable, so require the mutation authority fenced from the workload, atomic default-deny updates, and fail-closed on lease loss rather than calling the boundary irreversible. "Confirms effective enforcement", not "reads rules back". Bind Attested identity to sandbox and claim id/generation. Tighten the execution-domain invariant (admission-visible named capability + audit, whole descendant trees). Distinguish structural pre-Running ordering from dynamic post-termination rejection; note adoption must remove legacy bypass paths. Lifecycle: drop the "establishes before supervisor boot" contradiction; add shutdown to the supervisor sequence; rewrite the in-pod migration appendix (policy-free attach, netns/rules at claim/bind, agent().wait(), wait_terminated, idempotent shutdown). The RFC is implemented only after the in-pod backend passes its release gates. Topology: narrow node-enforcer claims to "network-setup capabilities off the pod"; nuance NetworkPolicy (coarse L3/L4 defense in depth, not a proxy replacement); drop the proxy-policy-reprograms-boundary claim; label the containment table a kernel-compromise ceiling. Grounding: re-pin to the RFC's parent a5161d0 (process.rs and proxy.rs line numbers refreshed; others unchanged and reverified). Signed-off-by: Jordan Ganoff <jordan.ganoff@docker.com>

This removes a lot of detail to keep the RFC succinct.

The RFC should describe the end state, and not intermediate milestones or POCs.

- Introduced topology nomenclature to align with how we've been talking about this in other conversations - Simplified the explanation in the proposal section - Clarified binary identity - Simplified the initial contract to require the minimal interface necessary to satisfy all known topologies - Confirmed this will work with the proposed mxc (RFC 0013) proposal

- Repin codebase-grounding.md to 8eacb47 (sidecar supervisor topology, NVIDIA#2076); update capabilities line numbers (1534→2538, 1540→2544), init container line numbers (191→423, 993→1506, 1185→2113), and remove the stale "no native sidecars today" claim. Add openshell-network-init and openshell-supervisor-network sidecar entries and expand the rg pattern. - Add Implementation column to topology-matrix.md; mark Co-located/in-pod and Same-pod composite as implemented (original topology and NVIDIA#2076 respectively); remaining patterns noted as proposed. Signed-off-by: Jordan Ganoff <jordan.ganoff@docker.com>

All line numbers and function names verified against the post-sidecar state of the codebase (commit 8eacb47, NVIDIA#2076). Changes: - process.rs: ProcessHandle::spawn 440→527, netns param 446→535; drop_privileges call sites 603/700→710/812, enforcement 613/705→721/818-819; enter_netns_and_sandbox now documented at ssh.rs:1245 - CLONE_NEWNET call sites: process.rs:589→695, ssh.rs:619/1186→653/1262, supervisor_session.rs:610→735, netns/mod.rs:363→342 (226 unchanged) - CLONE_NEWNS: was one unshare at :393; now unshare at :449 and a new setns at :480 added for sidecar mount-namespace entry - nft fail-open: line 264→265, return Ok(()) range 272-277→277; note that the sidecar path (netns/mod.rs:477) requires nft and returns an error if absent, fixing the invariant bug for the sidecar topology - nft_ruleset.rs: policy accept 41→53; accept rules 43-49→56-92; reject rules now at 106+ - VM driver MASQUERADE: runtime.rs:417/436→418/437 - Agent command: main.rs:331→601; sleep infinity driver.rs:1886→2937, clarify it is set via SANDBOX_COMMAND env var - OPA evaluation: proxy.rs:1611 / evaluate_opa_tcp renamed to authorize_egress_intent at proxy.rs:1955; NetworkInput built at :2032 - openshell.proto: clarify no lifecycle Attach; note AttachSandboxProvider (provider record attachment, not isolation lifecycle) - README.md appendix: update pinned commit reference a5161d0→8eacb477 Signed-off-by: Jordan Ganoff <jordan.ganoff@docker.com>

russellb

Reviewed this against the in-progress cni-sidecar supervisor topology (#2606), which moves nftables installation from an in-pod NET_ADMIN init container to a node CNI DaemonSet. It lines up well with this RFC — it's a concrete instance of the "Delegated backend components" row, and the direction (privilege out of the agent container, one supervisor operating the boundary) is exactly what we built toward.

Most of my notes are inline. The substantive one is invariant 6: as written it assumes the backend can detect standing-enforcement loss at runtime, which a node-delegated backend structurally can't do from inside the pod. I think the contract needs a small amount of give for delegated backends there and in the Ready/confirmation wording; everything else is either a clean fit or a mechanical reshape on our side.

Signed-off-by: Jordan Ganoff <jordan.ganoff@docker.com>

jganoff · 2026-08-22T04:12:39Z

@drew @russellb I pushed commit 6f892f5 with a broader revision based on the review feedback and subsequent design discussions. The update intends to simplify the mental model and normative contract, clarify the responsibilities of the logical supervisor and isolation backend, and explicitly covers delegated and multi-process topologies. It also aligns descriptor, lifecycle, network-mediation, binary-identity, and failure semantics throughout.

Please let me know how your testing goes with this and if there's any other changes you'd like to see.

Keep RFC 0012 focused on placing binary-aware enforcement behind the Isolation Backend contract instead of requiring a new connection-initiation capture mechanism. Define identity as trusted resolution for an accepted connection before policy evaluation, retain fail-closed attribution requirements, and document the current authorization-time procfs behavior. This preserves RFC 0002's binary gating baseline while allowing future backends to provide stronger identity resolution without changing the contract. Signed-off-by: Jordan Ganoff <jordan.ganoff@docker.com>

jganoff · 2026-08-22T21:54:46Z

I realized I had inadvertently changed the contract for how binary identity is resolved, and that shouldn't be part of RFC 0012. 86dc3f2 preserves RFC 0002's binary gating baseline while allowing future backends to provide stronger identity resolution without changing the contract.

mayawang · 2026-09-10T04:23:50Z

We're planning to implement RFC 12 for Agent Substrate, which lands as a delegated backend: a thin in-process implementation carrying its own transport, with delegation staying topology-private.

My question is what conformance can cover for that shape. It's specified per backend and topology, so a delegated topology does get tested. What I can't tell is how much of its behavior the suite reaches. If the transport inside the topology is backend-private, the tests exercise the in-process trait while the load-bearing part happens over a link they never cross, and two delegated backends could pass the same suite while behaving differently in ways it stays silent on.

So the concrete question: does conformance for a delegated topology stop at the trait boundary, or is there a point where the delegated transport becomes versioned contract surface? The alternatives section says

A transport-bearing surface is not precluded: it is versioned contract surface, added when a concrete delegated backend requires it.

We may well be that concrete backend, so it would help to know what "requires" means in practice: a backend that can't be built without it, or one where conformance coverage argues for it. If it's the second, we can bring the specifics from our side rather than leave it abstract.

Happy to write it up as a proposal if that's more useful than a question.

drew · 2026-09-10T05:19:57Z

We're planning to implement RFC 12 for Agent Substrate, which lands as a delegated backend: a thin in-process implementation carrying its own transport, with delegation staying topology-private.

My question is what conformance can cover for that shape. It's specified per backend and topology, so a delegated topology does get tested. What I can't tell is how much of its behavior the suite reaches. If the transport inside the topology is backend-private, the tests exercise the in-process trait while the load-bearing part happens over a link they never cross, and two delegated backends could pass the same suite while behaving differently in ways it stays silent on.

So the concrete question: does conformance for a delegated topology stop at the trait boundary, or is there a point where the delegated transport becomes versioned contract surface? The alternatives section says

A transport-bearing surface is not precluded: it is versioned contract surface, added when a concrete delegated backend requires it.

We may well be that concrete backend, so it would help to know what "requires" means in practice: a backend that can't be built without it, or one where conformance coverage argues for it. If it's the second, we can bring the specifics from our side rather than leave it abstract.

Happy to write it up as a proposal if that's more useful than a question.

@mayawang if it's not too much trouble a proposal would be great. We have an initial prototype of this RFC starting here, #3151. We could quickly prototype your proposal against that PR to make sure we have everything we need in the interface.

You might also be interested in this doc, https://docs.google.com/document/d/11iR15xUgncckT50aBsDQgDVfCWrUU3nIh6Woc8IYaxI/edit?tab=t.0#heading=h.bwtw9rv4ec9q. We discussed this at our last community call and it outlines how we want to think about a concrete implementation (#3151) of the RFC for our existing drivers.

We also have an initial set of conformance tests here, https://github.com/NVIDIA/OpenShell/tree/main/crates/openshell-conformance. Our conformance tests will let you standup a OpenShell deployment in any specific configuration. Once that deployment is available we can run tests against that configuration. I would expect this to be a good starting point to ensure that the substrate implementation is conformant.

If you notice anything missing in the conformance tests we should be able to include additional tests as well.

Addresses PR feedback: Removed the implementation plan and codebase grounding, renamed the forwarding primitive to BoundaryLoopbackConnector, and clarified that conformance tests exercise the deployed topology, including delegated components and their transport.

jganoff · 2026-09-10T18:24:33Z

Thanks, @drew and @mayawang. I removed the implementation plan and codebase grounding, renamed the forwarding primitive to BoundaryLoopbackConnector, and clarified that conformance tests exercise the deployed topology, including delegated components and their transport. Let me know if that addresses the concern @mayawang.

I'd like to leave the TCP/DNS interface details open for implementation work (see comment above).

jganoff requested review from a team, derekwaynecarr, maxamillion and mrunalp as code owners June 29, 2026 15:22

jganoff force-pushed the jganoff/rfc-0012-isolation-backend branch from efaeff7 to c31cbda Compare June 29, 2026 17:13

drew added the rfc label Jun 29, 2026

drew added this to OpenShell Roadmap Jun 29, 2026

drew changed the title ~~docs(rfc): RFC 0012 Isolation Backend interface~~ rfc-0012: Isolation Backend interface Jun 29, 2026

drew moved this to In progress in OpenShell Roadmap Jun 29, 2026

jganoff force-pushed the jganoff/rfc-0012-isolation-backend branch from bb9cefb to b0b6a44 Compare June 29, 2026 20:08

jganoff mentioned this pull request Jun 30, 2026

feat: establish the Isolation Backend interface #1737

Open

2 tasks

ericcurtin approved these changes Jul 1, 2026

View reviewed changes

jganoff force-pushed the jganoff/rfc-0012-isolation-backend branch from e19656f to 0924382 Compare July 5, 2026 21:53

drew mentioned this pull request Jul 7, 2026

fix(supervisor): fail fast with diagnostic error when CAP_SETPCAP is unavailable #2075

Closed

7 tasks

ericcurtin mentioned this pull request Jul 14, 2026

feat: native Linux compute driver (openshell-driver-oci) #2255

Open

github-actions Bot added the state:stale Inactive item at risk of automatic closure. label Jul 21, 2026

mrunalp reviewed Jul 28, 2026

View reviewed changes

Comment thread rfc/0012-isolation-backend/codebase-grounding.md Outdated

mrunalp reviewed Jul 28, 2026

View reviewed changes

Comment thread rfc/0012-isolation-backend/topology-matrix.md Outdated

jganoff added 2 commits August 3, 2026 14:56

jganoff added 6 commits August 3, 2026 14:56

docs(rfc): simplify isolation backend contract

beb1f84

This removes a lot of detail to keep the RFC succinct.

removed POC and intermediate stage references

5fa4b1f

The RFC should describe the end state, and not intermediate milestones or POCs.

jganoff force-pushed the jganoff/rfc-0012-isolation-backend branch from 0924382 to 3f5bc31 Compare August 3, 2026 21:58

russellb reviewed Aug 12, 2026

View reviewed changes

docs(rfc): address feedback and clean ups

6f892f5

Signed-off-by: Jordan Ganoff <jordan.ganoff@docker.com>

jganoff requested a review from sjenning as a code owner August 22, 2026 04:00

jganoff requested a review from russellb August 22, 2026 04:01

jganoff mentioned this pull request Aug 24, 2026

feat(kubernetes): add cni-sidecar supervisor topology #2606

Open

7 tasks

This was referenced Aug 25, 2026

feat(firecracker): prototype host-supervised isolation #2934

Draft

feat(vm): run the supervisor outside the guest workload #2945

Open

feat(docker): isolate workloads behind the host supervisor #2965

Open

github-actions Bot removed the state:stale Inactive item at risk of automatic closure. label Aug 28, 2026

drew reviewed Sep 10, 2026

View reviewed changes

drew approved these changes Sep 10, 2026

View reviewed changes

drew enabled auto-merge September 10, 2026 19:25

Uh oh!

Conversation

jganoff commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Checklist

Uh oh!

copy-pr-bot Bot commented Jun 29, 2026

Uh oh!

github-actions Bot commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jganoff commented Jun 29, 2026

Uh oh!

drew commented Jul 2, 2026

Uh oh!

ericcurtin commented Jul 2, 2026

Uh oh!

jganoff commented Jul 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 21, 2026

Uh oh!

Uh oh!

Uh oh!

russellb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jganoff commented Aug 22, 2026

Uh oh!

jganoff commented Aug 22, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mayawang commented Sep 10, 2026

Uh oh!

drew commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jganoff commented Sep 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

jganoff commented Jun 29, 2026 •

edited

Loading

github-actions Bot commented Jun 29, 2026 •

edited

Loading

jganoff commented Jul 5, 2026 •

edited

Loading

drew commented Sep 10, 2026 •

edited

Loading