feat(sandbox): support sandbox-initiated supervisor connections

[drew]

Summary

New dedicated Kubernetes sandbox generations open their protected control connection to the supervisor. Existing running legacy generations keep their current connection; stop/start upgrades them automatically while preserving the boundary Service's UID and ClusterIP. No legacy rollout flag is required.

Only TCP dialing changes. TLS and gRPC roles, pinned runtime identity, supervisor bearer authentication, and isolation-backend lifecycle remain the same. This is an independent precursor to the shared-supervisor design in #4377.

Related Issue

Closes #4652

Changes

• Record the current generation's connection mode on each Sandbox. Missing metadata identifies a legacy generation. Creation and restart use runtime-initiated automatically; reconciliation does not migrate a running generation. Gateway recovery that recreates both Pods also upgrades the generation.
• Upgrade a legacy boundary Service in place only after the old Pods are gone and the controller is suspended. Guard the Service patch with its resourceVersion, preserve its allocated address, and recover if a previous attempt updated the Service before recording the new mode. Grant Service patch permission through the appropriate Helm Role or ClusterRole.
• Add backend-private SandboxTransport::RuntimeInitiatedTcp and BoundaryListener::RuntimeInitiatedTlsTcp variants, preserving legacy encodings and public RPCs.
• Select one BoundaryTransport for discovery and attachment. All stream sources share pinned TLS authentication; the private listener bounds pending handshakes and authenticated spares. Reuse existing cancellation, reconnect, and authenticated serving paths, with a spare connection for credential replacement.
• Share Kubernetes generation installation between creation and restart. Publish the supervisor endpoint before readiness, admit only the paired control flow through a per-sandbox NetworkPolicy, and validate Service/mode/policy drift during bootstrap and normal reconciliation. Existing namespace fences remain intact.
• Document automatic migration and image compatibility. New generations require compatible gateway, supervisor, and runtime binaries. Docker, Podman, and VM retain their current transports.

Testing

Rebased onto main at 35a3390db. Adapted reverse control to the published typed Sandbox Protocol and shared transport descriptor introduced on main. Existing unit suites and the commit checks passed; Kubernetes conformance passed 8/8, and the unchanged full Kubernetes E2E rerun passed 100/100 with four existing skips. The first full run had one SSH restart timing failure; its isolated rerun and the full rerun passed without source changes, so the cause remains unestablished. Earlier checks below predate this rebase. CI will validate the pushed commits.

• Existing unit tests retained with minimal fixture updates; no new Rust unit tests. The Kubernetes driver passes 297 library tests and 1 binary test. Scoped gateway/driver Clippy, formatting, and Markdown lint pass.
• Helm lint passes for all variants. Existing chart checks pass: 339 gateway tests, 6 workspace tests, split ownership, Rust TOML loader validation, and resource coherence.
• Targeted manual mixed-version Kubernetes E2E on k3d/kube-router: genuine baseline images from a4c848adb; unchanged running legacy Pod across gateway upgrade; automatic migration of stopped legacy sandboxes; stop/start migration of a running legacy sandbox; new and legacy generations side by side; retry after interruption between Service and Sandbox updates; exec; 300 KB upload/download; network allow/deny and control-port denial; gateway restart. Migrated Service UIDs/ClusterIPs and namespace fence UID/resourceVersion/spec remain unchanged. Fixtures were cleaned up.
• Earlier compatibility checks verified rejection of the new descriptor by an incompatible old runtime and new-mode operation with the Helm SSH ingress policy disabled.

Cluster verification covers kube-router. Other CNIs and a forced mid-session boundary-network partition were not exercised; the targeted manual exercise was used in place of the full Kubernetes conformance lane. A procfs listener-closure assertion failed once in an earlier sandbox suite run, then passed 30 isolated runs and full suite reruns without code changes; its cause was not established.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Runtime documentation and relevant operator skills updated

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-4656.docs.buildwithfern.com/openshell