diff --git a/.agents/skills/helm-dev-environment/SKILL.md b/.agents/skills/helm-dev-environment/SKILL.md index 9fdf315c3c..94826b0412 100644 --- a/.agents/skills/helm-dev-environment/SKILL.md +++ b/.agents/skills/helm-dev-environment/SKILL.md @@ -81,7 +81,16 @@ The Skaffold flow builds distinct `gateway`, `sandbox`, and `supervisor` images and deploys the OpenShell Helm chart. The Kubernetes driver creates a capability-free workload Pod and a directly managed capability-free supervisor Pod. One namespace-wide NetworkPolicy denies direct egress from every OpenShell -workload Pod. The +workload Pod. New sandbox generations use `runtime-initiated` control: the sandbox +runtime dials its supervisor through `os-supervisor-`. The legacy +`os-boundary-` Service keeps selecting the workload for old gateways during +rolling upgrades. New gateways create both Services and bind the control-mode +annotation to its runtime generation. The driver adds a per-sandbox +`os-control-` NetworkPolicy for that path only. Existing running generations +keep their connection until replaced; stop/start and gateway recovery that +recreates both Pods upgrade legacy generations automatically. Use compatible +gateway, supervisor, and sandbox runtime images together; there is no legacy +rollout flag. The `pkiInitJob` hook (a pre-install Job that runs `openshell-gateway generate-certs`) generates gateway and CLI TLS secrets on first install. Supervisor Pods project only `ca.crt` and authenticate gateway RPCs with sandbox bearer tokens. User diff --git a/crates/openshell-driver-docker/README.md b/crates/openshell-driver-docker/README.md index ee77408fdc..2f64fd05d1 100644 --- a/crates/openshell-driver-docker/README.md +++ b/crates/openshell-driver-docker/README.md @@ -108,10 +108,16 @@ LSM decisions remain authoritative. | `network_mode = host` on the supervisor | Lets the trusted supervisor reach the gateway's primary loopback listener and originate approved upstream connections. | | `restart_policy = no` | Keeps canonical main-process exit terminal. | | `PidsLimit` | Applies the configured sandbox PID budget. Omit `sandbox_pids_limit` to use OpenShell's default. Explicit zero is invalid. | -| Private named volumes | One carries the authenticated sandbox/supervisor channel. The other is mounted only into the supervisor and contains its JWT and private gateway credentials. | +| Private named volumes | Separate volumes carry sandbox bootstrap keys, the runtime-initiated Unix socket, and supervisor-only credentials. The supervisor mounts bootstrap keys read-only and only the socket volume writable. | | In-memory `/run/openshell-supervisor-ca` tmpfs | Holds only the public supervisor CA certificate and trust bundle without making all of `/run` writable. | | CDI GPU request | Assigns the exact validated CDI devices requested by driver config or count-based selection. | +New containers dial the supervisor at `/.openshell/channel/control/supervisor/control.sock`. +The runtime remains the TLS/gRPC server; the supervisor verifies its pinned +identity and supplies the existing bearer credential. Saved descriptors retain +the legacy Unix listener for existing containers, including across restart. +New generations require matching supervisor and sandbox runtime binaries. + ## Stop, Start, and Delete Stop terminates the supervisor companion and stops the workload container diff --git a/crates/openshell-driver-docker/src/isolation.rs b/crates/openshell-driver-docker/src/isolation.rs index a817ec484b..9935ff0cbe 100644 --- a/crates/openshell-driver-docker/src/isolation.rs +++ b/crates/openshell-driver-docker/src/isolation.rs @@ -109,7 +109,7 @@ impl DockerBoundarySpec { auth_epoch: self.auth_epoch, gateway_id: self.gateway_id, verification_keys: self.verification_keys, - listener: BoundaryListener::Unix { + listener: BoundaryListener::RuntimeInitiatedUnix { socket_path: self.listener_socket, tls: self.sandbox_tls, }, @@ -124,7 +124,7 @@ impl DockerBoundarySpec { generation: self.generation, session_id: self.session_id, workload_identity: self.workload_identity, - transport: SandboxTransport::Unix { + transport: SandboxTransport::RuntimeInitiatedUnix { socket_path: self.control_socket, }, tls: self.supervisor_tls, diff --git a/crates/openshell-driver-docker/src/lib.rs b/crates/openshell-driver-docker/src/lib.rs index 059ba1535c..bf00b2bfb8 100644 --- a/crates/openshell-driver-docker/src/lib.rs +++ b/crates/openshell-driver-docker/src/lib.rs @@ -107,7 +107,8 @@ const SUPERVISOR_UID: u32 = 65_534; const SUPERVISOR_GID: u32 = 65_534; const BOUNDARY_MOUNT_PATH: &str = "/.openshell/channel"; const BOUNDARY_CONFIG_MOUNT_PATH: &str = "/.openshell/channel/sandbox/bootstrap.json"; -const BOUNDARY_SOCKET_MOUNT_PATH: &str = "/.openshell/channel/sandbox/control.sock"; +const BOUNDARY_CONTROL_MOUNT_PATH: &str = "/.openshell/channel/control"; +const BOUNDARY_SOCKET_MOUNT_PATH: &str = "/.openshell/channel/control/supervisor/control.sock"; const BOUNDARY_CERTIFICATE_MOUNT_PATH: &str = "/.openshell/channel/sandbox/server.crt"; const BOUNDARY_PRIVATE_KEY_MOUNT_PATH: &str = "/.openshell/channel/sandbox/server.key"; const SUPERVISOR_STATE_MOUNT_PATH: &str = "/.openshell/supervisor"; @@ -1309,7 +1310,9 @@ impl DockerComputeDriver { Status::unavailable("volume admission lookup failed") } })?; - if name == docker_channel_volume_name_by_id(sandbox_id, &self.config) { + if name == docker_channel_volume_name_by_id(sandbox_id, &self.config) + || name == docker_control_volume_name_by_id(sandbox_id, &self.config) + { if volume.driver != "local" || !volume.options.is_empty() || volume.labels.get(LABEL_SANDBOX_ID) != Some(sandbox_id) @@ -4126,28 +4129,36 @@ fn docker_supervisor_volume_name_by_id( format!("openshell-supervisor-{}", &digest[..32]) } +fn docker_control_volume_name_by_id( + sandbox_id: &str, + config: &DockerDriverRuntimeConfig, +) -> String { + docker_channel_volume_name_by_id(sandbox_id, config).replacen( + "openshell-channel-", + "openshell-control-", + 1, + ) +} + async fn create_docker_channel_volume( docker: &Docker, sandbox: &DriverSandbox, config: &DockerDriverRuntimeConfig, ) -> Result<(), Status> { - create_docker_runtime_volume( - docker, - sandbox, - config, + let names = [ docker_channel_volume_name(sandbox, config), - ) - .await?; - if let Err(error) = create_docker_runtime_volume( - docker, - sandbox, - config, docker_supervisor_volume_name(sandbox, config), - ) - .await - { - let _ = remove_docker_volume(docker, &docker_channel_volume_name(sandbox, config)).await; - return Err(error); + docker_control_volume_name_by_id(&sandbox.id, config), + ]; + for (index, name) in names.iter().enumerate() { + if let Err(error) = + create_docker_runtime_volume(docker, sandbox, config, name.clone()).await + { + for created in &names[..index] { + let _ = remove_docker_volume(docker, created).await; + } + return Err(error); + } } Ok(()) } @@ -4213,6 +4224,11 @@ async fn remove_docker_channel_volume_by_id( docker, &docker_supervisor_volume_name_by_id(sandbox_id, config), ) + .await?; + remove_docker_volume( + docker, + &docker_control_volume_name_by_id(sandbox_id, config), + ) .await } @@ -4472,6 +4488,8 @@ fn docker_sandbox_bundle_archive( identity.uid, identity.gid, )?; + // The control volume is read-only in the runtime container; the + // supervisor stager initializes it. append_docker_archive_file( &mut archive, ".openshell/runtime/openshell-sandbox", @@ -4551,7 +4569,8 @@ async fn stage_docker_sandbox_bundle( bollard::body_full(Bytes::from(archive)), ) .await - .map_err(|error| Status::internal(format!("stage Docker sandbox bundle: {error}"))) + .map_err(|error| Status::internal(format!("stage Docker sandbox bundle: {error}")))?; + Ok(()) } fn decode_docker_launch_authentication( @@ -4964,6 +4983,32 @@ async fn stage_docker_supervisor_bundle( Some(RemoveContainerOptionsBuilder::default().force(true).build()), ) .await; + let runtime_initiated = read_docker_runtime_descriptor(&sandbox.id, config).await? + .is_some_and(|descriptor| matches!(descriptor.transport, openshell_sandbox_backend::boundary_protocol::SandboxTransport::RuntimeInitiatedUnix { .. })); + let mut mounts = vec![Mount { + target: Some(SUPERVISOR_STATE_MOUNT_PATH.to_string()), + source: Some(docker_supervisor_volume_name(sandbox, config)), + typ: Some(MountTypeEnum::VOLUME), + read_only: Some(false), + volume_options: Some(MountVolumeOptions { + no_copy: Some(true), + ..Default::default() + }), + ..Default::default() + }]; + if runtime_initiated { + mounts.push(Mount { + target: Some(BOUNDARY_CONTROL_MOUNT_PATH.to_string()), + source: Some(docker_control_volume_name_by_id(&sandbox.id, config)), + typ: Some(MountTypeEnum::VOLUME), + read_only: Some(false), + volume_options: Some(MountVolumeOptions { + no_copy: Some(true), + ..Default::default() + }), + ..Default::default() + }); + } let created = docker .create_container( Some( @@ -4985,17 +5030,7 @@ async fn stage_docker_supervisor_bundle( )), host_config: Some(HostConfig { network_mode: Some("none".to_string()), - mounts: Some(vec![Mount { - target: Some(SUPERVISOR_STATE_MOUNT_PATH.to_string()), - source: Some(docker_supervisor_volume_name(sandbox, config)), - typ: Some(MountTypeEnum::VOLUME), - read_only: Some(false), - volume_options: Some(MountVolumeOptions { - no_copy: Some(true), - ..Default::default() - }), - ..Default::default() - }]), + mounts: Some(mounts), cap_drop: Some(vec!["ALL".to_string()]), security_opt: Some(vec!["no-new-privileges:true".to_string()]), ..Default::default() @@ -5013,14 +5048,50 @@ async fn stage_docker_supervisor_bundle( .path(SUPERVISOR_STATE_MOUNT_PATH) .copy_uidgid("true") .build(); - let result = docker - .upload_to_container( - &created.id, - Some(options), - bollard::body_full(Bytes::from(archive)), - ) - .await - .map_err(|error| Status::internal(format!("stage Docker supervisor bundle: {error}"))); + let result: Result<(), Status> = async { + docker + .upload_to_container( + &created.id, + Some(options), + bollard::body_full(Bytes::from(archive)), + ) + .await + .map_err(|error| { + Status::internal(format!("stage Docker supervisor bundle: {error}")) + })?; + if runtime_initiated { + // Initialize the socket volume using the supervisor's staging user. + // Docker preserves the existing volume root's ownership, so use + // a child directory created by the supervisor's staging user. + let mut archive = tar::Builder::new(Vec::new()); + append_docker_archive_directory( + &mut archive, + "supervisor", + 0o711, + SUPERVISOR_UID, + SUPERVISOR_GID, + )?; + let bytes = archive.into_inner().map_err(|error| { + Status::internal(format!("finish control socket archive: {error}")) + })?; + let options = UploadToContainerOptionsBuilder::default() + .path(BOUNDARY_CONTROL_MOUNT_PATH) + .copy_uidgid("true") + .build(); + docker + .upload_to_container( + &created.id, + Some(options), + bollard::body_full(Bytes::from(bytes)), + ) + .await + .map_err(|error| { + Status::internal(format!("stage Docker control socket volume: {error}")) + })?; + } + Ok(()) + } + .await; let cleanup = docker .remove_container( &created.id, @@ -5245,6 +5316,8 @@ async fn spawn_docker_control_process( &config.upstream_proxy, config.proxy_ca_bundle.is_some(), )); + let runtime_initiated = read_docker_runtime_descriptor(&sandbox.id, config).await? + .is_some_and(|descriptor| matches!(descriptor.transport, openshell_sandbox_backend::boundary_protocol::SandboxTransport::RuntimeInitiatedUnix { .. })); let mut supervisor_mounts = vec![ Mount { target: Some(BOUNDARY_MOUNT_PATH.to_string()), @@ -5269,6 +5342,20 @@ async fn spawn_docker_control_process( ..Default::default() }, ]; + if runtime_initiated { + supervisor_mounts.push(Mount { + target: Some(BOUNDARY_CONTROL_MOUNT_PATH.to_string()), + source: Some(docker_control_volume_name_by_id(&sandbox.id, config)), + typ: Some(MountTypeEnum::VOLUME), + read_only: Some(false), + volume_options: Some(MountVolumeOptions { + no_copy: Some(true), + ..Default::default() + }), + ..Default::default() + }); + } + if let Some(socket) = config.provider_spiffe_workload_api_socket.as_ref() { let parent = socket.parent().ok_or_else(|| { Status::failed_precondition("provider SPIFFE socket has no parent directory") @@ -5931,6 +6018,19 @@ fn build_container_create_body_for_image( }), ..Default::default() }); + // The runtime only dials the supervisor-owned socket; it never writes here. + user_mounts.push(Mount { + target: Some(BOUNDARY_CONTROL_MOUNT_PATH.to_string()), + source: Some(docker_control_volume_name_by_id(&sandbox.id, config)), + typ: Some(MountTypeEnum::VOLUME), + read_only: Some(true), + volume_options: Some(MountVolumeOptions { + no_copy: Some(true), + ..Default::default() + }), + ..Default::default() + }); + let user_bind_strings = docker_driver_bind_strings(driver_config)?; let device_requests = gpu_device_ids.map(|device_ids| { vec![DeviceRequest { diff --git a/crates/openshell-driver-docker/src/tests.rs b/crates/openshell-driver-docker/src/tests.rs index 0159ecd99f..b679a7f5a6 100644 --- a/crates/openshell-driver-docker/src/tests.rs +++ b/crates/openshell-driver-docker/src/tests.rs @@ -2056,7 +2056,7 @@ fn build_container_create_body_includes_driver_config_mounts() { .mounts .expect("driver config mounts should be set"); - assert_eq!(mounts.len(), 3); + assert_eq!(mounts.len(), 4); assert_eq!(mounts[0].typ, Some(MountTypeEnum::VOLUME)); assert_eq!(mounts[0].source.as_deref(), Some("work-nfs")); assert_eq!(mounts[0].target.as_deref(), Some("/sandbox/work")); diff --git a/crates/openshell-driver-kubernetes/README.md b/crates/openshell-driver-kubernetes/README.md index 8c2f24a7a4..9a9d6dca18 100644 --- a/crates/openshell-driver-kubernetes/README.md +++ b/crates/openshell-driver-kubernetes/README.md @@ -103,6 +103,61 @@ them. The cluster CNI must enforce ingress and egress `NetworkPolicy` for every sandbox namespace. Kubernetes accepts policy objects without confirming enforcement, so operators must verify CNI support before running sandboxes. +### Control connection direction + +Every new runtime generation uses `runtime-initiated` control. The Sandbox +annotation `openshell.ai/sandbox-runtime-control-connection` records the mode, +bound to `openshell.ai/sandbox-runtime-control-generation`. A missing marker or +a different current runtime generation identifies a legacy, +`supervisor-initiated` generation, which keeps working until replaced. Stop/start +upgrades it automatically: with both old Pods removed, the driver adds the +supervisor Service and records the new mode and generation +before releasing the replacement workload. Gateway recovery that recreates both +Pods also upgrades the generation. Reconciliation never migrates a running +generation. An unknown annotation value fails closed; there is no rollout flag. + +Only the dial direction differs. In both modes the sandbox runtime presents the +generation-pinned TLS server identity and serves gRPC, and the supervisor +verifies that identity, acts as the gRPC client, and sends the Sandbox Protocol +JWT. The per-sandbox Service `os-boundary-` always selects the paired +workload Pod. A separate `os-supervisor-` Service always selects the paired +supervisor Pod. New gateways create both Services, retaining the legacy +Service's UID, ClusterIP, and selector across restarts so an older gateway can +still replace the generation during a rolling upgrade. + +For `runtime-initiated`: + +- The bootstrap Secret gives the runtime the Service ClusterIP and port. The + runtime retries with capped backoff until the supervisor listens, so neither + Pod waits on the other during bootstrap. The Service sets + `publishNotReadyAddresses` because the supervisor becomes Ready only after + the runtime attaches through it. The workload still cannot reach the control + port. +- A per-sandbox `NetworkPolicy` `os-control-` selects only this sandbox's + two Pods and admits one flow, from its workload Pod to its supervisor Pod on + `boundary_port` over TCP, as workload egress and supervisor ingress. + Supervisors may already be ingress-isolated by the managed SSH ingress policy; + when that policy is disabled, this one isolates them. Exec, file sync, and + logs need no other supervisor ingress because the supervisor opens those + connections. The CNI must still allow kubelet readiness probes. The + namespace-wide policies are unchanged. +- The policy's UID and resourceVersion are bound into the boundary resource + claims and outer-fence evidence. Reconciliation suspends the workload if the + policy disappears, diverges, or changes generation, or if the active control + Service no longer selects the role the recorded mode requires. +- Both Services and the control policy are owned by the Sandbox resource and are + deleted only with that sandbox. + +Deploy a gateway, supervisor image, and sandbox runtime image that all support +`runtime-initiated` before using it. Older binaries reject the new descriptor +shapes and fail closed. + +During a mixed-version rollout, an old gateway can restart a sandbox by +creating its own compatible supervisor/runtime pair through the legacy +Service. It preserves unknown annotations, so the mode marker is accepted only +for its recorded generation. Do not replace an individual generation's +supervisor or runtime binary with one that cannot read its descriptor. + Each sandbox generation uses two immutable bootstrap Secrets. A trusted init container stages the sandbox bootstrap into memory, and the sandbox removes it before starting untrusted code. The other Secret is mounted only by the @@ -110,7 +165,8 @@ supervisor; when `proxy_ca_bundle` is configured it also carries the operator's corporate proxy CA bundle, which the gateway reads from its own filesystem so the anchor stays in the gateway's trust domain rather than the sandbox namespace. The TLS channel binds the namespace, Sandbox CR, workload Pod, -supervisor Pod, and shared network-policy identities. Stop deletes the workload +supervisor Pod, and shared network-policy identities, plus the per-sandbox +control policy for `runtime-initiated` sandboxes. Stop deletes the workload and supervisor Pods. Start rotates both Secrets and creates a new supervisor Pod before releasing a new workload Pod. It also renders the trusted init container again from gateway config, so a sandbox created by an older gateway diff --git a/crates/openshell-driver-kubernetes/src/config.rs b/crates/openshell-driver-kubernetes/src/config.rs index ec37ed61b1..14adb9ee01 100644 --- a/crates/openshell-driver-kubernetes/src/config.rs +++ b/crates/openshell-driver-kubernetes/src/config.rs @@ -74,11 +74,52 @@ pub const DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME: &str = "default"; /// Default storage size for the workspace PVC. pub const DEFAULT_WORKSPACE_STORAGE_SIZE: &str = "2Gi"; +/// Which side opens the Sandbox Protocol TCP connection. +/// +/// Internal state for the current runtime generation. A missing Sandbox +/// annotation identifies a legacy generation; every new generation uses +/// `RuntimeInitiated`. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default, Serialize, Deserialize)] +#[serde(rename_all = "kebab-case")] +pub enum KubernetesControlConnection { + /// The sandbox runtime dials its supervisor's per-sandbox Service. + #[default] + RuntimeInitiated, + /// The supervisor dials the sandbox runtime's listener (original mode). + SupervisorInitiated, +} + +impl KubernetesControlConnection { + /// Stable spelling used in configuration and Sandbox annotations. + #[must_use] + pub const fn as_str(self) -> &'static str { + match self { + Self::RuntimeInitiated => "runtime-initiated", + Self::SupervisorInitiated => "supervisor-initiated", + } + } +} + +impl FromStr for KubernetesControlConnection { + type Err = String; + + fn from_str(value: &str) -> Result { + match value { + "runtime-initiated" => Ok(Self::RuntimeInitiated), + "supervisor-initiated" => Ok(Self::SupervisorInitiated), + other => Err(format!( + "invalid sandbox control connection '{other}'; expected runtime-initiated or supervisor-initiated" + )), + } + } +} + /// Driver-owned requirements for the Kubernetes sandbox runtime. #[derive(Debug, Clone, Serialize, Deserialize)] #[serde(default, deny_unknown_fields)] pub struct KubernetesSandboxRuntimeConfig { - /// TCP port exposed by the workload boundary to its paired control pod. + /// TCP port of the Sandbox Protocol listener: the workload boundary for + /// supervisor-initiated sandboxes, the supervisor for runtime-initiated ones. pub boundary_port: u16, } diff --git a/crates/openshell-driver-kubernetes/src/driver.rs b/crates/openshell-driver-kubernetes/src/driver.rs index f30b49f501..f53e05eb1d 100644 --- a/crates/openshell-driver-kubernetes/src/driver.rs +++ b/crates/openshell-driver-kubernetes/src/driver.rs @@ -5,17 +5,20 @@ use crate::config::{ DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME, DEFAULT_SANDBOX_UID, DEFAULT_WORKSPACE_STORAGE_SIZE, - KubernetesComputeConfig, OperatorNamespaceAllowlist, WorkspaceMode, is_dns_1123_label, - managed_namespace, managed_namespace_prefix, validate_managed_namespace_name, + KubernetesComputeConfig, KubernetesControlConnection, OperatorNamespaceAllowlist, + WorkspaceMode, is_dns_1123_label, managed_namespace, managed_namespace_prefix, + validate_managed_namespace_name, }; use crate::isolation::{ - BOUNDARY_PAIR_LABEL, BOUNDARY_ROLE_LABEL, KubernetesSandboxRuntimeBoundarySpec, + BOUNDARY_PAIR_LABEL, BOUNDARY_ROLE_LABEL, KubernetesControlPath, + KubernetesSandboxRuntimeBoundarySpec, }; use crate::lifecycle::LifecycleGates; use crate::sandbox_runtime::{ BOUNDARY_CERTIFICATE_PATH, BOUNDARY_CONFIG_PATH, BOUNDARY_PRIVATE_KEY_PATH, ClientTlsMaterial, SUPERVISOR_TERMINATION_GRACE_PERIOD_SECONDS, SandboxRuntimeNames, SupervisorClientTls, - boundary_service, generate_proxy_ca_material, image_pull_secret_copy, sandbox_bootstrap_secret, + boundary_service, boundary_service_matches_mode, control_policy, generate_proxy_ca_material, + image_pull_secret_copy, sandbox_bootstrap_secret, sandbox_owner_reference as sandbox_runtime_sandbox_owner_reference, supervisor_bootstrap_secret, supervisor_pod, workload_fence, }; @@ -104,6 +107,16 @@ const ANNOTATION_SANDBOX_RUNTIME_NETWORK_POLICY_UID: &str = "openshell.ai/sandbox-runtime-network-policy-uid"; const ANNOTATION_SANDBOX_RUNTIME_NETWORK_POLICY_GENERATION: &str = "openshell.ai/sandbox-runtime-network-policy-generation"; +/// Connection mode of the current runtime generation. Absent on legacy +/// generations, which remain supervisor-initiated until they are replaced. +const ANNOTATION_SANDBOX_RUNTIME_CONTROL_CONNECTION: &str = + "openshell.ai/sandbox-runtime-control-connection"; +const ANNOTATION_SANDBOX_RUNTIME_CONTROL_GENERATION: &str = + "openshell.ai/sandbox-runtime-control-generation"; +const ANNOTATION_SANDBOX_RUNTIME_CONTROL_POLICY_UID: &str = + "openshell.ai/sandbox-runtime-control-policy-uid"; +const ANNOTATION_SANDBOX_RUNTIME_CONTROL_POLICY_GENERATION: &str = + "openshell.ai/sandbox-runtime-control-policy-generation"; #[derive(Clone, Copy, Debug, PartialEq, Eq)] enum SandboxRuntimeBootstrapPhase { @@ -138,13 +151,71 @@ fn boundary_service_authority( namespace: &str, names: &SandboxRuntimeNames, boundary_port: u16, + mode: KubernetesControlConnection, ) -> String { format!( - "{}.{}.svc:{boundary_port}", - names.boundary_service, namespace + "{}.{namespace}.svc:{boundary_port}", + names.control_service(mode), ) } +/// Read the current generation's mode without migrating a running workload. +fn sandbox_runtime_control_connection( + object: &DynamicObject, +) -> Result { + let Some(annotations) = object.metadata.annotations.as_ref() else { + return Ok(KubernetesControlConnection::SupervisorInitiated); + }; + let Some(value) = annotations.get(ANNOTATION_SANDBOX_RUNTIME_CONTROL_CONNECTION) else { + return Ok(KubernetesControlConnection::SupervisorInitiated); + }; + let mode = value.parse().map_err(KubernetesDriverError::Precondition)?; + // Old gateways rotate the runtime generation but preserve unknown + // annotations. Their replacement uses legacy control, not this stale mode. + if annotations + .get(ANNOTATION_SANDBOX_RUNTIME_CONTROL_GENERATION) + .is_none_or(|generation| { + Some(generation) != annotations.get(ANNOTATION_SANDBOX_RUNTIME_GENERATION) + }) + { + return Ok(KubernetesControlConnection::SupervisorInitiated); + } + Ok(mode) +} + +/// Operator-provided bootstrap bytes read once per runtime generation. +struct StagedBootstrapMaterial { + upstream_proxy_ca_bundle: Option>, + client_tls: Option, +} + +/// Stored coordinates of a validated runtime-initiated control policy. +struct VerifiedControlPolicy { + uid: String, + resource_version: String, + generation: i64, +} + +impl VerifiedControlPolicy { + fn from_policy(policy: &NetworkPolicy) -> Result { + let missing = |field: &str| { + KubernetesDriverError::Message(format!("control NetworkPolicy has no {field}")) + }; + Ok(Self { + uid: policy.metadata.uid.clone().ok_or_else(|| missing("UID"))?, + resource_version: policy + .metadata + .resource_version + .clone() + .ok_or_else(|| missing("resourceVersion"))?, + generation: policy + .metadata + .generation + .ok_or_else(|| missing("generation"))?, + }) + } +} + #[derive(Debug, thiserror::Error)] pub enum KubernetesDriverError { #[error("sandbox already exists")] @@ -2120,6 +2191,15 @@ impl KubernetesComputeDriver { ANNOTATION_SANDBOX_RUNTIME_LOG_LEVEL.to_string(), log_level.clone(), ); + let control_connection = KubernetesControlConnection::RuntimeInitiated; + annotations.insert( + ANNOTATION_SANDBOX_RUNTIME_CONTROL_CONNECTION.to_string(), + control_connection.as_str().to_string(), + ); + annotations.insert( + ANNOTATION_SANDBOX_RUNTIME_CONTROL_GENERATION.to_string(), + generation.clone(), + ); for key in [ crate::config::ANNOTATION_SCC_UID_RANGE, crate::config::ANNOTATION_SCC_SUPPLEMENTAL_GROUPS, @@ -2181,7 +2261,7 @@ impl KubernetesComputeDriver { &agent_sandbox_api, &created, &proxy_names, - &generation, + control_connection, resolved_user_id, resolved_group_id, &main_process_spec, @@ -2202,6 +2282,156 @@ impl KubernetesComputeDriver { Ok(runtime_identity) } + /// Create or validate this sandbox's control egress policy and return the + /// stored coordinates bound into its boundary evidence. + async fn ensure_sandbox_runtime_control_policy( + &self, + namespace: &str, + names: &SandboxRuntimeNames, + sandbox_id: &str, + owner: OwnerReference, + ) -> Result { + let policies: Api = Api::namespaced(self.client.clone(), namespace); + let expected = control_policy( + namespace, + names, + sandbox_id, + self.config.sandbox_runtime.boundary_port, + owner, + ); + create_or_validate_sandbox_runtime_fence(&policies, &expected).await?; + // Bind the coordinates of the object that was actually validated. + let stored = tokio::time::timeout(KUBE_API_TIMEOUT, policies.get(&names.control_policy)) + .await + .map_err(|_| { + KubernetesDriverError::Message( + "timed out reading sandbox-runtime control NetworkPolicy".to_string(), + ) + })? + .map_err(KubernetesDriverError::from_kube)?; + validate_sandbox_runtime_fence(&stored, &expected)?; + VerifiedControlPolicy::from_policy(&stored) + } + + fn sandbox_runtime_control_path( + &self, + namespace: &str, + names: &SandboxRuntimeNames, + service_ip: std::net::IpAddr, + control_policy: Option, + ) -> KubernetesControlPath { + let port = self.config.sandbox_runtime.boundary_port; + let unspecified = std::net::SocketAddr::new( + if service_ip.is_ipv6() { + std::net::IpAddr::V6(std::net::Ipv6Addr::UNSPECIFIED) + } else { + std::net::IpAddr::V4(std::net::Ipv4Addr::UNSPECIFIED) + }, + port, + ); + let mode = if control_policy.is_some() { + KubernetesControlConnection::RuntimeInitiated + } else { + KubernetesControlConnection::SupervisorInitiated + }; + let control_authority = boundary_service_authority(namespace, names, port, mode); + let service_address = std::net::SocketAddr::new(service_ip, port); + match control_policy { + None => KubernetesControlPath::SupervisorInitiated { + boundary_listener: unspecified, + control_authority, + control_address: service_address, + }, + Some(policy) => KubernetesControlPath::RuntimeInitiated { + control_authority, + supervisor_address: service_address, + supervisor_listen: unspecified, + control_policy_uid: policy.uid, + control_policy_resource_version: policy.resource_version, + }, + } + } + + /// Fail closed when a sandbox's boundary Service or control policy no + /// longer matches the mode recorded for its generation. This is the reconcile + /// pass's only Service read; a missing Service is drift once the sandbox + /// should be running. Returns `Precondition` for drift that must suspend. + async fn verify_sandbox_runtime_control_path( + &self, + object: &DynamicObject, + sandbox_id: &str, + namespace: &str, + require_service: bool, + ) -> Result<(), KubernetesDriverError> { + let mode = sandbox_runtime_control_connection(object)?; + let names = SandboxRuntimeNames::new(sandbox_id); + let service_name = names.control_service(mode); + let services = Api::::namespaced(self.client.clone(), namespace); + let service = tokio::time::timeout(KUBE_API_TIMEOUT, services.get_opt(service_name)) + .await + .map_err(|_| { + KubernetesDriverError::Message( + "timed out reading sandbox-runtime boundary Service".to_string(), + ) + })? + .map_err(KubernetesDriverError::from_kube)?; + match service { + None if require_service => { + return Err(KubernetesDriverError::Precondition(format!( + "sandbox-runtime boundary Service {service_name} is missing and its allocated address cannot be safely reconstructed", + ))); + } + Some(service) if !boundary_service_matches_mode(&service, sandbox_id, mode) => { + return Err(KubernetesDriverError::Precondition(format!( + "boundary Service {} no longer selects the {} listener", + service_name, + mode.as_str() + ))); + } + _ => {} + } + if mode == KubernetesControlConnection::SupervisorInitiated { + return Ok(()); + } + let policies = Api::::namespaced(self.client.clone(), namespace); + let stored = + tokio::time::timeout(KUBE_API_TIMEOUT, policies.get_opt(&names.control_policy)) + .await + .map_err(|_| { + KubernetesDriverError::Message( + "timed out reading sandbox-runtime control NetworkPolicy".to_string(), + ) + })? + .map_err(KubernetesDriverError::from_kube)? + .ok_or_else(|| { + KubernetesDriverError::Precondition(format!( + "control NetworkPolicy {} is missing", + names.control_policy + )) + })?; + // Ownership is not part of enforcement; compare the spec and labels. + let expected = control_policy( + namespace, + &names, + sandbox_id, + self.config.sandbox_runtime.boundary_port, + OwnerReference::default(), + ); + validate_sandbox_runtime_fence(&stored, &expected)?; + if !sandbox_runtime_policy_generation_matches( + &stored, + object, + ANNOTATION_SANDBOX_RUNTIME_CONTROL_POLICY_UID, + ANNOTATION_SANDBOX_RUNTIME_CONTROL_POLICY_GENERATION, + ) { + return Err(KubernetesDriverError::Precondition(format!( + "control NetworkPolicy {} changed after it was bound", + names.control_policy + ))); + } + Ok(()) + } + async fn create_sandbox_runtime_fence( &self, namespace: &str, @@ -2481,126 +2711,61 @@ impl KubernetesComputeDriver { sandbox_api: &AgentSandboxApi, sandbox_cr: &DynamicObject, names: &SandboxRuntimeNames, - _generation: &str, + control_connection: KubernetesControlConnection, agent_uid: u32, agent_gid: u32, main_process_spec: &str, log_level: &str, ) -> Result { - // Read once per call so the whole generation stages, mounts, and - // references the same bytes, and so a bundle that became unreadable - // fails before any Service, Pod, or Secret is created. - let upstream_proxy_ca_bundle = - read_staged_upstream_proxy_ca_bundle(self.config.proxy_ca_bundle.as_deref()).await?; - let client_tls = self.read_client_tls_material().await?; + let staged = self.read_staged_bootstrap_material().await?; + let launch_authentication = sandbox + .spec + .as_ref() + .filter(|spec| !spec.launch_authentication.is_empty()) + .ok_or_else(|| { + KubernetesDriverError::Precondition( + "Kubernetes sandbox launch authentication is required".to_string(), + ) + }) + .and_then(|spec| decode_launch_authentication(&spec.launch_authentication))?; let cr_uid = sandbox_cr.metadata.uid.as_deref().ok_or_else(|| { KubernetesDriverError::Message("created Sandbox CR has no UID".to_string()) })?; - let namespace_uid = Api::::all(self.client.clone()) - .get(namespace) - .await - .map_err(KubernetesDriverError::from_kube)? - .metadata - .uid - .ok_or_else(|| { - KubernetesDriverError::Message("sandbox namespace has no UID".to_string()) - })?; let dependent_owner = sandbox_runtime_sandbox_owner_reference( cr_name, cr_uid, &sandbox_api.resource.api_version, false, ); - let services: Api = Api::namespaced(self.client.clone(), namespace); - let service = services - .create( - &PostParams::default(), - &boundary_service( - namespace, - names, - &sandbox.id, - self.config.sandbox_runtime.boundary_port, - dependent_owner.clone(), - ), - ) - .await - .map_err(KubernetesDriverError::from_kube)?; - let service_ip: std::net::IpAddr = service - .spec - .and_then(|spec| spec.cluster_ip) - .filter(|ip| ip != "None") - .ok_or_else(|| { - KubernetesDriverError::Message("boundary Service has no ClusterIP".to_string()) - })? - .parse() - .map_err(|error| { - KubernetesDriverError::Message(format!( - "invalid boundary Service ClusterIP: {error}" - )) - })?; - - let pods: Api = Api::namespaced(self.client.clone(), namespace); - let supervisor = pods - .create( - &PostParams::default(), - &supervisor_pod( - namespace, - names, - &sandbox.id, - &sandbox.name, - &self.config.gateway_id, - &self.config.supervisor_image, - self.config.supervisor_image_pull_policy, - &self.config.service_account_name, - agent_uid, - agent_gid, - &self.generation_image_pull_secret_names(names), - &self.config.grpc_endpoint, - self.config.supervisor_client_tls(), - main_process_spec, - log_level, - self.config.effective_sa_token_ttl_secs(), - self.config.https_proxy.as_deref(), - self.config.no_proxy.as_deref(), - self.config - .proxy_auth_secret_name - .as_deref() - .zip(self.config.proxy_auth_secret_key.as_deref()), - self.config.proxy_auth_allow_insecure == Some(true), - self.config.proxy_connect_by_hostname == Some(true), - upstream_proxy_ca_bundle.is_some(), - self.config.provider_spiffe_enabled().then_some( - self.config - .provider_spiffe_workload_api_socket_path - .as_str(), - ), - dependent_owner.clone(), - &self.supervisor_otlp_environment(), - ) - .map_err(KubernetesDriverError::Message)?, + // Preserve the legacy workload endpoint even for newly created + // sandboxes: an old gateway may handle their next start mid-rollout. + for mode in [ + KubernetesControlConnection::SupervisorInitiated, + KubernetesControlConnection::RuntimeInitiated, + ] { + self.ensure_sandbox_runtime_service( + namespace, + names, + &sandbox.id, + mode, + dependent_owner.clone(), ) - .await - .map_err(KubernetesDriverError::from_kube)?; - let supervisor_uid = supervisor.metadata.uid.ok_or_else(|| { - KubernetesDriverError::Message("supervisor Pod has no UID".to_string()) - })?; - - let policies: Api = Api::namespaced(self.client.clone(), namespace); - let fence = policies - .get(&names.workload_policy) - .await - .map_err(KubernetesDriverError::from_kube)?; - let fence_uid = fence.metadata.uid.ok_or_else(|| { - KubernetesDriverError::Message("workload NetworkPolicy has no UID".to_string()) - })?; - let fence_generation = fence.metadata.generation.ok_or_else(|| { - KubernetesDriverError::Message("workload NetworkPolicy has no generation".to_string()) - })?; - let fence_resource_version = fence.metadata.resource_version.ok_or_else(|| { - KubernetesDriverError::Message( - "workload NetworkPolicy has no resourceVersion".to_string(), + .await?; + } + let supervisor_uid = self + .create_sandbox_runtime_supervisor( + namespace, + names, + &sandbox.id, + &sandbox.name, + dependent_owner, + agent_uid, + agent_gid, + main_process_spec, + log_level, + staged.upstream_proxy_ca_bundle.is_some(), ) - })?; + .await?; // Release the Sandbox CR only far enough for the controller to create // the workload Pod. The Pod remains unschedulable because its template @@ -2611,57 +2776,6 @@ impl KubernetesComputeDriver { }) .await?; - let workload_pod = self - .wait_for_bootstrap_workload_pod(&pods, cr_name, cr_uid) - .await?; - let admission_object = sandbox_api - .api - .get(cr_name) - .await - .map_err(KubernetesDriverError::from_kube)?; - self.admit_stored_resources(&admission_object) - .await - .map_err(admission_error)?; - Self::validate_capability_free_workload_pod( - &workload_pod, - cr_uid, - &sandbox.id, - agent_uid, - agent_gid, - &names.sandbox_secret, - )?; - let workload_pod_uid = - workload_pod.metadata.uid.clone().ok_or_else(|| { - KubernetesDriverError::Message("workload Pod has no UID".to_string()) - })?; - let workload_pod_name = workload_pod.metadata.name.clone().ok_or_else(|| { - KubernetesDriverError::Message("workload Pod has no name".to_string()) - })?; - patch_dynamic_object_with_resource_version_retry(&sandbox_api.api, cr_name, |version| { - serde_json::json!({ - "metadata": { - "resourceVersion": version, - "annotations": { - ANNOTATION_SANDBOX_RUNTIME_WORKLOAD_UID: workload_pod_uid.clone(), - ANNOTATION_SANDBOX_RUNTIME_SUPERVISOR_UID: supervisor_uid.clone(), - ANNOTATION_SANDBOX_RUNTIME_NETWORK_POLICY_UID: fence_uid.clone(), - ANNOTATION_SANDBOX_RUNTIME_NETWORK_POLICY_GENERATION: fence_generation.to_string(), - } - } - }) - }) - .await?; - - let launch_authentication = sandbox - .spec - .as_ref() - .filter(|spec| !spec.launch_authentication.is_empty()) - .ok_or_else(|| { - KubernetesDriverError::Precondition( - "Kubernetes sandbox launch authentication is required".to_string(), - ) - }) - .and_then(|spec| decode_launch_authentication(&spec.launch_authentication))?; let mut child_env = sandbox .spec .as_ref() @@ -2673,144 +2787,22 @@ impl KubernetesComputeDriver { child_env.extend(spec.environment.clone()); } child_env.retain(|name, _| !name.starts_with("OPENSHELL_")); - let host_gateway_ip = self.config.host_gateway_ip.parse().ok(); - let session_id = launch_authentication.supervisor.session_id; - let tls = generate_sandbox_tls_material(session_id) - .map_err(|error| KubernetesDriverError::Message(error.to_string()))?; - let verification_keys = - gateway_verification_keys(&launch_authentication.verification_keys)?; - let proxy_ca = generate_proxy_ca_material().map_err(KubernetesDriverError::Message)?; - let workload_identity = ResolvedWorkloadIdentity::new( - agent_uid, - agent_gid, - Vec::new(), - "kubernetes-config".to_string(), - format!("sandbox:{cr_uid}"), - ) - .map_err(|error| KubernetesDriverError::Message(error.to_string()))?; - let provisioned = KubernetesSandboxRuntimeBoundarySpec { - boundary_id: sandbox.id.clone(), - generation: launch_authentication - .supervisor - .runtime_generation - .to_string(), - session_id, - session_rotation: launch_authentication.supervisor.session_rotation, - auth_epoch: launch_authentication.supervisor.auth_epoch, - gateway_id: launch_authentication.gateway_id, - verification_keys, - namespace_uid, - sandbox_resource_uid: cr_uid.to_string(), - workload_pod_uid: workload_pod_uid.clone(), - workload_pod_uid_path: PathBuf::from(SANDBOX_POD_UID_PATH), - supervisor_pod_uid: supervisor_uid.clone(), - egress_policy_uid: fence_uid.clone(), - egress_policy_resource_version: fence_resource_version.clone(), - boundary_listener: std::net::SocketAddr::new( - if service_ip.is_ipv6() { - std::net::IpAddr::V6(std::net::Ipv6Addr::UNSPECIFIED) - } else { - std::net::IpAddr::V4(std::net::Ipv4Addr::UNSPECIFIED) - }, - self.config.sandbox_runtime.boundary_port, - ), - control_authority: boundary_service_authority( - namespace, - names, - self.config.sandbox_runtime.boundary_port, - ), - control_address: std::net::SocketAddr::new( - service_ip, - self.config.sandbox_runtime.boundary_port, - ), - sandbox_tls: SandboxTlsServerConfig { - certificate_chain_path: PathBuf::from(BOUNDARY_CERTIFICATE_PATH), - private_key_path: PathBuf::from(BOUNDARY_PRIVATE_KEY_PATH), - }, - supervisor_tls: SandboxTlsClientConfig { - server_name: tls.server_name.clone(), - trust_anchor_pem: tls.trust_anchor_pem.clone(), - }, - host_gateway_ip, - workload_identity, - child_env, - } - .provision() - .map_err(|error| KubernetesDriverError::Message(error.to_string()))?; - let descriptor = provisioned - .runtime_descriptor - .backend_descriptor() - .map_err(|error| KubernetesDriverError::Message(error.to_string()))?; - let workload_owner = runtime_pod_owner(&workload_pod_name, &workload_pod_uid); - let supervisor_owner = runtime_pod_owner(&names.supervisor_pod, &supervisor_uid); - let sandbox_secret = sandbox_bootstrap_secret( + self.install_sandbox_runtime_generation( namespace, - names, - &sandbox.id, - provisioned - .boundary_config - .encode() - .map_err(|error| KubernetesDriverError::Message(error.to_string()))?, - tls.certificate_chain_pem.into_bytes(), - tls.private_key_pem.into_bytes(), - workload_owner.clone(), - ); - let supervisor_secret = supervisor_bootstrap_secret( - namespace, - names, + cr_name, + sandbox_api, &sandbox.id, - descriptor.payload, - serde_json::to_vec(&launch_authentication.supervisor).map_err(|error| { - KubernetesDriverError::Message(format!("encode supervisor auth bundle: {error}")) - })?, - proxy_ca.certificate_pem.into_bytes(), - proxy_ca.private_key_pem.into_bytes(), - upstream_proxy_ca_bundle.clone(), - client_tls, - supervisor_owner.clone(), - ); - let secrets = Api::::namespaced(self.client.clone(), namespace); - secrets - .create(&PostParams::default(), &sandbox_secret) - .await - .map_err(KubernetesDriverError::from_kube)?; - secrets - .create(&PostParams::default(), &supervisor_secret) - .await - .map_err(KubernetesDriverError::from_kube)?; - self.create_generation_image_pull_secrets( - namespace, + cr_uid, names, - &sandbox.id, - vec![workload_owner, supervisor_owner], - ) - .await?; - - pods.patch( - &names.supervisor_pod, - &PatchParams::default(), - &Patch::Merge(&serde_json::json!({"spec": {"schedulingGates": []}})), - ) - .await - .map_err(KubernetesDriverError::from_kube)?; - pods.patch( - &workload_pod_name, - &PatchParams::default(), - &Patch::Merge(&serde_json::json!({"spec": {"schedulingGates": []}})), + control_connection, + &supervisor_uid, + agent_uid, + agent_gid, + child_env, + &launch_authentication, + staged, ) - .await - .map_err(KubernetesDriverError::from_kube)?; - patch_dynamic_object_with_resource_version_retry(&sandbox_api.api, cr_name, |version| { - sandbox_runtime_bootstrap_phase_patch(version, SandboxRuntimeBootstrapPhase::Released) - }) .await?; - spawn_sandbox_runtime_bootstrap_completion( - pods.clone(), - sandbox_api.api.clone(), - names.supervisor_pod.clone(), - cr_name.to_string(), - Some(cr_uid.to_string()), - ); // Return while the CR remains explicitly bootstrapping. The gateway // can now commit the sandbox configuration required by a policy-less // control process without deadlocking behind this driver call. Only @@ -2824,6 +2816,118 @@ impl KubernetesComputeDriver { )) } + /// Add an endpoint without changing the destination of an existing one. + async fn ensure_sandbox_runtime_service( + &self, + namespace: &str, + names: &SandboxRuntimeNames, + sandbox_id: &str, + mode: KubernetesControlConnection, + owner: OwnerReference, + ) -> Result<(), KubernetesDriverError> { + let services: Api = Api::namespaced(self.client.clone(), namespace); + let expected = boundary_service( + namespace, + names, + sandbox_id, + self.config.sandbox_runtime.boundary_port, + mode, + owner, + ); + let service = match services.create(&PostParams::default(), &expected).await { + Ok(service) => service, + Err(kube::Error::Api(error)) if error.code == 409 => services + .get(names.control_service(mode)) + .await + .map_err(KubernetesDriverError::from_kube)?, + Err(error) => return Err(KubernetesDriverError::from_kube(error)), + }; + if !boundary_service_matches_mode(&service, sandbox_id, mode) { + return Err(KubernetesDriverError::Precondition(format!( + "control Service {} does not select the {} listener", + names.control_service(mode), + mode.as_str(), + ))); + } + Ok(()) + } + + /// Read operator-provided bootstrap material once per generation, so the + /// whole generation stages, mounts, and references the same bytes, and so + /// unreadable material fails before any Service, Pod, or Secret is created. + async fn read_staged_bootstrap_material( + &self, + ) -> Result { + Ok(StagedBootstrapMaterial { + upstream_proxy_ca_bundle: read_staged_upstream_proxy_ca_bundle( + self.config.proxy_ca_bundle.as_deref(), + ) + .await?, + client_tls: self.read_client_tls_material().await?, + }) + } + + /// Create one generation's gated supervisor Pod and return its UID. + #[allow(clippy::too_many_arguments, clippy::similar_names)] + async fn create_sandbox_runtime_supervisor( + &self, + namespace: &str, + names: &SandboxRuntimeNames, + sandbox_id: &str, + sandbox_name: &str, + owner: OwnerReference, + agent_uid: u32, + agent_gid: u32, + main_process_spec: &str, + log_level: &str, + upstream_proxy_ca_bundle_staged: bool, + ) -> Result { + let pods: Api = Api::namespaced(self.client.clone(), namespace); + pods.create( + &PostParams::default(), + &supervisor_pod( + namespace, + names, + sandbox_id, + sandbox_name, + &self.config.gateway_id, + &self.config.supervisor_image, + self.config.supervisor_image_pull_policy, + &self.config.service_account_name, + agent_uid, + agent_gid, + &self.generation_image_pull_secret_names(names), + &self.config.grpc_endpoint, + self.config.supervisor_client_tls(), + main_process_spec, + log_level, + self.config.effective_sa_token_ttl_secs(), + self.config.https_proxy.as_deref(), + self.config.no_proxy.as_deref(), + self.config + .proxy_auth_secret_name + .as_deref() + .zip(self.config.proxy_auth_secret_key.as_deref()), + self.config.proxy_auth_allow_insecure == Some(true), + self.config.proxy_connect_by_hostname == Some(true), + upstream_proxy_ca_bundle_staged, + self.config.provider_spiffe_enabled().then_some( + self.config + .provider_spiffe_workload_api_socket_path + .as_str(), + ), + owner, + &self.supervisor_otlp_environment(), + ) + .map_err(KubernetesDriverError::Message)?, + ) + .await + .map_err(KubernetesDriverError::from_kube)? + .metadata + .uid + .ok_or_else(|| KubernetesDriverError::Message("supervisor Pod has no UID".to_string())) + } + #[allow(clippy::too_many_arguments, clippy::similar_names)] async fn install_sandbox_runtime_generation( &self, @@ -2833,19 +2937,14 @@ impl KubernetesComputeDriver { sandbox_id: &str, cr_uid: &str, names: &SandboxRuntimeNames, - _generation: &str, + control_connection: KubernetesControlConnection, supervisor_uid: &str, agent_uid: u32, agent_gid: u32, child_env: std::collections::HashMap, launch_authentication: &openshell_core::jwt::SandboxLaunchAuthentication, + staged: StagedBootstrapMaterial, ) -> Result<(), KubernetesDriverError> { - // Read once per call so the whole generation stages, mounts, and - // references the same bytes, and so a bundle that became unreadable - // fails before any Service, Pod, or Secret is created. - let upstream_proxy_ca_bundle = - read_staged_upstream_proxy_ca_bundle(self.config.proxy_ca_bundle.as_deref()).await?; - let client_tls = self.read_client_tls_material().await?; let namespace_uid = Api::::all(self.client.clone()) .get(namespace) .await @@ -2856,10 +2955,20 @@ impl KubernetesComputeDriver { KubernetesDriverError::Message("sandbox namespace has no UID".to_string()) })?; let services = Api::::namespaced(self.client.clone(), namespace); - let service_ip: std::net::IpAddr = services - .get(&names.boundary_service) + let service_name = names.control_service(control_connection); + let service = services + .get(service_name) .await - .map_err(KubernetesDriverError::from_kube)? + .map_err(KubernetesDriverError::from_kube)?; + // Running generations must retain the Service role prepared for them. + if !boundary_service_matches_mode(&service, sandbox_id, control_connection) { + return Err(KubernetesDriverError::Precondition(format!( + "boundary Service {} does not select the {} listener recorded for this sandbox", + service_name, + control_connection.as_str() + ))); + } + let service_ip: std::net::IpAddr = service .spec .and_then(|spec| spec.cluster_ip) .filter(|ip| ip != "None") @@ -2888,6 +2997,23 @@ impl KubernetesComputeDriver { "workload NetworkPolicy has no resourceVersion".to_string(), ) })?; + let control_policy = match control_connection { + KubernetesControlConnection::SupervisorInitiated => None, + KubernetesControlConnection::RuntimeInitiated => Some( + self.ensure_sandbox_runtime_control_policy( + namespace, + names, + sandbox_id, + sandbox_runtime_sandbox_owner_reference( + cr_name, + cr_uid, + &sandbox_api.resource.api_version, + false, + ), + ) + .await?, + ), + }; let pods = Api::::namespaced(self.client.clone(), namespace); let workload_pod = self @@ -2916,17 +3042,20 @@ impl KubernetesComputeDriver { let workload_pod_name = workload_pod.metadata.name.clone().ok_or_else(|| { KubernetesDriverError::Message("workload Pod has no name".to_string()) })?; + let mut annotations = serde_json::json!({ + ANNOTATION_SANDBOX_RUNTIME_WORKLOAD_UID: workload_pod_uid.clone(), + ANNOTATION_SANDBOX_RUNTIME_SUPERVISOR_UID: supervisor_uid, + ANNOTATION_SANDBOX_RUNTIME_NETWORK_POLICY_UID: fence_uid.clone(), + ANNOTATION_SANDBOX_RUNTIME_NETWORK_POLICY_GENERATION: fence_generation.to_string(), + }); + if let Some(policy) = &control_policy { + annotations[ANNOTATION_SANDBOX_RUNTIME_CONTROL_POLICY_UID] = policy.uid.clone().into(); + annotations[ANNOTATION_SANDBOX_RUNTIME_CONTROL_POLICY_GENERATION] = + policy.generation.to_string().into(); + } patch_dynamic_object_with_resource_version_retry(&sandbox_api.api, cr_name, |version| { serde_json::json!({ - "metadata": { - "resourceVersion": version, - "annotations": { - ANNOTATION_SANDBOX_RUNTIME_WORKLOAD_UID: workload_pod_uid.clone(), - ANNOTATION_SANDBOX_RUNTIME_SUPERVISOR_UID: supervisor_uid, - ANNOTATION_SANDBOX_RUNTIME_NETWORK_POLICY_UID: fence_uid.clone(), - ANNOTATION_SANDBOX_RUNTIME_NETWORK_POLICY_GENERATION: fence_generation.to_string(), - } - } + "metadata": {"resourceVersion": version, "annotations": annotations.clone()} }) }) .await?; @@ -2963,22 +3092,11 @@ impl KubernetesComputeDriver { supervisor_pod_uid: supervisor_uid.to_string(), egress_policy_uid: fence_uid.clone(), egress_policy_resource_version: fence_resource_version.clone(), - boundary_listener: std::net::SocketAddr::new( - if service_ip.is_ipv6() { - std::net::IpAddr::V6(std::net::Ipv6Addr::UNSPECIFIED) - } else { - std::net::IpAddr::V4(std::net::Ipv4Addr::UNSPECIFIED) - }, - self.config.sandbox_runtime.boundary_port, - ), - control_authority: boundary_service_authority( + control_path: self.sandbox_runtime_control_path( namespace, names, - self.config.sandbox_runtime.boundary_port, - ), - control_address: std::net::SocketAddr::new( service_ip, - self.config.sandbox_runtime.boundary_port, + control_policy, ), sandbox_tls: SandboxTlsServerConfig { certificate_chain_path: PathBuf::from(BOUNDARY_CERTIFICATE_PATH), @@ -3022,8 +3140,8 @@ impl KubernetesComputeDriver { })?, proxy_ca.certificate_pem.into_bytes(), proxy_ca.private_key_pem.into_bytes(), - upstream_proxy_ca_bundle.clone(), - client_tls, + staged.upstream_proxy_ca_bundle, + staged.client_tls, supervisor_owner.clone(), ); let secrets = Api::::namespaced(self.client.clone(), namespace); @@ -3201,11 +3319,7 @@ impl KubernetesComputeDriver { encoded_authentication: &[u8], encoded_expected_runtime_identity: &str, ) -> Result { - // Read once per call so the whole generation stages, mounts, and - // references the same bytes, and so a bundle that became unreadable - // fails before any Service, Pod, or Secret is created. - let upstream_proxy_ca_bundle = - read_staged_upstream_proxy_ca_bundle(self.config.proxy_ca_bundle.as_deref()).await?; + let staged = self.read_staged_bootstrap_material().await?; let generation = openshell_core::sandbox_generation::SandboxGenerationId::parse( encoded_generation.to_string(), ) @@ -3301,16 +3415,17 @@ impl KubernetesComputeDriver { &namespace, ); let pods = Api::::namespaced(self.client.clone(), &namespace); - if pods + let workload_exists = pods .get_opt(cr_name) .await .map_err(KubernetesDriverError::from_kube)? - .is_some() - { + .is_some(); + if workload_exists || sandbox_runtime_should_run(&object) { if !encoded_authentication.is_empty() { // Gateway restart creates a fresh in-memory launch session. - // Recreate both Pods so neither side retains credentials for - // the gateway instance that was replaced. + // Suspend the controller and recreate both Pods so neither + // side retains credentials for the replaced gateway. Suspend + // even if the old workload Pod disappeared before this call. self.stop_sandbox_inner(sandbox_id).await?; return Box::pin(self.start_sandbox_runtime_generation( sandbox_id, @@ -3325,6 +3440,7 @@ impl KubernetesComputeDriver { )); } + let control_connection = KubernetesControlConnection::RuntimeInitiated; let names = SandboxRuntimeNames::for_generation(sandbox_id, generation.as_str()); self.create_sandbox_runtime_fence(&namespace, &names) .await?; @@ -3332,6 +3448,36 @@ impl KubernetesComputeDriver { stale_generations.push(generation.as_str()); self.delete_sandbox_runtime_supervisor(sandbox_id, &namespace, &stale_generations) .await?; + // The legacy Service keeps selecting the workload for old gateways. + // Add the supervisor endpoint when upgrading a legacy sandbox. + let services = Api::::namespaced(self.client.clone(), &namespace); + let service = services + .get(&names.boundary_service) + .await + .map_err(KubernetesDriverError::from_kube)?; + if !boundary_service_matches_mode( + &service, + sandbox_id, + KubernetesControlConnection::SupervisorInitiated, + ) { + return Err(KubernetesDriverError::Precondition(format!( + "boundary Service {} no longer selects the workload", + names.boundary_service + ))); + } + self.ensure_sandbox_runtime_service( + &namespace, + &names, + sandbox_id, + control_connection, + sandbox_runtime_sandbox_owner_reference( + cr_name, + cr_uid, + &sandbox_api.resource.api_version, + false, + ), + ) + .await?; let main_process_spec = required_sandbox_annotation(&object, ANNOTATION_SANDBOX_RUNTIME_MAIN_PROCESS_SPEC)?; let log_level = required_sandbox_annotation(&object, ANNOTATION_SANDBOX_RUNTIME_LOG_LEVEL)?; @@ -3357,55 +3503,25 @@ impl KubernetesComputeDriver { agent_gid, ), )?; - let supervisor = pods - .create( - &PostParams::default(), - &supervisor_pod( - &namespace, - &names, - sandbox_id, - &sandbox_name, - &self.config.gateway_id, - &self.config.supervisor_image, - self.config.supervisor_image_pull_policy, - &self.config.service_account_name, - agent_uid, - agent_gid, - &self.generation_image_pull_secret_names(&names), - &self.config.grpc_endpoint, - self.config.supervisor_client_tls(), - &main_process_spec, - &log_level, - self.config.effective_sa_token_ttl_secs(), - self.config.https_proxy.as_deref(), - self.config.no_proxy.as_deref(), - self.config - .proxy_auth_secret_name - .as_deref() - .zip(self.config.proxy_auth_secret_key.as_deref()), - self.config.proxy_auth_allow_insecure == Some(true), - self.config.proxy_connect_by_hostname == Some(true), - upstream_proxy_ca_bundle.is_some(), - self.config.provider_spiffe_enabled().then_some( - self.config - .provider_spiffe_workload_api_socket_path - .as_str(), - ), - sandbox_runtime_sandbox_owner_reference( - cr_name, - cr_uid, - &sandbox_api.resource.api_version, - false, - ), - &self.supervisor_otlp_environment(), - ) - .map_err(KubernetesDriverError::Message)?, + let supervisor_uid = self + .create_sandbox_runtime_supervisor( + &namespace, + &names, + sandbox_id, + &sandbox_name, + sandbox_runtime_sandbox_owner_reference( + cr_name, + cr_uid, + &sandbox_api.resource.api_version, + false, + ), + agent_uid, + agent_gid, + &main_process_spec, + &log_level, + staged.upstream_proxy_ca_bundle.is_some(), ) - .await - .map_err(KubernetesDriverError::from_kube)?; - let supervisor_uid = supervisor.metadata.uid.ok_or_else(|| { - KubernetesDriverError::Message("replacement supervisor Pod has no UID".to_string()) - })?; + .await?; let mut volumes = object .data @@ -3446,6 +3562,8 @@ impl KubernetesComputeDriver { ANNOTATION_SANDBOX_RUNTIME_BOOTSTRAP_SESSION: launch_authentication.supervisor.session_id.to_string(), ANNOTATION_SANDBOX_RUNTIME_GENERATION: generation.as_str(), ANNOTATION_SANDBOX_RUNTIME_SUPERVISOR_UID: supervisor_uid.clone(), + ANNOTATION_SANDBOX_RUNTIME_CONTROL_CONNECTION: control_connection.as_str(), + ANNOTATION_SANDBOX_RUNTIME_CONTROL_GENERATION: generation.as_str(), }); running_patch["spec"]["podTemplate"]["spec"] = restart_pod_template_spec( &volumes, @@ -3465,12 +3583,13 @@ impl KubernetesComputeDriver { sandbox_id, cr_uid, &names, - generation.as_str(), + control_connection, &supervisor_uid, agent_uid, agent_gid, child_env, &launch_authentication, + staged, ) .await } @@ -3992,6 +4111,30 @@ impl KubernetesComputeDriver { .await; continue; } + if workload_may_run { + let require_service = !sandbox_runtime_bootstrap_in_progress(&object); + match self + .verify_sandbox_runtime_control_path( + &object, + &sandbox_id, + namespace, + require_service, + ) + .await + { + Ok(()) => {} + Err(KubernetesDriverError::Precondition(error)) => { + warn!(sandbox_id, %error, "sandbox-runtime control path drifted; suspending workload"); + self.suspend_sandbox_runtime_after_dependency_failure(&lookup_api, &object) + .await; + continue; + } + Err(error) => { + warn!(sandbox_id, %error, "could not verify sandbox-runtime control path; reconciliation will retry"); + continue; + } + } + } if sandbox_runtime_bootstrap_in_progress(&object) { if sandbox_runtime_control_availability(&self.client, namespace, &sandbox_id).await == SandboxRuntimeControlAvailability::Available @@ -4395,24 +4538,8 @@ impl KubernetesComputeDriver { .delete_sandbox_runtime_supervisor(sandbox_id, namespace, generations) .await; } + // The boundary Service was checked by verify_sandbox_runtime_control_path. let names = SandboxRuntimeNames::new(sandbox_id); - let services = Api::::namespaced(self.client.clone(), namespace); - let service_exists = - tokio::time::timeout(KUBE_API_TIMEOUT, services.get_opt(&names.boundary_service)) - .await - .map_err(|_| { - KubernetesDriverError::Message( - "timed out reading sandbox-runtime boundary Service".to_string(), - ) - })? - .map_err(KubernetesDriverError::from_kube)? - .is_some(); - if !service_exists { - return Err(KubernetesDriverError::Precondition(format!( - "sandbox-runtime boundary Service {} is missing and its allocated address cannot be safely reconstructed", - names.boundary_service - ))); - } let pods = Api::::namespaced(self.client.clone(), namespace); tokio::time::timeout( KUBE_API_TIMEOUT, @@ -7118,7 +7245,7 @@ fn validate_sandbox_runtime_fence( } else { let name = expected.metadata.name.as_deref().unwrap_or_default(); Err(KubernetesDriverError::Precondition(format!( - "sandbox-runtime workload fence {name} exists but does not match the intended enforcement" + "sandbox-runtime NetworkPolicy {name} exists but does not match the intended enforcement" ))) } } @@ -7163,16 +7290,29 @@ fn sandbox_runtime_fence_matches(existing: &NetworkPolicy, expected: &NetworkPol fn sandbox_runtime_namespace_fence_generation_matches( policy: &NetworkPolicy, sandbox: &DynamicObject, +) -> bool { + sandbox_runtime_policy_generation_matches( + policy, + sandbox, + ANNOTATION_SANDBOX_RUNTIME_NETWORK_POLICY_UID, + ANNOTATION_SANDBOX_RUNTIME_NETWORK_POLICY_GENERATION, + ) +} + +/// Whether `policy` is the exact object and generation recorded on the +/// sandbox under the given annotation keys. +fn sandbox_runtime_policy_generation_matches( + policy: &NetworkPolicy, + sandbox: &DynamicObject, + uid_annotation: &str, + generation_annotation: &str, ) -> bool { let Some(annotations) = sandbox.metadata.annotations.as_ref() else { return false; }; - policy.metadata.uid.as_deref() - == annotations - .get(ANNOTATION_SANDBOX_RUNTIME_NETWORK_POLICY_UID) - .map(String::as_str) + policy.metadata.uid.as_deref() == annotations.get(uid_annotation).map(String::as_str) && annotations - .get(ANNOTATION_SANDBOX_RUNTIME_NETWORK_POLICY_GENERATION) + .get(generation_annotation) .and_then(|generation| generation.parse::().ok()) == policy.metadata.generation } @@ -8075,9 +8215,55 @@ mod tests { let names = SandboxRuntimeNames::new("sandbox-1"); assert_eq!( - boundary_service_authority("workspace-a", &names, 5500), + boundary_service_authority( + "workspace-a", + &names, + 5500, + KubernetesControlConnection::SupervisorInitiated, + ), "os-boundary-sandbox-1.workspace-a.svc:5500" ); + assert_eq!( + boundary_service_authority( + "workspace-a", + &names, + 5500, + KubernetesControlConnection::RuntimeInitiated, + ), + "os-supervisor-sandbox-1.workspace-a.svc:5500" + ); + } + + #[test] + fn control_mode_does_not_survive_an_old_gateways_generation_replacement() { + let mut object: DynamicObject = serde_json::from_value(serde_json::json!({ + "apiVersion": "agents.x-k8s.io/v1beta1", + "kind": "Sandbox", + "metadata": {"annotations": { + ANNOTATION_SANDBOX_RUNTIME_GENERATION: "new-generation", + ANNOTATION_SANDBOX_RUNTIME_CONTROL_GENERATION: "new-generation", + ANNOTATION_SANDBOX_RUNTIME_CONTROL_CONNECTION: "runtime-initiated", + }}, + })) + .unwrap(); + assert_eq!( + sandbox_runtime_control_connection(&object).unwrap(), + KubernetesControlConnection::RuntimeInitiated + ); + let annotations = object.metadata.annotations.as_mut().unwrap(); + annotations.insert( + ANNOTATION_SANDBOX_RUNTIME_GENERATION.into(), + "old-gateway-generation".into(), + ); + assert_eq!( + sandbox_runtime_control_connection(&object).unwrap(), + KubernetesControlConnection::SupervisorInitiated + ); + object.metadata.annotations.as_mut().unwrap().insert( + ANNOTATION_SANDBOX_RUNTIME_CONTROL_CONNECTION.into(), + "unknown".into(), + ); + assert!(sandbox_runtime_control_connection(&object).is_err()); } #[tokio::test] diff --git a/crates/openshell-driver-kubernetes/src/isolation.rs b/crates/openshell-driver-kubernetes/src/isolation.rs index f4ae89b38b..c9da7f3e3a 100644 --- a/crates/openshell-driver-kubernetes/src/isolation.rs +++ b/crates/openshell-driver-kubernetes/src/isolation.rs @@ -29,6 +29,8 @@ use openshell_sandbox_backend::boundary_protocol::{ }; use serde::Serialize; +use crate::config::KubernetesControlConnection; + #[derive(Serialize)] struct KubernetesOuterFenceEvidence<'a> { network_policy_uid: &'a str, @@ -36,6 +38,29 @@ struct KubernetesOuterFenceEvidence<'a> { ingress_isolated: bool, egress_isolated: bool, egress_rule_count: u32, + /// Present only for runtime-initiated control. Omitted otherwise so the + /// supervisor-initiated evidence encoding and digest are unchanged. + #[serde(skip_serializing_if = "Option::is_none")] + runtime_control_policy: Option>, +} + +/// The stored per-pair control policy bound into this generation. The driver +/// supplies it only after verifying the stored object equals +/// [`runtime_control_policy`] for this pair and port, so its only added path +/// is the runtime's connection to its own supervisor. +#[derive(Serialize)] +struct KubernetesControlPolicyEvidence<'a> { + network_policy_uid: &'a str, + network_policy_resource_version: &'a str, + port: u16, +} + +impl KubernetesControlPolicyEvidence<'_> { + fn is_bound(&self) -> bool { + !self.network_policy_uid.is_empty() + && !self.network_policy_resource_version.is_empty() + && self.port != 0 + } } impl KubernetesOuterFenceEvidence<'_> { @@ -50,6 +75,15 @@ impl KubernetesOuterFenceEvidence<'_> { "Kubernetes provisioning evidence is incomplete".to_string(), )); } + if self + .runtime_control_policy + .as_ref() + .is_some_and(|control| !control.is_bound()) + { + return Err(BackendError::Descriptor( + "Kubernetes control policy evidence is incomplete".into(), + )); + } let encoded = serde_json::to_vec(self).map_err(|error| { BackendError::Descriptor(format!("encode Kubernetes outer fence evidence: {error}")) })?; @@ -174,6 +208,101 @@ fn role_labels(role: &str) -> BTreeMap { BTreeMap::from([(BOUNDARY_ROLE_LABEL.to_string(), role.to_string())]) } +/// Labels selecting one role of one workload/supervisor pair. +#[must_use] +pub fn pair_role_labels(pair: &str, role: &str) -> BTreeMap { + BTreeMap::from([ + (BOUNDARY_PAIR_LABEL.to_string(), pair.to_string()), + (BOUNDARY_ROLE_LABEL.to_string(), role.to_string()), + ]) +} + +/// Pod role the per-sandbox boundary Service selects for a connection mode. +#[must_use] +pub const fn boundary_service_role(mode: KubernetesControlConnection) -> &'static str { + match mode { + KubernetesControlConnection::SupervisorInitiated => WORKLOAD_ROLE, + KubernetesControlConnection::RuntimeInitiated => SUPERVISOR_ROLE, + } +} + +/// Per-sandbox allowance for runtime-initiated control. +/// +/// Policies are additive: this admits exactly one flow, from this pair's +/// workload Pod to this pair's supervisor Pod on the control port, at both +/// enforcement points. Egress is required because the namespace fence denies +/// all workload egress; ingress is required because supervisors may be +/// ingress-isolated (for example by the managed SSH ingress policy). The +/// selector matches only this pair, so the extra rules can name only the pair +/// itself. It never adds DNS, gateway, API-server, or address-block access and +/// leaves the namespace-wide policies untouched. +#[must_use] +pub fn runtime_control_policy(namespace: &str, name: &str, pair: &str, port: u16) -> NetworkPolicy { + let pair_role = |role| { + Some(vec![NetworkPolicyPeer { + pod_selector: Some(LabelSelector { + match_labels: Some(pair_role_labels(pair, role)), + ..Default::default() + }), + ..Default::default() + }]) + }; + let control_port = || { + Some(vec![NetworkPolicyPort { + port: Some(IntOrString::Int(i32::from(port))), + protocol: Some("TCP".to_string()), + ..Default::default() + }]) + }; + NetworkPolicy { + metadata: ObjectMeta { + name: Some(name.to_string()), + namespace: Some(namespace.to_string()), + ..Default::default() + }, + spec: Some(NetworkPolicySpec { + pod_selector: LabelSelector { + match_labels: Some(BTreeMap::from([( + BOUNDARY_PAIR_LABEL.to_string(), + pair.to_string(), + )])), + ..Default::default() + }, + policy_types: Some(vec!["Ingress".to_string(), "Egress".to_string()]), + ingress: Some(vec![NetworkPolicyIngressRule { + from: pair_role(WORKLOAD_ROLE), + ports: control_port(), + }]), + egress: Some(vec![NetworkPolicyEgressRule { + to: pair_role(SUPERVISOR_ROLE), + ports: control_port(), + }]), + }), + } +} + +/// Which side opens the Sandbox Protocol connection, with the verified +/// Kubernetes coordinates that mode needs. +pub enum KubernetesControlPath { + /// The supervisor dials the workload's listener through the Service. + SupervisorInitiated { + boundary_listener: SocketAddr, + control_authority: String, + control_address: SocketAddr, + }, + /// The workload runtime dials the supervisor through the Service. + RuntimeInitiated { + control_authority: String, + /// Service address the runtime dials. + supervisor_address: SocketAddr, + /// Supervisor bind address. + supervisor_listen: SocketAddr, + /// Verified per-sandbox control policy coordinates. + control_policy_uid: String, + control_policy_resource_version: String, + }, +} + /// Driver-owned inputs that bind one workload/supervisor pair to one boundary. /// /// The driver constructs this only after Kubernetes has assigned every UID and @@ -195,9 +324,7 @@ pub struct KubernetesSandboxRuntimeBoundarySpec { pub supervisor_pod_uid: String, pub egress_policy_uid: String, pub egress_policy_resource_version: String, - pub boundary_listener: SocketAddr, - pub control_authority: String, - pub control_address: SocketAddr, + pub control_path: KubernetesControlPath, pub sandbox_tls: SandboxTlsServerConfig, pub supervisor_tls: SandboxTlsClientConfig, pub host_gateway_ip: Option, @@ -215,7 +342,7 @@ impl KubernetesSandboxRuntimeBoundarySpec { /// Produce both sides of the common protocol from one observed Kubernetes /// resource set so a stale or recreated object cannot be attached. pub fn provision(self) -> Result { - let resource_claims = BTreeMap::from([ + let mut resource_claims = BTreeMap::from([ ("kubernetes.namespace_uid".to_string(), self.namespace_uid), ( "kubernetes.sandbox_resource_uid".to_string(), @@ -238,6 +365,55 @@ impl KubernetesSandboxRuntimeBoundarySpec { self.egress_policy_resource_version, ), ]); + let (listener, transport) = match self.control_path { + KubernetesControlPath::SupervisorInitiated { + boundary_listener, + control_authority, + control_address, + } => ( + BoundaryListener::TlsTcp { + address: boundary_listener, + tls: self.sandbox_tls, + }, + SandboxTransport::Tcp { + authority: control_authority, + addresses: vec![control_address], + }, + ), + KubernetesControlPath::RuntimeInitiated { + control_authority, + supervisor_address, + supervisor_listen, + control_policy_uid, + control_policy_resource_version, + } => { + resource_claims.insert( + "kubernetes.control_policy_uid".to_string(), + control_policy_uid, + ); + resource_claims.insert( + "kubernetes.control_policy_resource_version".to_string(), + control_policy_resource_version, + ); + ( + BoundaryListener::RuntimeInitiatedTlsTcp { + authority: control_authority.clone(), + addresses: vec![supervisor_address], + tls: self.sandbox_tls, + }, + SandboxTransport::RuntimeInitiatedTcp { + authority: control_authority, + listen_address: supervisor_listen, + }, + ) + } + }; + let control_port = match &transport { + SandboxTransport::RuntimeInitiatedTcp { listen_address, .. } => { + Some(listen_address.port()) + } + _ => None, + }; let provisioning_evidence = KubernetesOuterFenceEvidence { network_policy_uid: &resource_claims["kubernetes.egress_policy_uid"], network_policy_resource_version: &resource_claims @@ -245,6 +421,12 @@ impl KubernetesSandboxRuntimeBoundarySpec { ingress_isolated: true, egress_isolated: true, egress_rule_count: 0, + runtime_control_policy: control_port.map(|port| KubernetesControlPolicyEvidence { + network_policy_uid: &resource_claims["kubernetes.control_policy_uid"], + network_policy_resource_version: &resource_claims + ["kubernetes.control_policy_resource_version"], + port, + }), } .project(&self.generation)?; Ok(KubernetesSandboxRuntimeBoundaryProvisioning { @@ -256,10 +438,7 @@ impl KubernetesSandboxRuntimeBoundarySpec { auth_epoch: self.auth_epoch, gateway_id: self.gateway_id, verification_keys: self.verification_keys, - listener: BoundaryListener::TlsTcp { - address: self.boundary_listener, - tls: self.sandbox_tls, - }, + listener, resource_claims: resource_claims.clone(), resource_claim_files: BTreeMap::from([( "kubernetes.workload_pod_uid".to_string(), @@ -274,10 +453,7 @@ impl KubernetesSandboxRuntimeBoundarySpec { generation: self.generation, session_id: self.session_id, workload_identity: self.workload_identity, - transport: SandboxTransport::Tcp { - authority: self.control_authority, - addresses: vec![self.control_address], - }, + transport, tls: self.supervisor_tls, host_gateway_ip: self.host_gateway_ip, resource_claims, @@ -300,6 +476,7 @@ mod tests { ingress_isolated: true, egress_isolated: true, egress_rule_count: 0, + runtime_control_policy: None, }, KubernetesOuterFenceEvidence { network_policy_uid: "uid", @@ -307,6 +484,7 @@ mod tests { ingress_isolated: true, egress_isolated: true, egress_rule_count: 0, + runtime_control_policy: None, }, KubernetesOuterFenceEvidence { network_policy_uid: "uid", @@ -314,6 +492,7 @@ mod tests { ingress_isolated: false, egress_isolated: true, egress_rule_count: 0, + runtime_control_policy: None, }, KubernetesOuterFenceEvidence { network_policy_uid: "uid", @@ -321,6 +500,7 @@ mod tests { ingress_isolated: true, egress_isolated: false, egress_rule_count: 0, + runtime_control_policy: None, }, KubernetesOuterFenceEvidence { network_policy_uid: "uid", @@ -328,6 +508,7 @@ mod tests { ingress_isolated: true, egress_isolated: true, egress_rule_count: 1, + runtime_control_policy: None, }, ] { assert!(evidence.project("generation-1").is_err()); @@ -353,9 +534,11 @@ mod tests { supervisor_pod_uid: "supervisor-pod-uid".to_string(), egress_policy_uid: "network-policy-uid".to_string(), egress_policy_resource_version: "1945".to_string(), - boundary_listener: "0.0.0.0:5500".parse().expect("valid listener"), - control_authority: "os-boundary-sandbox.default.svc:5500".to_string(), - control_address: "10.42.0.7:5500".parse().expect("valid target"), + control_path: KubernetesControlPath::SupervisorInitiated { + boundary_listener: "0.0.0.0:5500".parse().expect("valid listener"), + control_authority: "os-boundary-sandbox.default.svc:5500".to_string(), + control_address: "10.42.0.7:5500".parse().expect("valid target"), + }, sandbox_tls: SandboxTlsServerConfig { certificate_chain_path: PathBuf::from("/run/boundary/tls.crt"), private_key_path: PathBuf::from("/run/boundary/tls.key"), diff --git a/crates/openshell-driver-kubernetes/src/lib.rs b/crates/openshell-driver-kubernetes/src/lib.rs index aa980db44f..da17212a39 100644 --- a/crates/openshell-driver-kubernetes/src/lib.rs +++ b/crates/openshell-driver-kubernetes/src/lib.rs @@ -12,8 +12,9 @@ mod sandbox_runtime; pub use config::{ DEFAULT_GATEWAY_ID, DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME, DEFAULT_WORKSPACE_STORAGE_SIZE, - KubernetesComputeConfig, KubernetesImagePullPolicy, KubernetesSandboxRuntimeConfig, - ManagedSshIngressConfig, WorkspaceMode, managed_namespace_prefix, + KubernetesComputeConfig, KubernetesControlConnection, KubernetesImagePullPolicy, + KubernetesSandboxRuntimeConfig, ManagedSshIngressConfig, WorkspaceMode, + managed_namespace_prefix, }; pub use driver::{KubernetesComputeDriver, KubernetesDriverError}; pub use grpc::ComputeDriverService; diff --git a/crates/openshell-driver-kubernetes/src/lifecycle_tests.rs b/crates/openshell-driver-kubernetes/src/lifecycle_tests.rs index 9d70b4340e..16c8f82981 100644 --- a/crates/openshell-driver-kubernetes/src/lifecycle_tests.rs +++ b/crates/openshell-driver-kubernetes/src/lifecycle_tests.rs @@ -175,6 +175,23 @@ mod lifecycle_reconciliation { workload(), supervisor(), workload(), + // The recorded mode (none: supervisor-initiated) must still match + // the role the boundary Service selects. + ( + http::Method::GET, + "/api/v1/namespaces/openshell/services/os-boundary-sandbox-1", + kube_test_response( + http::StatusCode::OK, + serde_json::json!({ + "apiVersion": "v1", "kind": "Service", + "metadata": {"name": "os-boundary-sandbox-1"}, + "spec": {"selector": { + BOUNDARY_PAIR_LABEL: "sandbox-1", + BOUNDARY_ROLE_LABEL: "workload" + }} + }), + ), + ), ( http::Method::GET, "/api/v1/namespaces/openshell/pods/os-supervisor-sandbox-1", diff --git a/crates/openshell-driver-kubernetes/src/sandbox_runtime.rs b/crates/openshell-driver-kubernetes/src/sandbox_runtime.rs index d3bc1b27ec..fec98f9c93 100644 --- a/crates/openshell-driver-kubernetes/src/sandbox_runtime.rs +++ b/crates/openshell-driver-kubernetes/src/sandbox_runtime.rs @@ -19,9 +19,14 @@ use k8s_openapi::apimachinery::pkg::util::intstr::IntOrString; use kube::core::ObjectMeta; use rcgen::{CertificateParams, DnType, IsCa, KeyPair, KeyUsagePurpose}; +use k8s_openapi::api::networking::v1::NetworkPolicy; + +use crate::config::KubernetesControlConnection; +#[cfg(test)] +use crate::isolation::BOUNDARY_ROLE_LABEL; use crate::isolation::{ - BOUNDARY_PAIR_LABEL, BOUNDARY_ROLE_LABEL, KubernetesSandboxRuntimeNetworkFence, - KubernetesSandboxRuntimeNetworkFenceSpec, + KubernetesSandboxRuntimeNetworkFence, KubernetesSandboxRuntimeNetworkFenceSpec, + boundary_service_role, runtime_control_policy, }; pub const SANDBOX_SECRET_COMPONENT: &str = "sandbox-bootstrap"; @@ -108,6 +113,9 @@ pub struct SandboxRuntimeNames { pub sandbox_secret: String, pub supervisor_secret: String, pub boundary_service: String, + pub supervisor_service: String, + /// Per-sandbox egress allowance used only by runtime-initiated control. + pub control_policy: String, pub supervisor_pod: String, pub workload_policy: String, pub supervisor_policy: String, @@ -122,6 +130,8 @@ impl SandboxRuntimeNames { sandbox_secret: format!("os-sandbox-{suffix}"), supervisor_secret: format!("os-supervisor-{suffix}"), boundary_service: format!("os-boundary-{suffix}"), + supervisor_service: format!("os-supervisor-{suffix}"), + control_policy: format!("os-control-{suffix}"), supervisor_pod: format!("os-supervisor-{suffix}"), workload_policy: NAMESPACE_WORKLOAD_POLICY_NAME.to_string(), supervisor_policy: NAMESPACE_SUPERVISOR_EGRESS_POLICY_NAME.to_string(), @@ -129,6 +139,17 @@ impl SandboxRuntimeNames { } } + /// The legacy boundary Service always selects the workload. New runtimes + /// dial a separate supervisor Service, so old gateways can still restart + /// the pair without understanding runtime-initiated control. + #[must_use] + pub fn control_service(&self, mode: KubernetesControlConnection) -> &str { + match mode { + KubernetesControlConnection::SupervisorInitiated => &self.boundary_service, + KubernetesControlConnection::RuntimeInitiated => &self.supervisor_service, + } + } + /// Return stable companion names plus generation-specific immutable /// bootstrap Secret names. #[must_use] @@ -176,24 +197,28 @@ pub fn workload_fence( .provision() } +/// Per-sandbox Sandbox Protocol endpoint. Its stable `ClusterIP` is allocated +/// before either Pod runs. Each connection mode has a separate Service name +/// and fixed selector, so legacy gateways keep their workload endpoint. #[must_use] pub fn boundary_service( namespace: &str, names: &SandboxRuntimeNames, sandbox_id: &str, boundary_port: u16, + mode: KubernetesControlConnection, owner: OwnerReference, ) -> Service { Service { metadata: ObjectMeta { - name: Some(names.boundary_service.clone()), + name: Some(names.control_service(mode).to_string()), namespace: Some(namespace.to_string()), owner_references: Some(vec![owner]), labels: Some(common_labels(sandbox_id, "boundary-service")), ..Default::default() }, spec: Some(ServiceSpec { - selector: Some(pair_labels(sandbox_id, "workload")), + selector: Some(pair_labels(sandbox_id, boundary_service_role(mode))), ports: Some(vec![ServicePort { name: Some("boundary".to_string()), protocol: Some("TCP".to_string()), @@ -201,12 +226,61 @@ pub fn boundary_service( target_port: Some(IntOrString::Int(i32::from(boundary_port))), ..Default::default() }]), + publish_not_ready_addresses: publishes_not_ready_supervisor(mode), ..Default::default() }), ..Default::default() } } +/// A runtime-initiated supervisor becomes Ready only after the runtime has +/// attached through this Service, so the Service must route to it before it is +/// Ready. TLS and the Sandbox Protocol bearer, not readiness, authenticate the +/// endpoint. Supervisor-initiated Services keep their original encoding. +const fn publishes_not_ready_supervisor(mode: KubernetesControlConnection) -> Option { + match mode { + KubernetesControlConnection::RuntimeInitiated => Some(true), + KubernetesControlConnection::SupervisorInitiated => None, + } +} + +/// Whether an existing boundary Service still selects the role required by +/// `mode` for this sandbox's pair. +#[must_use] +pub fn boundary_service_matches_mode( + service: &Service, + sandbox_id: &str, + mode: KubernetesControlConnection, +) -> bool { + let Some(spec) = service.spec.as_ref() else { + return false; + }; + let publishes_not_ready = spec.publish_not_ready_addresses == Some(true); + spec.selector.as_ref() == Some(&pair_labels(sandbox_id, boundary_service_role(mode))) + && publishes_not_ready == (publishes_not_ready_supervisor(mode) == Some(true)) +} + +/// Per-sandbox control egress policy, owned by the Sandbox resource so it is +/// garbage-collected with that sandbox only. +#[must_use] +pub fn control_policy( + namespace: &str, + names: &SandboxRuntimeNames, + sandbox_id: &str, + boundary_port: u16, + owner: OwnerReference, +) -> NetworkPolicy { + let mut policy = runtime_control_policy( + namespace, + &names.control_policy, + &pair_label_value(sandbox_id), + boundary_port, + ); + policy.metadata.owner_references = Some(vec![owner]); + policy.metadata.labels = Some(common_labels(sandbox_id, "sandbox-control-egress")); + policy +} + #[allow(clippy::too_many_arguments, clippy::similar_names)] pub fn supervisor_pod( namespace: &str, @@ -632,13 +706,7 @@ pub fn sandbox_owner_reference( } fn pair_labels(sandbox_id: &str, role: &str) -> BTreeMap { - BTreeMap::from([ - ( - BOUNDARY_PAIR_LABEL.to_string(), - pair_label_value(sandbox_id), - ), - (BOUNDARY_ROLE_LABEL.to_string(), role.to_string()), - ]) + crate::isolation::pair_role_labels(&pair_label_value(sandbox_id), role) } fn common_labels(sandbox_id: &str, component: &str) -> BTreeMap { @@ -726,11 +794,42 @@ mod tests { #[test] fn service_selects_only_the_workload_boundary() { let names = SandboxRuntimeNames::new("4b67c0d0-1111-2222-3333-444444444444"); - let service = boundary_service("sandbox", &names, "pair", 5500, owner()); + let legacy = boundary_service( + "sandbox", + &names, + "pair", + 5500, + KubernetesControlConnection::SupervisorInitiated, + owner(), + ); assert_eq!( - service.spec.unwrap().selector.unwrap()[BOUNDARY_ROLE_LABEL], + legacy.spec.as_ref().unwrap().selector.as_ref().unwrap()[BOUNDARY_ROLE_LABEL], "workload" ); + let runtime_initiated = boundary_service( + "sandbox", + &names, + "pair", + 5500, + KubernetesControlConnection::RuntimeInitiated, + owner(), + ); + assert_eq!( + legacy.metadata.name.as_deref(), + Some(names.boundary_service.as_str()) + ); + assert_eq!( + runtime_initiated.metadata.name.as_deref(), + Some(names.supervisor_service.as_str()) + ); + assert_ne!(legacy.metadata.name, runtime_initiated.metadata.name); + assert_ne!( + legacy.spec.as_ref().unwrap().publish_not_ready_addresses, + Some(true) + ); + let spec = runtime_initiated.spec.unwrap(); + assert_eq!(spec.selector.unwrap()[BOUNDARY_ROLE_LABEL], "supervisor"); + assert_eq!(spec.publish_not_ready_addresses, Some(true)); } #[test] diff --git a/crates/openshell-driver-podman/README.md b/crates/openshell-driver-podman/README.md index f7ceed653b..f582cb03b3 100644 --- a/crates/openshell-driver-podman/README.md +++ b/crates/openshell-driver-podman/README.md @@ -7,6 +7,13 @@ REST API over a Unix socket. Each sandbox has two independent containers: - `openshell-supervisor` evaluates policy, holds gateway credentials, and proxies approved egress in a separate companion container. +New containers dial the supervisor over a separate private Unix socket volume +at `/.openshell/channel/control/control.sock`; workload networking remains +`none`. The supervisor's bootstrap-key mount stays read-only. Restart metadata +records the direction, with missing metadata retaining the legacy runtime +listener. Deploy matching supervisor and sandbox runtime binaries for new +containers. + The driver provisions placement, identity, credentials, transport, and lifecycle. The shared isolation interface supplies exec, attach, signal, terminate, binary identity, DNS, TCP, and loopback-forwarding semantics. diff --git a/crates/openshell-driver-podman/src/container.rs b/crates/openshell-driver-podman/src/container.rs index 9c3841d000..8e7a9f4228 100644 --- a/crates/openshell-driver-podman/src/container.rs +++ b/crates/openshell-driver-podman/src/container.rs @@ -1677,6 +1677,12 @@ pub fn build_isolation_specs( options: vec!["rw".into(), "z".into()], }); + workload.volumes.push(NamedVolume { + name: crate::isolation::control_volume_name(&input.sandbox.id), + dest: crate::isolation::CONTROL_ROOT.into(), + options: vec!["rw".into(), "z".into()], + }); + supervisor.name = crate::isolation::supervisor_name(&input.sandbox.id); supervisor .labels @@ -1732,6 +1738,11 @@ pub fn build_isolation_specs( dest: crate::isolation::CHANNEL_ROOT.into(), options: vec!["ro".into(), "z".into()], }]; + supervisor.volumes.push(NamedVolume { + name: crate::isolation::control_volume_name(&input.sandbox.id), + dest: crate::isolation::CONTROL_ROOT.into(), + options: vec!["rw".into(), "z".into()], + }); supervisor.mounts.retain(|mount| { trusted_mount(&mount.destination) && mount.destination != openshell_core::container_paths::NETNS_MOUNT_ROOT @@ -2061,7 +2072,7 @@ mod tests { assert_eq!(specs.supervisor.secrets.len(), 1); assert_eq!(specs.supervisor.secrets[0].source, "jwt"); assert_eq!(specs.supervisor.secrets[0].uid, 1000); - assert_eq!(specs.supervisor.volumes.len(), 1); + assert_eq!(specs.supervisor.volumes.len(), 2); assert!(specs.workload.volumes[0].options.contains(&"rw".into())); assert!(!specs.workload.volumes[0].options.contains(&"nocopy".into())); assert!(specs.supervisor.volumes[0].options.contains(&"ro".into())); diff --git a/crates/openshell-driver-podman/src/driver.rs b/crates/openshell-driver-podman/src/driver.rs index b6bbe6a36f..06d4d7b6fd 100644 --- a/crates/openshell-driver-podman/src/driver.rs +++ b/crates/openshell-driver-podman/src/driver.rs @@ -829,7 +829,9 @@ impl PodmanComputeDriver { return Err(missing()); } let workspace_volume = name == container::volume_name(sandbox_id); - if workspace_volume || name == crate::isolation::channel_volume_name(sandbox_id) + if workspace_volume + || name == crate::isolation::channel_volume_name(sandbox_id) + || name == crate::isolation::control_volume_name(sandbox_id) { let owned = volume.labels.as_ref().is_some_and(|labels| { labels.get(LABEL_SANDBOX_ID) == Some(sandbox_id) @@ -1064,6 +1066,7 @@ impl PodmanComputeDriver { .resolve_workload_identity(sandbox, &resolved_image.id, &resolved_image.oci_user) .await?; let channel_volume = crate::isolation::channel_volume_name(&sandbox.id); + let control_volume = crate::isolation::control_volume_name(&sandbox.id); let mut runtime_config = self.config.clone(); runtime_config.sandbox_runtime_image = self .client @@ -1157,6 +1160,7 @@ impl PodmanComputeDriver { let cleanup_created = || async { if channel_owned.load(std::sync::atomic::Ordering::Relaxed) { let _ = self.client.remove_volume(&channel_volume).await; + let _ = self.client.remove_volume(&control_volume).await; } let _ = self.client.remove_volume(&vol_name).await; cleanup_sandbox_resolver_secret(&self.client, &resolver_secret_name).await; @@ -1240,6 +1244,9 @@ impl PodmanComputeDriver { .create_owned_volume(&channel_volume, &sandbox.id, &sandbox.workspace, None) .await?; channel_owned.store(true, std::sync::atomic::Ordering::Relaxed); + self.client + .create_owned_volume(&control_volume, &sandbox.id, &sandbox.workspace, None) + .await?; let workload_id = self.client.create_typed_container(&specs.workload).await?; created_workload = Some(workload_id.clone()); self.client.verify_isolation_fence(&workload_id).await?; @@ -1260,6 +1267,7 @@ impl PodmanComputeDriver { let generation = uuid::Uuid::new_v4().to_string(); let archives = crate::isolation::bootstrap_archives( crate::isolation::BootstrapArchivesInput { + runtime_initiated: true, sandbox_id: &sandbox.id, container_id: &workload_id, generation: &generation, @@ -1280,6 +1288,15 @@ impl PodmanComputeDriver { archives.channel, ) .await?; + if let Some(control) = archives.control { + self.client + .copy_to_container( + &workload_id, + crate::isolation::CONTROL_ROOT, + control, + ) + .await?; + } if managed_workspace { self.client .copy_to_container(&workload_id, "/sandbox", archives.workspace) @@ -1611,6 +1628,7 @@ impl PodmanComputeDriver { let restart_metadata = crate::isolation::restart_metadata_from_slice(&bundle)?; let archives = crate::isolation::bootstrap_archives(crate::isolation::BootstrapArchivesInput { + runtime_initiated: restart_metadata.runtime_initiated, sandbox_id, container_id: &container_id, generation: generation.as_str(), @@ -1633,6 +1651,11 @@ impl PodmanComputeDriver { archives.channel, ) .await?; + if let Some(control) = archives.control { + self.client + .copy_to_container(&container_id, crate::isolation::CONTROL_ROOT, control) + .await?; + } self.client .copy_to_container(&supervisor, "/", archives.supervisor) .await?; @@ -1687,6 +1710,10 @@ impl PodmanComputeDriver { let Some(container_id) = self.find_container_id(sandbox_id).await? else { debug!(sandbox_id = %sandbox_id, "Sandbox container not found (already deleted)"); + let _ = self + .client + .remove_volume(&crate::isolation::control_volume_name(sandbox_id)) + .await; let vol = container::volume_name(sandbox_id); if let Err(e) = self.client.remove_volume(&vol).await { warn!(sandbox_id = %sandbox_id, volume = %vol, error = %e, "Failed to remove workspace volume"); @@ -1722,6 +1749,14 @@ impl PodmanComputeDriver { Err(e) => return Err(ComputeDriverError::from(e)), }; + if let Err(error) = self + .client + .remove_volume(&crate::isolation::control_volume_name(sandbox_id)) + .await + && !matches!(error, PodmanApiError::NotFound(_)) + { + warn!(%sandbox_id, %error, "Failed to remove private control volume"); + } // Remove workspace volume. if let Err(error) = self .client @@ -2527,6 +2562,7 @@ mod tests { uploads, [ "/libpod/containers/workload/archive?path=%2F.openshell%2Fchannel", + "/libpod/containers/workload/archive?path=%2F.openshell%2Fchannel%2Fcontrol", "/libpod/containers/workload/archive?path=%2Fsandbox", "/libpod/containers/supervisor/archive?path=%2F", ] @@ -3526,6 +3562,7 @@ mod tests { StubResponse::new(StatusCode::NO_CONTENT, ""), // remove channel if detached // list_containers returns empty (container already gone) StubResponse::new(StatusCode::OK, "[]"), + StubResponse::new(StatusCode::NO_CONTENT, ""), // control volume // remove_volume StubResponse::new(StatusCode::NO_CONTENT, ""), ], @@ -3545,7 +3582,7 @@ mod tests { .clone(); assert!(requests[2].contains("/libpod/containers/json")); assert_eq!( - requests[3], + requests[4], format!( "DELETE {}", api_path(&format!("/libpod/volumes/{volume_name}")) @@ -3662,6 +3699,7 @@ mod tests { ) .unwrap(); let bundle = serde_json::to_vec(&crate::isolation::RestartMetadata { + runtime_initiated: false, generation: "generation-1".to_string(), workload_identity: identity, child_env: HashMap::new(), @@ -3810,6 +3848,13 @@ mod tests { sandbox_id, None, )); + responses.push(StubResponse::new(StatusCode::NOT_FOUND, "")); + responses.push(StubResponse::new(StatusCode::CREATED, "{}")); + responses.push(owned_volume_response( + &crate::isolation::control_volume_name(sandbox_id), + sandbox_id, + None, + )); responses } @@ -3843,6 +3888,7 @@ mod tests { created_response("workload"), fence_response(), StubResponse::new(StatusCode::OK, "").with_archive_members(channel_archive_members()), + StubResponse::new(StatusCode::OK, "").with_archive_members(&["."]), // control volume StubResponse::new(StatusCode::OK, "").with_archive_members(&["."]), created_response("supervisor"), StubResponse::new(StatusCode::OK, ""), // supervisor archive @@ -3905,6 +3951,7 @@ mod tests { .chain([ StubResponse::new(StatusCode::INTERNAL_SERVER_ERROR, "create failed"), StubResponse::new(StatusCode::NO_CONTENT, ""), // channel + StubResponse::new(StatusCode::NO_CONTENT, ""), // control StubResponse::new(StatusCode::NO_CONTENT, ""), // workspace StubResponse::new(StatusCode::NO_CONTENT, ""), // resolver secret StubResponse::new(StatusCode::NO_CONTENT, ""), // proxy secret @@ -3950,12 +3997,13 @@ mod tests { "create-start-fail", create_setup_responses(true, sandbox_id) .into_iter() - .chain(create_launch_responses().into_iter().take(7)) + .chain(create_launch_responses().into_iter().take(8)) .chain([ StubResponse::new(StatusCode::INTERNAL_SERVER_ERROR, "supervisor start failed"), StubResponse::new(StatusCode::NO_CONTENT, ""), // supervisor StubResponse::new(StatusCode::NO_CONTENT, ""), // workload StubResponse::new(StatusCode::NO_CONTENT, ""), // channel + StubResponse::new(StatusCode::NO_CONTENT, ""), // control StubResponse::new(StatusCode::NO_CONTENT, ""), // workspace StubResponse::new(StatusCode::NO_CONTENT, ""), // resolver secret StubResponse::new(StatusCode::NO_CONTENT, ""), // proxy secret @@ -4001,7 +4049,8 @@ mod tests { vec![ StubResponse::new(StatusCode::NO_CONTENT, ""), // remove companion StubResponse::new(StatusCode::NO_CONTENT, ""), // remove channel if detached - StubResponse::new(StatusCode::OK, "[]"), // list_containers (not found) + StubResponse::new(StatusCode::OK, "[]"), + StubResponse::new(StatusCode::NO_CONTENT, ""), // control volume StubResponse::new(StatusCode::NO_CONTENT, ""), // remove volume StubResponse::new(StatusCode::NO_CONTENT, ""), // remove token secret StubResponse::new(StatusCode::NO_CONTENT, ""), // remove resolver secret @@ -4055,6 +4104,7 @@ mod tests { StubResponse::new(StatusCode::OK, list_body), // single timed remove_container operation StubResponse::new(StatusCode::NO_CONTENT, ""), + StubResponse::new(StatusCode::NO_CONTENT, ""), // control volume // channel volume, now detached StubResponse::new(StatusCode::NO_CONTENT, ""), // remove_volume @@ -4085,7 +4135,7 @@ mod tests { ) ); assert_eq!( - requests[5], + requests[6], format!( "DELETE {}", api_path(&format!("/libpod/volumes/{volume_name}")) diff --git a/crates/openshell-driver-podman/src/grpc.rs b/crates/openshell-driver-podman/src/grpc.rs index a6d208f707..25cfd09d7a 100644 --- a/crates/openshell-driver-podman/src/grpc.rs +++ b/crates/openshell-driver-podman/src/grpc.rs @@ -683,6 +683,7 @@ mod tests { StubResponse::new(StatusCode::NO_CONTENT, ""), // channel // list_containers returns empty (container already gone) StubResponse::new(StatusCode::OK, "[]"), + StubResponse::new(StatusCode::NO_CONTENT, ""), // control volume // remove_volume StubResponse::new(StatusCode::NO_CONTENT, ""), ], @@ -711,7 +712,7 @@ mod tests { .clone(); assert!(requests[2].contains("/libpod/containers/json")); assert_eq!( - requests[3], + requests[4], format!( "DELETE {}", api_path(&format!("/libpod/volumes/{volume_name}")) diff --git a/crates/openshell-driver-podman/src/isolation.rs b/crates/openshell-driver-podman/src/isolation.rs index 0ee226b446..6da06a5773 100644 --- a/crates/openshell-driver-podman/src/isolation.rs +++ b/crates/openshell-driver-podman/src/isolation.rs @@ -26,6 +26,8 @@ pub const BOOTSTRAP_PATH: &str = "/.openshell/channel/sandbox/bootstrap.json"; pub const RUNTIME_DESCRIPTOR_PATH: &str = "/.openshell/supervisor/runtime-descriptor.json"; pub const AUTH_BUNDLE_PATH: &str = "/.openshell/supervisor/auth.json"; pub const RESTART_METADATA_PATH: &str = "/.openshell/supervisor/restart-metadata.json"; +pub const CONTROL_ROOT: &str = "/.openshell/channel/control"; +const RUNTIME_INITIATED_SOCKET: &str = "/.openshell/channel/control/control.sock"; const SOCKET_PATH: &str = "/.openshell/channel/sandbox/control.sock"; #[derive(Serialize)] @@ -54,6 +56,9 @@ impl PodmanOuterFenceEvidence<'_> { pub fn supervisor_name(id: &str) -> String { format!("openshell-supervisor-{id}") } +pub fn control_volume_name(id: &str) -> String { + format!("openshell-control-{id}") +} pub fn channel_volume_name(id: &str) -> String { format!("openshell-channel-{id}") } @@ -171,6 +176,7 @@ pub fn resolve_identity( pub struct BootstrapArchives { pub channel: Vec, + pub control: Option>, pub workspace: Vec, pub supervisor: Vec, } @@ -178,11 +184,14 @@ pub struct BootstrapArchives { #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] pub struct RestartMetadata { pub(crate) generation: String, + #[serde(default)] + pub(crate) runtime_initiated: bool, pub(crate) workload_identity: ResolvedWorkloadIdentity, pub(crate) child_env: HashMap, } pub struct BootstrapArchivesInput<'a> { + pub runtime_initiated: bool, pub sandbox_id: &'a str, pub container_id: &'a str, pub generation: &'a str, @@ -199,6 +208,7 @@ pub fn bootstrap_archives( input: BootstrapArchivesInput<'_>, ) -> Result { let BootstrapArchivesInput { + runtime_initiated, sandbox_id, container_id, generation, @@ -255,12 +265,22 @@ pub fn bootstrap_archives( auth_epoch: launch_authentication.supervisor.auth_epoch, gateway_id: launch_authentication.gateway_id.clone(), verification_keys, - listener: BoundaryListener::Unix { - socket_path: PathBuf::from(SOCKET_PATH), - tls: SandboxTlsServerConfig { + listener: { + let tls = SandboxTlsServerConfig { certificate_chain_path: PathBuf::from("/.openshell/channel/sandbox/server.crt"), private_key_path: PathBuf::from("/.openshell/channel/sandbox/server.key"), - }, + }; + if runtime_initiated { + BoundaryListener::RuntimeInitiatedUnix { + socket_path: RUNTIME_INITIATED_SOCKET.into(), + tls, + } + } else { + BoundaryListener::Unix { + socket_path: SOCKET_PATH.into(), + tls, + } + } }, resource_claims: resource_claims.clone(), resource_claim_files: BTreeMap::new(), @@ -272,8 +292,14 @@ pub fn bootstrap_archives( boundary_id: sandbox_id.into(), generation: runtime_generation, session_id, - transport: SandboxTransport::Unix { - socket_path: PathBuf::from(SOCKET_PATH), + transport: if runtime_initiated { + SandboxTransport::RuntimeInitiatedUnix { + socket_path: RUNTIME_INITIATED_SOCKET.into(), + } + } else { + SandboxTransport::Unix { + socket_path: SOCKET_PATH.into(), + } }, tls: SandboxTlsClientConfig { server_name: tls.server_name, @@ -311,6 +337,7 @@ pub fn bootstrap_archives( &serde_json::to_vec(&launch_authentication.supervisor).map_err(invalid)?, )?; let restart_metadata = RestartMetadata { + runtime_initiated, generation: generation.to_string(), workload_identity: identity.clone(), child_env, @@ -319,7 +346,18 @@ pub fn bootstrap_archives( RESTART_METADATA_PATH, &serde_json::to_vec(&restart_metadata).map_err(invalid)?, )?; + let control = if runtime_initiated { + let mut archive = Archive::new(identity); + // The containers can use different rootless user-namespace mappings. + // Only this socket volume is writable; keys stay in the read-only + // bootstrap volume. The sticky bit protects existing socket inodes. + archive.directory(".", 0o1777, false)?; + Some(archive.finish()?) + } else { + None + }; Ok(BootstrapArchives { + control, channel, workspace: workspace.finish()?, supervisor: supervisor.finish()?, @@ -510,6 +548,7 @@ mod tests { let authentication = authentication(); let child_env = HashMap::from([("PATH".to_string(), "/agent/bin".to_string())]); let archives = bootstrap_archives(BootstrapArchivesInput { + runtime_initiated: false, sandbox_id: "sandbox", container_id: "container", generation: "generation-1", diff --git a/crates/openshell-driver-vm/README.md b/crates/openshell-driver-vm/README.md index fa05c48efc..deb4ca9dae 100644 --- a/crates/openshell-driver-vm/README.md +++ b/crates/openshell-driver-vm/README.md @@ -4,6 +4,14 @@ Standalone libkrun-backed [`ComputeDriver`](../../proto/compute_driver.proto) for OpenShell. The gateway spawns this binary as a subprocess and talks to it over the `openshell.compute.v1.ComputeDriver` Unix-socket surface. `openshell-supervisor` runs as a native host process, while `openshell-sandbox` runs as capability-free PID 1 inside each microVM and applies guest-local isolation over virtio-vsock. +New VM generations initiate the control connection to host CID 2 over vsock. +Libkrun forwards that guest connection to the supervisor's protected Unix +socket; QEMU uses a host vsock listener on a per-guest port and checks the guest +CID before TLS authentication. Existing protocol roles and authorization stay +unchanged. Running legacy generations retain their original descriptors; +newly booted generations require matching embedded runtime and supervisor +binaries. + The driver embeds libkrun, libkrunfw, the guest OCI unpacker, the portable guest sandbox, and the custom kernel runtime. Each sandbox boots from a cached immutable bootstrap ext4 root disk plus a per-sandbox writable overlay disk. When the requested sandbox image differs from the bootstrap image, the driver prepares a read-only image ext4 disk inside a bootstrap VM and mounts that unpacked rootfs as the sandbox lowerdir. ## How it fits together diff --git a/crates/openshell-driver-vm/src/driver.rs b/crates/openshell-driver-vm/src/driver.rs index d0309a2efd..7bd6f39a91 100644 --- a/crates/openshell-driver-vm/src/driver.rs +++ b/crates/openshell-driver-vm/src/driver.rs @@ -1633,14 +1633,20 @@ impl VmDriver { trust_anchor_pem: channel_tls.trust_anchor_pem.clone(), }; let transport = if plan.backend == VmBackend::Qemu { - SandboxTransport::Vsock { - guest_cid: plan.vsock_cid.ok_or_else(|| { - Status::internal("QEMU launch plan is missing a guest vsock CID") - })?, - port: VM_CONTROL_PORT, + let guest_cid = plan + .vsock_cid + .ok_or_else(|| Status::internal("QEMU launch plan is missing a guest vsock CID"))?; + SandboxTransport::RuntimeInitiatedVsock { + guest_cid, + // Each QEMU guest connects to a distinct host port. The CID + // allocator already guarantees uniqueness among live guests. + port: VM_CONTROL_PORT + .checked_add(guest_cid) + .filter(|port| *port != u32::MAX) + .ok_or_else(|| Status::resource_exhausted("VM control port space exhausted"))?, } } else { - SandboxTransport::Unix { + SandboxTransport::RuntimeInitiatedUnix { socket_path: control_socket.clone(), } }; @@ -1730,6 +1736,7 @@ impl VmDriver { } else { let _ = tokio::fs::remove_file(&control_socket).await; command + .arg("--vm-vsock-runtime-initiated") .arg("--vm-vsock-control-port") .arg(VM_CONTROL_PORT.to_string()) .arg("--vm-vsock-control-socket") @@ -7111,7 +7118,10 @@ fn inject_guest_boundary_bundle( let tls = match &config.listener { BoundaryListener::Unix { tls, .. } | BoundaryListener::TlsTcp { tls, .. } - | BoundaryListener::Vsock { tls, .. } => tls.clone(), + | BoundaryListener::Vsock { tls, .. } + | BoundaryListener::RuntimeInitiatedTlsTcp { tls, .. } + | BoundaryListener::RuntimeInitiatedUnix { tls, .. } + | BoundaryListener::RuntimeInitiatedVsock { tls, .. } => tls.clone(), }; let encoded_config = config .encode() diff --git a/crates/openshell-driver-vm/src/isolation/mod.rs b/crates/openshell-driver-vm/src/isolation/mod.rs index dc7426d229..f52456b50e 100644 --- a/crates/openshell-driver-vm/src/isolation/mod.rs +++ b/crates/openshell-driver-vm/src/isolation/mod.rs @@ -100,9 +100,23 @@ impl VmBoundarySpec { auth_epoch: self.auth_epoch, gateway_id: self.gateway_id, verification_keys: self.verification_keys, - listener: BoundaryListener::Vsock { - control_port: self.control_port, - tls: self.sandbox_tls, + listener: match &self.transport { + SandboxTransport::RuntimeInitiatedVsock { port, .. } => { + BoundaryListener::RuntimeInitiatedVsock { + host_port: *port, + tls: self.sandbox_tls, + } + } + SandboxTransport::RuntimeInitiatedUnix { .. } => { + BoundaryListener::RuntimeInitiatedVsock { + host_port: self.control_port, + tls: self.sandbox_tls, + } + } + _ => BoundaryListener::Vsock { + control_port: self.control_port, + tls: self.sandbox_tls, + }, }, resource_claims: resource_claims.clone(), resource_claim_files: BTreeMap::new(), diff --git a/crates/openshell-driver-vm/src/main.rs b/crates/openshell-driver-vm/src/main.rs index 796aad68d8..fb7c4accf2 100644 --- a/crates/openshell-driver-vm/src/main.rs +++ b/crates/openshell-driver-vm/src/main.rs @@ -263,6 +263,9 @@ struct Args { #[arg(long, hide = true)] vm_vsock_control_socket: Option, + #[arg(long, hide = true)] + vm_vsock_runtime_initiated: bool, + #[arg(long, hide = true)] vm_mount: Vec, } @@ -673,7 +676,7 @@ fn build_vm_launch_config(args: &Args) -> std::result::Result Some(VsockPortMap { guest_port, host_socket, - host_initiated: true, + host_initiated: !args.vm_vsock_runtime_initiated, }), (None, None) => None, _ => { diff --git a/crates/openshell-driver-vm/src/runtime.rs b/crates/openshell-driver-vm/src/runtime.rs index fbccba270a..ad300d98cf 100644 --- a/crates/openshell-driver-vm/src/runtime.rs +++ b/crates/openshell-driver-vm/src/runtime.rs @@ -345,7 +345,9 @@ fn run_libkrun_vm(config: &VmLaunchConfig) -> Result<(), String> { vm.disable_implicit_vsock()?; vm.add_vsock(0)?; if let Some(port_map) = &config.vsock_port_map { - let _ = std::fs::remove_file(&port_map.host_socket); + if port_map.host_initiated { + let _ = std::fs::remove_file(&port_map.host_socket); + } vm.add_vsock_port(port_map)?; } diff --git a/crates/openshell-sandbox-backend/Cargo.toml b/crates/openshell-sandbox-backend/Cargo.toml index 1f5908662d..15a0dd29f0 100644 --- a/crates/openshell-sandbox-backend/Cargo.toml +++ b/crates/openshell-sandbox-backend/Cargo.toml @@ -41,8 +41,6 @@ uuid = { workspace = true } [target.'cfg(unix)'.dependencies] libc = "0.2" - -[target.'cfg(target_os = "linux")'.dependencies] rustix = { workspace = true, features = ["fs", "process"] } [lints] diff --git a/crates/openshell-sandbox-backend/README.md b/crates/openshell-sandbox-backend/README.md index 534c4c4589..a63d52ebde 100644 --- a/crates/openshell-sandbox-backend/README.md +++ b/crates/openshell-sandbox-backend/README.md @@ -102,6 +102,29 @@ retain generation, session, resource, identity, and provisioning evidence checks ## Boundary transport establishment -The client connects to the configured Unix socket, TCP, or vsock endpoint. It -owns TLS peer verification, session bearer authentication, generation checks, -and reconnection handling for policy discovery and attachment. +`SandboxTransport` belongs to this backend's runtime descriptor. Core retains +only operator-registered endpoints; `DelegatedLaunch` converts those endpoints +into the backend's transport without changing their serialized representation. + +The supervisor chooses one `BoundaryTransport` per runtime descriptor and uses +clones of it for policy discovery and attachment. It dials the descriptor's +endpoint, uses a backend-supplied `BoundaryTransportConnector` for raw stream +establishment, or, for `SandboxTransport::RuntimeInitiatedTcp`, accepts the +connections the sandbox runtime opens. Every stream from every source completes +the same pinned TLS client handshake before it carries a request; the shared +client also keeps session bearer authentication, generation checks, and +reconnection handling. + +For runtime-initiated connections, the supervisor authenticates each accepted +socket before queueing it. Concurrent unauthenticated handshakes are bounded in +number and duration, and at most two authenticated spares wait for a client, so +a new connection cannot displace a live channel. The runtime keeps one spare +open, so reconnects and credential rotation never wait for a new dial. + +The new variants leave existing encodings unchanged. Binaries that predate them +reject runtime-initiated descriptors instead of falling back to another mode. + +Runtime-initiated transport supports TCP, protected Unix sockets, and host +vsock listeners. These share bounded TLS handshakes, the authenticated spare +queue, and the same generation-pinned runtime identity check. The legacy +supervisor-dialed transports remain available for existing descriptors. diff --git a/crates/openshell-sandbox-backend/src/boundary_protocol.rs b/crates/openshell-sandbox-backend/src/boundary_protocol.rs index 4bbcaca879..65527d5eb1 100644 --- a/crates/openshell-sandbox-backend/src/boundary_protocol.rs +++ b/crates/openshell-sandbox-backend/src/boundary_protocol.rs @@ -238,7 +238,88 @@ impl<'de> Deserialize<'de> for SupervisorInstanceId { #[error("supervisor instance ID must be a non-nil canonical lowercase UUID")] pub struct SupervisorInstanceIdError; -pub use openshell_core::isolation_registration::SandboxTransport; +/// Backend-owned transport for a driver-provisioned runtime. The serialized +/// variants preserve existing runtime descriptors independently of operator +/// endpoint registration in core. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(tag = "kind", rename_all = "kebab-case", deny_unknown_fields)] +pub enum SandboxTransport { + Unix { + socket_path: PathBuf, + }, + Tcp { + /// Logical HTTP authority used for routing. + authority: String, + /// Concrete connection candidates pinned by the operator or driver. + addresses: Vec, + }, + Vsock { + guest_cid: u32, + port: u32, + }, + /// The supervisor accepts a connection from its sandbox runtime. TLS and + /// gRPC roles stay unchanged. + RuntimeInitiatedTcp { + authority: String, + listen_address: std::net::SocketAddr, + }, + RuntimeInitiatedUnix { + socket_path: PathBuf, + }, + RuntimeInitiatedVsock { + guest_cid: u32, + port: u32, + }, +} + +impl SandboxTransport { + pub fn validate(&self) -> Result<(), &'static str> { + let valid = match self { + Self::Unix { socket_path } | Self::RuntimeInitiatedUnix { socket_path } => { + socket_path.is_absolute() + } + Self::Tcp { + authority, + addresses, + } => { + !authority.is_empty() + && !addresses.is_empty() + && addresses + .iter() + .all(|address| address.port() != 0 && !address.ip().is_unspecified()) + } + Self::RuntimeInitiatedTcp { + authority, + listen_address, + } => !authority.is_empty() && listen_address.port() != 0, + Self::Vsock { guest_cid, port } | Self::RuntimeInitiatedVsock { guest_cid, port } => { + *guest_cid >= 3 && *guest_cid != u32::MAX && *port != 0 && *port != u32::MAX + } + }; + if valid { + Ok(()) + } else { + Err("invalid boundary control transport endpoint") + } + } +} + +impl From for SandboxTransport { + fn from(endpoint: openshell_core::isolation_registration::SandboxTransport) -> Self { + use openshell_core::isolation_registration::SandboxTransport as RegisteredEndpoint; + match endpoint { + RegisteredEndpoint::Unix { socket_path } => Self::Unix { socket_path }, + RegisteredEndpoint::Tcp { + authority, + addresses, + } => Self::Tcp { + authority, + addresses, + }, + RegisteredEndpoint::Vsock { guest_cid, port } => Self::Vsock { guest_cid, port }, + } + } +} /// Supervisor-side, generation-pinned TLS server authentication. #[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] @@ -354,6 +435,25 @@ pub enum BoundaryListener { control_port: u32, tls: SandboxTlsServerConfig, }, + /// The boundary dials the supervisor's listener and serves TLS and gRPC + /// over that outbound connection. Only driver-resolved addresses are used. + RuntimeInitiatedTlsTcp { + /// Stable logical Service authority, for diagnostics. + authority: String, + /// Driver-resolved supervisor endpoints, tried in order. + addresses: Vec, + tls: SandboxTlsServerConfig, + }, + /// Dial the supervisor over the driver's protected shared Unix socket. + RuntimeInitiatedUnix { + socket_path: PathBuf, + tls: SandboxTlsServerConfig, + }, + /// Dial the host supervisor through the VM's vsock device. + RuntimeInitiatedVsock { + host_port: u32, + tls: SandboxTlsServerConfig, + }, } /// Public verification key staged in the sandbox's immutable auth bundle. diff --git a/crates/openshell-sandbox-backend/src/delegated.rs b/crates/openshell-sandbox-backend/src/delegated.rs index 349141e406..96e2047ecb 100644 --- a/crates/openshell-sandbox-backend/src/delegated.rs +++ b/crates/openshell-sandbox-backend/src/delegated.rs @@ -12,9 +12,7 @@ use openshell_isolation_interface::contract::{ }; use serde::{Deserialize, Serialize}; -use crate::boundary_protocol::{ - SandboxRuntimeDescriptor, SandboxTlsClientConfig, SandboxTransport, -}; +use crate::boundary_protocol::{SandboxRuntimeDescriptor, SandboxTlsClientConfig}; /// Delivered by the trusted driver separately from its opaque backend payload. /// Endpoint selection belongs to operator configuration, not launch data. @@ -37,7 +35,7 @@ impl DelegatedLaunch { /// Combine driver-owned identity and TLS with the operator-owned endpoint. pub fn runtime_descriptor( self, - transport: SandboxTransport, + transport: openshell_core::isolation_registration::SandboxTransport, ) -> Result { validate_backend_name(&self.backend_name)?; let descriptor = SandboxRuntimeDescriptor { @@ -45,7 +43,7 @@ impl DelegatedLaunch { generation: self.generation, session_id: self.session_id, workload_identity: self.workload_identity, - transport, + transport: transport.into(), tls: self.tls, host_gateway_ip: self.host_gateway_ip, resource_claims: self.resource_claims, diff --git a/crates/openshell-sandbox-backend/src/lib.rs b/crates/openshell-sandbox-backend/src/lib.rs index dd25b36f4b..e38e216d02 100644 --- a/crates/openshell-sandbox-backend/src/lib.rs +++ b/crates/openshell-sandbox-backend/src/lib.rs @@ -17,6 +17,7 @@ pub mod isolation_protocol; pub mod mediation; pub mod protobuf; mod runtime; +mod runtime_initiated; pub mod sandbox_auth; pub mod stream_adapter; @@ -26,6 +27,7 @@ pub use runtime::{ ConfirmedBoundary, ExecSession, NetworkMediationSource, OpenShellRuntimeBackend, ProcessAttachment, ReadyBoundary, RunningBoundary, }; +pub use runtime::{BoundaryTransport, BoundaryTransportConnector}; /// Stable isolation backend name implemented by `openshell-sandbox`. pub const BACKEND_NAME: &str = openshell_core::isolation_registration::BUILTIN_BACKEND_NAME; diff --git a/crates/openshell-sandbox-backend/src/runtime.rs b/crates/openshell-sandbox-backend/src/runtime.rs index bcda27830c..4bfa0c5a6f 100644 --- a/crates/openshell-sandbox-backend/src/runtime.rs +++ b/crates/openshell-sandbox-backend/src/runtime.rs @@ -72,17 +72,88 @@ fn begin_recovery_window( /// Internal raw transport establishment; shared code still authenticates /// the resulting stream with generation-pinned TLS and Sandbox Protocol JWTs. #[async_trait] -trait BoundaryTransportConnector: std::fmt::Debug + Send + Sync { +pub trait BoundaryTransportConnector: std::fmt::Debug + Send + Sync { async fn connect( &self, descriptor: &SandboxRuntimeDescriptor, ) -> Result; } -/// Concrete RPC client used for both in-tree and endpoint-registered backends. +/// How the supervisor obtains each physical Sandbox Protocol connection. +/// +/// Choose it once per runtime descriptor and share clones between policy +/// discovery and attachment. Every stream, whatever its source, completes the +/// descriptor's pinned TLS handshake in [`authenticate_boundary_stream`] +/// before it carries a request. +#[derive(Clone, Debug, Default)] +pub struct BoundaryTransport(TransportSource); + +#[derive(Clone, Debug, Default)] +enum TransportSource { + /// Dial the descriptor's endpoint. + #[default] + Dial, + /// Obtain raw streams from a backend-owned connector. + Connector(Arc), + /// Accept streams the sandbox runtime opens to this supervisor. + RuntimeInitiated(Arc), +} + +impl BoundaryTransport { + /// Select the transport the descriptor requires. A runtime-initiated + /// descriptor binds its control listener, so call this inside the + /// supervisor's Tokio runtime. + pub fn for_descriptor(descriptor: &SandboxRuntimeDescriptor) -> Result { + Ok(Self(match descriptor.transport { + SandboxTransport::RuntimeInitiatedTcp { .. } + | SandboxTransport::RuntimeInitiatedUnix { .. } + | SandboxTransport::RuntimeInitiatedVsock { .. } => TransportSource::RuntimeInitiated( + crate::runtime_initiated::RuntimeInitiatedListener::bind(descriptor)?, + ), + _ => TransportSource::Dial, + })) + } + + /// Establish raw streams through a backend-owned connector. + #[must_use] + pub fn connector(connector: Arc) -> Self { + Self(TransportSource::Connector(connector)) + } + + /// Open one authenticated physical connection, retrying within the + /// connect window while the remote boundary starts. + async fn connect( + &self, + descriptor: &SandboxRuntimeDescriptor, + ) -> Result { + let connector = match &self.0 { + TransportSource::Dial => None, + TransportSource::Connector(connector) => Some(connector.as_ref()), + TransportSource::RuntimeInitiated(listener) => { + return tokio::time::timeout(CONNECT_RETRY_TIMEOUT, listener.accept(descriptor)) + .await + .map_err(|_| { + BackendError::Unavailable( + "timed out waiting for the sandbox runtime to connect".to_string(), + ) + })?; + } + }; + let deadline = tokio::time::Instant::now() + CONNECT_RETRY_TIMEOUT; + loop { + match connect_boundary_using_connector(descriptor, connector).await { + Ok(stream) => return Ok(stream), + Err(error) if tokio::time::Instant::now() >= deadline => return Err(error), + Err(_) => tokio::time::sleep(Duration::from_millis(25)).await, + } + } + } +} + +/// Host-side `OpenShell` Sandbox Protocol implementation registered with the supervisor. #[derive(Debug)] pub struct OpenShellRuntimeBackend { - connector: Option>, + transport: BoundaryTransport, ca_file_paths: Arc>>, provider_credentials: openshell_core::provider_credentials::ProviderCredentialState, sandbox_bearer: openshell_core::jwt::SessionBearerTokenSlot, @@ -110,10 +181,11 @@ impl OpenShellRuntimeBackend { pub async fn discover_policy( descriptor: SandboxRuntimeDescriptor, bearer: openshell_core::jwt::SessionBearerTokenSlot, + transport: BoundaryTransport, ) -> Result<(Option, bool), BackendError> { - BoundaryClient::new(descriptor, bearer) - .discover_policy() - .await + let mut client = BoundaryClient::new(descriptor, bearer); + client.transport = transport; + client.discover_policy().await } pub fn new( @@ -122,7 +194,7 @@ impl OpenShellRuntimeBackend { sandbox_bearer: openshell_core::jwt::SessionBearerTokenSlot, ) -> Self { Self { - connector: None, + transport: BoundaryTransport::default(), ca_file_paths, provider_credentials, sandbox_bearer, @@ -164,6 +236,13 @@ impl OpenShellRuntimeBackend { client.discover_policy().await } + /// Use the transport selected for the descriptor this backend will attach. + #[must_use] + pub fn with_transport(mut self, transport: BoundaryTransport) -> Self { + self.transport = transport; + self + } + pub fn backend_name(&self) -> &str { self.delegated .as_ref() @@ -204,7 +283,7 @@ impl OpenShellRuntimeBackend { self.backend_name().to_string(), descriptor.payload.clone(), ); - client.connector = self.connector.clone(); + client.transport = self.transport.clone(); let client = Arc::new(client); let response = client .call_idempotent(Request::Attach { @@ -294,7 +373,7 @@ pub fn validate_runtime_coordinates( Ok(()) } -fn validate_client_tls(tls: &SandboxTlsClientConfig) -> Result<(), BackendError> { +pub fn validate_client_tls(tls: &SandboxTlsClientConfig) -> Result<(), BackendError> { rustls::pki_types::ServerName::try_from(tls.server_name.clone()).map_err(|error| { BackendError::Descriptor(format!( "boundary TLS server name {:?} is invalid: {error}", @@ -1058,7 +1137,7 @@ async fn dispatch_client_mediation_frame( } struct BoundaryClient { - connector: Option>, + transport: BoundaryTransport, runtime_descriptor: SandboxRuntimeDescriptor, backend_name: String, driver_descriptor: Vec, @@ -1250,7 +1329,7 @@ impl BoundaryClient { driver_descriptor: Vec, ) -> Self { Self { - connector: None, + transport: BoundaryTransport::default(), runtime_descriptor, backend_name, driver_descriptor, @@ -1880,14 +1959,16 @@ impl BoundaryClient { credential: &BoundaryCredential, ) -> Result<(tonic::transport::Channel, Arc), BackendError> { let runtime_descriptor = self.runtime_descriptor.clone(); - let connector = self.connector.clone(); + let source = self.transport.clone(); let transport = Arc::new(TransportAbort::default()); let connector_transport = transport.clone(); let authority = match &runtime_descriptor.transport { - SandboxTransport::Tcp { authority, .. } => authority.as_str(), - SandboxTransport::Unix { .. } | SandboxTransport::Vsock { .. } => { - "boundary.openshell.internal" - } + SandboxTransport::Tcp { authority, .. } + | SandboxTransport::RuntimeInitiatedTcp { authority, .. } => authority.as_str(), + SandboxTransport::Unix { .. } + | SandboxTransport::Vsock { .. } + | SandboxTransport::RuntimeInitiatedUnix { .. } + | SandboxTransport::RuntimeInitiatedVsock { .. } => "boundary.openshell.internal", }; let endpoint = tonic::transport::Endpoint::from_shared(format!("http://{authority}")) .map_err(|error| { @@ -1902,7 +1983,7 @@ impl BoundaryClient { let channel = endpoint .connect_with_connector(tower::service_fn(move |_: tonic::transport::Uri| { let runtime_descriptor = runtime_descriptor.clone(); - let connector = connector.clone(); + let source = source.clone(); let abort = connector_transport.clone(); async move { // A channel is one authenticated connection. Never let it @@ -1910,7 +1991,8 @@ impl BoundaryClient { if abort.is_aborted() || abort.connected.swap(true, Ordering::AcqRel) { return Err(AbortableTransport::aborted_error()); } - connect_boundary_with_retry(&runtime_descriptor, connector.clone()) + source + .connect(&runtime_descriptor) .await .map(|inner| TokioIo::new(AbortableTransport { inner, abort })) .map_err(|error| std::io::Error::other(error.to_string())) @@ -2004,20 +2086,6 @@ impl BoundaryClient { } } -async fn connect_boundary_with_retry( - runtime_descriptor: &SandboxRuntimeDescriptor, - connector: Option>, -) -> Result { - let deadline = tokio::time::Instant::now() + CONNECT_RETRY_TIMEOUT; - loop { - match connect_boundary_using_connector(runtime_descriptor, connector.as_deref()).await { - Ok(stream) => return Ok(stream), - Err(error) if tokio::time::Instant::now() >= deadline => return Err(error), - Err(_) => tokio::time::sleep(Duration::from_millis(25)).await, - } - } -} - #[cfg(test)] async fn connect_boundary_once( runtime_descriptor: &SandboxRuntimeDescriptor, @@ -2064,9 +2132,24 @@ async fn connect_boundary_using_connector( Box::new(stream) } SandboxTransport::Vsock { guest_cid, port } => connect_host_vsock(*guest_cid, *port)?, + SandboxTransport::RuntimeInitiatedTcp { .. } + | SandboxTransport::RuntimeInitiatedUnix { .. } + | SandboxTransport::RuntimeInitiatedVsock { .. } => { + return Err(BackendError::Descriptor( + "a runtime-initiated transport cannot be dialed".to_string(), + )); + } } }; - let tls = &runtime_descriptor.tls; + authenticate_boundary_stream(&runtime_descriptor.tls, stream).await +} + +/// The only Sandbox Protocol client handshake: verify the generation-pinned +/// server identity before a stream from any source can carry a request. +pub async fn authenticate_boundary_stream( + tls: &SandboxTlsClientConfig, + stream: BoundaryDuplexStream, +) -> Result { let server_name = rustls::pki_types::ServerName::try_from(tls.server_name.clone()).map_err(|error| { BackendError::Descriptor(format!( @@ -2227,7 +2310,7 @@ fn control_read_error(error: std::io::Error) -> BackendError { } } -fn enable_boundary_tcp_keepalive(stream: &tokio::net::TcpStream) { +pub fn enable_boundary_tcp_keepalive(stream: &tokio::net::TcpStream) { let keepalive = socket2::TcpKeepalive::new() .with_time(Duration::from_secs(30)) .with_interval(Duration::from_secs(10)); @@ -2528,6 +2611,7 @@ mod tests { let result = OpenShellRuntimeBackend::discover_policy( tls_runtime_descriptor(address, client_tls), test_bearer(&"a".repeat(32)), + BoundaryTransport::default(), ) .await; assert!( @@ -2570,6 +2654,7 @@ mod tests { let result = OpenShellRuntimeBackend::discover_policy( tls_runtime_descriptor(address, client_tls), test_bearer(&"a".repeat(32)), + BoundaryTransport::default(), ) .await; assert!( @@ -3574,7 +3659,7 @@ mod tests { tls_runtime_descriptor(address, certificate.client_tls), test_bearer(&"a".repeat(32)), ); - client.connector = Some(connector); + client.transport = BoundaryTransport::connector(connector); let response = tokio::time::timeout(Duration::from_secs(5), client.exchange(Request::Confirm)) .await diff --git a/crates/openshell-sandbox-backend/src/runtime/test_support.rs b/crates/openshell-sandbox-backend/src/runtime/test_support.rs index 464bcc83bf..8d29407300 100644 --- a/crates/openshell-sandbox-backend/src/runtime/test_support.rs +++ b/crates/openshell-sandbox-backend/src/runtime/test_support.rs @@ -723,7 +723,7 @@ fn client(peer: Peer) -> Arc { .unwrap(); let mut client = BoundaryClient::with_backend(descriptor, bearer, crate::BACKEND_NAME.into(), vec![]); - client.connector = Some(Arc::new(Connector { + client.transport = crate::BoundaryTransport::connector(Arc::new(Connector { peer, address, task, @@ -861,7 +861,7 @@ pub fn backend( ) .unwrap(); let mut backend = crate::OpenShellRuntimeBackend::new(ca, credentials, bearer); - backend.connector.clone_from(&peer.connector); + backend.transport.clone_from(&peer.transport); backend.delegated = Some(crate::runtime::DelegatedRuntime { backend_name: name, descriptor, diff --git a/crates/openshell-sandbox-backend/src/runtime_initiated.rs b/crates/openshell-sandbox-backend/src/runtime_initiated.rs new file mode 100644 index 0000000000..8f2bf94e2c --- /dev/null +++ b/crates/openshell-sandbox-backend/src/runtime_initiated.rs @@ -0,0 +1,309 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Supervisor listener for runtime-initiated Sandbox Protocol connections. +//! +//! Only connection establishment is reversed. Each accepted socket completes the +//! same pinned TLS client handshake as a dialed connection before it is queued, +//! and every request still carries the Sandbox Protocol bearer token. A peer +//! address is never trusted. + +use std::net::SocketAddr; +use std::time::Duration; + +use openshell_isolation_interface::contract::{BackendError, BoundaryDuplexStream}; +use tokio::sync::mpsc; + +use crate::boundary_protocol::{ + SandboxRuntimeDescriptor, SandboxTlsClientConfig, SandboxTransport, +}; + +/// Unauthenticated sockets may hold at most this many TLS handshakes. +/// Further sockets are closed immediately rather than queued. +const MAX_PENDING_HANDSHAKES: usize = 8; +/// Upper bound on the time an unauthenticated peer may occupy a slot. +const HANDSHAKE_TIMEOUT: Duration = Duration::from_secs(5); +/// Authenticated connections not yet claimed by a client. The runtime keeps +/// one spare open, so a small bound only absorbs reconnect races. +const MAX_QUEUED_CONNECTIONS: usize = 2; + +/// Accepts runtime-initiated connections for exactly one sandbox session. +/// +/// Live channels own their streams; the queue holds only unclaimed, +/// authenticated spares, so no new connection can displace a live owner. +/// Dropping the listener closes the socket, pending handshakes, and spares. +#[derive(Debug)] +pub struct RuntimeInitiatedListener { + tls: SandboxTlsClientConfig, + spares: tokio::sync::Mutex>, + accept_task: tokio::task::AbortHandle, +} + +impl RuntimeInitiatedListener { + /// Bind the descriptor's listen address and start authenticating peers. + pub fn bind( + descriptor: &SandboxRuntimeDescriptor, + ) -> Result, BackendError> { + let runtime = tokio::runtime::Handle::try_current().map_err(|_| { + BackendError::Unavailable("runtime-initiated listener requires a Tokio runtime".into()) + })?; + let _entered = runtime.enter(); + crate::runtime::validate_client_tls(&descriptor.tls)?; + descriptor + .transport + .validate() + .map_err(|error| BackendError::Descriptor(error.into()))?; + let listener = match &descriptor.transport { + SandboxTransport::RuntimeInitiatedTcp { listen_address, .. } => { + Listener::Tcp(bind_tcp(*listen_address).map_err(unavailable)?) + } + #[cfg(unix)] + SandboxTransport::RuntimeInitiatedUnix { socket_path } => { + Listener::bind_unix(socket_path).map_err(unavailable)? + } + #[cfg(target_os = "linux")] + SandboxTransport::RuntimeInitiatedVsock { guest_cid, port } => { + Listener::bind_vsock(*guest_cid, *port).map_err(unavailable)? + } + _ => { + return Err(BackendError::Descriptor( + "unsupported runtime-initiated transport on this platform".into(), + )); + } + }; + Self::start(listener, &descriptor.tls) + } + + fn start( + listener: Listener, + tls: &SandboxTlsClientConfig, + ) -> Result, BackendError> { + let runtime = tokio::runtime::Handle::try_current().map_err(|_| { + BackendError::Unavailable("runtime-initiated listener requires a Tokio runtime".into()) + })?; + let (sender, spares) = mpsc::channel(MAX_QUEUED_CONNECTIONS); + let accept_task = runtime + .spawn(accept_loop(listener, tls.clone(), sender)) + .abort_handle(); + tracing::info!("Runtime-initiated Sandbox Protocol listener ready"); + Ok(std::sync::Arc::new(Self { + tls: tls.clone(), + spares: tokio::sync::Mutex::new(spares), + accept_task, + })) + } + + /// Wait for one authenticated runtime connection for `descriptor`. A + /// listener bound for another session is refused. + pub(crate) async fn accept( + &self, + descriptor: &SandboxRuntimeDescriptor, + ) -> Result { + if descriptor.tls != self.tls { + return Err(BackendError::Descriptor( + "runtime-initiated listener is bound to a different sandbox session".to_string(), + )); + } + self.spares.lock().await.recv().await.ok_or_else(|| { + BackendError::Unavailable("runtime-initiated control listener stopped".to_string()) + }) + } +} + +impl Drop for RuntimeInitiatedListener { + fn drop(&mut self) { + self.accept_task.abort(); + } +} + +fn bind_tcp(address: SocketAddr) -> std::io::Result { + let socket = socket2::Socket::new( + socket2::Domain::for_address(address), + socket2::Type::STREAM, + Some(socket2::Protocol::TCP), + )?; + socket.set_reuse_address(true)?; + socket.set_nonblocking(true)?; + socket.bind(&address.into())?; + socket.listen(64)?; + tokio::net::TcpListener::from_std(socket.into()) +} + +/// Authenticate peers concurrently, bounded in count and time, and queue only +/// streams that completed the pinned handshake. A full queue closes the new +/// spare; the runtime dials again once a spare is used or lost. +async fn accept_loop( + listener: Listener, + tls: SandboxTlsClientConfig, + spares: mpsc::Sender, +) { + let pending = std::sync::Arc::new(tokio::sync::Semaphore::new(MAX_PENDING_HANDSHAKES)); + // Owned here so aborting this task also cancels every pending handshake. + let mut handshakes = tokio::task::JoinSet::new(); + loop { + tokio::select! { + accepted = listener.accept() => { + let stream = match accepted { + Ok(accepted) => accepted, + Err(error) => { + // Descriptor exhaustion and similar errors are transient. + tracing::warn!(%error, "runtime-initiated control accept failed"); + tokio::time::sleep(Duration::from_millis(100)).await; + continue; + } + }; + let Ok(permit) = pending.clone().try_acquire_owned() else { + tracing::debug!("runtime-initiated handshake limit reached; closing connection"); + continue; + }; + let tls = tls.clone(); + let spares = spares.clone(); + handshakes.spawn(async move { + let _permit = permit; + let authenticated = tokio::time::timeout( + HANDSHAKE_TIMEOUT, + crate::runtime::authenticate_boundary_stream(&tls, stream), + ) + .await; + match authenticated { + Ok(Ok(stream)) => { + if spares.try_send(stream).is_err() { + tracing::debug!("runtime-initiated spare queue full; closing connection"); + } + } + Ok(Err(error)) => { + tracing::debug!(%error, "rejected unauthenticated runtime-initiated connection"); + } + Err(_) => tracing::debug!("runtime-initiated TLS handshake timed out"), + } + }); + } + Some(_) = handshakes.join_next(), if !handshakes.is_empty() => {} + } + } +} + +fn unavailable(error: std::io::Error) -> BackendError { + BackendError::Unavailable(format!("runtime-initiated control listener: {error}")) +} + +#[derive(Debug)] +enum Listener { + Tcp(tokio::net::TcpListener), + #[cfg(unix)] + Unix { + listener: tokio::net::UnixListener, + path: std::path::PathBuf, + inode: u64, + }, + #[cfg(target_os = "linux")] + Vsock { + listener: tokio::io::unix::AsyncFd, + guest_cid: u32, + }, +} + +impl Listener { + #[cfg(unix)] + fn bind_unix(path: &std::path::Path) -> std::io::Result { + use std::os::unix::fs::{FileTypeExt, MetadataExt, PermissionsExt}; + // The driver owns the containing directory. Never replace a live socket, + // another user's inode, or a non-socket (including symlinks). + match std::fs::symlink_metadata(path) { + Ok(metadata) => { + if !metadata.file_type().is_socket() + || metadata.uid() != rustix::process::geteuid().as_raw() + { + return Err(std::io::Error::new( + std::io::ErrorKind::PermissionDenied, + "refusing to replace unowned control socket", + )); + } + let probe = + socket2::Socket::new(socket2::Domain::UNIX, socket2::Type::STREAM, None)?; + probe.set_nonblocking(true)?; + match probe.connect(&socket2::SockAddr::unix(path)?) { + Err(error) if error.kind() == std::io::ErrorKind::ConnectionRefused => { + std::fs::remove_file(path)?; + } + _ => { + return Err(std::io::Error::new( + std::io::ErrorKind::AddrInUse, + "control socket is still active", + )); + } + } + } + Err(error) if error.kind() == std::io::ErrorKind::NotFound => {} + Err(error) => return Err(error), + } + let listener = tokio::net::UnixListener::bind(path)?; + // Runtime and supervisor can have different UIDs. TLS pins the runtime; + // the protected mount path is inaccessible to workload code. + std::fs::set_permissions(path, std::fs::Permissions::from_mode(0o666))?; + let inode = std::fs::symlink_metadata(path)?.ino(); + Ok(Self::Unix { + listener, + path: path.into(), + inode, + }) + } + + #[cfg(target_os = "linux")] + fn bind_vsock(guest_cid: u32, port: u32) -> std::io::Result { + let socket = socket2::Socket::new(socket2::Domain::VSOCK, socket2::Type::STREAM, None)?; + socket.set_nonblocking(true)?; + socket.bind(&socket2::SockAddr::vsock(libc::VMADDR_CID_ANY, port))?; + socket.listen(64)?; + Ok(Self::Vsock { + listener: tokio::io::unix::AsyncFd::new(socket)?, + guest_cid, + }) + } + + async fn accept(&self) -> std::io::Result { + match self { + Self::Tcp(listener) => { + let (stream, _) = listener.accept().await?; + openshell_core::net::set_tcp_nodelay_best_effort(&stream); + crate::runtime::enable_boundary_tcp_keepalive(&stream); + Ok(Box::new(stream)) + } + #[cfg(unix)] + Self::Unix { listener, .. } => Ok(Box::new(listener.accept().await?.0)), + #[cfg(target_os = "linux")] + Self::Vsock { + listener, + guest_cid, + } => loop { + let mut ready = listener.readable().await?; + match ready.try_io(|fd| fd.get_ref().accept()) { + Ok(Ok((socket, address))) => { + if address.as_vsock_address().map(|(cid, _)| cid) != Some(*guest_cid) { + continue; + } + socket.set_nonblocking(true)?; + let stream = tokio::net::UnixStream::from_std( + std::os::unix::net::UnixStream::from(socket), + )?; + return Ok(Box::new(stream)); + } + Ok(Err(error)) => return Err(error), + Err(_) => {} + } + }, + } + } +} + +impl Drop for Listener { + fn drop(&mut self) { + #[cfg(unix)] + if let Self::Unix { path, inode, .. } = self { + use std::os::unix::fs::MetadataExt; + if std::fs::symlink_metadata(&*path).is_ok_and(|metadata| metadata.ino() == *inode) { + let _ = std::fs::remove_file(path); + } + } + } +} diff --git a/crates/openshell-sandbox/README.md b/crates/openshell-sandbox/README.md index 3ee57c085f..1cde277dd8 100644 --- a/crates/openshell-sandbox/README.md +++ b/crates/openshell-sandbox/README.md @@ -17,3 +17,12 @@ the kernel then tracks the lifetime through workload-owned descriptors. Native the last workload reference to a listener removes it without waiting for registry collection. Broker source descriptors always use close-on-exec independently of the workload's requested descriptor flags. + +A boundary configured with `RuntimeInitiatedTlsTcp` binds no listener. It dials +only the driver-provided supervisor addresses, which must be concrete, +non-loopback, and share one port, and serves TLS and gRPC over that outbound +connection. It keeps one spare connection that the supervisor has not yet used +and dials a replacement when the supervisor starts HTTP/2 on it or the spare +closes. Live connections keep the same disconnect, freeze, and recovery-deadline +handling as listener mode, and the workload still cannot connect to the control +port. diff --git a/crates/openshell-sandbox/src/boundary_server.rs b/crates/openshell-sandbox/src/boundary_server.rs index a0c3bf5bc5..49e5690083 100644 --- a/crates/openshell-sandbox/src/boundary_server.rs +++ b/crates/openshell-sandbox/src/boundary_server.rs @@ -92,6 +92,15 @@ mod linux { const FORCE_KILL_REAP_TIMEOUT: Duration = Duration::from_secs(2); const MAX_PENDING_HANDSHAKES: usize = 32; const MAX_CONTROL_CONNECTIONS: usize = 128; + /// Bounds one runtime-initiated TCP connect attempt. + const RUNTIME_DIAL_TIMEOUT: Duration = Duration::from_secs(5); + const RUNTIME_DIAL_BACKOFF_MIN: Duration = Duration::from_millis(100); + const RUNTIME_DIAL_BACKOFF_MAX: Duration = Duration::from_secs(2); + /// A spare that dies this quickly without being claimed counts as a failure. + const RUNTIME_SPARE_MIN_LIFETIME: Duration = Duration::from_secs(1); + /// Consecutive dial failures are expected while the supervisor starts; + /// report a persistent failure at most this often. + const RUNTIME_DIAL_WARNING_INTERVAL: Duration = Duration::from_secs(30); const MAX_REPLAY_LEDGER_ENTRIES: usize = 4096; const MAX_RETAINED_EXEC_PROCESSES: usize = 64; // Three stdio pipes require six descriptors before fork, and Rust's @@ -239,7 +248,14 @@ mod linux { .map_err(|error| format!("start sandbox workload launcher: {error}"))?; let protected_control_port = match &config.listener { BoundaryListenerConfig::TlsTcp { address, .. } => Some(address.port()), - BoundaryListenerConfig::Unix { .. } | BoundaryListenerConfig::Vsock { .. } => None, + // Validation requires every supervisor address to share one port. + BoundaryListenerConfig::RuntimeInitiatedTlsTcp { addresses, .. } => { + addresses.first().map(std::net::SocketAddr::port) + } + BoundaryListenerConfig::Unix { .. } + | BoundaryListenerConfig::Vsock { .. } + | BoundaryListenerConfig::RuntimeInitiatedUnix { .. } + | BoundaryListenerConfig::RuntimeInitiatedVsock { .. } => None, }; let network_broker = NetworkBroker::start(listener, protected_control_port) .map_err(|error| format!("start sandbox network broker: {error}"))?; @@ -326,6 +342,7 @@ mod linux { } match &config.listener { BoundaryListenerConfig::Unix { socket_path, tls } + | BoundaryListenerConfig::RuntimeInitiatedUnix { socket_path, tls } if !socket_path.is_absolute() || !tls_paths_are_absolute(tls) => { return Err("boundary Unix socket path must be absolute".to_string()); @@ -353,7 +370,19 @@ mod linux { } => { return Err("boundary control port must be nonzero".to_string()); } + BoundaryListenerConfig::RuntimeInitiatedVsock { host_port, tls } + if *host_port == 0 || *host_port == u32::MAX || !tls_paths_are_absolute(tls) => + { + return Err("runtime-initiated vsock requires a valid host port and absolute certificate paths".into()); + } + BoundaryListenerConfig::RuntimeInitiatedTlsTcp { + authority, + addresses, + tls, + } => validate_runtime_initiated_targets(authority, addresses, tls)?, BoundaryListenerConfig::Unix { .. } + | BoundaryListenerConfig::RuntimeInitiatedUnix { .. } + | BoundaryListenerConfig::RuntimeInitiatedVsock { .. } | BoundaryListenerConfig::TlsTcp { .. } | BoundaryListenerConfig::Vsock { .. } => {} } @@ -391,6 +420,39 @@ mod linux { Ok(()) } + /// The dial target is driver-resolved bootstrap state, never workload + /// input. Loopback is refused because workloads share the loopback + /// interface and could otherwise receive the runtime's connection. + fn validate_runtime_initiated_targets( + authority: &str, + addresses: &[std::net::SocketAddr], + tls: &openshell_sandbox_backend::boundary_protocol::SandboxTlsServerConfig, + ) -> Result<(), String> { + let Some(first) = addresses.first() else { + return Err("runtime-initiated control requires a supervisor address".to_string()); + }; + if authority.is_empty() || !tls_paths_are_absolute(tls) { + return Err( + "runtime-initiated control requires an authority and absolute certificate paths" + .to_string(), + ); + } + for address in addresses { + let ip = address.ip().to_canonical(); + if address.port() == 0 + || address.port() != first.port() + || ip.is_unspecified() + || ip.is_loopback() + { + return Err( + "runtime-initiated supervisor addresses must be concrete, non-loopback, and share one nonzero port" + .to_string(), + ); + } + } + Ok(()) + } + fn tls_paths_are_absolute( tls: &openshell_sandbox_backend::boundary_protocol::SandboxTlsServerConfig, ) -> bool { @@ -487,31 +549,48 @@ mod linux { Ok(()) } + /// Shut down after SIGTERM/SIGINT; `None` while no termination is pending. + fn take_termination(runtime: &BoundaryRuntime) -> Option> { + if !BOUNDARY_TERMINATION_REQUESTED.load(Ordering::Acquire) { + return None; + } + let signal = BOUNDARY_TERMINATION_SIGNAL.load(Ordering::Acquire); + let before_supervisor_confirmation = !matches!( + *lock(&runtime.supervisor_connection), + SupervisorConnectionState::Connected(_) + ); + runtime.shutdown(); + if before_supervisor_confirmation { + return Some(Err(format!( + "sandbox boundary received {} before supervisor confirmation", + boundary_termination_signal_name(signal) + ))); + } + tracing::info!( + signal = boundary_termination_signal_name(signal), + "Sandbox boundary received termination signal" + ); + Some(Ok(())) + } + fn serve(config: &BoundaryListenerConfig, runtime: Arc) -> Result<(), String> { + let dial_tls = match config { + BoundaryListenerConfig::RuntimeInitiatedTlsTcp { tls, .. } + | BoundaryListenerConfig::RuntimeInitiatedUnix { tls, .. } + | BoundaryListenerConfig::RuntimeInitiatedVsock { tls, .. } => Some(tls), + _ => None, + }; + if let Some(tls) = dial_tls { + return serve_runtime_initiated(config, tls, runtime); + } let listener = ControlListener::bind(config) .map_err(|error| format!("bind boundary control listener: {error}"))?; let active_connections = Arc::new(AtomicUsize::new(0)); let pending_handshakes = Arc::new(tokio::sync::Semaphore::new(MAX_PENDING_HANDSHAKES)); tracing::info!(?config, "Boundary control listener ready"); loop { - if BOUNDARY_TERMINATION_REQUESTED.load(Ordering::Acquire) { - let signal = BOUNDARY_TERMINATION_SIGNAL.load(Ordering::Acquire); - let before_supervisor_confirmation = !matches!( - *lock(&runtime.supervisor_connection), - SupervisorConnectionState::Connected(_) - ); - runtime.shutdown(); - if before_supervisor_confirmation { - return Err(format!( - "sandbox boundary received {} before supervisor confirmation", - boundary_termination_signal_name(signal) - )); - } - tracing::info!( - signal = boundary_termination_signal_name(signal), - "Sandbox boundary received termination signal" - ); - return Ok(()); + if let Some(result) = take_termination(&runtime) { + return result; } match listener.accept() { Ok(stream) => { @@ -556,6 +635,198 @@ mod linux { } } + /// Runtime-initiated control: dial the supervisor instead of listening. + /// TLS server and gRPC server roles are the same as for a listener. + fn serve_runtime_initiated( + config: &BoundaryListenerConfig, + tls: &openshell_sandbox_backend::boundary_protocol::SandboxTlsServerConfig, + runtime: Arc, + ) -> Result<(), String> { + let server_config = Arc::new( + load_tls_server_config(tls) + .map_err(|error| format!("load boundary TLS identity: {error}"))?, + ); + tracing::info!(?config, "Boundary control dialer ready"); + let dialer = runtime.process_runtime.spawn(run_runtime_initiated_control( + config.clone(), + server_config, + runtime.clone(), + Arc::new(AtomicUsize::new(0)), + )); + loop { + if let Some(result) = take_termination(&runtime) { + dialer.abort(); + return result; + } + std::thread::sleep(Duration::from_millis(10)); + } + } + + /// Keep exactly one unclaimed spare connection open to the supervisor. + /// + /// The supervisor claims a spare by starting HTTP/2 on it; then the next + /// spare is dialed so credential rotation and recovery, which open a new + /// connection before retiring the old one, never wait on this loop. TLS + /// failures back off like dial failures. Established spares are served + /// like accepted connections: same identity, slot limit, and disconnect, + /// freeze, and termination handling. + async fn run_runtime_initiated_control( + config: BoundaryListenerConfig, + server_config: Arc, + runtime: Arc, + active_connections: Arc, + ) { + let mut backoff = RUNTIME_DIAL_BACKOFF_MIN; + let mut warn_at: Option = None; + loop { + let established = + match tokio::time::timeout(RUNTIME_DIAL_TIMEOUT, dial_supervisor(&config)) + .await + .unwrap_or_else(|_| { + Err(io::Error::new( + io::ErrorKind::TimedOut, + "supervisor dial timed out", + )) + }) { + Ok(stream) => ControlStream::PendingTls { + stream, + server_config: server_config.clone(), + } + .establish_async(&runtime.process_runtime) + .await + .map_err(|error| format!("authenticate boundary transport: {error}")), + Err(error) => Err(format!("dial supervisor: {error}")), + }; + let healthy = match established { + Ok(stream) => { + warn_at = None; + let established_at = tokio::time::Instant::now(); + let (claimed, claim) = tokio::sync::oneshot::channel(); + let served = serve_authenticated_connection( + stream, + runtime.clone(), + active_connections.clone(), + Some(claimed), + ); + runtime.process_runtime.spawn(async move { + if let Err(error) = served.await { + tracing::debug!(%error, "Boundary control connection ended"); + } + }); + // A used spare, or one that stayed open, needs no backoff. + claim.await.is_ok() || established_at.elapsed() >= RUNTIME_SPARE_MIN_LIFETIME + } + Err(error) => { + // Failures are expected while the supervisor starts; report + // a persistent one at most once per interval. + let now = tokio::time::Instant::now(); + if now >= *warn_at.get_or_insert(now + RUNTIME_DIAL_WARNING_INTERVAL) { + warn_at = Some(now + RUNTIME_DIAL_WARNING_INTERVAL); + tracing::warn!(%error, "Runtime-initiated control cannot reach the supervisor; retrying"); + } else { + tracing::debug!(%error, "Runtime-initiated control dial failed; retrying"); + } + false + } + }; + if healthy { + backoff = RUNTIME_DIAL_BACKOFF_MIN; + } else { + tokio::time::sleep(backoff).await; + backoff = (backoff * 2).min(RUNTIME_DIAL_BACKOFF_MAX); + } + } + } + + /// Dial only driver-provided supervisor addresses from the bootstrap. + async fn dial_supervisor(config: &BoundaryListenerConfig) -> io::Result { + match config { + BoundaryListenerConfig::RuntimeInitiatedTlsTcp { addresses, .. } => { + let stream = + openshell_core::net::connect_tcp_nodelay_best_effort(addresses).await?; + let keepalive = socket2::TcpKeepalive::new() + .with_time(Duration::from_secs(10)) + .with_interval(Duration::from_secs(5)) + .with_retries(3); + let _ = socket2::SockRef::from(&stream).set_tcp_keepalive(&keepalive); + Ok(PlainControlStream::Tcp(stream.into_std()?)) + } + BoundaryListenerConfig::RuntimeInitiatedUnix { socket_path, .. } => { + Ok(PlainControlStream::Unix( + tokio::net::UnixStream::connect(socket_path) + .await? + .into_std()?, + )) + } + BoundaryListenerConfig::RuntimeInitiatedVsock { host_port, .. } => { + let socket = + socket2::Socket::new(socket2::Domain::VSOCK, socket2::Type::STREAM, None)?; + socket.set_nonblocking(true)?; + match socket.connect(&socket2::SockAddr::vsock(libc::VMADDR_CID_HOST, *host_port)) { + Ok(()) => {} + Err(error) if error.raw_os_error() == Some(libc::EINPROGRESS) => {} + Err(error) => return Err(error), + } + let stream = + tokio::net::UnixStream::from_std(std::os::unix::net::UnixStream::from(socket))?; + stream.writable().await?; + if let Some(error) = stream.take_error()? { + return Err(error); + } + Ok(PlainControlStream::Unix(stream.into_std()?)) + } + _ => Err(io::Error::new( + io::ErrorKind::InvalidInput, + "control transport does not dial", + )), + } + } + + /// Signals once the supervisor sends its first application bytes. + struct ClaimOnFirstRead { + inner: openshell_isolation_interface::contract::BoundaryDuplexStream, + claimed: Option>, + } + + impl tokio::io::AsyncRead for ClaimOnFirstRead { + fn poll_read( + mut self: Pin<&mut Self>, + context: &mut Context<'_>, + buffer: &mut tokio::io::ReadBuf<'_>, + ) -> Poll> { + let before = buffer.filled().len(); + let result = Pin::new(&mut self.inner).poll_read(context, buffer); + if matches!(result, Poll::Ready(Ok(()))) + && buffer.filled().len() > before + && let Some(claimed) = self.claimed.take() + { + let _ = claimed.send(()); + } + result + } + } + + impl tokio::io::AsyncWrite for ClaimOnFirstRead { + fn poll_write( + mut self: Pin<&mut Self>, + context: &mut Context<'_>, + buffer: &[u8], + ) -> Poll> { + Pin::new(&mut self.inner).poll_write(context, buffer) + } + + fn poll_flush(mut self: Pin<&mut Self>, context: &mut Context<'_>) -> Poll> { + Pin::new(&mut self.inner).poll_flush(context) + } + + fn poll_shutdown( + mut self: Pin<&mut Self>, + context: &mut Context<'_>, + ) -> Poll> { + Pin::new(&mut self.inner).poll_shutdown(context) + } + } + fn boundary_termination_signal_name(signal: i32) -> &'static str { match signal { libc::SIGTERM => "SIGTERM", @@ -575,10 +846,30 @@ mod linux { .await .map_err(|error| format!("authenticate boundary transport: {error}"))?; drop(pending); + serve_authenticated_connection(stream, runtime, active_connections, None).await + } + + /// Serve one TLS-established connection, accepted or runtime-initiated. + /// `claimed`, used by runtime-initiated spares, fires when the supervisor + /// first sends data. + async fn serve_authenticated_connection( + stream: ControlStream, + runtime: Arc, + active_connections: Arc, + claimed: Option>, + ) -> Result<(), String> { let Some(_slot) = acquire_control_connection_slot(&active_connections) else { return Err("authenticated control connection limit reached".to_string()); }; - serve_grpc(stream.into_tokio()?, runtime, SandboxConnectionId::new()).await + let stream = stream.into_tokio()?; + let stream = match claimed { + Some(claimed) => Box::new(ClaimOnFirstRead { + inner: stream, + claimed: Some(claimed), + }), + None => stream, + }; + serve_grpc(stream, runtime, SandboxConnectionId::new()).await } async fn serve_grpc( @@ -3364,6 +3655,12 @@ mod linux { server_config, }) } + BoundaryListenerConfig::RuntimeInitiatedTlsTcp { .. } + | BoundaryListenerConfig::RuntimeInitiatedUnix { .. } + | BoundaryListenerConfig::RuntimeInitiatedVsock { .. } => Err(io::Error::new( + io::ErrorKind::InvalidInput, + "runtime-initiated control dials the supervisor and binds no listener", + )), BoundaryListenerConfig::TlsTcp { address, tls } => { let (listener, reject_local_peers) = Self::bind_tcp(*address)?; listener.set_nonblocking(true)?; diff --git a/crates/openshell-supervisor/src/backend_setup.rs b/crates/openshell-supervisor/src/backend_setup.rs index fbf770b905..fe11068b52 100644 --- a/crates/openshell-supervisor/src/backend_setup.rs +++ b/crates/openshell-supervisor/src/backend_setup.rs @@ -231,11 +231,24 @@ impl BackendSetup for OpenShellBackendSetup { gid: descriptor.workload_identity.gid, }), }; - Ok((identity, Box::new(OpenShellLaunch(descriptor)))) + let transport = openshell_sandbox_backend::BoundaryTransport::for_descriptor(&descriptor)?; + Ok(( + identity, + Box::new(OpenShellLaunch { + descriptor, + transport, + }), + )) } } -struct OpenShellLaunch(openshell_sandbox_backend::boundary_protocol::SandboxRuntimeDescriptor); +/// Launch data plus the transport chosen for it. Discovery and the attached +/// backend use clones of one transport, so a runtime-initiated descriptor binds +/// its control listener once for the supervisor's lifetime. +struct OpenShellLaunch { + descriptor: openshell_sandbox_backend::boundary_protocol::SandboxRuntimeDescriptor, + transport: openshell_sandbox_backend::BoundaryTransport, +} #[tonic::async_trait] impl PreparedBackend for OpenShellLaunch { @@ -243,23 +256,30 @@ impl PreparedBackend for OpenShellLaunch { &self, bearer: SessionBearerTokenSlot, ) -> std::result::Result<(Option, bool), BackendError> { - openshell_sandbox_backend::OpenShellRuntimeBackend::discover_policy(self.0.clone(), bearer) - .await + openshell_sandbox_backend::OpenShellRuntimeBackend::discover_policy( + self.descriptor.clone(), + bearer, + self.transport.clone(), + ) + .await } fn build( self: Box, services: BackendServices, ) -> std::result::Result { - let payload = self.0.backend_descriptor()?.payload; + let payload = self.descriptor.backend_descriptor()?.payload; Ok(BuiltBackend { payload, proxy_listener: None, - backend: Arc::new(openshell_sandbox_backend::OpenShellRuntimeBackend::new( - services.ca_file_paths, - services.provider_credentials, - services.sandbox_bearer, - )), + backend: Arc::new( + openshell_sandbox_backend::OpenShellRuntimeBackend::new( + services.ca_file_paths, + services.provider_credentials, + services.sandbox_bearer, + ) + .with_transport(self.transport), + ), }) } } diff --git a/e2e/rust/tests/kubernetes_runtime_reconcile.rs b/e2e/rust/tests/kubernetes_runtime_reconcile.rs index ccba4da773..f93b2b8448 100644 --- a/e2e/rust/tests/kubernetes_runtime_reconcile.rs +++ b/e2e/rust/tests/kubernetes_runtime_reconcile.rs @@ -9,6 +9,7 @@ use std::process::Stdio; +use openshell_e2e::harness::binary::openshell_cmd; use openshell_e2e::harness::cli::run_cli; use openshell_e2e::harness::sandbox::SandboxGuard; @@ -74,6 +75,112 @@ fn bootstrap_image(resource: &serde_json::Value, index: usize) -> &serde_json::V &resource["spec"]["podTemplate"]["spec"]["initContainers"][index]["image"] } +async fn service(resource: &serde_json::Value, prefix: &str) -> serde_json::Value { + let sandbox_id = resource["metadata"]["labels"]["openshell.ai/sandbox-id"] + .as_str() + .expect("sandbox ID"); + serde_json::from_str( + &kubectl(&[ + "-n", + &namespace(), + "get", + "service", + &format!("{prefix}-{sandbox_id}"), + "-o", + "json", + ]) + .await, + ) + .expect("parse Service") +} + +async fn legacy_lifecycle(endpoint: &str, operation: &str, name: &str) { + let output = openshell_cmd() + .args(["--gateway-endpoint", endpoint, "sandbox", operation, name]) + .output() + .await + .expect("legacy gateway lifecycle command"); + assert!( + output.status.success(), + "legacy {operation} failed: {}{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); +} + +/// Run with an old gateway sharing the fixture database, signing keys and +/// cluster, with its own matching supervisor/runtime images. The configured +/// gateway endpoint must target the new gateway. +#[tokio::test] +#[ignore = "requires OPENSHELL_E2E_LEGACY_GATEWAY_ENDPOINT and a mixed-version gateway fixture"] +async fn old_and_new_gateways_can_restart_the_same_sandbox() { + let legacy = + std::env::var("OPENSHELL_E2E_LEGACY_GATEWAY_ENDPOINT").expect("legacy gateway endpoint"); + let mut sandbox = SandboxGuard::create_keep( + &["sh", "-c", "echo mixed-ready; exec sleep infinity"], + "mixed-ready", + ) + .await + .expect("create sandbox through new gateway"); + let (created, _) = sandbox_resource(&sandbox.name).await; + let boundary = service(&created, "os-boundary").await; + let supervisor = service(&created, "os-supervisor").await; + assert_eq!( + boundary["spec"]["selector"]["openshell.ai/boundary-role"], + "workload" + ); + assert_eq!( + supervisor["spec"]["selector"]["openshell.ai/boundary-role"], + "supervisor" + ); + assert_ne!( + boundary["spec"]["clusterIP"], + supervisor["spec"]["clusterIP"] + ); + + legacy_lifecycle(&legacy, "stop", &sandbox.name).await; + legacy_lifecycle(&legacy, "start", &sandbox.name).await; + let (restarted, _) = sandbox_resource(&sandbox.name).await; + let annotations = &restarted["metadata"]["annotations"]; + assert_ne!( + annotations["openshell.ai/sandbox-runtime-generation"], + annotations["openshell.ai/sandbox-runtime-control-generation"] + ); + assert!( + sandbox + .exec(&["printf", "old-ready"]) + .await + .expect("old pair exec") + .contains("old-ready") + ); + + for operation in ["stop", "start"] { + let (output, code) = run_cli(&["sandbox", operation, &sandbox.name]).await; + assert_eq!(code, 0, "new gateway {operation}: {output}"); + } + assert!( + sandbox + .exec(&["printf", "new-ready"]) + .await + .expect("new pair exec") + .contains("new-ready") + ); + let (restarted, _) = sandbox_resource(&sandbox.name).await; + let annotations = &restarted["metadata"]["annotations"]; + assert_eq!( + annotations["openshell.ai/sandbox-runtime-generation"], + annotations["openshell.ai/sandbox-runtime-control-generation"] + ); + let unchanged = service(&restarted, "os-boundary").await; + assert_eq!(unchanged["metadata"]["uid"], boundary["metadata"]["uid"]); + assert_eq!( + unchanged["spec"]["clusterIP"], + boundary["spec"]["clusterIP"] + ); + assert_eq!(unchanged["spec"]["selector"], boundary["spec"]["selector"]); + sandbox.cleanup().await; +} + #[tokio::test] async fn start_refreshes_stale_bootstrap_image() { let mut sandbox = SandboxGuard::create_keep( @@ -95,6 +202,20 @@ async fn start_refreshes_stale_bootstrap_image() { // An older gateway stored its own runtime image in the resource. let (before, index) = sandbox_resource(&sandbox.name).await; + let boundary = service(&before, "os-boundary").await; + let supervisor = service(&before, "os-supervisor").await; + // A legacy sandbox has no supervisor Service. Adding it on start must + // preserve the workload Service that old gateways use. + kubectl(&[ + "-n", + &namespace(), + "delete", + "service", + supervisor["metadata"]["name"] + .as_str() + .expect("supervisor Service name"), + ]) + .await; let patch = serde_json::json!([{ "op": "replace", "path": format!("/spec/podTemplate/spec/initContainers/{index}/image"), @@ -127,6 +248,20 @@ async fn start_refreshes_stale_bootstrap_image() { ); let (after, after_index) = sandbox_resource(&sandbox.name).await; assert_eq!(after["metadata"]["uid"], before["metadata"]["uid"]); + let unchanged = service(&after, "os-boundary").await; + assert_eq!(unchanged["metadata"]["uid"], boundary["metadata"]["uid"]); + assert_eq!( + unchanged["spec"]["clusterIP"], + boundary["spec"]["clusterIP"] + ); + assert_eq!( + unchanged["spec"]["selector"]["openshell.ai/boundary-role"], + "workload" + ); + assert_eq!( + service(&after, "os-supervisor").await["spec"]["selector"]["openshell.ai/boundary-role"], + "supervisor" + ); assert_eq!( bootstrap_image(&after, after_index), bootstrap_image(&before, index) diff --git a/skills/debug-openshell-cluster/SKILL.md b/skills/debug-openshell-cluster/SKILL.md index ddf042c67c..31eb0c158a 100644 --- a/skills/debug-openshell-cluster/SKILL.md +++ b/skills/debug-openshell-cluster/SKILL.md @@ -335,6 +335,13 @@ Common findings: - Sandbox runtime image exits before printing `openshell-sandbox --version`: verify the configured image contains a static executable at `/openshell-sandbox`. - A sandbox with explicit `protocol: tcp` endpoints fails before workload readiness: inspect supervisor and backend logs for policy rejection, network-mediation initialization, protected-channel setup, or listener failures. A backend that cannot enforce the admitted policy must reject it before starting workload code. - Supervisor runtime validation fails: verify `supervisor_image` contains an `/openshell-supervisor` executable from the same release as the sandbox runtime, and that the dynamic loader and shared libraries it links against are available inside that image. `docker run --rm --network none --entrypoint /openshell-supervisor --version` should print that release; a `no such file or directory` error for a binary that exists means the loader or a library is missing. The supervisor runs from its own image and does not need to be static; only `/openshell-sandbox` must be. +- New Docker and Podman sandboxes dial the supervisor over a separate private + Unix socket volume; the workload keeps `network=none`. Check both containers' + mounts and socket-directory ownership if attachment times out. Existing + containers retain their saved legacy transport through restart. New VM + generations dial host CID 2 over vsock; libkrun forwards to a protected host + Unix socket, while QEMU uses a per-guest host port. Keep supervisor and runtime + binaries compatible when creating new generations. - Sandbox startup reports an `isolation control contract` failure: inspect the supervisor and sandbox-runtime image versions and update them together. An older runtime or incompatible isolation capability set fails before workload startup. See the published Sandbox Runtimes documentation for trusted image override precedence. - The sandbox fails its enforcement probe: inspect the sandbox log for the exact nested seccomp user-notification, task-memory, Landlock, loopback DNS, or socket-injection check that failed. A runtime may return `ENOSYS` for `process_vm_readv` and `process_vm_writev` while satisfying the production parent-to-workload-child task-memory probe through `/proc//mem`; only failure of both backends is fatal. Do not add capabilities or switch to an unconfined seccomp profile; use a runtime whose default profile permits the unprivileged probe. - A GPU sandbox fails because Docker reports no discovered NVIDIA CDI devices: verify `.DiscoveredDevices` contains entries such as `nvidia.com/gpu=all`, verify `/etc/cdi` or `/var/run/cdi` contains a generated NVIDIA spec, and check that `nvidia-cdi-refresh.service` and `nvidia-cdi-refresh.path` from NVIDIA Container Toolkit are enabled and healthy. The service is a one-shot unit, so `inactive (dead)` can be normal after a successful run; use `systemctl status` and `journalctl` to distinguish success from a skipped or failed refresh. Restart `nvidia-cdi-refresh.service` to regenerate missing or stale CDI specs, then restart or reload Docker and re-check `docker info`. @@ -1010,10 +1017,50 @@ per-sandbox Service, split immutable bootstrap Secrets, and a gated supervisor Pod before releasing either Pod. The supervisor Pod runs `/openshell-supervisor`. Both Pods use the same resolved non-root identity, request no capabilities, drop `ALL`, disable -privilege escalation, and use `RuntimeDefault` seccomp. The supervisor reaches -the sandbox over per-sandbox TLS with server-certificate verification plus -bootstrap-token client authentication, and owns gateway policy, provider -credentials, DNS, and mediated upstream connections. +privilege escalation, and use `RuntimeDefault` seccomp. The supervisor and +sandbox talk over per-sandbox TLS with server-certificate verification plus +bootstrap-token client authentication; the supervisor owns gateway policy, +provider credentials, DNS, and mediated upstream connections. + +Each sandbox records which side opens that connection in the Sandbox annotation +`openshell.ai/sandbox-runtime-control-connection`, bound to +`openshell.ai/sandbox-runtime-control-generation`. A missing marker or a +different current runtime generation means +`supervisor-initiated`: the `os-boundary-` Service selects the workload Pod +and the supervisor dials it. With `runtime-initiated` (used for new +generations), a separate `os-supervisor-` Service selects the supervisor Pod, the sandbox runtime dials +it, and a per-sandbox NetworkPolicy `os-control-` admits only that +connection at both the workload (egress) and the supervisor (ingress). The +Service publishes the supervisor before it is Ready, because readiness follows +the runtime's attach. If the runtime logs `cannot reach the supervisor`, check +that the CNI applied both directions of `os-control-`. +Stop/start upgrades legacy sandboxes to runtime-initiated control, as does +gateway recovery that recreates both Pods. Existing running generations keep +their current connection until replaced. Check the mode first when a sandbox +never becomes ready, especially after changing images: + +```bash +kubectl -n get sandbox \ + -o jsonpath='{.metadata.annotations.openshell\.ai/sandbox-runtime-control-connection}{"\n"}' +kubectl -n get service os-boundary- -o jsonpath='{.spec.selector}{"\n"}' +kubectl -n get service os-supervisor- -o jsonpath='{.spec.selector}{"\n"}' +kubectl -n get networkpolicy os-control- -o yaml +``` + +A `runtime-initiated` sandbox needs a gateway, supervisor image, and sandbox +runtime image that all support the mode. An older supervisor or runtime fails +to decode its launch descriptor, so the sandbox cannot start or restart. +Old gateways can restart sandboxes by creating their own compatible pair; +`os-boundary-` always keeps its workload selector. New gateways also create +it for new sandboxes, alongside the supervisor Service. Upgrade the images +together within each generation. There is no legacy rollout flag. +Adding the supervisor endpoint on restart requires `create` and `get` on +Services in the sandbox namespace. For a `Forbidden` restart error, check the +gateway ServiceAccount with `kubectl auth can-i create services --namespace + --as system:serviceaccount::`. +Do not edit the annotation, Service selector, or `os-control-` policy: +reconciliation suspends a sandbox whose Service or control policy no longer +matches its recorded mode. Inspect all driver-managed resources when a Kubernetes sandbox remains Starting or loses readiness: