From 5561549b642cd00aac1a98329cffa202a694b056 Mon Sep 17 00:00:00 2001 From: Florent Benoit Date: Tue, 13 Oct 2026 09:33:58 +0200 Subject: [PATCH 1/2] feat(vm): add host bind mount support via virtiofs (libkrun) Introduce host directory bind mounts for libkrun VM sandboxes using the virtiofs4 API with simplified permission semantics. The gateway accepts `enable_bind_mounts` in the `[openshell.drivers.vm]` TOML table and forwards it to the VM driver subprocess. Mounts are specified per-sandbox in driver_config.mounts[] with source, target, optional read_only (default: read-only), and an optional `type` that must be `bind`, so Docker and Podman bind entries (without `selinux_label`) parse unchanged. At sandbox validation the driver rejects mounts unless bind mounts are enabled and resource admission is disabled. It rejects GPU and other QEMU launches, sources with control characters, and more mounts than libkrun has interrupt lines for (32 on aarch64, 3 on x86_64), and checks targets against reserved VM-internal directories and each other (no nested targets). The QEMU check runs again once the launch backend is resolved. Host source directories are checked at provisioning, before image preparation, so a briefly missing source fails provisioning visibly instead of denying recovery of a persisted sandbox. Missing, inaccessible, symlinked, non-directory, and unreadable sources are rejected: libkrun does not follow a final symlink, and on macOS it panics during guest boot when it cannot open the share root. The driver generates virtiofs tags and always writes a guest-side manifest into the overlay disk, together with the init drop-ins, so an image-supplied manifest can never drive guest mounts. The launcher parses its tab-separated --vm-mount arguments from the right, so a source can never shift the tag or mode, and the driver logs each shared directory at launch. Guest init mounts the shares after every /sandbox ownership fixup and the init drop-ins, so recursive chowns never reach host files. It refuses targets that traverse symlinks in the image, hands newly created mount point parents under /sandbox to the sandbox user, and skips an empty manifest. krun_add_virtiofs4 is resolved optionally, so older libkrun runtimes still boot sandboxes without mounts. The kernel config lists CONFIG_FUSE_FS and CONFIG_VIRTIO_FS explicitly; the pinned libkrunfw base config already enables both. A vm_bind_mount e2e test checks a read-write share round-trips data with the host, accepts a new file, directory, and nested file that the host then sees, and that a read-only share rejects writes. libkrun on rootless Linux refuses file creation by a guest user whose UID/GID differ from its own, so the VM e2e gateway runs the sandbox as the gateway user there. It enables bind mounts like the Docker and Podman e2e gateways. The runtimes docs cover the mount limit, the symlink restriction, the guest-kernel exposure, and the rootless Linux UID/GID note. The debug-openshell-cluster skill lists VM bind mount checks. Signed-off-by: Florent Benoit --- crates/openshell-core/src/driver_mounts.rs | 3 +- crates/openshell-driver-vm/README.md | 1 + .../runtime/kernel/openshell.kconfig | 6 + .../scripts/openshell-vm-sandbox-init.sh | 70 ++ crates/openshell-driver-vm/src/driver.rs | 866 +++++++++++++++++- crates/openshell-driver-vm/src/ffi.rs | 12 + crates/openshell-driver-vm/src/lib.rs | 3 +- crates/openshell-driver-vm/src/main.rs | 81 +- crates/openshell-driver-vm/src/runtime.rs | 46 +- crates/openshell-gateway/src/vm.rs | 8 + docs/how-it-works/gateways/configuration.mdx | 5 + docs/how-it-works/sandboxes/runtimes.mdx | 30 + e2e/rust/Cargo.toml | 5 + e2e/rust/e2e-vm.sh | 23 + e2e/rust/tests/vm_bind_mount.rs | 119 +++ skills/debug-openshell-cluster/SKILL.md | 8 + 16 files changed, 1271 insertions(+), 15 deletions(-) create mode 100644 e2e/rust/tests/vm_bind_mount.rs diff --git a/crates/openshell-core/src/driver_mounts.rs b/crates/openshell-core/src/driver_mounts.rs index bf6ba9f361..f786f76ce4 100644 --- a/crates/openshell-core/src/driver_mounts.rs +++ b/crates/openshell-core/src/driver_mounts.rs @@ -239,7 +239,8 @@ pub fn path_is_or_under(path: &Path, parent: &Path) -> bool { path == parent || path.starts_with(parent) } -fn paths_overlap(left: &Path, right: &Path) -> bool { +/// Return true when either path is the other or contains it. +pub fn paths_overlap(left: &Path, right: &Path) -> bool { path_is_or_under(left, right) || path_is_or_under(right, left) } diff --git a/crates/openshell-driver-vm/README.md b/crates/openshell-driver-vm/README.md index 4d6454e810..fa05c48efc 100644 --- a/crates/openshell-driver-vm/README.md +++ b/crates/openshell-driver-vm/README.md @@ -174,6 +174,7 @@ Select the VM driver with `--compute-driver vm`, `OPENSHELL_COMPUTE_DRIVER=vm`, | `proxy_connect_by_hostname` | unset | Send hostnames rather than validated IPs in CONNECT. Last resort for proxies whose ACLs reject IP CONNECT targets. | | `proxy_ca_bundle` | unset | Gateway-host PEM CA bundle trusted for the corporate proxy and TLS-intercepted server certificates. The driver validates it and stages it at a fixed non-secret guest path in the protected overlay. Requires `https_proxy`. | | `provider_spiffe_workload_api_tcp_endpoint` | unset | Explicit guest-reachable `tcp:IP:port` SPIFFE Workload API listener for provider token exchange. It requires `provider_spiffe_allow_guest_tcp = true`; a host UNIX socket is never silently exposed to a VM guest. | +| `enable_bind_mounts` | `false` | Allow per-sandbox `driver_config.vm.mounts` host directory shares (libkrun virtiofs only). The driver writes a `tag\ttarget\tmode` manifest to `/.openshell/mounts.manifest` in the overlay, and guest init mounts each share after `/sandbox` ownership fixups. Rejected while resource admission is enabled. | The proxy settings are operator-owned and deployment-level: they are not accepted through `template.driver_config.vm`, and the driver passes them only to native host control. Every present-but-invalid value is fatal at gateway or sandbox startup rather than degrading to a direct dial. diff --git a/crates/openshell-driver-vm/runtime/kernel/openshell.kconfig b/crates/openshell-driver-vm/runtime/kernel/openshell.kconfig index d4493abd7b..c3573759f1 100644 --- a/crates/openshell-driver-vm/runtime/kernel/openshell.kconfig +++ b/crates/openshell-driver-vm/runtime/kernel/openshell.kconfig @@ -15,6 +15,12 @@ CONFIG_VIRTIO_BLK=y CONFIG_EXT4_FS=y CONFIG_EXT4_USE_FOR_EXT2=y +# Host-directory sharing via virtiofs (used for VM driver bind mounts). +# The pinned libkrunfw base config already enables both; they are listed here +# because bind mounts depend on them. +CONFIG_FUSE_FS=y +CONFIG_VIRTIO_FS=y + # Cgroups used for process supervision and resource limits. CONFIG_CGROUPS=y CONFIG_CGROUP_DEVICE=y diff --git a/crates/openshell-driver-vm/scripts/openshell-vm-sandbox-init.sh b/crates/openshell-driver-vm/scripts/openshell-vm-sandbox-init.sh index 7975f60cdb..0db25514b5 100644 --- a/crates/openshell-driver-vm/scripts/openshell-vm-sandbox-init.sh +++ b/crates/openshell-driver-vm/scripts/openshell-vm-sandbox-init.sh @@ -544,6 +544,74 @@ run_openshell_init_dropins() { done < <(LC_ALL=C sort -u "$manifest") } +# Create a virtiofs mount point under ROOT_PREFIX without following symlinks, +# so an image cannot redirect a validated target onto another guest path. +# Directories created under /sandbox are handed to the sandbox user. +prepare_virtiofs_target() { + local target="$1" owner="$2" + local path="${ROOT_PREFIX:-}" component + local -a components + IFS=/ read -r -a components <<<"${target#/}" + for component in "${components[@]}"; do + [ -n "$component" ] || continue + path="${path}/${component}" + if [ -L "$path" ]; then + ts >&2 "FATAL: virtiofs mount target ${target} traverses symlink ${path#"${ROOT_PREFIX:-}"}" + exit 1 + fi + if [ ! -e "$path" ]; then + if ! mkdir "$path"; then + ts >&2 "FATAL: failed to create virtiofs mount point ${target}" + exit 1 + fi + case "${path#"${ROOT_PREFIX:-}"}" in + /sandbox/*) + if ! chown "$owner" "$path"; then + ts >&2 "FATAL: failed to hand virtiofs mount point parent ${path#"${ROOT_PREFIX:-}"} to ${owner}" + exit 1 + fi + ;; + esac + elif [ ! -d "$path" ]; then + ts >&2 "FATAL: virtiofs mount target ${target} is not a directory" + exit 1 + fi + done + printf '%s\n' "$path" +} + +# Runs after every /sandbox ownership fixup and the root-run init drop-ins so +# nothing in init walks into host-backed shares. +mount_virtiofs_shares() { + local manifest + manifest="$(root_path /.openshell/mounts.manifest)" + # The driver always writes the manifest; it is empty without mounts. + [ -s "$manifest" ] || return 0 + + ts "mounting virtiofs shares" + local tag target mode mount_opts guest_target owner + owner="$(sandbox_owner)" + while IFS=$'\t' read -r tag target mode; do + [ -n "$tag" ] || continue + case "$mode" in + ro) mount_opts="-o ro" ;; + rw) mount_opts="" ;; + *) + ts "FATAL: unknown virtiofs mount mode '${mode}' for tag ${tag}" + exit 1 + ;; + esac + guest_target="$(prepare_virtiofs_target "$target" "$owner")" || exit 1 + # shellcheck disable=SC2086 + if mount -t virtiofs $mount_opts "$tag" "$guest_target"; then + ts " mounted virtiofs ${tag} -> ${target} (${mode})" + else + ts "FATAL: failed to mount virtiofs ${tag} at ${target}" + exit 1 + fi + done < "$manifest" +} + run_post_overlay_setup() { # Source QEMU-injected environment variables if present. The file lives in # the overlay upperdir so the cached bootstrap rootfs remains immutable. @@ -617,6 +685,8 @@ fi run_openshell_init_dropins +mount_virtiofs_shares + if [ -n "${OPENSHELL_SANDBOX_ID:-}" ]; then ts "OPENSHELL_SANDBOX_ID=${OPENSHELL_SANDBOX_ID}" fi diff --git a/crates/openshell-driver-vm/src/driver.rs b/crates/openshell-driver-vm/src/driver.rs index 38d801216d..aa7d9417df 100644 --- a/crates/openshell-driver-vm/src/driver.rs +++ b/crates/openshell-driver-vm/src/driver.rs @@ -44,6 +44,7 @@ use oci_client::manifest::{ use oci_client::secrets::RegistryAuth; use oci_client::{Reference, RegistryOperation}; use openshell_core::UpstreamProxyConfig; +use openshell_core::driver_mounts; use openshell_core::gpu::{ driver_gpu_requirements, effective_driver_gpu_count, validate_specific_gpu_device_request, }; @@ -120,6 +121,41 @@ const DEFAULT_ROOTFS_TAR_MAX_BYTES: u64 = 10 * 1024 * 1024 * 1024; const ROOTFS_TAR_STAGING_DIR: &str = "rootfs-tar-staging"; const VM_CONSOLE_DIAGNOSTIC_BYTES: u64 = 8 * 1024; +/// Mirrors the Docker/Podman `type` tag so their bind entries parse +/// unchanged; virtiofs can only share host directories. +#[derive(Debug, Clone, Copy, serde::Deserialize)] +#[serde(rename_all = "lowercase")] +enum VmMountType { + Bind, +} + +#[derive(Debug, Clone, serde::Deserialize)] +#[serde(deny_unknown_fields)] +struct VmMountConfig { + #[serde(default, rename = "type")] + #[allow(dead_code)] + mount_type: Option, + source: String, + target: String, + #[serde(default = "default_read_only")] + read_only: bool, +} + +impl VmMountConfig { + /// Source without trailing slashes. A trailing slash makes both the + /// symlink check and libkrun's `O_NOFOLLOW` open resolve a final symlink. + fn host_source(&self) -> &str { + match self.source.trim_end_matches('/') { + "" => "/", + trimmed => trimmed, + } + } +} + +fn default_read_only() -> bool { + true +} + #[derive(Debug, Clone, Default, serde::Deserialize)] #[serde(default, deny_unknown_fields)] struct VmSandboxDriverConfig { @@ -129,6 +165,8 @@ struct VmSandboxDriverConfig { )] gpu_device_ids: Option>, rootfs_tar_path: Option, + #[serde(default)] + mounts: Vec, } impl VmSandboxDriverConfig { @@ -251,6 +289,7 @@ enum GuestImagePayloadSource { #[derive(Clone, serde::Serialize, serde::Deserialize)] #[serde(deny_unknown_fields)] +#[allow(clippy::struct_excessive_bools)] pub struct VmDriverConfig { /// Permit caller-supplied driver JSON. Does not waive resource admission. #[serde(default)] @@ -306,6 +345,9 @@ pub struct VmDriverConfig { /// Maximum rootfs tar file size in bytes. Defaults to 10 GiB. #[serde(default, skip_serializing_if = "Option::is_none")] pub rootfs_tar_max_bytes: Option, + + #[serde(default)] + pub enable_bind_mounts: bool, } /// Redacting `Debug` so a proxy URL or credential path never reaches a log. @@ -368,6 +410,7 @@ impl std::fmt::Debug for VmDriverConfig { ) .field("rootfs_tar_staging_dir", &self.rootfs_tar_staging_dir) .field("rootfs_tar_max_bytes", &self.rootfs_tar_max_bytes) + .field("enable_bind_mounts", &self.enable_bind_mounts) .finish() } } @@ -407,6 +450,7 @@ impl Default for VmDriverConfig { sandbox_gid: None, rootfs_tar_staging_dir: None, rootfs_tar_max_bytes: None, + enable_bind_mounts: false, } } } @@ -1094,8 +1138,14 @@ impl VmDriver { sandbox, )?; validate_vm_sandbox(sandbox, self.config.gpu_enabled)?; - let has_rootfs_tar = - VmSandboxDriverConfig::from_sandbox(sandbox).is_ok_and(|c| c.rootfs_tar_path.is_some()); + let driver_config = + VmSandboxDriverConfig::from_sandbox(sandbox).map_err(Status::invalid_argument)?; + validate_vm_driver_mounts( + &driver_config.mounts, + &self.config, + sandbox_requests_gpu(sandbox), + )?; + let has_rootfs_tar = driver_config.rootfs_tar_path.is_some(); if self.resolved_sandbox_image(sandbox).is_none() && !has_rootfs_tar { return Err(Status::failed_precondition( "vm sandboxes require template.image, rootfs_tar_path in driver_config, or a configured default sandbox image", @@ -1314,14 +1364,11 @@ impl VmDriver { .as_ref() .map_or(&[], |spec| spec.launch_authentication.as_slice()), )?; - let is_gpu = sandbox - .spec - .as_ref() - .and_then(|spec| spec.resource_requirements.as_ref()) - .and_then(|requirements| driver_gpu_requirements(Some(requirements))) - .is_some(); + let is_gpu = sandbox_requests_gpu(&sandbox); let driver_config = VmSandboxDriverConfig::from_sandbox(&sandbox).map_err(Status::invalid_argument)?; + // Check host sources before the costly image and overlay preparation. + validate_vm_mount_sources(&driver_config.mounts).await?; let driver_config_had_rootfs_tar = driver_config.rootfs_tar_path.is_some(); let rootfs_tar_path = match driver_config.rootfs_tar_path { Some(raw) if overlay_preparation == OverlayPreparation::Fresh => { @@ -1497,7 +1544,16 @@ impl VmDriver { return Err(err); } - if let Err(err) = Self::validate_launch_plan_backend(is_gpu, &plan) { + if let Err(err) = Self::validate_launch_plan_backend(is_gpu, &plan).and_then(|()| { + // Cannot fail today: `validate_sandbox` already ran and non-GPU + // QEMU is blocked above. Keeps virtiofs mounts off QEMU once that + // guard is lifted. + validate_vm_driver_mounts( + &driver_config.mounts, + &self.config, + plan.backend == VmBackend::Qemu, + ) + }) { self.lifecycle_extensions .after_launch_failed( &sandbox, @@ -1533,7 +1589,14 @@ impl VmDriver { }); } - if let Err(err) = inject_guest_init_dropins(&overlay_disk, &plan.guest_init_dropins) { + // The mount manifest is always written, even when empty, so an + // image-supplied manifest in the lowerdir can never drive guest mounts. + if let Err(err) = inject_guest_init_dropins(&overlay_disk, &plan.guest_init_dropins) + .and_then(|()| { + inject_guest_mount_manifest(&overlay_disk, &driver_config.mounts) + .map_err(|error| Status::internal(format!("inject VM mount manifest: {error}"))) + }) + { self.lifecycle_extensions .after_launch_failed(&sandbox, &state_dir, LaunchAbortReason::GuestPrepareFailed) .await; @@ -1675,6 +1738,21 @@ impl VmDriver { for env in sandbox_owner_state.guest_environment() { command.arg("--vm-env").arg(env); } + for (i, mount) in driver_config.mounts.iter().enumerate() { + command.arg("--vm-mount").arg(format!( + "{}\t{}\t{}", + mount.host_source(), + virtiofs_tag(i), + vm_mount_mode(mount.read_only) + )); + info!( + sandbox_id = %sandbox.id, + source = %mount.host_source(), + target = %mount.target, + mode = vm_mount_mode(mount.read_only), + "vm driver: sharing host directory with sandbox" + ); + } info!( sandbox_id = %sandbox.id, @@ -4983,6 +5061,183 @@ fn validate_vm_sandbox_template(template: &SandboxTemplate) -> Result<(), Status Ok(()) } +/// Each share is one virtio device and libkrun has a fixed number of interrupt +/// lines; a share beyond them breaks VM start. The VM's own devices take up +/// to 8 (balloon, rng, console, root filesystem, three disks, vsock), plus an +/// RTC on `aarch64`. `x86_64` has 11 lines. `aarch64` has 128 on HVF but only +/// 64 usable on KVM (the vGIC is created with 96 SPIs and libkrun allocates +/// from GSI 32), so 32 leaves headroom on both. +const VM_MAX_BIND_MOUNTS: usize = if cfg!(target_arch = "aarch64") { 32 } else { 3 }; + +const VM_RESERVED_GUEST_PATHS: &[&str] = &[ + "/.openshell", + "/.openshell-bootstrap", + "/overlay", + "/lower", + "/newroot", + "/image-cache", + "/srv", + "/proc", + "/sys", + "/dev", + "/tmp", +]; + +const VIRTIOFS_TAG_PREFIX: &str = "osfs"; + +fn virtiofs_tag(index: usize) -> String { + format!("{VIRTIOFS_TAG_PREFIX}{index}") +} + +fn vm_mount_mode(read_only: bool) -> &'static str { + if read_only { "ro" } else { "rw" } +} + +/// Guest manifest consumed by `mount_virtiofs_shares`: `tag\ttarget\tmode`. +fn render_mount_manifest(mounts: &[VmMountConfig]) -> String { + let mut manifest = String::new(); + for (i, mount) in mounts.iter().enumerate() { + writeln!( + manifest, + "{}\t{}\t{}", + virtiofs_tag(i), + driver_mounts::normalize_mount_target(&mount.target), + vm_mount_mode(mount.read_only) + ) + .expect("write to String cannot fail"); + } + manifest +} + +fn sandbox_requests_gpu(sandbox: &Sandbox) -> bool { + sandbox + .spec + .as_ref() + .and_then(|spec| spec.resource_requirements.as_ref()) + .and_then(|requirements| driver_gpu_requirements(Some(requirements))) + .is_some() +} + +fn validate_no_control_chars(value: &str, field: &str) -> Result<(), String> { + if value.chars().any(char::is_control) { + return Err(format!("{field} must not contain control characters")); + } + Ok(()) +} + +#[allow(clippy::result_large_err)] +fn validate_vm_driver_mounts( + mounts: &[VmMountConfig], + config: &VmDriverConfig, + is_qemu: bool, +) -> Result<(), Status> { + if mounts.is_empty() { + return Ok(()); + } + if is_qemu { + return Err(Status::failed_precondition( + "vm bind mounts require the libkrun backend; GPU sandboxes and other QEMU launches cannot use them", + )); + } + if !config.enable_bind_mounts { + return Err(Status::failed_precondition( + "vm bind mounts require enable_bind_mounts = true in the VM driver configuration", + )); + } + config + .resource_admission + .reject_unlabelable("host bind mount")?; + if mounts.len() > VM_MAX_BIND_MOUNTS { + return Err(Status::invalid_argument(format!( + "vm sandboxes support at most {VM_MAX_BIND_MOUNTS} bind mounts, got {}", + mounts.len() + ))); + } + let mut targets: Vec = Vec::with_capacity(mounts.len()); + for mount in mounts { + driver_mounts::validate_absolute_mount_source(&mount.source, "vm mount source") + .map_err(Status::invalid_argument)?; + validate_no_control_chars(&mount.source, "vm mount source") + .map_err(Status::invalid_argument)?; + driver_mounts::validate_container_mount_target(&mount.target) + .map_err(Status::invalid_argument)?; + let normalized = driver_mounts::normalize_mount_target(&mount.target); + driver_mounts::validate_workspace_mount_target( + &normalized, + driver_mounts::DEFAULT_WORKSPACE_ROOT, + ) + .map_err(Status::invalid_argument)?; + for reserved in VM_RESERVED_GUEST_PATHS { + if driver_mounts::paths_overlap(Path::new(&normalized), Path::new(reserved)) { + return Err(Status::invalid_argument(format!( + "vm mount target '{}' conflicts with VM-internal path '{reserved}'", + mount.target + ))); + } + } + // Nested shares would make guest init create directories inside an + // already-mounted host share, so targets must be disjoint. + if let Some(existing) = targets.iter().find(|existing| { + driver_mounts::paths_overlap(Path::new(&normalized), Path::new(existing)) + }) { + return Err(Status::invalid_argument(if *existing == normalized { + format!("duplicate vm driver_config mount target '{normalized}'") + } else { + format!("vm mount target '{normalized}' overlaps mount target '{existing}'") + })); + } + targets.push(normalized); + } + Ok(()) +} + +/// Host filesystem checks, run at provisioning rather than in +/// `validate_sandbox` so a briefly missing source (for example an NFS share +/// not yet mounted after a reboot) fails provisioning visibly instead of +/// denying recovery of a persisted sandbox. +#[allow(clippy::result_large_err)] +async fn validate_vm_mount_sources(mounts: &[VmMountConfig]) -> Result<(), Status> { + for mount in mounts { + // libkrun opens the share root without following a final symlink. + let metadata = match tokio::fs::symlink_metadata(mount.host_source()).await { + Ok(metadata) => metadata, + Err(err) if err.kind() == std::io::ErrorKind::NotFound => { + return Err(Status::failed_precondition(format!( + "vm mount source path does not exist: {}", + mount.source + ))); + } + Err(err) => { + return Err(Status::failed_precondition(format!( + "cannot access vm mount source {}: {err}", + mount.source + ))); + } + }; + if metadata.file_type().is_symlink() { + return Err(Status::invalid_argument(format!( + "vm mount source must be a real directory, not a symbolic link: {}", + mount.source + ))); + } + if !metadata.is_dir() { + return Err(Status::invalid_argument(format!( + "vm mount source must be a directory, not a file: {}", + mount.source + ))); + } + // On macOS libkrun opens the share root during guest boot and panics + // if it cannot, so an unreadable source must fail here instead. + if let Err(err) = tokio::fs::read_dir(mount.host_source()).await { + return Err(Status::failed_precondition(format!( + "cannot access vm mount source {}: {err}", + mount.source + ))); + } + } + Ok(()) +} + #[allow(clippy::result_large_err)] fn validate_gpu_request(sandbox: &Sandbox, gpu_enabled: bool) -> Result<(), Status> { let spec = sandbox @@ -6862,6 +7117,17 @@ fn inject_guest_boundary_bundle( Ok(()) } +fn inject_guest_mount_manifest( + overlay_disk: &Path, + mounts: &[VmMountConfig], +) -> Result<(), String> { + let manifest = render_mount_manifest(mounts); + let guest_path = overlay_upper_path("/.openshell/mounts.manifest"); + write_rootfs_image_file(overlay_disk, &guest_path, manifest.as_bytes())?; + set_rootfs_image_file_mode(overlay_disk, &guest_path, 0o644)?; + Ok(()) +} + fn guest_boundary_config_path(generation: &str) -> String { format!("{GUEST_BOUNDARY_CONFIG_DIR}/bootstrap-{generation}.json") } @@ -12895,4 +13161,584 @@ mod tests { assert!(alpha.join(SANDBOX_STOPPED_FILE).exists()); } + + // ── VM mount config tests ────────────────────────────────────────── + + fn mount_test_config(enable_bind_mounts: bool) -> VmDriverConfig { + let mut config = VmDriverConfig { + enable_bind_mounts, + ..VmDriverConfig::default() + }; + config.resource_admission.enabled = false; + config + } + + #[test] + fn vm_mount_validation_rejects_when_resource_admission_enabled() { + let dir = std::env::temp_dir(); + let mounts = vec![VmMountConfig { + mount_type: None, + source: dir.display().to_string(), + target: "/sandbox/data".to_string(), + read_only: true, + }]; + let mut config = mount_test_config(true); + config.resource_admission.enabled = true; + let err = validate_vm_driver_mounts(&mounts, &config, false).unwrap_err(); + assert_eq!(err.code(), Code::FailedPrecondition); + assert!(err.message().contains("resource admission")); + } + + #[test] + fn vm_mount_config_deserializes_with_default_readonly() { + let json = serde_json::json!({ + "mounts": [{"source": "/host/data", "target": "/sandbox/data"}] + }); + let config: VmSandboxDriverConfig = serde_json::from_value(json).unwrap(); + assert_eq!(config.mounts.len(), 1); + assert_eq!(config.mounts[0].source, "/host/data"); + assert_eq!(config.mounts[0].target, "/sandbox/data"); + assert!(config.mounts[0].read_only); + } + + #[test] + fn vm_mount_config_accepts_docker_style_bind_type() { + let json = serde_json::json!({ + "mounts": [{"type": "bind", "source": "/host/data", "target": "/sandbox/data"}] + }); + let config: VmSandboxDriverConfig = serde_json::from_value(json).unwrap(); + assert!(matches!( + config.mounts[0].mount_type, + Some(VmMountType::Bind) + )); + assert!(config.mounts[0].read_only); + } + + #[test] + fn vm_mount_config_rejects_non_bind_types() { + for mount_type in ["volume", "tmpfs", "image"] { + let json = serde_json::json!({ + "mounts": [{"type": mount_type, "source": "x", "target": "/sandbox/data"}] + }); + let err = serde_json::from_value::(json).unwrap_err(); + assert!( + err.to_string().contains("expected `bind`"), + "{mount_type}: {err}" + ); + } + } + + #[test] + fn vm_mount_config_deserializes_explicit_readwrite() { + let json = serde_json::json!({ + "mounts": [{"source": "/host/data", "target": "/sandbox/data", "read_only": false}] + }); + let config: VmSandboxDriverConfig = serde_json::from_value(json).unwrap(); + assert!(!config.mounts[0].read_only); + } + + #[test] + fn vm_mount_validation_requires_enable_bind_mounts() { + let dir = std::env::temp_dir(); + let mounts = vec![VmMountConfig { + mount_type: None, + source: dir.display().to_string(), + target: "/sandbox/data".to_string(), + read_only: true, + }]; + let err = validate_vm_driver_mounts(&mounts, &mount_test_config(false), false).unwrap_err(); + assert_eq!(err.code(), Code::FailedPrecondition); + assert!(err.message().contains("enable_bind_mounts")); + } + + #[test] + fn vm_mount_validation_allows_when_enabled() { + let dir = std::env::temp_dir(); + let mounts = vec![VmMountConfig { + mount_type: None, + source: dir.display().to_string(), + target: "/sandbox/data".to_string(), + read_only: true, + }]; + assert!(validate_vm_driver_mounts(&mounts, &mount_test_config(true), false).is_ok()); + } + + #[test] + fn vm_mount_validation_rejects_relative_source() { + let mounts = vec![VmMountConfig { + mount_type: None, + source: "relative/path".to_string(), + target: "/sandbox/data".to_string(), + read_only: true, + }]; + let err = validate_vm_driver_mounts(&mounts, &mount_test_config(true), false).unwrap_err(); + assert_eq!(err.code(), Code::InvalidArgument); + } + + #[test] + fn vm_mount_validation_rejects_control_chars_in_source() { + // The launcher receives mounts as tab-separated arguments. + for source in ["/srv/data\tvfs9\trw", "/srv/data\n/etc"] { + let mounts = vec![VmMountConfig { + mount_type: None, + source: source.to_string(), + target: "/sandbox/data".to_string(), + read_only: true, + }]; + let err = + validate_vm_driver_mounts(&mounts, &mount_test_config(true), false).unwrap_err(); + assert_eq!(err.code(), Code::InvalidArgument); + assert!(err.message().contains("control characters")); + } + } + + #[test] + fn vm_mount_validation_rejects_reserved_openshell_target() { + let dir = std::env::temp_dir(); + let mounts = vec![VmMountConfig { + mount_type: None, + source: dir.display().to_string(), + target: "/opt/openshell/data".to_string(), + read_only: true, + }]; + let err = validate_vm_driver_mounts(&mounts, &mount_test_config(true), false).unwrap_err(); + assert_eq!(err.code(), Code::InvalidArgument); + } + + #[test] + fn vm_mount_validation_rejects_vm_internal_paths() { + let dir = std::env::temp_dir(); + for reserved in VM_RESERVED_GUEST_PATHS { + let mounts = vec![VmMountConfig { + mount_type: None, + source: dir.display().to_string(), + target: reserved.to_string(), + read_only: true, + }]; + let err = + validate_vm_driver_mounts(&mounts, &mount_test_config(true), false).unwrap_err(); + assert_eq!( + err.code(), + Code::InvalidArgument, + "expected rejection for target {reserved}" + ); + assert!( + err.message().contains("VM-internal path"), + "expected VM-internal path message for {reserved}, got: {}", + err.message() + ); + } + } + + #[test] + fn vm_mount_validation_rejects_duplicate_targets() { + let dir = std::env::temp_dir(); + let mounts = vec![ + VmMountConfig { + mount_type: None, + source: dir.display().to_string(), + target: "/sandbox/data".to_string(), + read_only: true, + }, + VmMountConfig { + mount_type: None, + source: dir.display().to_string(), + target: "/sandbox/data".to_string(), + read_only: false, + }, + ]; + let err = validate_vm_driver_mounts(&mounts, &mount_test_config(true), false).unwrap_err(); + assert_eq!(err.code(), Code::InvalidArgument); + assert!(err.message().contains("duplicate")); + } + + #[tokio::test] + async fn vm_mount_validation_defers_missing_source_to_provisioning() { + let mounts = vec![VmMountConfig { + mount_type: None, + source: "/nonexistent/openshell-vm-mount-source".to_string(), + target: "/sandbox/data".to_string(), + read_only: true, + }]; + assert!(validate_vm_driver_mounts(&mounts, &mount_test_config(true), false).is_ok()); + let err = validate_vm_mount_sources(&mounts).await.unwrap_err(); + assert_eq!(err.code(), Code::FailedPrecondition); + } + + #[test] + fn vm_mount_validation_rejects_nested_targets() { + let dir = std::env::temp_dir().display().to_string(); + for (first, second) in [ + ("/sandbox/a", "/sandbox/a/b"), + ("/sandbox/a/b", "/sandbox/a"), + ] { + let mounts = vec![ + VmMountConfig { + mount_type: None, + source: dir.clone(), + target: first.to_string(), + read_only: true, + }, + VmMountConfig { + mount_type: None, + source: dir.clone(), + target: second.to_string(), + read_only: true, + }, + ]; + let err = + validate_vm_driver_mounts(&mounts, &mount_test_config(true), false).unwrap_err(); + assert_eq!(err.code(), Code::InvalidArgument); + assert!(err.message().contains("overlaps"), "{first} vs {second}"); + } + } + + #[test] + fn vm_mount_validation_allows_sibling_targets() { + let dir = std::env::temp_dir().display().to_string(); + let mounts = vec![ + VmMountConfig { + mount_type: None, + source: dir.clone(), + target: "/sandbox/a".to_string(), + read_only: true, + }, + VmMountConfig { + mount_type: None, + source: dir, + target: "/sandbox/ab".to_string(), + read_only: true, + }, + ]; + assert!(validate_vm_driver_mounts(&mounts, &mount_test_config(true), false).is_ok()); + } + + #[test] + fn vm_mount_validation_empty_passes() { + assert!(validate_vm_driver_mounts(&[], &mount_test_config(false), false).is_ok()); + } + + #[test] + fn vm_mount_validation_rejects_qemu_backend() { + let dir = std::env::temp_dir(); + let mounts = vec![VmMountConfig { + mount_type: None, + source: dir.display().to_string(), + target: "/sandbox/data".to_string(), + read_only: true, + }]; + let err = validate_vm_driver_mounts(&mounts, &mount_test_config(true), true).unwrap_err(); + assert_eq!(err.code(), Code::FailedPrecondition); + assert!(err.message().contains("QEMU")); + } + + #[test] + fn vm_mount_validation_rejects_too_many_mounts() { + let dir = std::env::temp_dir().display().to_string(); + let mounts = |count: usize| -> Vec { + (0..count) + .map(|i| VmMountConfig { + mount_type: None, + source: dir.clone(), + target: format!("/sandbox/data{i}"), + read_only: true, + }) + .collect() + }; + let config = mount_test_config(true); + assert!(validate_vm_driver_mounts(&mounts(VM_MAX_BIND_MOUNTS), &config, false).is_ok()); + let err = + validate_vm_driver_mounts(&mounts(VM_MAX_BIND_MOUNTS + 1), &config, false).unwrap_err(); + assert_eq!(err.code(), Code::InvalidArgument); + assert!(err.message().contains("at most"), "{}", err.message()); + } + + #[cfg(unix)] + #[tokio::test] + async fn vm_mount_validation_rejects_unreadable_source() { + use std::os::unix::fs::PermissionsExt; + + if rustix::process::geteuid().is_root() { + return; // root bypasses directory permissions + } + let source = tempfile::tempdir().unwrap(); + fs::set_permissions(source.path(), fs::Permissions::from_mode(0o000)).unwrap(); + let mounts = vec![VmMountConfig { + mount_type: None, + source: source.path().display().to_string(), + target: "/sandbox/data".to_string(), + read_only: true, + }]; + let result = validate_vm_mount_sources(&mounts).await; + // Restore access before asserting so the temp dir can always be removed. + fs::set_permissions(source.path(), fs::Permissions::from_mode(0o700)).unwrap(); + let err = result.unwrap_err(); + assert_eq!(err.code(), Code::FailedPrecondition); + assert!(err.message().contains("cannot access"), "{}", err.message()); + } + + #[cfg(unix)] + #[tokio::test] + async fn vm_mount_validation_rejects_symlink_source() { + let parent = tempfile::tempdir().unwrap(); + let real = parent.path().join("real"); + let link = parent.path().join("link"); + fs::create_dir(&real).unwrap(); + std::os::unix::fs::symlink(&real, &link).unwrap(); + // A trailing slash would make the stat resolve the link. + for suffix in ["", "/"] { + let mounts = vec![VmMountConfig { + mount_type: None, + source: format!("{}{suffix}", link.display()), + target: "/sandbox/data".to_string(), + read_only: true, + }]; + let err = validate_vm_mount_sources(&mounts).await.unwrap_err(); + assert_eq!(err.code(), Code::InvalidArgument); + assert!(err.message().contains("symbolic link"), "{}", err.message()); + } + } + + #[tokio::test] + async fn vm_mount_validation_rejects_file_source() { + let file = tempfile::NamedTempFile::new().unwrap(); + let mounts = vec![VmMountConfig { + mount_type: None, + source: file.path().display().to_string(), + target: "/sandbox/data".to_string(), + read_only: true, + }]; + let err = validate_vm_mount_sources(&mounts).await.unwrap_err(); + assert_eq!(err.code(), Code::InvalidArgument); + assert!(err.message().contains("must be a directory")); + } + + #[tokio::test] + async fn vm_mount_validation_reports_inaccessible_source() { + // A path below a regular file fails stat with something other than + // NotFound, even as root. + let file = tempfile::NamedTempFile::new().unwrap(); + let mounts = vec![VmMountConfig { + mount_type: None, + source: file.path().join("child").display().to_string(), + target: "/sandbox/data".to_string(), + read_only: true, + }]; + let err = validate_vm_mount_sources(&mounts).await.unwrap_err(); + assert_eq!(err.code(), Code::FailedPrecondition); + assert!(err.message().contains("cannot access"), "{}", err.message()); + } + + #[tokio::test] + async fn provisioning_checks_mount_sources_before_resolving_images() { + let directory = tempfile::tempdir().unwrap(); + let mut driver = test_driver_with_extensions(LifecycleExtensionRegistry::new()); + driver.config.state_dir = directory.path().to_path_buf(); + let sandbox = Sandbox { + id: "sb-mount-source".to_string(), + name: "mount-source".to_string(), + spec: Some(SandboxSpec { + launch_authentication: test_launch_authentication("mount-source").0, + template: Some(SandboxTemplate { + driver_config: Some(mounts_driver_config( + "/nonexistent/openshell-vm-mount-source", + "/sandbox/data", + )), + ..Default::default() + }), + ..Default::default() + }), + ..Default::default() + }; + let state_dir = directory.path().join("sandbox"); + driver.registry.lock().await.insert( + sandbox.id.clone(), + SandboxRecord { + snapshot: sandbox.clone(), + state_dir: state_dir.clone(), + process: None, + provisioning_task: None, + preparation: None, + gpu_bdf: None, + deleting: false, + }, + ); + let mut events = driver.events.subscribe(); + let error = driver + .provision_sandbox_inner( + sandbox, + "invalid image reference".to_string(), + state_dir, + None, + OverlayPreparation::Fresh, + ) + .await + .unwrap_err(); + assert!( + error.message().contains("does not exist"), + "{}", + error.message() + ); + assert!(matches!( + events.try_recv(), + Err(broadcast::error::TryRecvError::Empty) + )); + } + + /// Runs `prepare_virtiofs_target` from the guest init script against a + /// host directory standing in for the guest root. + #[cfg(unix)] + fn run_prepare_virtiofs_target(root: &Path, target: &str) -> std::process::Output { + let script = include_str!("../scripts/openshell-vm-sandbox-init.sh"); + let start = script + .find("prepare_virtiofs_target() {") + .expect("init script defines prepare_virtiofs_target"); + let end = start + script[start..].find("\n}\n").expect("function end") + 3; + let program = format!( + "set -euo pipefail\nts() {{ echo \"$@\"; }}\n{}\nprepare_virtiofs_target \"$1\" \"$(id -u):$(id -g)\"\n", + &script[start..end] + ); + std::process::Command::new("bash") + .args(["-c", &program, "test", target]) + .env("ROOT_PREFIX", root) + .output() + .unwrap() + } + + #[cfg(unix)] + #[test] + fn vm_init_prepare_virtiofs_target_creates_missing_directories() { + let root = tempfile::tempdir().unwrap(); + fs::create_dir(root.path().join("sandbox")).unwrap(); + let output = run_prepare_virtiofs_target(root.path(), "/sandbox/a/b"); + assert!(output.status.success(), "{output:?}"); + let expected = root.path().join("sandbox/a/b"); + assert!(expected.is_dir()); + assert_eq!( + String::from_utf8_lossy(&output.stdout).trim(), + expected.display().to_string() + ); + } + + #[cfg(unix)] + #[test] + fn vm_init_prepare_virtiofs_target_refuses_symlinks() { + let root = tempfile::tempdir().unwrap(); + let elsewhere = tempfile::tempdir().unwrap(); + fs::create_dir(root.path().join("sandbox")).unwrap(); + std::os::unix::fs::symlink(elsewhere.path(), root.path().join("sandbox/link")).unwrap(); + let output = run_prepare_virtiofs_target(root.path(), "/sandbox/link/x"); + assert!(!output.status.success()); + assert!(String::from_utf8_lossy(&output.stderr).contains("traverses symlink")); + assert!(!elsewhere.path().join("x").exists()); + } + + #[cfg(unix)] + #[test] + fn vm_init_prepare_virtiofs_target_refuses_non_directory() { + let root = tempfile::tempdir().unwrap(); + fs::create_dir(root.path().join("sandbox")).unwrap(); + fs::write(root.path().join("sandbox/file"), b"").unwrap(); + let output = run_prepare_virtiofs_target(root.path(), "/sandbox/file/x"); + assert!(!output.status.success()); + assert!(String::from_utf8_lossy(&output.stderr).contains("is not a directory")); + } + + fn mounts_driver_config(source: &str, target: &str) -> Struct { + let string = |s: &str| Value { + kind: Some(Kind::StringValue(s.to_string())), + }; + let mount = Struct { + fields: [ + ("source".to_string(), string(source)), + ("target".to_string(), string(target)), + ] + .into_iter() + .collect(), + }; + Struct { + fields: std::iter::once(( + "mounts".to_string(), + Value { + kind: Some(Kind::ListValue(prost_types::ListValue { + values: vec![Value { + kind: Some(Kind::StructValue(mount)), + }], + })), + }, + )) + .collect(), + } + } + + #[tokio::test] + async fn recovery_denied_when_bind_mounts_are_revoked() { + let directory = tempfile::tempdir().unwrap(); + let source = tempfile::tempdir().unwrap(); + let mut driver = test_driver_with_extensions(LifecycleExtensionRegistry::new()); + driver.config.state_dir = directory.path().to_path_buf(); + driver.config.default_image = "invalid image reference".to_string(); + driver.config.allow_driver_config = true; + driver.config.resource_admission.enabled = false; + driver.config.enable_bind_mounts = true; + let sandbox = Sandbox { + id: "sb-mount-restore".to_string(), + name: "mount-restore".to_string(), + spec: Some(SandboxSpec { + launch_authentication: test_launch_authentication("mount-restore").0, + template: Some(SandboxTemplate { + driver_config: Some(mounts_driver_config( + &source.path().display().to_string(), + "/sandbox/data", + )), + ..Default::default() + }), + ..Default::default() + }), + ..Default::default() + }; + decode_launch_authentication(&sandbox.spec.as_ref().unwrap().launch_authentication) + .expect("credentials are valid, so only the mount gate can deny recovery"); + driver + .validate_sandbox(&sandbox) + .expect("mounts are admitted while enabled"); + + driver.config.enable_bind_mounts = false; + let err = driver.validate_sandbox(&sandbox).unwrap_err(); + assert!( + err.message().contains("enable_bind_mounts"), + "{}", + err.message() + ); + let state_dir = sandbox_state_dir(directory.path(), &sandbox.id).unwrap(); + create_private_dir_all(&state_dir).await.unwrap(); + write_sandbox_request(&state_dir, &sandbox).await.unwrap(); + let accepted = driver + .restore_persisted_sandbox(sandbox, state_dir, true, &tracing::Span::current()) + .await; + assert!(!accepted, "revoked bind mounts must deny restoration"); + assert!(driver.registry.lock().await.is_empty()); + } + + #[test] + fn vm_mount_manifest_renders_tab_separated_entries() { + let mounts = [ + VmMountConfig { + mount_type: None, + source: "/host/a".to_string(), + target: "/sandbox/a".to_string(), + read_only: true, + }, + VmMountConfig { + mount_type: None, + source: "/host/b".to_string(), + target: "/sandbox/b".to_string(), + read_only: false, + }, + ]; + assert_eq!( + render_mount_manifest(&mounts), + "osfs0\t/sandbox/a\tro\nosfs1\t/sandbox/b\trw\n" + ); + } } diff --git a/crates/openshell-driver-vm/src/ffi.rs b/crates/openshell-driver-vm/src/ffi.rs index f84ea35743..db1deb619c 100644 --- a/crates/openshell-driver-vm/src/ffi.rs +++ b/crates/openshell-driver-vm/src/ffi.rs @@ -54,6 +54,14 @@ type KrunDisableImplicitVsock = unsafe extern "C" fn(ctx_id: u32) -> i32; type KrunAddVsock = unsafe extern "C" fn(ctx_id: u32, tsi_features: u32) -> i32; type KrunAddVsockPort2 = unsafe extern "C" fn(ctx_id: u32, port: u32, filepath: *const c_char, listen: bool) -> i32; +type KrunAddVirtiofs4 = unsafe extern "C" fn( + ctx_id: u32, + tag: *const c_char, + path: *const c_char, + shm_size: u64, + read_only: bool, + semantics: u32, +) -> i32; // Field names mirror the libkrun C API symbol names (`krun_*`); preserving // the prefix keeps the FFI binding 1:1 with the upstream library. @@ -72,6 +80,9 @@ pub struct LibKrun { pub krun_disable_implicit_vsock: KrunDisableImplicitVsock, pub krun_add_vsock: KrunAddVsock, pub krun_add_vsock_port2: KrunAddVsockPort2, + /// Load error kept so runtimes without virtiofs still boot sandboxes + /// that request no bind mounts, and mount requests report why. + pub krun_add_virtiofs4: Result, } static LIBKRUN: OnceLock = OnceLock::new(); @@ -134,6 +145,7 @@ impl LibKrun { )?, krun_add_vsock: load_symbol(library, b"krun_add_vsock\0", &libkrun_path)?, krun_add_vsock_port2: load_symbol(library, b"krun_add_vsock_port2\0", &libkrun_path)?, + krun_add_virtiofs4: load_symbol(library, b"krun_add_virtiofs4\0", &libkrun_path), }) } } diff --git a/crates/openshell-driver-vm/src/lib.rs b/crates/openshell-driver-vm/src/lib.rs index 22f45c96e9..66041ea034 100644 --- a/crates/openshell-driver-vm/src/lib.rs +++ b/crates/openshell-driver-vm/src/lib.rs @@ -47,5 +47,6 @@ pub use lifecycle::{ }; #[cfg(feature = "compute-driver")] pub use runtime::{ - VM_RUNTIME_DIR_ENV, VmBackend, VmLaunchConfig, VsockPortMap, configured_runtime_dir, run_vm, + VM_RUNTIME_DIR_ENV, VmBackend, VmLaunchConfig, VmMount, VsockPortMap, configured_runtime_dir, + run_vm, }; diff --git a/crates/openshell-driver-vm/src/main.rs b/crates/openshell-driver-vm/src/main.rs index 0fbaf7b2fe..d08243a01f 100644 --- a/crates/openshell-driver-vm/src/main.rs +++ b/crates/openshell-driver-vm/src/main.rs @@ -9,7 +9,7 @@ use openshell_core::proto::compute::v1::compute_driver_server::ComputeDriverServ #[cfg(target_os = "macos")] use openshell_driver_vm::{VM_RUNTIME_DIR_ENV, configured_runtime_dir}; use openshell_driver_vm::{ - VmBackend, VmDriver, VmDriverConfig, VmLaunchConfig, VsockPortMap, procguard, run_vm, + VmBackend, VmDriver, VmDriverConfig, VmLaunchConfig, VmMount, VsockPortMap, procguard, run_vm, }; use std::io; use std::net::SocketAddr; @@ -227,6 +227,9 @@ struct Args { #[arg(long, env = "OPENSHELL_VM_PROXY_CA_BUNDLE")] proxy_ca_bundle: Option, + #[arg(long, env = "OPENSHELL_VM_ENABLE_BIND_MOUNTS", default_value_t = false)] + enable_bind_mounts: bool, + #[arg(long, env = "OPENSHELL_VM_ROOTFS_TAR_STAGING_DIR")] rootfs_tar_staging_dir: Option, @@ -247,6 +250,9 @@ struct Args { #[arg(long, hide = true)] vm_vsock_control_socket: Option, + + #[arg(long, hide = true)] + vm_mount: Vec, } #[tokio::main] @@ -329,6 +335,7 @@ async fn main() -> Result<()> { sandbox_gid: args.sandbox_gid, rootfs_tar_staging_dir: args.rootfs_tar_staging_dir.clone(), rootfs_tar_max_bytes: args.rootfs_tar_max_bytes, + enable_bind_mounts: args.enable_bind_mounts, }) .await .map_err(|err| miette::miette!("{err}"))?; @@ -618,6 +625,12 @@ fn build_vm_launch_config(args: &Args) -> std::result::Result return Err(format!("unknown VM backend: {other}")), }; + let mounts = args + .vm_mount + .iter() + .map(|m| parse_vm_mount_arg(m)) + .collect::, _>>()?; + Ok(VmLaunchConfig { root_disk, overlay_disk, @@ -651,6 +664,36 @@ fn build_vm_launch_config(args: &Args) -> std::result::Result std::result::Result { + // Split from the right: the driver generates the tag and mode, so a tab + // in the source can never shift them. + let mut parts = arg.rsplitn(3, '\t'); + let mode = parts + .next() + .ok_or_else(|| format!("invalid --vm-mount format: {arg}"))?; + let tag = parts + .next() + .ok_or_else(|| format!("invalid --vm-mount format (missing tag): {arg}"))?; + let source = parts + .next() + .ok_or_else(|| format!("invalid --vm-mount format (missing source): {arg}"))?; + let read_only = match mode { + "ro" => true, + "rw" => false, + _ => { + return Err(format!( + "invalid --vm-mount mode '{mode}': expected 'ro' or 'rw'" + )); + } + }; + Ok(VmMount { + host_path: PathBuf::from(source), + tag: tag.to_string(), + read_only, }) } @@ -710,7 +753,7 @@ fn maybe_reexec_internal_vm_with_runtime_env() -> Result<()> { mod tests { use super::{ Args, ComputeDriverListenMode, PeerCredentials, authorize_peer_credentials, - compute_driver_listen_mode, + compute_driver_listen_mode, parse_vm_mount_arg, }; use clap::Parser; use std::path::PathBuf; @@ -950,4 +993,38 @@ mod tests { } ); } + + #[test] + fn parse_vm_mount_arg_parses_readonly() { + let m = parse_vm_mount_arg("/host/src\tosfs0\tro").unwrap(); + assert_eq!(m.host_path, PathBuf::from("/host/src")); + assert_eq!(m.tag, "osfs0"); + assert!(m.read_only); + } + + #[test] + fn parse_vm_mount_arg_parses_readwrite() { + let m = parse_vm_mount_arg("/host/src\tosfs1\trw").unwrap(); + assert!(!m.read_only); + } + + #[test] + fn parse_vm_mount_arg_keeps_tag_and_mode_when_source_has_tab() { + let m = parse_vm_mount_arg("/host/src\tvfs9\trw\tosfs0\tro").unwrap(); + assert_eq!(m.host_path, PathBuf::from("/host/src\tvfs9\trw")); + assert_eq!(m.tag, "osfs0"); + assert!(m.read_only); + } + + #[test] + fn parse_vm_mount_arg_rejects_unknown_mode() { + let err = parse_vm_mount_arg("/host/src\tosfs0\treadonly").unwrap_err(); + assert!(err.contains("expected 'ro' or 'rw'")); + } + + #[test] + fn parse_vm_mount_arg_rejects_missing_fields() { + assert!(parse_vm_mount_arg("/host/src\tosfs0").is_err()); + assert!(parse_vm_mount_arg("/host/src").is_err()); + } } diff --git a/crates/openshell-driver-vm/src/runtime.rs b/crates/openshell-driver-vm/src/runtime.rs index 5556a44448..fbccba270a 100644 --- a/crates/openshell-driver-vm/src/runtime.rs +++ b/crates/openshell-driver-vm/src/runtime.rs @@ -32,6 +32,14 @@ pub struct VsockPortMap { pub host_initiated: bool, } +/// A host directory shared into the VM guest via virtiofs. +#[derive(Debug, Clone)] +pub struct VmMount { + pub tag: String, + pub host_path: PathBuf, + pub read_only: bool, +} + pub struct VmLaunchConfig { pub root_disk: PathBuf, pub overlay_disk: PathBuf, @@ -49,6 +57,7 @@ pub struct VmLaunchConfig { pub gpu_bdf: Option, pub vsock_cid: Option, pub vsock_port_map: Option, + pub mounts: Vec, } pub fn run_vm(config: &VmLaunchConfig) -> Result<(), String> { @@ -59,6 +68,9 @@ pub fn run_vm(config: &VmLaunchConfig) -> Result<(), String> { } fn run_qemu_vm(config: &VmLaunchConfig) -> Result<(), String> { + if !config.mounts.is_empty() { + return Err("virtiofs mounts are not yet supported with the QEMU backend".to_string()); + } let gpu_bdf = config .gpu_bdf .as_deref() @@ -307,7 +319,8 @@ fn run_libkrun_vm(config: &VmLaunchConfig) -> Result<(), String> { // Arm procguard before forking libkrun so the VM worker cannot outlive // the launcher. No network helper is started: the only host/guest data - // path is the protected vsock mapping below. + // paths are the protected vsock mapping below and any operator-enabled + // virtiofs shares. if let Err(err) = procguard::die_with_parent_cleanup(procguard_kill_children) { return Err(format!("procguard arm failed: {err}")); } @@ -336,6 +349,10 @@ fn run_libkrun_vm(config: &VmLaunchConfig) -> Result<(), String> { vm.add_vsock_port(port_map)?; } + for mount in &config.mounts { + vm.add_virtiofs(&mount.tag, &mount.host_path, mount.read_only)?; + } + vm.set_console_output(&config.console_output)?; let env = libkrun_guest_env(config); @@ -660,6 +677,32 @@ impl VmContext { ) } + fn add_virtiofs(&self, tag: &str, host_path: &Path, read_only: bool) -> Result<(), String> { + // KRUN_SEMANTICS_LINUX_SIMPLIFIED: guest ownership is not recorded on + // host files and permission bits are stored on the host directly, not + // in extended attributes. Only the macOS passthrough applies that; on + // Linux the mode just disables attribute caching and idmapped mounts. + const SEMANTICS_SIMPLIFIED: u32 = 1; + let tag_c = CString::new(tag).map_err(|e| format!("invalid virtiofs tag: {e}"))?; + let path_c = path_to_cstring(host_path)?; + let add_virtiofs4 = *self.krun.krun_add_virtiofs4.as_ref().map_err(|err| { + format!("VM bind mounts need a libkrun runtime with virtiofs support: {err}") + })?; + check( + unsafe { + add_virtiofs4( + self.ctx_id, + tag_c.as_ptr(), + path_c.as_ptr(), + 0, + read_only, + SEMANTICS_SIMPLIFIED, + ) + }, + "krun_add_virtiofs4", + ) + } + fn start_enter(&self) -> i32 { unsafe { (self.krun.krun_start_enter)(self.ctx_id) } } @@ -776,6 +819,7 @@ mod tests { gpu_bdf: Some("0000:01:00.0".to_string()), vsock_cid: Some(4), vsock_port_map: None, + mounts: Vec::new(), } } diff --git a/crates/openshell-gateway/src/vm.rs b/crates/openshell-gateway/src/vm.rs index d0c00cd786..c5f4ca7f8b 100644 --- a/crates/openshell-gateway/src/vm.rs +++ b/crates/openshell-gateway/src/vm.rs @@ -122,6 +122,10 @@ pub struct VmComputeConfig { pub provider_spiffe_workload_api_tcp_endpoint: Option, #[serde(default)] pub provider_spiffe_allow_guest_tcp: bool, + + /// Allow bind-mounting host directories into VM sandboxes. + #[serde(default)] + pub enable_bind_mounts: bool, } impl VmComputeConfig { @@ -254,6 +258,7 @@ impl Default for VmComputeConfig { proxy_ca_bundle: None, provider_spiffe_workload_api_tcp_endpoint: None, provider_spiffe_allow_guest_tcp: false, + enable_bind_mounts: false, } } } @@ -575,6 +580,9 @@ pub async fn spawn( command.arg("--guest-tls-ca").arg(tls.ca); } append_vm_proxy_and_spiffe_args(&mut command, vm_config); + if vm_config.enable_bind_mounts { + command.arg("--enable-bind-mounts"); + } let mut child = command.spawn().map_err(|e| { Error::execution(format!( diff --git a/docs/how-it-works/gateways/configuration.mdx b/docs/how-it-works/gateways/configuration.mdx index 30a75f7f6a..2551291347 100644 --- a/docs/how-it-works/gateways/configuration.mdx +++ b/docs/how-it-works/gateways/configuration.mdx @@ -1314,6 +1314,11 @@ overlay_disk_mib = 4096 # the exposure; host-only sockets are never exposed automatically. # provider_spiffe_workload_api_tcp_endpoint = "tcp:192.0.2.10:8081" # provider_spiffe_allow_guest_tcp = true +# Unsafe operator override. Host bind mounts expose gateway-host paths inside +# VM sandboxes and can negate OpenShell isolation and filesystem controls. +# Requires allow_driver_config = true and +# [openshell.drivers.vm.resource_admission] enabled = false; libkrun only. +# enable_bind_mounts = false # Where the gateway stages rootfs tar archives for `--from ./rootfs.tar`. # Defaults to /rootfs-tar-staging. The gateway creates one # request-scoped subdirectory per staging slot and removes it after use. diff --git a/docs/how-it-works/sandboxes/runtimes.mdx b/docs/how-it-works/sandboxes/runtimes.mdx index cc76cf98d2..9db9194a8f 100644 --- a/docs/how-it-works/sandboxes/runtimes.mdx +++ b/docs/how-it-works/sandboxes/runtimes.mdx @@ -230,6 +230,36 @@ VM sandboxes have no network interface. All traffic flows through the OpenShell If the gateway is briefly unavailable while a VM starts, its supervisor retries the session connection. The sandbox stays in provisioning until the gateway accepts the session. +### MicroVM Mounts + +The VM driver shares gateway-host directories into the guest with virtiofs. Each entry in `mounts` takes `source` (absolute host directory), `target`, optional `read_only` (default `true`), and optional `type`, which must be `bind` so Docker and Podman bind entries work unchanged: + +```shell +openshell sandbox create \ + --driver-config-json '{"vm":{"mounts":[{"type":"bind","source":"/srv/datasets","target":"/sandbox/datasets"}]}}' \ + -- claude +``` + +A sandbox takes at most 32 mounts on an arm64 gateway host and three on an x86_64 host. Single-file sources, sources that are symbolic links, `selinux_label`, and other mount types are not supported. VM mounts work only with the libkrun backend, so GPU sandboxes cannot use them. + + +VM bind mounts expose host files to sandbox workloads. Additionally, libkrun does not confine a compromised guest kernel to the configured source directory; it may access other host files permitted by the gateway user's permissions. VM mounts require the same opt-in as [Docker bind mounts](#docker-mounts): + +```toml +[openshell.drivers.vm] +allow_driver_config = true +enable_bind_mounts = true + +[openshell.drivers.vm.resource_admission] +enabled = false +``` + + + +Targets cannot replace the workspace root, OpenShell paths, or VM-internal paths such as `/tmp`, `/proc`, `/dev`, and `/srv`, and cannot be nested inside one another. The guest refuses a target that passes through a symlink in the image. + +On Linux with a non-root gateway, the sandbox can create files in a read-write mount only when its UID and GID match the gateway user's. Set `sandbox_uid` and `sandbox_gid` under `[openshell.drivers.vm]` to match. Otherwise the sandbox cannot create files, and its access to existing files follows the host permission bits for its numeric UID and GID. + ## Kubernetes Driver The Kubernetes driver runs sandboxes as pods in a sandbox namespace. It requires the [Agent Sandbox](https://github.com/kubernetes-sigs/agent-sandbox) controller. Install the gateway with the Helm chart; see [Kubernetes setup](/kubernetes/setup). diff --git a/e2e/rust/Cargo.toml b/e2e/rust/Cargo.toml index 01315baef5..7659015625 100644 --- a/e2e/rust/Cargo.toml +++ b/e2e/rust/Cargo.toml @@ -69,6 +69,11 @@ name = "rootfs_tar" path = "tests/rootfs_tar.rs" required-features = ["e2e-vm"] +[[test]] +name = "vm_bind_mount" +path = "tests/vm_bind_mount.rs" +required-features = ["e2e-vm"] + [[test]] name = "docker_preflight" path = "tests/docker_preflight.rs" diff --git a/e2e/rust/e2e-vm.sh b/e2e/rust/e2e-vm.sh index ebb160c987..d74359e0c3 100755 --- a/e2e/rust/e2e-vm.sh +++ b/e2e/rust/e2e-vm.sh @@ -281,6 +281,13 @@ gateway_id = "${GATEWAY_NAME}" [openshell.drivers.vm] EOF +# A rootless Linux libkrun refuses file creation on read-write bind mounts by +# a guest UID/GID other than its own, so vm_bind_mount runs the sandbox as the +# gateway user there. +VM_IDENTITY_ARGS=() +if [ "$(uname -s)" = "Linux" ] && [ "$(id -u)" != "0" ]; then + VM_IDENTITY_ARGS=(--sandbox-uid "$(id -u)" --sandbox-gid "$(id -g)") +fi if [ "${OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER:-0}" = "1" ]; then printf 'socket_path = "%s"\n' "${DRIVER_SOCKET}" >>"${GATEWAY_CONFIG}" else @@ -288,8 +295,20 @@ else grpc_endpoint = "https://host.openshell.internal:${HOST_PORT}" driver_dir = "${DRIVER_DIR}" state_dir = "${RUN_STATE_DIR}" +enable_bind_mounts = true EOF + if [ "${#VM_IDENTITY_ARGS[@]}" -gt 0 ]; then + printf 'sandbox_uid = %s\nsandbox_gid = %s\n' "$(id -u)" "$(id -g)" >>"${GATEWAY_CONFIG}" + fi fi +# Host bind mounts are exercised by vm_bind_mount, mirroring the Docker and +# Podman e2e gateways. An external driver must acknowledge the same policy. +# Keep [openshell.drivers.vm] the last table: vm_corporate_proxy appends bare +# keys to it. +cat >>"${GATEWAY_CONFIG}" <"${DRIVER_LOG}" 2>&1 & DRIVER_PID=$! e2e_wait_for_socket \ @@ -410,6 +432,7 @@ else run_e2e_test ephemeral_cleanup run_e2e_test host_gateway_alias run_e2e_test vm_overlay + run_e2e_test vm_bind_mount run_e2e_test vm_gateway_start run_e2e_test vm_corporate_proxy fi diff --git a/e2e/rust/tests/vm_bind_mount.rs b/e2e/rust/tests/vm_bind_mount.rs new file mode 100644 index 0000000000..069c7d9d66 --- /dev/null +++ b/e2e/rust/tests/vm_bind_mount.rs @@ -0,0 +1,119 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! VM driver host bind mounts over virtiofs: a read-write share round-trips +//! data with the host, accepts new files and directories, and a read-only +//! share rejects writes. +//! +//! On Linux, libkrun running as a non-root user refuses file creation by a +//! guest user whose UID/GID differ from its own, so `e2e-vm.sh` sets the +//! sandbox UID/GID to the gateway user's there. +//! +//! Needs the libkrun backend (macOS HVF or Linux KVM); QEMU rejects virtiofs +//! mounts. The e2e VM gateway configures no GPU or QEMU-only lifecycle +//! extension, so this non-GPU sandbox always launches on libkrun. + +use std::fs; +use std::os::unix::fs::PermissionsExt; + +use openshell_e2e::harness::sandbox::SandboxGuard; + +const RW_TARGET: &str = "/sandbox/e2e-bind"; +const RO_TARGET: &str = "/sandbox/e2e-bind-ro"; + +fn shared_host_dir(prefix: &str) -> tempfile::TempDir { + let dir = tempfile::Builder::new() + .prefix(prefix) + .tempdir() + .expect("create bind mount host dir"); + fs::set_permissions(dir.path(), fs::Permissions::from_mode(0o777)) + .expect("make bind mount host dir writable by sandbox user"); + let input = dir.path().join("input.txt"); + fs::write(&input, "host-bind-ok").expect("seed bind mount host dir"); + fs::set_permissions(&input, fs::Permissions::from_mode(0o666)) + .expect("make bind mount input readable by sandbox user"); + dir +} + +fn seed_world_writable(path: &std::path::Path) { + fs::write(path, "").expect("seed bind mount output file"); + fs::set_permissions(path, fs::Permissions::from_mode(0o666)) + .expect("make bind mount output writable by sandbox user"); +} + +#[tokio::test] +async fn vm_bind_mount() { + let rw_dir = shared_host_dir("openshell-e2e-vm-bind-rw-"); + let ro_dir = shared_host_dir("openshell-e2e-vm-bind-ro-"); + seed_world_writable(&rw_dir.path().join("output.txt")); + + // `type: bind` keeps the entry identical to Docker/Podman bind mounts. + let driver_config = serde_json::json!({ + "vm": { + "mounts": [ + { + "type": "bind", + "source": rw_dir.path(), + "target": RW_TARGET, + "read_only": false + }, + { + "source": ro_dir.path(), + "target": RO_TARGET + } + ] + } + }) + .to_string(); + + let script = format!( + "set -eu; \ + test \"$(cat {RW_TARGET}/input.txt)\" = host-bind-ok; \ + test \"$(cat {RO_TARGET}/input.txt)\" = host-bind-ok; \ + printf sandbox-bind-ok > {RW_TARGET}/output.txt; \ + printf sandbox-created > {RW_TARGET}/created.txt; \ + mkdir {RW_TARGET}/created-dir; \ + printf sandbox-nested > {RW_TARGET}/created-dir/nested.txt; \ + if printf x > {RO_TARGET}/input.txt 2>/dev/null; then echo ro-writable; exit 1; fi; \ + echo vm-bind-ok" + ); + let mut sandbox = SandboxGuard::create(&[ + "--driver-config-json", + &driver_config, + "--", + "sh", + "-lc", + &script, + ]) + .await + .expect("sandbox create with VM bind mounts"); + + assert!( + sandbox.create_output.contains("vm-bind-ok"), + "sandbox should read both shares, write and create in the rw share, and be denied on the ro share:\n{}", + sandbox.create_output + ); + + sandbox.cleanup().await; + let output = fs::read_to_string(rw_dir.path().join("output.txt")) + .expect("read sandbox output from rw host dir"); + assert_eq!(output, "sandbox-bind-ok"); + let created = + fs::read_to_string(rw_dir.path().join("created.txt")).expect("read sandbox-created file"); + assert_eq!(created, "sandbox-created"); + assert!( + rw_dir.path().join("created-dir").is_dir(), + "sandbox-created directory must exist on the host" + ); + let nested = fs::read_to_string(rw_dir.path().join("created-dir/nested.txt")) + .expect("read file inside sandbox-created directory"); + assert_eq!(nested, "sandbox-nested"); + // Overwriting an existing world-writable file is the write libkrun would + // otherwise allow, so only the read-only share can be what refuses it. + let ro_input = + fs::read_to_string(ro_dir.path().join("input.txt")).expect("read input from ro host dir"); + assert_eq!( + ro_input, "host-bind-ok", + "read-only share must not receive writes" + ); +} diff --git a/skills/debug-openshell-cluster/SKILL.md b/skills/debug-openshell-cluster/SKILL.md index ffedd4504a..8e9ab204fe 100644 --- a/skills/debug-openshell-cluster/SKILL.md +++ b/skills/debug-openshell-cluster/SKILL.md @@ -1117,6 +1117,14 @@ Use the VM driver logs and host diagnostics available in the user's environment. - Host virtualization support is enabled. - The sandbox supervisor can establish its authenticated gateway session. +For `vm.mounts` (virtiofs host bind mounts) failures, also verify: + +- `[openshell.drivers.vm]` sets `allow_driver_config = true` and `enable_bind_mounts = true`; an external VM driver needs `--enable-bind-mounts` and `allow_driver_config` in `--admission-config-json`. +- `[openshell.drivers.vm.resource_admission]` sets `enabled = false`; bind mounts are rejected while resource admission is on. +- The libkrun runtime exports `krun_add_virtiofs4`; otherwise the driver reports "VM bind mounts need a libkrun runtime with virtiofs support". The driver extracts its runtime under `$XDG_DATA_HOME/openshell/vm-runtime//` (default `~/.local/share`). Check with `nm -D /libkrun.so | grep krun_add_virtiofs4` on Linux or `nm -gU /libkrun.dylib | grep krun_add_virtiofs4` on macOS. +- The sandbox launches on libkrun. GPU sandboxes and QEMU launches do not support VM mounts. +- On Linux with a non-root gateway, file creation in a read-write mount fails unless `sandbox_uid`/`sandbox_gid` match the gateway user's UID/GID. Compare `id -u`/`id -g` of the gateway user with the configured values. + Then run: ```bash From 8093a7ac75bb4dc141979cdac6b4269c197d1a6e Mon Sep 17 00:00:00 2001 From: Florent Benoit Date: Wed, 14 Oct 2026 10:42:38 +0200 Subject: [PATCH 2/2] fix(nix): refresh vm-runtime hashes to the libkrun v1.19.4 runtime The Nix vm-runtime package fetches the rolling vm-runtime release with fixed hashes pinned on 2026-09-16. The release was rebuilt on 2026-09-21 after the libkrun v1.19.4 / libkrunfw v5.6.1 bump (#3451), but the hashes were not updated, so CI kept substituting the 2026-09-15 runtime (libkrun v1.17.4) from the binary cache. That libkrun lacks krun_add_virtiofs4, which VM bind mounts need. Point the hashes at the current release assets. Signed-off-by: Florent Benoit --- nix/pkgs/vm-runtime.nix | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/nix/pkgs/vm-runtime.nix b/nix/pkgs/vm-runtime.nix index eb3819c09f..bf3d2a50a7 100644 --- a/nix/pkgs/vm-runtime.nix +++ b/nix/pkgs/vm-runtime.nix @@ -13,7 +13,7 @@ let { x86_64-linux = { platform = "linux-x86_64"; - hash = "sha256-kA6ZQz53geBp2zHx8tv+D8VcGkSrnC74Op/60H4fmck="; + hash = "sha256-mZwSUhs9IZ0pPLxzVs4AbRLdYWkUlHwP4lmfsbeiUGw="; artifacts = [ "libkrun.so" "libkrunfw.so.5" @@ -22,7 +22,7 @@ let }; aarch64-linux = { platform = "linux-aarch64"; - hash = "sha256-7FNqUtC6ixfw202EyoYCgZOLdyxJYfg/YzdlkTT/+sM="; + hash = "sha256-Yk5Rxn6JWxMu82Vkyx3eSAxh5/mnfUwcNDNeDIQJz4c="; artifacts = [ "libkrun.so" "libkrunfw.so.5" @@ -31,7 +31,7 @@ let }; aarch64-darwin = { platform = "darwin-aarch64"; - hash = "sha256-cPfogd7QiPFk76kR1JJeKLaWwFksAZ9BeW/LsmDgsjo="; + hash = "sha256-Sm0zwSkctPETKd46zqOqDO02jLapQ3ST1hafqIVkVjU="; artifacts = [ "libkrun.dylib" "libkrunfw.5.dylib"