Repository navigation
Conversation
|
Confirming from the requester side: this matches what we asked for in #1509 — opt-in volume mounts for MicroVM sandboxes. The core ask there was the ability to add virtiofs devices beyond the rootfs in a libkrun VM, behind an explicit operator opt-in, with the guest-side mount handled during the driver's boot sequence. That's what this implements. Thanks for picking it up. |
d7236e6 to
3c81133
Compare
|
I rebased the PR on top of the main branch due to conflicts |
bd1d7b2 to
135f022
Compare
|
rebased and added e2e tests |
135f022 to
606a784
Compare
|
rebased to fix conflict |
606a784 to
71ddc0f
Compare
|
rebased |
|
when will this get merged? we are sort of stuck since we need to mount a host filepath for our usecase |
|
@sideDesert I'm waiting for a review 🚀 |
elezar
left a comment
elezar
left a comment
There was a problem hiding this comment.
Requesting changes primarily because file-creation coverage and successful VM end-to-end runs on Linux KVM and macOS HVF are still missing. I found no confirmed implementation defect in the reviewed paths; this review does not require adding launcher confinement in this PR. The inline comments request clarification of the documented host exposure and validation of the file-creation workflow.
Please update the VM troubleshooting section in skills/debug-openshell-cluster/SKILL.md alongside this driver configuration change, as required by the repository instructions. Add checks for allow_driver_config, enable_bind_mounts, disabled resource admission, a runtime providing krun_add_virtiofs4, unsupported GPU/QEMU launches, and UID/GID mismatch when file creation fails on rootless Linux.
71ddc0f to
073f217
Compare
Introduce host directory bind mounts for libkrun VM sandboxes using the virtiofs4 API with simplified permission semantics. The gateway accepts `enable_bind_mounts` in the `[openshell.drivers.vm]` TOML table and forwards it to the VM driver subprocess. Mounts are specified per-sandbox in driver_config.mounts[] with source, target, optional read_only (default: read-only), and an optional `type` that must be `bind`, so Docker and Podman bind entries (without `selinux_label`) parse unchanged. At sandbox validation the driver rejects mounts unless bind mounts are enabled and resource admission is disabled. It rejects GPU and other QEMU launches, sources with control characters, and more mounts than libkrun has interrupt lines for (32 on aarch64, 3 on x86_64), and checks targets against reserved VM-internal directories and each other (no nested targets). The QEMU check runs again once the launch backend is resolved. Host source directories are checked at provisioning, before image preparation, so a briefly missing source fails provisioning visibly instead of denying recovery of a persisted sandbox. Missing, inaccessible, symlinked, non-directory, and unreadable sources are rejected: libkrun does not follow a final symlink, and on macOS it panics during guest boot when it cannot open the share root. The driver generates virtiofs tags and always writes a guest-side manifest into the overlay disk, together with the init drop-ins, so an image-supplied manifest can never drive guest mounts. The launcher parses its tab-separated --vm-mount arguments from the right, so a source can never shift the tag or mode, and the driver logs each shared directory at launch. Guest init mounts the shares after every /sandbox ownership fixup and the init drop-ins, so recursive chowns never reach host files. It refuses targets that traverse symlinks in the image, hands newly created mount point parents under /sandbox to the sandbox user, and skips an empty manifest. krun_add_virtiofs4 is resolved optionally, so older libkrun runtimes still boot sandboxes without mounts. The kernel config lists CONFIG_FUSE_FS and CONFIG_VIRTIO_FS explicitly; the pinned libkrunfw base config already enables both. A vm_bind_mount e2e test checks a read-write share round-trips data with the host, accepts a new file, directory, and nested file that the host then sees, and that a read-only share rejects writes. libkrun on rootless Linux refuses file creation by a guest user whose UID/GID differ from its own, so the VM e2e gateway runs the sandbox as the gateway user there. It enables bind mounts like the Docker and Podman e2e gateways. The runtimes docs cover the mount limit, the symlink restriction, the guest-kernel exposure, and the rootless Linux UID/GID note. The debug-openshell-cluster skill lists VM bind mount checks. Signed-off-by: Florent Benoit <fbenoit@redhat.com>
073f217 to
73c9361
Compare
|
@elezar I've updated my PR Docs
E2E coverage (
Skill
Test runs against the current revision I've launched e2e on my mac and setup a custom runner on GitHub for Linux KVM
you can find GitHub custom launch for Linux KVM run at https://github.com/benoitf/OpenShell/actions/runs/38360705661 |
elezar
left a comment
elezar
left a comment
There was a problem hiding this comment.
The updated PR addresses all three requests from my previous review: the warning now explicitly describes the guest-kernel exposure beyond the configured source, the bind-mount e2e test covers creation of a file, directory, and nested file with host-side verification, and the VM troubleshooting skill includes the requested checks.
I verified passing Linux KVM vm_bind_mount runs for both the managed and external-driver paths. The runner commit differs from this PR head only by the custom CI workflow, so the tested source matches. The author also reports a passing macOS HVF run; I have not independently reproduced that run. No remaining blocking findings from this review.
Minor follow-up: please update the PR description's testing checklist, which still says the VM bind-mount e2e test has not been run.
I've updated the checkbox |
|
@elezar could you add the ok to test instruction please ? |
|
/ok-to-test 73c9361 |
|
Label |
This branch has not been deployed
5 participants
Summary
Add host directory bind mount support for libkrun VM sandboxes via the virtiofs4 API with simplified permission semantics. Operators enable the feature with
enable_bind_mounts = truein[openshell.drivers.vm], and sandbox creators specify mounts indriver_config.mounts[].note: it depends on a new release of kernel published at https://github.com/NVIDIA/OpenShell/actions/workflows/release-vm-kernel.yml for pre-built vm (with newer libkrun)
Related Issue
#2585
Changes
openshell-gateway): acceptenable_bind_mountsinVmComputeConfigand forward it as--enable-bind-mountsto the VM driver subprocess.openshell-driver-vm): add an optionalkrun_add_virtiofs4binding, so older libkrun runtimes still boot sandboxes without mounts.openshell-driver-vm): newVmMounttype andadd_virtiofs()calling virtiofs4 with simplified semantics; QEMU refuses mounts.openshell-driver-vm):driver_config.mounts[]takessource,target, optionalread_only(default read-only) and optionaltype: bind, so Docker and Podman bind entries (withoutselinux_label) parse unchanged. Mounts requireallow_driver_config,enable_bind_mountsand disabled resource admission. The driver rejects GPU and other QEMU launches, sources with control characters, reserved VM-internal targets, nested or duplicate targets, and more mounts than libkrun has interrupt lines for (32 on aarch64, 3 on x86_64). The QEMU check runs again after backend resolution.openshell-driver-vm): host sources are checked before image preparation. Missing, inaccessible, symlinked, non-directory and unreadable sources are rejected. The driver generates virtiofs tags, always writes the guest mount manifest into the overlay (so an image-supplied manifest can never drive mounts), logs each shared directory, and forwards--vm-mountarguments that the launcher parses from the right.mount_virtiofs_shares()mounts the shares after every/sandboxownership fixup and the init drop-ins, refuses targets that traverse a symlink in the image, and skips an empty manifest.CONFIG_FUSE_FSandCONFIG_VIRTIO_FSexplicitly; the pinned libkrunfw base config already enables both.enable_bind_mountsindocs/how-it-works/gateways/configuration.mdx; a "MicroVM Mounts" section indocs/how-it-works/sandboxes/runtimes.mdxcovering the opt-in, the mount limit, unsupported sources, the guest-kernel exposure, and the rootless Linux UID/GID note.vm_bind_mounttest; the VM e2e gateway enables bind mounts like the Docker and Podman e2e gateways.Known limits
sandbox_uid/sandbox_gid).sourceagainst a compromised guest kernel; the launcher serves it as the gateway user. Confining the launcher (for example with Landlock) is not part of this change.Testing
type: bind)enable_bind_mountsand disabled resource admission; rejects QEMU, relative sources, control characters, reserved paths, nested and duplicate targets, too many mounts)prepare_virtiofs_targetin the guest init script (creates directories, refuses symlinks and non-directories)enable_bind_mountsis revoked--vm-mountargument parsing and manifest renderingcargo test -p openshell-driver-vm,cargo fmt --checkandcargo clippyon the driver crate (macOS arm64)vm_bind_mounte2e on Linux KVM (GitHub runner) and macOS HVF (my machine)Checklist
configuration.mdx,runtimes.mdx)mise run pre-commitpasses on this revisionmise run testpasses on this revision