Repository navigation
VM driver authenticates with the registry on every boot, even for a cached digest-pinned image #4581
Description
User Story
We run the VM driver as the sandbox runtime behind a warm pool: every sandbox a user releases is replaced by booting a fresh one from the same image, so boots are frequent and a slow boot means fewer warm sandboxes when the next user asks for one.
On arm64 hosts pulling from nvcr.io, with the image's root disk already cached, about 1.9 s (p50 over 96 boots) of a 2.5 s re-warm goes to registry authentication and manifest resolution before the driver uses its cache. When the registry throttled our boot rate, every boot failed even though every host had the image cached. We pin the image by digest, so the registry cannot tell the driver anything new on those boots.
Measurements
Setup: openshell-driver-vm at v0.1.3 (e1f3c82c), arm64 hosts (144 cores, Linux 6.8 with 64 KiB pages), image on nvcr.io pinned by digest (arm64, 20 layers, 1.08 GiB compressed), root disk already cached. 96 boots on one host, timed from the driver's own log lines:
| Phase (driver log lines) | p50 | p90 | max |
|---|---|---|---|
create_sandbox received -> sandbox root disk plan resolved (registry auth + manifest + cache lookup) |
1,934 ms | 3,250 ms | 5,114 ms |
sandbox root disk plan resolved -> launcher spawned |
93 ms | 101 ms | 178 ms |
launcher spawned -> supervisor connected to the gateway |
372 ms | 490 ms | 782 ms |
| Total, create -> supervisor connected | 2,380 ms | 3,666 ms | 5,548 ms |
On a single boot the steps were: authenticating with registry -> fetching manifest digest 1,039 ms, -> manifest digest resolved 289 ms. With several boots in flight the registry phase grows (p90 3.25 s), because each boot authenticates on its own.
So about 81 % of a re-warm is registry traffic that cannot change the outcome for a cached, digest-pinned image. With the cache hit returned before authentication, we expect a re-warm of about 0.5 s on these hosts (launcher + VM + supervisor).
For scale, the one-time first boot on a host was 60 to 119 s: about 91 s pulling the layers and 26 s preparing the root disk.
Current behaviour
ensure_cached_registry_rootfs_image (crates/openshell-driver-vm/src/driver.rs)
authenticates with the registry, and for a tag resolves the manifest, before
it checks the root disk cache. It does this on every boot. Since #4342 a
digest-pinned reference skips the manifest request, but authentication still
happens on every boot, including cache hits.
Expected behaviour
For a reference that carries a digest, check the cache first: a cached root
disk for that digest is used with no registry request. Authentication and the
pull stay on the miss path; tags keep resolving through the registry.
Reproduction
- Start a VM sandbox from
<registry>/<image>@sha256:<digest>(the first boot pulls and builds the root disk). - Start a second sandbox from the same reference.
- The driver log of the second boot shows
authenticating with registrybeforeroot disk image cache hit; with the registry unreachable the second boot fails.
OpenShell version: v0.1.4 (5a7728c), also main at 909f336.
Proposed change
Return the cache hit before authenticating when the reference is digest-pinned
and its root disk is cached (a digest names the content, and since #4342 the
cache entry is bound to a verified manifest digest). Behaviour change: a host
whose credentials were revoked still boots digest-pinned images it already
cached, as a kubelet does with IfNotPresent and a digest. A change with a
test is ready to open as a PR once this issue is accepted.
Activity
cloudlesssmarthome commentedon Oct 16, 2026
+1, with an offline use case this would also fix. I run OpenShell's VM driver on a Mac for local agents that must keep working
when the internet is down. There is no Docker or Podman on the host, so sandboxes boot from a rootfs tar
(
openshell sandbox create --from ./rootfs.tar), and the VM driver gets its bootstrap image from the registry.On the dev build
0.1.5-dev.49+ga4c848adb, with the bootstrap image already cached and pinned by digest, I cut the gateway'snetwork. Each of these then failed in about 2 s with
failed to authenticate registry access … https://nvcr.io/v2/:openshell sandbox startof a stopped sandbox.Deleting a sandbox still worked. So with no container engine on the host, a cached VM sandbox cannot start or restart without
registry access.
Reading #4582 (head
596bc12), it fixes this too: with no local engine,ensure_cached_bootstrap_rootfs_imagefalls throughto
ensure_cached_registry_rootfs_image, which now returns the cached disk beforeclient.authfor a digest-pinned reference,and the restore at gateway start reaches the same function through
prepare_images_in_worker. So withbootstrap_imagepinnedby digest, create, start and restore should all work offline once the bootstrap is cached. I'd be glad to see #4582 reviewed
and merged, and I'll re-run the offline test above on the first dev build that carries it.
One follow-up, lower priority: the bootstrap still needs one registry download before a host can work offline. Would you take
a way to provide it without a registry, the way
--from ./rootfs.taralready does for the sandbox image (for examplebootstrap_imagenaming a local OCI layout or tar)? An offline host could then be set up entirely from files.