Integrate vendor VFIO vGPUs into the instance lifecycle

[yummybomb]

Summary

Top layer of the vendor VFIO vGPU stack (generalize-vgpu-device #322 ← hypervisor-liveness #363 ← vendor-vfio-backend #364 ← this). The backend itself is in #364; this layer wires it into the instance lifecycle and adds the release guards that make reusable-VF assignments safe:

• Live-instance claim guard on release — an assignment goes stale when its release succeeds but the metadata save does not (or start fails between the release and its first save). Since vendor VFIO reuses the same VF path across assignments, honoring a stale claim could clear a VF out from under the live instance that owns it now. Every release consults live instance metadata (liveness verified via Harden hypervisor process liveness checks #363's socket-ownership check) and drops stale metadata without touching the device.
• Assignment durability — assignments are tagged with the owning instance ID, persisted before booting a started instance without carrying over the previous VMM identity, and retained in metadata when rollback release fails in create/start so later release paths can still find the device. Retention persistence is a single attempt: when it fails, the response reports the assignment as unretained and the periodic reconciler releases the VF once the grace period and open-VFIO-handle checks prove it free.
• Fail-closed startup reconciliation — the protected set of live VFs is built from ListInstancesForReconcile, which fails on any unreadable metadata instead of silently skipping it. Recent assignments receive bounded startup protection when their PID is absent or stale; when the inventory is unavailable, vendor VFIO reconciliation is skipped entirely while mdev reconciliation still runs.
• Hypervisor scope — hypervisor selection remains a caller policy, preserving the framework layer's existing mdev behavior. The production rollout continues to place vGPU instances on QEMU; the Cloud Hypervisor vendor-VFIO limitation (upstream cloud-hypervisor#7572) is documented in lib/devices/GPU.md (in the base PR Add the vendor VFIO vGPU device backend #364).
• Failure reporting — a failed create or start whose vGPU release also failed surfaces as vgpu_cleanup_pending with retained/unretained guidance; the retention record keeps identity fields (name, image, hypervisor) so it lists as a recognizable, deletable instance. The instance API exposes gpu.device_path, the assignment identity on vendor VFIO hosts where mdev_uuid is empty.

Testing

• go build ./..., go vet ./... clean
• go test -race passes for lib/instances targeted suites, lib/devices, lib/resources, lib/builds (TestCreateInstanceWithNetwork needs image pulls + iptables and fails in this environment on the unmodified base as well)

---

Note

High Risk
Changes core GPU assignment, release, and hypervisor teardown paths on production hosts; incorrect reconciliation or release ordering could leak VFs or disrupt live VMs.

Overview
Vendor VFIO vGPUs are wired into create/start/stop/delete: assignments are persisted with GPUAssignedAt, released through the instance manager, and protected on release by scanning live instance metadata so a stale claim cannot clear a VF owned by another instance.

When create or start rollback cannot release a vGPU, the manager retains a delete-only metadata stub (GPURetainedForCleanup); start, fork, snapshot, and restore refuse that record until delete succeeds. API create/start map VGPUCleanupPendingError to 500 vgpu_cleanup_pending with retained vs unretained guidance; instance responses can include gpu.device_path.

Startup mdev-only reconcile is replaced by StartVGPUReconciler: an initial pass plus minute periodic cleanup that releases stale metadata claims, then sweeps orphaned mdev/vendor assignments (5-minute grace, live VFIO/socket protection, mdevs claimed in metadata). Listing metadata for reconcile fails closed—unreadable inventory skips the device sweep.

QEMU/hypervisor kill path for vGPU and passthrough devices SIGTERMs first (VFIOTermGrace) before SIGKILL to reduce VF wedge risk; docs and tests reflect that behavior. Metrics cover reconcile failures, stale releases, and retained assignments.

^{Reviewed by Cursor Bugbot for commit 081d9f7. Bugbot is set up for automated code reviews on this repo. Configure here.}

[yummybomb]

Trimmed over-engineering after an architectural review against the parent task (host kernel 6.8 / Ubuntu 24.04 support). Most of the removed weight came from earlier review-round additions rather than the original design: