desktop

This package boots desktop VMs and executes mouse and keyboard operations on them. It applies Operations — already resolved to absolute screen pixels — to a real desktop, and it starts, resets, pools, and proves the state of the VMs those desktops run on.

Two distributions

directory	distribution	handles
desktop/	desktop	the computer is wrong — black framebuffer, cursor did not land, reset did not rewind. Synchronous; the caller owns the lifetime.
(p-doom/desktop-fleet)	desktop-fleet	the computer is gone — Slurm preemption, a dead node, a stale heartbeat, a leaked process group, queue-or-fail.

desktop-fleet (now its own repository) may import desktop; desktop must never import it, and must never learn what a fleet is. That is what keeps it usable against one VM with no scheduler, and it is why the Pillow-only dependency floor below can hold.

Nothing here knows what an action grammar is. Callers resolve model output into absolute-pixel Operations before handing it to this package.

One Python runtime dependency: Pillow, imported in vm/readiness.py to downsample a screenshot and measure whether the framebuffer is still black. Everything else is stdlib, HTTP is urllib.request, and screenshots cross every boundary as raw bytes. Two external binaries are assumed: qemu-system-x86_64 and apptainer.

Usage

from desktop import DisplayGeometry, Operation
from desktop.vm import DesktopSession, QemuRuntime

runtime = QemuRuntime(image=Path("/images/guest.qcow2"))
with DesktopSession(runtime) as session:
    receipt = session.execute(
        (Operation("move_to", (640, 400)), Operation("click", ("left",)))
    )
    assert receipt.ok

    session.reset()   # ~4.5 s, and it proves the guest actually rewound

Layout

path	what
ir.py	Operation(kind: str, args: tuple) — open kind vocabulary, never an Enum. Includes an explicit drag(x0,y0,x1,y1) so a zero-extent drag survives resolution instead of collapsing into a no-op.
geometry.py	DisplayGeometry, scale_normalized_coordinate, anthropic_scale_coordinates — copied verbatim from Harbor, for callers resolving model coordinates.
execute/protocol.py	Validates operations and serializes the versioned action request. Defines the sole ExecutionReceipt.
execute/keymap.py	The key and pointer-button name tables, and the chord and transition helpers over them.
vm/runtime.py	The Runtime protocol: start, stop, checkpoint, restore, fork.
vm/qemu.py	QMP savevm/loadvm (13.6–16.6 s reboot-revert → 4.4–5.2 s restore), plus CoW fork. KVM or raise; -accel tcg only on explicit request.
vm/readiness.py	Whether the desktop is up or the framebuffer is still black, from the non-dark-pixel ratio and the luma standard deviation. The luma_sampler seam makes the rule testable against synthetic frames with no VM.
vm/factory.py	The constructor side: build_qemu_runtime, build_desktop_session, qemu_session_factory, build_desktop_pool. Plain functions, explicit config, no name registry and no plugin lookup — you pass an image path, not a provider name.
vm/session.py	The stable public desktop handle: actions, observations, trusted guest control, endpoints, and attested reset.
vm/pool.py	Prewarmed sessions, leases with activity timeouts, reset-before-ready reuse, and an inspectable status file.
vm/client.py	The one host client for /v1/actions, observations, and trusted guest commands. It verifies the installed executor hash.
vm/guest/executor.py	The self-contained XTEST executor installed into the guest image and invoked once per action.
vm/sandbox_protocol.py	NeMo-Gym's sandbox protocol including ConnectableProvider, plus our own Apptainer implementation.
vm/images/	Two tiers: KVM (full parity) and non-KVM (no OSWorld parity, ever — see that directory's README).
vm/image_build.py	The producer of the guest qcow2 every other module treats as a pinned input: boots the upstream image, installs the grader libraries and tools, verifies them from inside the guest, and publishes the image beside a manifest of what went in.

Provenance

Copied verbatim or near-verbatim, with attribution in each file: Harbor (geometry.py), NeMo-Gym (sandbox_protocol.py types), anthropics/claude-quickstarts computer-use-demo (images/desktop-nonkvm.def). trycua/cua was read as a reference only — the CoW fork and TCG fallback in vm/qemu.py are reimplementations of ideas, not code.