diff --git a/README.md b/README.md index 3ef0559..f44b8c8 100644 --- a/README.md +++ b/README.md @@ -42,7 +42,7 @@ Terminal, browser, and computer automation for Droids. Record demos, verify beha **Commands:** `/demo`, `/verify`, `/qa-test` -**Skills:** `droid-control` (orchestrator), `tuistory`, `true-input`, `agent-browser`, `droid-cli`, `pty-capture`, `capture`, `compose`, `verify`, `showcase` +**Skills:** `droid-control` (orchestrator), `tuistory`, `true-input`, `agent-browser`, `desktop-control`, `droid-cli`, `pty-capture`, `capture`, `compose`, `verify`, `showcase` See [plugins/droid-control/README.md](plugins/droid-control/README.md) for details. diff --git a/plugins/droid-control/.factory-plugin/plugin.json b/plugins/droid-control/.factory-plugin/plugin.json index 8f6c249..72a9699 100644 --- a/plugins/droid-control/.factory-plugin/plugin.json +++ b/plugins/droid-control/.factory-plugin/plugin.json @@ -1,5 +1,5 @@ { "name": "droid-control", "description": "Terminal, browser, and native desktop automation for testing, demos, QA, and computer-use tasks", - "version": "1.0.0" + "version": "1.1.0" } diff --git a/plugins/droid-control/ARCHITECTURE.md b/plugins/droid-control/ARCHITECTURE.md index cb0b699..a0383c0 100644 --- a/plugins/droid-control/ARCHITECTURE.md +++ b/plugins/droid-control/ARCHITECTURE.md @@ -12,7 +12,7 @@ The plugin is designed to keep a droid focused while it operates real software: - **Low context load:** load the Linux tuistory path without dragging in Windows KVM notes, macOS VM controls, browser automation, and Remotion internals. - **Evidence-first workflows:** every command starts by making commitments, then ends by verifying the artifact against those commitments. -- **Parallel execution:** before/after captures and render jobs can run in worker droids without sharing session names or output paths. +- **Parallel execution:** independent capture environments and render jobs can run in workers. Shared desktop input stays serialized; session names do not isolate focus. - **Clear ownership:** commands decide *what* must be produced; atom skills decide *how* to execute their slice. - **Platform specificity:** OS-specific mechanics live in platform subdocuments, not in global instructions. @@ -85,9 +85,10 @@ The parent droid keeps judgment. Workers get exact commands. | Work | Owner | Reason | |---|---|---| +| Short interactive desktop task | Parent | One controller owns observation, input, permission waits, and cleanup. | | Interpret PR / claim / QA goal | Parent | Requires context and judgment. | | Write the interaction script | Parent | Defines the proof story. | -| Capture baseline and candidate branches | Worker droids | Independent, mechanical, parallelizable. | +| Capture baseline and candidate branches | Workers only for independent environments | A shared desktop must be captured serially. | | Render Remotion video | Worker droid | Mechanical once props and clips are fixed. | | Verify commitments | Parent | Requires the original contract and evidence judgment. | @@ -161,13 +162,15 @@ skills/true-input/platforms/macos.md skills/pty-capture/platforms/linux.md skills/pty-capture/platforms/windows.md skills/pty-capture/platforms/macos.md -skills/desktop-control/platforms/linux.md -skills/desktop-control/platforms/windows.md -skills/desktop-control/platforms/macos.md +skills/desktop-control/SKILL.md ``` A Linux droid reads Linux Wayland instructions. A Windows VM byte-capture task reads Windows KVM instructions. The system does not rely on the droid to skim irrelevant sections correctly. +Desktop-control keeps routine setup, target selection, recovery, and recording rules in one compact entrypoint. It includes a host setup table rather than copied platform manuals. No user-home skill dependency or runtime documentation download is required. + +For ordinary desktop tasks, the driver verifies each action and reports directly. Capture/verify stages are loaded for formal evidence deliverables, and compose only when a produced artifact was requested. Explicit GUI-only or cua-only constraints take precedence over Electron's default browser route. + ## Extending the plugin Use the same composition rules when adding capability: diff --git a/plugins/droid-control/CUA-LICENSE.md b/plugins/droid-control/CUA-LICENSE.md new file mode 100644 index 0000000..b8b198c --- /dev/null +++ b/plugins/droid-control/CUA-LICENSE.md @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2025 Cua AI, Inc. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/plugins/droid-control/NOTICES.md b/plugins/droid-control/NOTICES.md index 0abef16..5fb48bd 100644 --- a/plugins/droid-control/NOTICES.md +++ b/plugins/droid-control/NOTICES.md @@ -1,6 +1,10 @@ # Third-Party Notices -This plugin depends on several third-party tools and libraries. They are not bundled -- each is installed separately by the user. Their respective licenses apply at the point of installation and use. +This plugin depends on several third-party tools and libraries. Executables are installed separately by the user. + +## Desktop automation documentation + +- **[Cua Driver](https://github.com/trycua/cua)** — desktop-control guidance is adapted from Cua under the [included MIT license](CUA-LICENSE.md). Driver code and binaries are not bundled. ## Video rendering diff --git a/plugins/droid-control/README.md b/plugins/droid-control/README.md index 590fbb8..0a62127 100644 --- a/plugins/droid-control/README.md +++ b/plugins/droid-control/README.md @@ -2,7 +2,7 @@ Terminal, browser, and computer automation plugin for Droids. -Droids can read and write code. This plugin lets them *operate* it: launch apps, type commands, click buttons, record what happens, and produce polished evidence. No human hands required. +Droids can read and write code. This plugin lets them *operate* it: launch apps, type commands, click buttons, record what happens, and produce evidence. OS permission grants and consequential actions still require the appropriate user or host authorization. ## What you get @@ -48,6 +48,10 @@ Or use the `/plugins` UI: Browse tab, select droid-control, install. Then open a Droid session and run `/demo`, `/verify`, or `/qa-test`. +For ordinary desktop work, ask directly: **“Using only cua, open Calculator and compute 17 × 23.”** Desktop-control runs the observe/act/verify loop without loading video-production stages. + +The [desktop-control skill](skills/desktop-control/SKILL.md) includes setup and operating guidance. Install the `cua-driver` executable if missing; no separate Cua skill installation is needed. Installed driver versions and Wayland compositors may support different capabilities. + ## Commands ### `/demo` @@ -69,7 +73,7 @@ Runs automated QA against terminal CLIs, web apps, or Electron apps. Accepts a U 1. **Commands** parse user intent into commitments. 2. **The orchestrator** routes by target, stage, and artifact needs. 3. **Atom skills** provide only the mechanics needed right now: drivers, target patterns, capture, compose, verify, and showcase polish. -4. **Workers** handle mechanical capture/render jobs while the parent droid keeps planning and verification context. +4. **Workers** handle independent capture/render jobs. The parent keeps short interactive desktop tasks, including observations, input, permission waits, and cleanup. 5. **Verify** checks the final evidence against the original commitments. For the full rationale and runtime pipeline, see [`ARCHITECTURE.md`](ARCHITECTURE.md). @@ -103,4 +107,4 @@ curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scr cd plugins/droid-control/remotion && npm install # Remotion video rendering ``` -Only install what you need for your use case. Terminal demos need tuistory, asciinema, agg, and ffmpeg. Web/Electron automation just needs agent-browser. Native desktop GUI automation just needs cua-driver. +Only install what you need, with approval. Terminal demos need tuistory, asciinema, agg, and ffmpeg. Web/Electron automation defaults to agent-browser; an explicit cua-only/native-input request uses desktop-control instead. Native desktop automation needs cua-driver plus the graphical session and OS permissions reported by its preflight. Recording and rendering have additional dependencies; they are not required for ordinary desktop tasks. diff --git a/plugins/droid-control/commands/qa-test.md b/plugins/droid-control/commands/qa-test.md index 482f24f..ccc4786 100644 --- a/plugins/droid-control/commands/qa-test.md +++ b/plugins/droid-control/commands/qa-test.md @@ -1,5 +1,5 @@ --- -description: Run an automated QA test flow against a terminal CLI or web/Electron app +description: Run an automated QA test flow against a terminal, browser, or native desktop app argument-hint: '"" or "" or " [-- focus area]" or ""' --- @@ -9,7 +9,7 @@ Load skills: **droid-control**. `$ARGUMENTS` can be: - **URL** (`https://app.factory.ai`, `localhost:3000`) → web app -- **Electron app name** (`Slack`, `VS Code`, `Figma`) → Electron app via CDP +- **Desktop app name** (`Slack`, `VS Code`, `Calculator`) → use the orchestrator's target route; explicit cua/native-input requirements override Electron's CDP default - **CLI command** (`droid-dev`, `htop`, `my-cli --flag`) → terminal TUI - **PR reference** (`11386`) with optional `-- focus area` → infer target from the diff - **Free-text description** ("test the login flow on staging") → infer target and flow @@ -48,6 +48,8 @@ If the user provides specific steps, use them. Otherwise, design a reasonable fl **Terminal**: launch app → wait for ready → snapshot → exercise primary features → verify output → snapshot → close. +**Native desktop / cua-only**: discover or launch the requested app → select an exact target → observe → act → verify each postcondition. Broaden to visible desktop capture/input only with authorization. Leave personal applications open unless closure is requested; end only the automation run. + If the flow is ambiguous or success criteria are unclear, ask the user. ## Capture diff --git a/plugins/droid-control/skills/capture/SKILL.md b/plugins/droid-control/skills/capture/SKILL.md index fd1e523..095e664 100644 --- a/plugins/droid-control/skills/capture/SKILL.md +++ b/plugins/droid-control/skills/capture/SKILL.md @@ -21,6 +21,8 @@ The command that invoked you should have provided: ## Recording lifecycle +For desktop-control, follow its [recording contract](../desktop-control/SKILL.md#recording); do not translate the terminal commands below into desktop commands. Routine desktop snapshots stay in the driver's observe/act/verify loop. + ### 1. Pre-flight Before recording anything: @@ -98,7 +100,7 @@ Film for a viewer with no context. You are a director, not an operator. - **Record before setup** -- the baseline state is act 1. - **Hold after state changes** -- 2-3 seconds so text is readable. Use `snapshot --trim` as natural verification beats. - **Verify between steps** -- `wait` or `snapshot` to confirm state before proceeding. Don't blindly fire the next key. -- **Verification IS evidence.** A snapshot that shows nothing changed after pressing ESC proves the session is frozen. A snapshot that shows an error message proves the command was blocked. Always snapshot after actions where the *absence* of a response is the point -- the viewer needs to see it too. +- **Verification IS evidence.** Capture the actual state after actions, including when no change is visible. One unchanged frame alone does not prove a frozen session or a dropped key; check the task's postcondition and timing. For comparison recordings, both branches run **identical interactions** -- only the behavior differs. @@ -120,7 +122,7 @@ Use readable key names (`Ctrl+C`, not `\x03`). Save alongside the recording (e.g ### 5. Close and verify raw outputs ```bash -$TCTL -s demo close # finalizes the .cast / stops recording +$TCTL -s "${RUN_ID}-demo" close # terminal session owned by this run ``` Before handing off, confirm every expected output file exists and is non-empty: @@ -137,7 +139,7 @@ Before handing off, confirm every expected output file exists and is non-empty: | Visual rendering | Screenshots: `$TCTL -s screenshot -o /tmp/proof-N.png` | | Keyboard encoding | PTY bytes: `${DROID_PLUGIN_ROOT}/scripts/capture-terminal-bytes.py --backend --combo ` | | Web/Electron | Screenshots: `agent-browser screenshot --annotate /tmp/proof-N.png` | -| Native desktop GUI | Window screenshots + AX trees: `cua-driver get_window_state '{...}' --screenshot-out-file ${RUN_DIR}/proof-N.png`; video via `cua-driver recording start/stop` | +| Native desktop GUI | Follow **desktop-control** for exact-window or authorized desktop state and recorder ownership | | Before/after | Run the same sequence on both branches at the same capture points | ## Outputs @@ -164,6 +166,6 @@ $TCTL -s snapshot --trim # check visible state $TCTL -s close # hard reset ``` -For browser: `agent-browser close`. +For an isolated browser owned by this run: `agent-browser close`. -Then re-launch and re-record. Partial recordings are not usable. +For desktop-control, reacquire state after interruption and coordinate with the recorder owner. Do not stop a shared daemon, close a personal app, or replay an uncertain input to recover a recording. Preserve partial artifacts as diagnostic evidence; label them incomplete rather than claiming they satisfy the deliverable. diff --git a/plugins/droid-control/skills/desktop-control/SKILL.md b/plugins/droid-control/skills/desktop-control/SKILL.md index ce971df..9b80e15 100644 --- a/plugins/droid-control/skills/desktop-control/SKILL.md +++ b/plugins/droid-control/skills/desktop-control/SKILL.md @@ -4,124 +4,74 @@ description: Background knowledge for droid-control workflows -- not invoked dir user-invocable: false --- -# Desktop-Control Driver +# Desktop Control -The orchestrator routed you here. Use these mechanics to execute your plan. +One controller operates an exact GUI target, observes each effect, and stops when the user's postcondition is proved. -Drive native desktop GUI apps through upstream [trycua/cua](https://github.com/trycua/cua) `cua-driver`: enumerate apps and windows, snapshot accessibility trees, click/type/scroll by `element_index` or pixel coordinates, and verify by re-snapshot -- all without bringing the target to the foreground. +## Act -## When to use - -- Automating a native desktop app (Finder, Notepad, System Settings, native editors) -- Driving native dialogs and security/permission sheets that no DOM or PTY can reach -- Visual QA of native UI: per-window screenshots, accessibility-tree assertions - -If the target is a terminal TUI, use **tuistory** or **true-input**. If it is a web page or an Electron app, use **agent-browser** -- CDP beats accessibility trees for anything Chromium-based. - -## Platform support - -| Platform | Upstream tier | Read | -|---|---|---| -| macOS | Production | [platforms/macos.md](platforms/macos.md) | -| Windows | Production | [platforms/windows.md](platforms/windows.md) | -| Linux | Pre-release (real caveats) | [platforms/linux.md](platforms/linux.md) | - -**Read the platform file for your target OS.** Each contains permissions, daemon launch, and platform-specific patterns and failure modes. - -## Prerequisites - -```bash -# one-time install: per-user, no sudo/admin -curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.sh | bash -# Windows (PowerShell): -# irm https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.ps1 | iex - -cua-driver doctor # platform probes: permissions, daemon, accessibility plumbing -cua-driver skills install # fetch the upstream skill pack to ~/.cua-driver/skills/cua-driver -``` - -The upstream pack (`~/.cua-driver/skills/cua-driver/SKILL.md` + your platform's doc) is the deep reference -- full tool surface, window-state behavior matrix, forbidden-command lists -- and it updates with the binary. **Read it before any nontrivial workflow.** This atom owns the droid-control integration: routing, run isolation, delegation, evidence handoff. - -## Daemon lifecycle - -`element_index` workflows **require the daemon**. Without it each CLI invocation is a fresh process and the per-`(pid, window_id)` element cache dies between calls. - -```bash -cua-driver serve # start the daemon (macOS needs the LaunchServices form -- see platforms/macos.md) -cua-driver status # daemon + socket health -cua-driver stop -``` +| Goal | Command / tool | +|---|---| +| Discover | `cua-driver list_apps`; use `launch_app` when launch is requested, then select the intended window from its response or `list_windows` | +| Observe | `get_window_state` with observed `pid`, `window_id`, and the run's `session`; use `query`, `max_elements`, or `max_depth` to bound large trees | +| Act | `click` / `type_text` with an exact `target` and fresh `element_token`; use `x,y` only from a valid target screenshot | +| Menu / geometry | Prefer `invoke_menu` with an observed menu path, or `set_window_frame`; verify the resulting window state | +| Verify | Fresh `get_window_state` / `get_desktop_state`, or `verify_state` for an expressible exact-window postcondition | +| Finish | Finalize any owned recording, then `end_session`; leave personal apps and shared services running unless closure was requested | -Permissions are checked and granted through the driver, not by hand-editing system settings (macOS-only gate; a no-op surface on Windows/Linux): +Use the CLI by default or an existing MCP connection. Replace example IDs, tokens, coordinates, and `RUN_ID` with current observations and a unique run label: ```bash -cua-driver permissions status # read-only; answers via the running daemon -cua-driver permissions grant # attributed prompt flow -- the correct way to grant +cua-driver get_window_state '{"pid":844,"window_id":10725,"session":"RUN_ID"}' +cua-driver click '{"target":{"kind":"window","pid":844,"window_id":10725},"element_token":"s0000002a:14","session":"RUN_ID"}' +# Observe again and verify the requested effect before another action. ``` -## Core loop - -Tool names are `snake_case` and invoked directly: `cua-driver ''`. (`cua-driver call ` is legacy; do not use it.) `cua-driver list-tools` for the inventory, `cua-driver describe ` for any schema. - -Every workflow is Discover -> Observe -> Act -> Verify against an explicit `(pid, window_id)`: +## Detect and setup ```bash -cua-driver launch_app '{"name":"TextEdit"}' -# -> {pid: 844, windows: [{window_id: 10725, ...}]} # list_windows only needed for long-lived pids -cua-driver get_window_state '{"pid":844,"window_id":10725}' --screenshot-out-file "${RUN_DIR}/before.png" -cua-driver click '{"pid":844,"window_id":10725,"element_index":14,"session":"'"${RUN_ID}"'-desktop"}' -cua-driver get_window_state '{"pid":844,"window_id":10725}' --screenshot-out-file "${RUN_DIR}/after.png" +command -v cua-driver +cua-driver --version +cua-driver status +cua-driver doctor +cua-driver describe click ``` -**Snapshot before AND after every action.** The pre-action `get_window_state` resolves the `element_index` you are about to use -- indices are per-snapshot, per `(pid, window_id)`, and stale ones fail with `No cached AX state`. The post-action snapshot is the evidence the action landed; without it a silent no-op looks like success. +No separate Cua skill installation is required. Preserve existing executable wrappers and service ownership. If the binary is missing, obtain approval before using the official [macOS/Linux installer](https://cua.ai/driver/install.sh) or [Windows installer](https://cua.ai/driver/install.ps1). Inspect unfamiliar live schemas; a client version does not identify an already-running daemon. -Addressing-mode preference: +| Host | Required setup | +|---|---| +| macOS | The responsible app/host needs Accessibility and Screen Recording grants; let the user run `cua-driver permissions grant` and approve prompts | +| Windows | The runtime must run in the interactive desktop session, not Session 0; user/host handles installation and security prompts | +| Linux | Run as the graphical user on its display/session bus; native Wayland may require `CUA_DRIVER_RS_ENABLE_WAYLAND=1` in the service environment. Compositor capture/input/video support varies | -1. **`element_index`** (default) -- semantic, works on hidden and backgrounded windows, no foreground change. -2. **Pixel** `click '{"pid":N,"window_id":W,"x":X,"y":Y}'` -- for surfaces the tree does not reach (canvases, custom-drawn controls). Coordinates are window-local screenshot pixels, top-left origin. -3. **Keyboard** (`press_key`, `hotkey`) and platform fallbacks -- last resort; see the platform files. +## Rules -## Run isolation (ground rule 5 -> cua sessions) +1. **Never substitute methods.** Cua-only/native-input excludes CDP, DOM, application APIs, and shell/media shortcuts, including for Electron. +2. **Never share desktop control.** Keep interactive observation, input, permission waits, and cleanup in the parent. Reuse one run label and artifact directory; repeat `session` on each supported CLI call. Labels do not isolate focus, app state, or snapshot caches. +3. **Never reuse stale or ambiguous targets.** A new snapshot invalidates old handles. Do not combine `target` with flat targeting fields; observation and semantic-only tools keep their own schemas. Re-resolve cold launches or vanished windows with bounded discovery, not repeated launches. +4. **Never infer pixels from absent evidence.** Read the actual image and its dimensions; account for resized previews/crops. Tree-only capture cannot ground pixel input. `capture_mode` does not repair capture failures. +5. **Never escalate implicitly.** Window background input is the default. Foreground delivery, temporary menu activation, desktop capture/input, service changes, and OS approvals require the appropriate user/host authorization. +6. **Never equate delivery with completion.** Reobserve after uncertain, partial, or interrupted input before retrying. Check the task's postcondition: selection is not playback; an unchanged frame is not proof of a freeze; a closed window is not proof of process exit. -cua sessions are the desktop equivalent of `tctl` session prefixes: a session owns its agent cursor, config overrides, and recording scope. Declare one per run, derived from the workflow's `RUN_ID`, and pass it on every action: +## Failure map -```bash -cua-driver start_session '{"session":"'"${RUN_ID}"'-desktop"}' -# ... every action carries "session":"${RUN_ID}-desktop" ... -cua-driver end_session '{"session":"'"${RUN_ID}"'-desktop"}' -``` +| Symptom | Next action | +|---|---| +| Missing binary, permission, or capability | Stop that route; resolve setup with the user rather than silently installing, restarting, or changing security settings | +| Sparse tree | Inspect `degraded_reason`; retry once for lazy initialization. Use pixels only if a valid image exists | +| Missing image / `surface_identity_unproven` | Use returned semantics if sufficient; otherwise request desktop scope or report the blocker. Do not relabel a crop as verified window capture | +| Permission wait | Let the user approve or deny; after approval reacquire state instead of replaying the timed-out action | +| `background_unavailable` | Reobserve; retry only the necessary action with `delivery_mode:"foreground"` if visible control is authorized | -Parallel workers each declare their **own session** and pass `creates_new_application_instance: true` to `launch_app` so each gets its own window. The element cache is keyed on `(pid, window_id)` and the cursor on `session`, so isolated workers cannot collide. +For an authorized desktop loop, use `get_desktop_state` → input with `target:{"kind":"desktop","display_id":"primary"}` → fresh desktop state. Keyboard input follows visible focus: stop if the user or another controller changes it. -## Delegation +## Recording -`cua-driver` is on PATH -- workers need no `${DROID_PLUGIN_ROOT}` resolution. As with the other drivers, give capture workers **exact commands** with the parent's run scope baked in: +Only when requested, use one persistent MCP connection: `get_recording_state({})` → `start_recording({"output_dir":"/absolute/unused/run-dir","record_video":true})` → authorized actions → `stop_recording({})`. -``` -Task prompt for a desktop capture worker: - "Run these commands in order. Report screenshot paths and any errors. - 1. cua-driver start_session '{"session":"1712345678-42-notepad"}' - 2. cua-driver launch_app '{"name":"Notepad","creates_new_application_instance":true}' - -> note the returned pid and window_id - 3. cua-driver get_window_state '{"pid":,"window_id":}' --screenshot-out-file /tmp/droid-run-1712345678-42-xxxx/before.png - 4. cua-driver type_text '{"pid":,"window_id":,"element_index":,"text":"hello","session":"1712345678-42-notepad"}' - 5. cua-driver get_window_state '{"pid":,"window_id":}' --screenshot-out-file /tmp/droid-run-1712345678-42-xxxx/after.png - 6. cua-driver end_session '{"session":"1712345678-42-notepad"}'" -``` +These recording tools have no public `session` parameter. Video is off by default; check `video_active` and `last_error`. The recorder is shared within its runtime, and manual stop is unconditional: coordinate with an existing owner rather than taking over. Finalize before disconnecting; inspect `last_video_path`, decode the video, and check its scope/dimensions/duration. A working PNG does not prove Wayland video support. ## Evidence handoff -| Proof type | How to capture | -|---|---| -| Window state | `get_window_state ... --screenshot-out-file ${RUN_DIR}/proof-N.png` (also keeps the PNG out of the tool response) | -| Full display | `cua-driver screenshot '{"out_file":"'"${RUN_DIR}"'/screen.png"}'` | -| Semantic assertions | `tree_markdown` from `get_window_state` (filter with `"query":"..."`) | -| Video | `cua-driver recording start` / `recording stop` -> session-scoped `recording.mp4` | - -Hand PNG/mp4 paths to **compose** / **verify** like any other driver output. Keep raw tool output alongside screenshots whenever GUI behavior is the thing under test. - -## Critical rules - -1. **Never change the user's frontmost app.** If a command says activate, foreground, raise, or make key -- stop; the per-pid event paths exist precisely so you do not need it. Platform forbidden-lists live in the upstream pack. -2. **Re-snapshot after every action and report what you observed**, not what you intended. An unchanged tree after an action is a finding, not a formality. -3. **Destructive actions need explicit user intent.** Do not delete files, send messages, or submit forms unless the workflow asked for exactly that. +Report ordinary task results directly. Load **capture** for recorded or multi-step evidence, **verify** for formal proof/QA, and **compose** only for a produced artifact. Include driver/host, exact target/input route, observed postcondition, raw evidence paths, and any limitation. Preserve partial recordings as incomplete evidence; review private content before sharing. diff --git a/plugins/droid-control/skills/desktop-control/platforms/linux.md b/plugins/droid-control/skills/desktop-control/platforms/linux.md deleted file mode 100644 index 13ac37d..0000000 --- a/plugins/droid-control/skills/desktop-control/platforms/linux.md +++ /dev/null @@ -1,57 +0,0 @@ -# Desktop-Control: Linux - -cua-driver on Linux enumerates windows via **X11**, walks semantic trees via **AT-SPI**, and injects input via **XSendEvent** (synthetic events targeted at a window XID -- no focus change, nothing leaks to the user's focused app). Upstream calls this tier pre-release, and it shows: the lifecycle (install, daemon, doctor, sessions, one-shot CLI), window discovery, and per-window screenshots are solid; Wayland-native enumeration, AT-SPI tree quality, and input delivery are not. Plan workflows around the reliable half. - -## Install and daemon - -Same installer and lifecycle as everywhere else (no sudo, `~/.cua-driver`): - -```bash -cua-driver doctor # trustworthy probes: catches missing DISPLAY, verifies X11 + AT-SPI before you waste a run -cua-driver serve # required for element_index workflows -cua-driver status -``` - -`cua-driver permissions` is a no-op surface on Linux. - -## The Wayland boundary - -Window enumeration is **X11-only**. On a modern Plasma/GNOME Wayland desktop, native-Wayland windows are invisible to `list_windows` -- which is most windows. - -- Targets running under **Xwayland** (or a plain X11 session) enumerate and screenshot fine. -- To drive an app that defaults to native Wayland, force its X11 backend at launch where the toolkit allows it: `QT_QPA_PLATFORM=xcb` (Qt), `GDK_BACKEND=x11` (GTK), `--ozone-platform=x11` (Chromium/Electron). -- If the target cannot be put on X11, desktop-control cannot see it -- fall back to **agent-browser** (web/Electron) or **true-input** (terminal emulators). - -## Semantic layer (AT-SPI) reliability - -AT-SPI trees can collapse: the registry's `GetChildren` may time out, and Qt apps can render as a single root node even with `QT_LINUX_ACCESSIBILITY_ALWAYS_ON=1`. When `get_window_state` returns a near-empty tree: - -```bash -cua-driver config set capture_mode vision # screenshot-only snapshots -``` - -and work the pixel path (`click '{"pid":N,"window_id":W,"x":X,"y":Y}'`) against the returned PNG. Don't burn turns re-snapshotting hoping the tree fills in -- on this tier, pixel-first is a legitimate default. - -## The toolkit boundary: synthetic input is silently dropped by Qt and GTK4 - -XSendEvent marks events with the `send_event` flag, and major toolkits **ignore flagged input entirely**. Verified on v0.5.1: Qt apps (kcalc) and GTK4 apps (zenity) no-op on *every* action -- pixel clicks, `press_key`, `type_text` -- while the driver reports success. There is no error to catch; only the post-action snapshot reveals it. - -Practical consequence: the Act stage only works against apps that honor synthetic events (verified: winit-based apps like alacritty; generally simpler/older X11 toolkits). **Probe before committing to a workflow**: send one cheap keystroke, re-snapshot, and check it rendered. If the target ignores synthetic input, desktop-control cannot act on it on this tier -- Observe (screenshots, window enumeration) still works, but route the interaction through **agent-browser** (web/Electron) or **true-input** (terminal) instead. - -## Text input is lossy even where it lands - -In apps that do accept synthetic input, typing drops and mangles characters: shifted symbols can inject as their unshifted key (`*` arriving as `8`), trailing characters get dropped (verified: `type_text "echo ok42"` rendered `echo ok4`), and `type_text_chars` with generous per-char delays still loses keystrokes. `hotkey` chords (including paste shortcuts) and middle-click paste do **not** land reliably, so the clipboard is not a workaround here. - -What works: short bursts plus verification. After every `type_text`, re-snapshot, compare the rendered text against what you sent, and repair the diff (`press_key` backspace, retype the missing tail). On Linux the post-action screenshot is not a formality -- it is the only way to know what actually arrived. - -## Failure modes - -| Symptom | Fix | -|---|---| -| Expected window missing from `list_windows` | Native-Wayland target -- relaunch it on the X11 backend (`QT_QPA_PLATFORM=xcb` / `GDK_BACKEND=x11` / `--ozone-platform=x11`) | -| Tree is a single root node / AT-SPI timeouts | `capture_mode vision` + pixel actions | -| Every action "succeeds" but nothing changes | Toolkit drops `send_event` input (Qt, GTK4) -- target is unreachable on this tier; use agent-browser or true-input for the interaction | -| Typed text arrives mangled or truncated | Verify-and-repair loop: re-snapshot, diff rendered text, backspace + retype the tail | -| `doctor` reports no DISPLAY | Run from the graphical session (or export the session's `DISPLAY`/`XAUTHORITY`), not a bare TTY/SSH context | - -Deep mechanics live in the upstream pack: `~/.cua-driver/skills/cua-driver/LINUX.md`. diff --git a/plugins/droid-control/skills/desktop-control/platforms/macos.md b/plugins/droid-control/skills/desktop-control/platforms/macos.md deleted file mode 100644 index e7c1645..0000000 --- a/plugins/droid-control/skills/desktop-control/platforms/macos.md +++ /dev/null @@ -1,64 +0,0 @@ -# Desktop-Control: macOS - -cua-driver on macOS posts events per-pid through Accessibility (AX) and captures via ScreenCaptureKit. Both are gated by TCC, and TCC attributes grants to the **app bundle that asks** -- which is why every flow below routes through `CuaDriver.app` instead of your terminal. - -## Permissions (TCC) - -```bash -cua-driver permissions grant # LaunchServices-routed: the Accessibility + Screen Recording dialogs - # attribute to com.trycua.driver, then it confirms the driver's own status -cua-driver permissions status # read-only via the daemon; reports `unknown` when no daemon is up -``` - -Do not grant by clicking through System Settings for your terminal app -- the daemon runs under the bundle identity, and terminal-attributed grants do nothing for it. The first real screen capture may trigger one extra consent sheet; accept it. - -## Daemon launch - -Launch from the logged-in GUI session so the daemon attaches to it with the bundle's TCC identity: - -```bash -open -n -g -a CuaDriver --args serve -cua-driver status -cua-driver stop -``` - -SSH-launched bare binaries often miss the GUI session and their AX/capture probes hang. (`cua-driver mcp` and CLI tool calls auto-proxy to a properly attributed daemon when one is reachable.) - -## Patterns - -**Reliable terminal command entry** -- when `type_text` or raw key posting drops characters in Terminal-class apps, route through the pasteboard: - -```bash -printf '%s' 'your command' | pbcopy -cua-driver hotkey '{"pid":,"window_id":,"keys":["cmd","v"]}' -cua-driver press_key '{"pid":,"window_id":,"key":"return"}' -``` - -**Native security / modal sheets** (SecurityAgent, Keychain prompts, auth dialogs) -- these often report `is_on_screen: false` even while visible. Locate by process, then enumerate everything: - -```bash -pgrep -fl SecurityAgent -cua-driver list_windows '{"pid":,"on_screen_only":false}' -cua-driver get_window_state '{"pid":,"window_id":}' -``` - -Only enter credentials in environments you own and were explicitly authorized to drive. - -**Menu commands / app shortcuts** -- pass `window_id` so AppKit routes the key equivalent to the target app instead of the frontmost one: - -```bash -cua-driver hotkey '{"pid":835,"window_id":79,"keys":["cmd","q"]}' -``` - -**Backgrounded / off-space windows** -- the driver acts on `(pid, window_id)` without raising. Enumerate with `on_screen_only: false` and target directly. - -## Failure modes - -| Symptom | Fix | -|---|---| -| AX write fails (`AXPress` returns `-25204`) on a system sheet | Fall back to `press_key` / `hotkey` / pixel `click` | -| ScreenCaptureKit error (e.g. SCK `-3801`) in `som`/`vision` capture | `cua-driver config set capture_mode ax` (tree-only, skips Screen Recording), or retry | -| Known dialog missing from `list_windows` results | Re-query with `"on_screen_only": false` | -| Probes hang / permissions report `unknown` | Daemon was launched without GUI attribution -- `cua-driver stop`, relaunch via `open -n -g -a CuaDriver --args serve` | - -Deep mechanics (no-foreground forbidden-list, AXMenuBar navigation, SkyLight click dispatch, Apple-Events browser bridge) live in the upstream pack: `~/.cua-driver/skills/cua-driver/MACOS.md`. diff --git a/plugins/droid-control/skills/desktop-control/platforms/windows.md b/plugins/droid-control/skills/desktop-control/platforms/windows.md deleted file mode 100644 index b55ae11..0000000 --- a/plugins/droid-control/skills/desktop-control/platforms/windows.md +++ /dev/null @@ -1,45 +0,0 @@ -# Desktop-Control: Windows - -cua-driver on Windows walks UI Automation (UIA) trees and dispatches actions through a layered UIA + `PostMessage` chain -- per-window message posting, not HID synthesis, so the user's foreground app is untouched. - -## Install and daemon - -The upstream installer is per-user (no admin elevation): binary under `%LOCALAPPDATA%\Programs\Cua\cua-driver\bin`, data and skill pack under `%USERPROFILE%\.cua-driver`, and an autostart task (`cua-driver autostart status|kick|disable`) registered for the daemon. - -```powershell -cua-driver doctor -cua-driver serve # required for element_index workflows -cua-driver status -cua-driver stop -``` - -`cua-driver permissions` is a no-op surface on Windows (TCC is a macOS concept) -- there is no grant dance. The real constraint is **Session 0 isolation**: anything launched by a service (including some SSH daemons) lives in a session with no interactive desktop, where window enumeration returns nothing. Tool calls auto-proxy to an interactive-session daemon when one is reachable; if results come back empty, confirm the daemon was started from the logged-in interactive session, not a service context. - -## JSON quoting (the PowerShell 5.1 footgun) - -Windows PowerShell 5.1 strips quotes around JSON field names in multi-field arguments, so positional JSON fails to parse. Pipe via stdin, or use PowerShell 7+ (`pwsh`): - -```powershell -'{"pid":1234,"window_id":5678}' | cua-driver get_window_state -``` - -From `cmd.exe`, escape inner quotes instead: `cua-driver get_window_state "{\"pid\":1234,\"window_id\":5678}"`. - -## Patterns - -**UWP / packaged apps** -- Store apps (Calculator, Settings) are hosted by `ApplicationFrameHost.exe`, so the visible window's pid is the host's, not the app process's. If `list_windows` against the app's own pid comes up empty, enumerate `ApplicationFrameHost.exe`'s windows and match by title. Classic Win32 apps (Notepad, Explorer) own their windows directly. - -**Minimized windows** -- `get_window_state` and element-index actions work in place, but `press_key` commits silently no-op (no message pump focus). Use `set_value` or element-index-click the commit-equivalent button instead. - -**Browsers / Electron** -- prefer **agent-browser**. If you must stay in desktop-control, launch the browser with `--remote-debugging-port=` and export `CUA_DRIVER_CDP_PORT=` so `execute_javascript` / `query_dom` can attach; UIA covers `get_text` either way. - -## Failure modes - -| Symptom | Fix | -|---|---| -| `UIA invoke failed` on an element | Try `click` with an explicit `action` (`show_menu`, `confirm`, ...) or fall through to a pixel click on the element's center | -| Empty window lists, blank screenshots | Session 0 daemon -- restart `cua-driver serve` from the interactive desktop session | -| Positional JSON "did not parse" errors | PowerShell 5.1 quote-stripping -- pipe JSON via stdin or use `pwsh` | -| Target window not under the app's pid | UWP hosting -- enumerate `ApplicationFrameHost.exe` windows | - -Deep mechanics (UIA tree semantics, click-dispatch layering, focus-steal vectors, UAC boundaries) live in the upstream pack: `~/.cua-driver/skills/cua-driver/WINDOWS.md`. diff --git a/plugins/droid-control/skills/droid-control/SKILL.md b/plugins/droid-control/skills/droid-control/SKILL.md index dc6c99f..5d3c9b5 100644 --- a/plugins/droid-control/skills/droid-control/SKILL.md +++ b/plugins/droid-control/skills/droid-control/SKILL.md @@ -1,18 +1,18 @@ --- name: droid-control -description: Control terminal TUIs and web/Electron apps for testing, demos, QA, and computer-use tasks. Use when you need to automate a CLI, drive a browser, record a demo, or capture proof artifacts. +description: Control terminal TUIs, browsers, and native desktop apps for testing, demos, QA, and computer-use tasks. Use when you need to operate an app, automate a CLI, record a demo, or capture proof. --- # Droid Control -Automate terminals and browsers. Three routing decisions, then atoms guide you the rest of the way. +Automate terminals, browsers, and desktop apps. Route by the user's requested method first, then load only the mechanics and evidence stages needed. ## Ground rules 1. **Real apps, real environments.** Non-deterministic behavior (LLM responses, network latency, variable output) is expected. Handle it with `wait` / `wait-idle`. Never substitute fixtures or mocked data. -2. **Commit to execution.** Once you've chosen a driver, run the plan. If something fails mid-run, recover and retry -- don't re-evaluate the approach. -3. **Atoms are self-contained.** Load one and follow its mechanics. No cross-referencing needed. -4. **`tctl` is the ONLY way to launch recorded sessions.** `tctl` manages recording by wrapping `asciinema rec` around the PTY — raw `tuistory` has no recording capability and never will. Never call `tuistory launch` directly; unknown flags crash `tuistory-relay`. Always resolve `TCTL` to its absolute filesystem path before use, especially when delegating to workers (they don't inherit `${DROID_PLUGIN_ROOT}`). +2. **Recover from evidence.** After a failed or uncertain action, observe current state before retrying. Honor method constraints and permission boundaries; a refusal does not authorize another driver or broader target. +3. **Atoms include their references.** Load linked material on demand. Desktop-control does not require a separately installed cua skill. +4. **`tctl` owns recorded terminal sessions.** It wraps `asciinema rec` around the PTY; browser and desktop drivers own their separate lifecycles. Never call `tuistory launch` directly. Resolve `TCTL` to an absolute path only for terminal workflows or worker handoffs. 5. **Isolate every run.** Multiple droids may be filming simultaneously on the same machine. Session names and output paths share a global namespace (`/tmp/tctl-sessions/`). At the start of every workflow, generate a run ID (`RUN_ID=$(date +%s)-$$` or similar) and use it as a prefix for all session names and a scoped temp directory for all output files: ```bash RUN_ID="$(date +%s)-$$" @@ -21,6 +21,7 @@ Automate terminals and browsers. Three routing decisions, then atoms guide you t # Output paths: ${RUN_DIR}/before.cast, ${RUN_DIR}/after.cast ``` Never use bare session names like `-s demo`, `-s before`, `-s after` — they will collide with concurrent runs. + Separate names and paths do not isolate shared desktop focus or keyboard input. Keep one controller for a visible desktop. ## Routing @@ -30,6 +31,7 @@ Three independent lookups. Do all three, then load the union of skills they prod | Target | Load these skills | |---|---| +| User explicitly requests cua-only, native GUI input, or desktop control (including Electron) | **desktop-control**; method constraints override the defaults below | | Droid CLI (`droid-dev`, `droid exec`) | **droid-cli** + tuistory backend via `${DROID_PLUGIN_ROOT}/bin/tctl` | | Droid CLI (real terminal proof) | **true-input** + **droid-cli** | | Other terminal TUI | tuistory backend via `${DROID_PLUGIN_ROOT}/bin/tctl` | @@ -38,7 +40,7 @@ Three independent lookups. Do all three, then load the union of skills they prod | Native desktop GUI app | **desktop-control** | | Raw terminal byte sequences | **true-input** + **pty-capture** | -**tuistory** is the default for terminal work. Use **true-input** only when you need real terminal rendering evidence. On Linux, desktop-control rides upstream's pre-release tier -- its platform file documents the Wayland/AT-SPI/input caveats and when to fall back to **agent-browser** or **true-input**. +**tuistory** is the default for terminal work. Use **true-input** when real terminal rendering evidence is needed. Desktop-control includes compositor-specific guidance; inspect live Cua capabilities rather than assuming all Linux targets are X11-only or abandoning the user's chosen method. ### 2. Stage route — what does the workflow need? @@ -46,9 +48,9 @@ Every workflow passes through stages. Load the atoms for each stage you'll use. | Stage | Skill | When to load | |---|---|---| -| Capture | **capture** | Always -- every workflow records or captures something | +| Capture | **capture** | Recording, scripted multi-step evidence, or a demo/QA deliverable; ordinary desktop operation uses the driver's observe/verify loop | | Compose | **compose** | When the deliverable is a produced artifact (video, annotated screenshots, comparison image) | -| Verify | **verify** | Always -- every deliverable gets checked against commitments | +| Verify | **verify** | Formal proof, demo, or QA deliverable; every action still needs verification even without this stage | ### 3. Artifact route — does compose need polish tools? @@ -89,14 +91,15 @@ Do not synthesize a "before" state to justify `side-by-side`. If there is no rea ## Delegation -The parent agent plans and orchestrates. Mechanical work runs in **worker subagents** via the Task tool. This keeps the parent's context clean and enables parallelism. +Keep short interactive desktop tasks in the parent: it owns observations, input, user permission waits, and cleanup. Delegate independent capture environments or rendering, not individual screenshots interleaved with another controller's input. ### What to delegate | Task | Delegate? | Why | |---|---|---| -| **Capture clip** (single layout) | YES | Worker runs the interaction script end-to-end and returns the `.cast` path | -| **Capture both clips** (comparison layout) | YES — `run_in_background=true` for each | Branches are independent; run in parallel | +| **Interactive shared desktop** | NO — parent | One controller owns focus, snapshots, input, permission waits, and cleanup | +| **Capture clip in an isolated terminal/browser environment** | YES | Worker owns the complete interaction and recording lifecycle | +| **Capture both clips** (comparison layout) | YES, only with independent environments | Worktrees and session labels alone do not isolate a desktop | | **Remotion render** | YES | Needs only props JSON, clip paths, output path. Runs `render-showcase.sh` (handles .cast conversion, fidelity profiles, duration detection, cleanup) | | Planning, interaction scripting | NO — parent | Requires PR context and editorial judgment | | Layout and prop construction | NO — parent | Requires editorial decisions about effects, timing, labels | @@ -143,7 +146,7 @@ Task prompt for a Remotion render worker: ### Parallel capture pattern (comparison flows only) -Only applicable when the Layout default table above selects `side-by-side`. For a `single` layout, launch one capture worker and skip this section. +Only applicable when the Layout default table selects `side-by-side` and the capture environments are independent. Serialize shared-desktop captures. For a single interactive desktop task, keep control in the parent. For before/after comparison demos, launch both capture workers simultaneously: @@ -209,7 +212,7 @@ Deterministic recipe for reproducing degraded transcript tails in the droid CLI | true-input | Windows (KVM) | `libvirt`, `qemu`, KVM VM with SPICE + SSH, `DROID_VM_*` env vars | `virt-manager` | | true-input | macOS (QEMU) | `qemu`, `socat`, macOS VM with SSH, `DROID_MAC_*` env vars | — | | agent-browser | All | `agent-browser` (+ `agent-browser install`) | — | -| desktop-control | All | `cua-driver` (+ daemon via `cua-driver serve`; macOS also `cua-driver permissions grant`) | upstream skill pack (`cua-driver skills install`) | +| desktop-control | All | `cua-driver` in the intended graphical session; approved OS permissions | Documentation is bundled; no separate skill install | | compose | All | `ffmpeg`, `ffprobe`, `agg` | — | | showcase | All | Node.js (>= 18), Chrome/Chromium | — | @@ -228,9 +231,8 @@ sudo apt-get install -y grim wf-recorder # optional: screenshots + v # agent-browser driver agent-browser install # one-time: downloads bundled Chromium -# desktop-control driver (Windows hosts: irm .../scripts/install.ps1 | iex) -curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.sh | bash -cua-driver skills install # upstream skill pack (deep tool reference) +# desktop-control: follow its setup instructions only if the binary +# is missing and installation is approved. No separate skill install. # compose + showcase (video rendering) sudo apt-get install -y ffmpeg # video processing (includes ffprobe) diff --git a/plugins/droid-control/skills/verify/SKILL.md b/plugins/droid-control/skills/verify/SKILL.md index dcabb19..38add81 100644 --- a/plugins/droid-control/skills/verify/SKILL.md +++ b/plugins/droid-control/skills/verify/SKILL.md @@ -15,6 +15,8 @@ You receive: 1. **Commitments** from the command's parse step -- the promises made about what the deliverable would contain 2. **Compose outputs** -- the finished artifact(s) and their metadata +Composition is optional: raw desktop screenshots/state and the driver's evidence handoff can be the deliverable. Verify the task's postcondition, not merely the action's exit status or `effect` field. An unavailable capture or permission wait is a blocker, not proof of failure or success. + ## Video deliverables ### Technical checks