Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/check-skills.yml
Original file line number Diff line number Diff line change
Expand Up @@ -83,3 +83,6 @@ jobs:
fi

echo "All skills are accounted for."

- name: Verify desktop-control documentation links
run: python3 -m unittest discover -s plugins/droid-control/tests -p 'test_desktop_control.py' -v
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ Terminal, browser, and computer automation for Droids. Record demos, verify beha

**Commands:** `/demo`, `/verify`, `/qa-test`

**Skills:** `droid-control` (orchestrator), `tuistory`, `true-input`, `agent-browser`, `droid-cli`, `pty-capture`, `capture`, `compose`, `verify`, `showcase`
**Skills:** `droid-control` (orchestrator), `tuistory`, `true-input`, `agent-browser`, `desktop-control`, `droid-cli`, `pty-capture`, `capture`, `compose`, `verify`, `showcase`

See [plugins/droid-control/README.md](plugins/droid-control/README.md) for details.

Expand Down
2 changes: 1 addition & 1 deletion plugins/droid-control/.factory-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"name": "droid-control",
"description": "Terminal, browser, and native desktop automation for testing, demos, QA, and computer-use tasks",
"version": "1.0.0"
"version": "1.1.0"
}
15 changes: 10 additions & 5 deletions plugins/droid-control/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ The plugin is designed to keep a droid focused while it operates real software:

- **Low context load:** load the Linux tuistory path without dragging in Windows KVM notes, macOS VM controls, browser automation, and Remotion internals.
- **Evidence-first workflows:** every command starts by making commitments, then ends by verifying the artifact against those commitments.
- **Parallel execution:** before/after captures and render jobs can run in worker droids without sharing session names or output paths.
- **Parallel execution:** independent capture environments and render jobs can run in workers. Shared desktop input stays serialized; session names do not isolate focus.
- **Clear ownership:** commands decide *what* must be produced; atom skills decide *how* to execute their slice.
- **Platform specificity:** OS-specific mechanics live in platform subdocuments, not in global instructions.

Expand Down Expand Up @@ -85,9 +85,10 @@ The parent droid keeps judgment. Workers get exact commands.

| Work | Owner | Reason |
|---|---|---|
| Short interactive desktop task | Parent | One controller owns observation, input, permission waits, and cleanup. |
| Interpret PR / claim / QA goal | Parent | Requires context and judgment. |
| Write the interaction script | Parent | Defines the proof story. |
| Capture baseline and candidate branches | Worker droids | Independent, mechanical, parallelizable. |
| Capture baseline and candidate branches | Workers only for independent environments | A shared desktop must be captured serially. |
| Render Remotion video | Worker droid | Mechanical once props and clips are fixed. |
| Verify commitments | Parent | Requires the original contract and evidence judgment. |

Expand Down Expand Up @@ -161,13 +162,17 @@ skills/true-input/platforms/macos.md
skills/pty-capture/platforms/linux.md
skills/pty-capture/platforms/windows.md
skills/pty-capture/platforms/macos.md
skills/desktop-control/platforms/linux.md
skills/desktop-control/platforms/windows.md
skills/desktop-control/platforms/macos.md
references/cua-driver/LINUX.md
references/cua-driver/WINDOWS.md
references/cua-driver/MACOS.md
```

A Linux droid reads Linux Wayland instructions. A Windows VM byte-capture task reads Windows KVM instructions. The system does not rely on the droid to skim irrelevant sections correctly.

Desktop-control includes plugin-maintained guidance adapted from Cua, with its license and attribution. Its entrypoint owns plugin setup and evidence handoff; the local references own driver mechanics. Maintain them with the plugin and check their links offline. No user-home skill dependency or runtime documentation download is required.

For ordinary desktop tasks, the driver verifies each action and reports directly. Capture/verify stages are loaded for formal evidence deliverables, and compose only when a produced artifact was requested. Explicit GUI-only or cua-only constraints take precedence over Electron's default browser route.

## Extending the plugin

Use the same composition rules when adding capability:
Expand Down
6 changes: 5 additions & 1 deletion plugins/droid-control/NOTICES.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
# Third-Party Notices

This plugin depends on several third-party tools and libraries. They are not bundled -- each is installed separately by the user. Their respective licenses apply at the point of installation and use.
This plugin depends on several third-party tools and libraries. Executables are installed separately by the user. The Cua documentation listed below is bundled; its license accompanies the copied files.

## Desktop automation documentation

- **[Cua Driver](https://github.com/trycua/cua)** — documentation under `references/cua-driver/` is adapted from Cua and maintained with this plugin under the [included MIT license](references/cua-driver/LICENSE.md). Driver code and binaries are not bundled. See [reference maintenance](references/README.md).

## Video rendering

Expand Down
10 changes: 7 additions & 3 deletions plugins/droid-control/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Terminal, browser, and computer automation plugin for Droids.

Droids can read and write code. This plugin lets them *operate* it: launch apps, type commands, click buttons, record what happens, and produce polished evidence. No human hands required.
Droids can read and write code. This plugin lets them *operate* it: launch apps, type commands, click buttons, record what happens, and produce evidence. OS permission grants and consequential actions still require the appropriate user or host authorization.

## What you get

Expand Down Expand Up @@ -48,6 +48,10 @@ Or use the `/plugins` UI: Browse tab, select droid-control, install.

Then open a Droid session and run `/demo`, `/verify`, or `/qa-test`.

For ordinary desktop work, ask directly: **“Using only cua, open Calculator and compute 17 × 23.”** Desktop-control runs the observe/act/verify loop without loading video-production stages.

The plugin includes Cua driver documentation and setup guidance. Install the `cua-driver` executable if missing; no separate cua skill installation is needed. The references are [maintained with this plugin](references/README.md). Installed driver versions and Wayland compositors may support different capabilities.

## Commands

### `/demo`
Expand All @@ -69,7 +73,7 @@ Runs automated QA against terminal CLIs, web apps, or Electron apps. Accepts a U
1. **Commands** parse user intent into commitments.
2. **The orchestrator** routes by target, stage, and artifact needs.
3. **Atom skills** provide only the mechanics needed right now: drivers, target patterns, capture, compose, verify, and showcase polish.
4. **Workers** handle mechanical capture/render jobs while the parent droid keeps planning and verification context.
4. **Workers** handle independent capture/render jobs. The parent keeps short interactive desktop tasks, including observations, input, permission waits, and cleanup.
5. **Verify** checks the final evidence against the original commitments.

For the full rationale and runtime pipeline, see [`ARCHITECTURE.md`](ARCHITECTURE.md).
Expand Down Expand Up @@ -103,4 +107,4 @@ curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scr
cd plugins/droid-control/remotion && npm install # Remotion video rendering
```

Only install what you need for your use case. Terminal demos need tuistory, asciinema, agg, and ffmpeg. Web/Electron automation just needs agent-browser. Native desktop GUI automation just needs cua-driver.
Only install what you need, with approval. Terminal demos need tuistory, asciinema, agg, and ffmpeg. Web/Electron automation defaults to agent-browser; an explicit cua-only/native-input request uses desktop-control instead. Native desktop automation needs cua-driver plus the graphical session and OS permissions reported by its preflight. Recording and rendering have additional dependencies; they are not required for ordinary desktop tasks.
6 changes: 4 additions & 2 deletions plugins/droid-control/commands/qa-test.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
description: Run an automated QA test flow against a terminal CLI or web/Electron app
description: Run an automated QA test flow against a terminal, browser, or native desktop app
argument-hint: '"<URL>" or "<app-name>" or "<PR-number> [-- focus area]" or "<description>"'
---

Expand All @@ -9,7 +9,7 @@ Load skills: **droid-control**.

`$ARGUMENTS` can be:
- **URL** (`https://app.factory.ai`, `localhost:3000`) → web app
- **Electron app name** (`Slack`, `VS Code`, `Figma`) → Electron app via CDP
- **Desktop app name** (`Slack`, `VS Code`, `Calculator`) → use the orchestrator's target route; explicit cua/native-input requirements override Electron's CDP default
- **CLI command** (`droid-dev`, `htop`, `my-cli --flag`) → terminal TUI
- **PR reference** (`11386`) with optional `-- focus area` → infer target from the diff
- **Free-text description** ("test the login flow on staging") → infer target and flow
Expand Down Expand Up @@ -48,6 +48,8 @@ If the user provides specific steps, use them. Otherwise, design a reasonable fl

**Terminal**: launch app → wait for ready → snapshot → exercise primary features → verify output → snapshot → close.

**Native desktop / cua-only**: discover or launch the requested app → select an exact target → observe → act → verify each postcondition. Broaden to visible desktop capture/input only with authorization. Leave personal applications open unless closure is requested; end only the automation run.

If the flow is ambiguous or success criteria are unclear, ask the user.

## Capture
Expand Down
27 changes: 27 additions & 0 deletions plugins/droid-control/references/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Desktop-control references

`cua-driver/` contains plugin-maintained guidance adapted from
[Cua](https://github.com/trycua/cua). Keep its [MIT license](cua-driver/LICENSE.md)
and attribution when editing. The executable is installed separately.

The [desktop-control entrypoint](../skills/desktop-control/SKILL.md) owns plugin
routing, setup, and evidence handoff. Load its linked mechanics references on
demand; they are documentation, not another skill to install.

## Maintenance

Edit these documents with the plugin. Keep shared action rules in
[WORKFLOW.md](cua-driver/WORKFLOW.md), lifecycle and authorization in
[RUNTIME.md](cua-driver/RUNTIME.md), and host-specific details in the platform
guides. Check changed examples against the intended driver's live schemas and
behavior rather than assuming every installation supports the same features.

From the plugin directory, check that the documentation's links and anchors
resolve after relocation:

```bash
python3 -m unittest discover -s tests -p 'test_desktop_control.py'
```

This static packaging check needs no Cua installation, personal skill directory,
GUI access, or network. It does not certify live desktop or recording behavior.
Loading
Loading