Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
147 changes: 147 additions & 0 deletions .github/workflows/pi-computer.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
name: PI and Computer backend

on:
workflow_dispatch:
push:
paths:
- '.github/workflows/pi-computer.yml'
- '.cargo/config.toml'
- 'openless-all/app/Cargo.toml'
- 'openless-all/app/Cargo.lock'
- 'openless-all/app/crates/openless-computer/**'
- 'openless-all/app/crates/openless-core/**'
- 'openless-all/app/pi-backend/**'
- 'openless-all/app/scripts/prepare-pi-backend*'
- 'openless-all/app/scripts/pi-node-entitlements.plist'
pull_request:
paths:
- '.github/workflows/pi-computer.yml'
- '.cargo/config.toml'
- 'openless-all/app/Cargo.toml'
- 'openless-all/app/Cargo.lock'
- 'openless-all/app/crates/openless-computer/**'
- 'openless-all/app/crates/openless-core/**'
- 'openless-all/app/pi-backend/**'
- 'openless-all/app/scripts/prepare-pi-backend*'
- 'openless-all/app/scripts/pi-node-entitlements.plist'

permissions:
contents: read

concurrency:
group: pi-computer-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true

jobs:
backend:
name: Backend (${{ matrix.os }})
runs-on: ${{ matrix.os }}
timeout-minutes: 45
strategy:
fail-fast: false
matrix:
os: [ubuntu-24.04, macos-latest, windows-latest]
defaults:
run:
shell: bash
working-directory: openless-all/app
env:
CARGO_TERM_COLOR: always
RUST_BACKTRACE: '1'

steps:
- uses: actions/checkout@v4

- uses: actions/setup-node@v4
with:
node-version: '22.23.2'
cache: npm
cache-dependency-path: openless-all/app/pi-backend/package-lock.json

- uses: dtolnay/rust-toolchain@stable

- uses: swatinem/rust-cache@v2
with:
workspaces: 'openless-all/app -> target'

- name: Install Linux native build dependencies
if: runner.os == 'Linux'
run: |
sudo apt-get update
sudo apt-get install -y --no-install-recommends \
build-essential clang pkg-config \
libxkbcommon-dev libxcb1-dev \
libpipewire-0.3-dev libwayland-dev libgbm-dev libegl1-mesa-dev

- name: Install PI dependencies
run: npm ci --prefix pi-backend --ignore-scripts --no-audit --no-fund

- name: Test PI tools and protocol
run: npm --prefix pi-backend test

- name: Test native Computer protocol
run: cargo test --locked -p openless-computer

- name: Test relevant application core modules
# Keep this job scoped to PI/Computer. The unrelated Whisper HTTP fixture
# tests are exercised by the existing core test workflow.
run: |
cargo test --locked -p openless-core coding_agent
cargo test --locked -p openless-core less_computer
cargo test --locked -p openless-core pi_backend

- name: Test packaging contract
run: node --test scripts/prepare-pi-backend.test.mjs

- name: Prepare standalone PI and Computer bundle
run: node scripts/prepare-pi-backend.mjs --force

- name: Verify bundled runtime without desktop access
# Run from a different directory to catch accidental source-tree lookup.
# Both flags only inspect installation health; no screenshot or input.
run: |
node --input-type=module <<'NODE'
import assert from 'node:assert/strict';
import { spawnSync } from 'node:child_process';
import { join, resolve } from 'node:path';
const bundle = resolve('src-tauri/resources/pi-backend');
const suffix = process.platform === 'win32' ? '.exe' : '';
const node = join(bundle, `node${suffix}`);
const computer = join(bundle, `openless-computer${suffix}`);
const options = {
cwd: process.env.RUNNER_TEMP,
encoding: 'utf8',
windowsHide: true,
timeout: 30000,
env: { ...process.env, OPENLESS_COMPUTER_BIN: computer },
};
function check(executable, args) {
const result = spawnSync(executable, args, options);
if (result.error) throw result.error;
assert.equal(result.status, 0, result.stderr || result.stdout);
const response = JSON.parse(result.stdout);
assert.equal(response.ok, true);
return response;
}
const health = check(node, [join(bundle, 'runtime/index.mjs'), '--health']);
const capabilities = check(computer, ['--capabilities']);
assert.equal(capabilities.data.protocol_version, 1);
assert.equal(capabilities.data.availability, 'not_probed');
console.log(JSON.stringify({ health, capabilities }));
NODE

- name: Archive standalone backend
# Tar preserves the executable bits needed by the bundled Node/helper on Unix.
run: |
mkdir -p pi-artifacts
tar -czf "pi-artifacts/openless-pi-backend-${{ runner.os }}-${{ runner.arch }}.tar.gz" \
-C src-tauri/resources pi-backend

- name: Upload standalone backend artifact
uses: actions/upload-artifact@v4
with:
name: openless-pi-backend-${{ runner.os }}-${{ runner.arch }}
path: openless-all/app/pi-artifacts/*.tar.gz
if-no-files-found: error
compression-level: 0
retention-days: 14
20 changes: 20 additions & 0 deletions .github/workflows/release-linux-egui.yml
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ jobs:
appstream \
build-essential \
cmake \
clang \
desktop-file-utils \
extra-cmake-modules \
fcitx5-modules-dev \
Expand All @@ -51,6 +52,10 @@ jobs:
libfcitx5core-dev \
libfcitx5utils-dev \
libopenblas-dev \
libpipewire-0.3-dev \
libxcb1-dev \
libgbm-dev \
libegl1-mesa-dev \
libssl-dev \
libwayland-dev \
libx11-dev \
Expand Down Expand Up @@ -78,6 +83,12 @@ jobs:
with:
components: clippy

- uses: actions/setup-node@v4
with:
node-version: '22.23.2'
cache: npm
cache-dependency-path: 'openless-all/app/pi-backend/package-lock.json'

- name: Cache Cargo
uses: swatinem/rust-cache@v2
with:
Expand Down Expand Up @@ -148,6 +159,15 @@ jobs:
working-directory: openless-all/app
run: cargo build --locked --release -p openless-linux-egui

- name: Build bundled PI and native Computer runtime
working-directory: openless-all/app
run: |
node --test scripts/prepare-pi-backend.test.mjs
node scripts/prepare-pi-backend.mjs --target x86_64-unknown-linux-gnu
src-tauri/resources/pi-backend/node --version
src-tauri/resources/pi-backend/node src-tauri/resources/pi-backend/runtime/index.mjs --health
src-tauri/resources/pi-backend/openless-computer --capabilities

- name: Resolve package version
id: version
shell: bash
Expand Down
6 changes: 5 additions & 1 deletion .github/workflows/release-tauri.yml
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ jobs:

- uses: actions/setup-node@v4
with:
node-version: "22"
node-version: "22.23.2"
cache: npm
cache-dependency-path: 'openless-all/app/package-lock.json'

Expand All @@ -94,6 +94,10 @@ jobs:
working-directory: 'openless-all/app'
run: npm ci

- name: Verify PI packaging contract
working-directory: 'openless-all/app'
run: node --test scripts/prepare-pi-backend.test.mjs

- name: Check updater signing availability
if: startsWith(github.ref, 'refs/tags/v') && endsWith(github.ref, '-tauri')
shell: bash
Expand Down
7 changes: 7 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,9 @@ video-materials/
node_modules/
dist/
target/
# Local build caches and diagnostic logs.
/openless-all/app/.cache/
/openless-all/app/*.log
# 本地从 CI / Actions 下载的 APK 及解压目录(非源码)
ci-artifacts/
*.apk
Expand Down Expand Up @@ -116,5 +119,9 @@ CapsWriter
.reasonix
/.codex

# Generated self-contained PI desktop payload and downloaded build archives.
openless-all/app/.cache/pi-backend/
openless-all/app/src-tauri/resources/pi-backend/

# Mimosa 安全钩子运行状态(不属版本库)
.mimosa/
6 changes: 6 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -256,6 +256,12 @@ On first launch, grant the permissions the app requests.

For the full end-user walkthrough, see [USAGE.md](USAGE.md).

## Bundled PI for Less Computer

The desktop app includes a private PI runtime and native Computer tools. Enable Less Computer and configure a vision-capable model to take screenshots, click, scroll, and type using text or voice requests. No separate PI, Node.js, or Computer MCP installation is required. Desktop control supports Windows, macOS (with system permissions), and Linux X11; Wayland input is currently unsupported. Existing external CLI providers remain available.

See the [PI setup and build guide](docs/less-computer-pi.md) for model configuration, permission modes, and platform requirements.

## Build from source (developers)

The active workspace lives in `openless-all/app/`. `crates/openless-core` is the framework-independent backend, `src-tauri` hosts macOS/Windows/Android, and `linux-egui` contains the native Linux UI and its platform adapters. Initialize submodules before a Tauri source build: its manifest resolves local path dependencies even when their target-specific code is not compiled. These include macOS ASR engines such as [`Open-Less/qwen-asr`](https://github.com/Open-Less/qwen-asr) under `src-tauri/vendor/`. The root Core/Linux workspace excludes `src-tauri`, so its independent checks do not parse the Tauri manifest or require those submodules. Start with the [documentation index](docs/index.md), [architecture](docs/architecture.md), and [source structure](docs/structure.md).
Expand Down
6 changes: 6 additions & 0 deletions README.zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -261,6 +261,12 @@ OpenLess 只做一件事:**把语音变成可用的书面文字(尤其是 AI 提

完整的终端用户指南见 [USAGE.md](USAGE.md)。

## Less Computer 内置 PI

桌面版新增随应用打包的 PI 后端,包含独立运行时和 Computer 原生工具,无需另外安装 PI、Node.js 或 Computer MCP。启用 Less Computer 并配置支持图像的模型后,即可通过文字或语音进行截图、点击、滚动和输入。支持 Windows、macOS 与 Linux X11;macOS 需授予系统权限,Wayland 暂不支持桌面控制。

模型配置、权限模式、构建命令与平台限制见 [Less Computer PI 使用说明](docs/less-computer-pi.md)。已有外部 CLI 后端可继续使用。

## 从源码构建(开发者)

活跃 workspace 位于 `openless-all/app/`:`crates/openless-core` 是框架无关后端,`src-tauri` 承载 macOS/Windows/Android,`linux-egui` 包含 Linux 原生 UI 与平台 Adapter。源码构建 Tauri 前需初始化子模块:即使不编译对应平台代码,Cargo 仍会解析 manifest 中的本地 path 依赖,其中包括 `src-tauri/vendor/` 下的 macOS ASR 引擎,如 [`Open-Less/qwen-asr`](https://github.com/Open-Less/qwen-asr)。根 Core/Linux workspace 排除了 `src-tauri`,其独立检查不解析 Tauri manifest,也不要求这些子模块。阅读入口为 [docs/index.md](docs/index.md)、[架构](docs/architecture.md)和[目录结构](docs/structure.md)。
Expand Down
98 changes: 98 additions & 0 deletions docs/less-computer-pi.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
# Less Computer:内置 PI 后端

桌面安装包包含 PI SDK、独立 Node.js 运行时和原生 `openless-computer` 工具。用户不需要另外安装 PI、Node.js、Python 或 Computer MCP。已有外部 Claude Code / OpenCode / Codex / dsh 配置仍可使用;新安装的默认 Agent 后端为 `pi-bundled`。

## 使用

1. 在设置中启用 Less Computer,后端选择 **PI**。
2. 配置模型服务凭据。可使用标准模型服务环境变量(例如 `OPENAI_API_KEY`),也可使用下面的封装 PI 配置。模型必须支持图像输入,才能理解桌面截图。
3. 选择“允许操作桌面与编辑文件”,打开 Less Computer 面板,输入或说出任务。选择“只读”时,仅允许查看文件和截图。
4. 运行中按 Esc 或使用取消按钮可停止当前任务及其子进程。关闭会话会清除续聊上下文。

模型名称使用 `provider/model` 格式;留空使用封装 PI 的配置。没有有效凭据时会返回配置错误,不会报告桌面任务成功。

## 模型配置

封装 PI 使用自己的配置目录,可用 `OPENLESS_PI_AGENT_DIR` 指定绝对路径。默认目录:

| 系统 | 目录 |
| --- | --- |
| Windows | `%APPDATA%\OpenLess\pi` |
| macOS | `~/Library/Application Support/OpenLess/pi` |
| Linux | `${XDG_CONFIG_HOME:-~/.config}/openless/pi` |

在该目录创建 UTF-8 编码的 `config.json`,例如使用 OpenAI 兼容服务:

```json
{
"provider": "openless",
"model": "your-vision-model",
"baseUrl": "https://your-provider.example/v1",
"api": "openai-completions",
"apiKeyEnv": "OPENAI_API_KEY"
}
```

凭据可通过环境变量提供;`config.json` 也支持 `apiKey` 字段,该字段会以明文存储在本机,请仅使用个人可读的目录,不要将真实配置放进源码或安装包。完整配置约定及标准 PI `models.json` / `auth.json` 支持见 [PI runtime 文档](../openless-all/app/pi-backend/README.md)。安装包不含模型凭据,也不会自动选择工作目录里的扩展或脚本。

## 平台范围

| 平台 | 原生操作 | 首次运行条件 |
| --- | --- | --- |
| Windows | 截图、显示器查询、鼠标、滚轮、按键、Unicode 文本 | 普通用户交互式桌面;不能跨越 UAC 安全桌面 |
| macOS | 同上 | 系统设置中允许屏幕录制与辅助功能 |
| Linux X11 | 同上 | 可访问当前 X11 会话 |
| Linux Wayland | 当前不提供桌面输入控制 | 返回明确的不支持信息;可切换到 X11 会话 |

本次内置运行时针对 Windows、macOS 和 Linux 桌面发行版。Android / iOS 不打包桌面运行时。平台条件由原生工具报告,工具失败会返回错误,不会退化成执行任意 shell 指令。

## 构建和分发

在 `openless-all/app` 中执行:

```sh
npm ci
node scripts/prepare-pi-backend.mjs
npm run build
```

准备脚本下载固定版本 Node 并验证固定 SHA-256,使用 lockfile 安装 PI 生产依赖,编译原生 Computer 工具,再进行无桌面操作的健康检查。只有检查通过才写入 `src-tauri/resources/pi-backend`。后续构建按源码指纹复用缓存,首次构建需要网络。

macOS / Windows 的 Tauri 开发和发布构建会自动调用准备脚本。安装资源包含:

```text
pi-backend/
node / node.exe
openless-computer / openless-computer.exe
NODE-LICENSE
manifest.json
runtime/
index.mjs
package.json
node_modules/
src/
```

Windows 资源位于应用可执行文件旁;macOS 位于 `Contents/Resources/pi-backend`;Linux 安装到 `usr/lib/openless/resources/pi-backend`。Linux 独立打包脚本和发布 workflow 会包含同一套资源及必要动态库。请在目标系统和 CPU 架构的构建机运行,不能把 Windows 的 npm 原生依赖复制进 macOS 或 Linux 安装包。

Windows 构建需要 MSVC C++ Build Tools 与 Windows SDK。Linux 原生工具需要 X11 / PipeWire / Wayland / GBM 开发库;完整依赖在 Linux 发布 workflow 中维护。macOS 构建会为 Node 添加 JIT 所需 entitlement,并对嵌套可执行文件签名。

Windows 一键构建完整安装包(同时编译输入法组件):

```powershell
./scripts/build-windows-pi.ps1
```

三平台后端测试和独立后端构建由 `.github/workflows/pi-computer.yml` 提供;工作流只生成构建产物,不发布版本。

## 开发验证

```sh
cargo test -p openless-core --lib
cargo test -p openless-computer
node --test scripts/prepare-pi-backend.test.mjs
npm --prefix pi-backend test
npm run build
```

`openless-computer --capabilities`、`runtime/index.mjs --health` 不截图、不点击、不输入,可用于安装包健康检查。后端请求通过 stdin JSON 传递,用户任务和凭据不进入命令行参数。原生工具的具体动作协议见其 [README](../openless-all/app/crates/openless-computer/README.md)。
Loading
Loading