diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 7397def..4d8cc24 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -246,6 +246,13 @@ jobs: MCP_REF: ${{ inputs.mcp-ref }} run: npm run bundle:mcp + # The GemDB Stats web build, downloaded from the GemDB Stats release + # pinned in vendor-pins.sh and refused unless its SHA-256 matches. No + # Flutter SDK here: GemDB Stats builds it and publishes it. Platform- + # neutral like the MCP payload, and fetched per leg for the same reason. + - name: Fetch the GemDB Stats web build + run: npm run bundle:stats + # Not a shipped artifact: GemDB installs Grail into whatever database the # user already has, so nothing here ships an extent. This one exists so # the tests that need Python but are not testing the file-in do not pay diff --git a/.gitignore b/.gitignore index 75cbc9a..45dacef 100644 --- a/.gitignore +++ b/.gitignore @@ -19,6 +19,10 @@ grail/ # but it is still the mcp_server repository's code, not this one's. mcp/ +# The GemDB Stats payload is a build artifact of scripts/bundle-stats.mjs: a +# Flutter web build from the GemDB_Stats repository, made hostable in a webview. +stats/ + # The extent the integration suite starts from, built by # scripts/build-test-extent.sh. A TEST artifact: no extent is shipped, because # a user's database is theirs and Grail is installed into it. diff --git a/.prettierignore b/.prettierignore index 10d7f24..c57fc33 100644 --- a/.prettierignore +++ b/.prettierignore @@ -1,5 +1,6 @@ out grail +stats *.vsix # Vendored from Jasper byte-for-byte so upstream fixes are a plain copy. diff --git a/CHANGELOG.md b/CHANGELOG.md index 0bb16ff..94b38f0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,13 @@ this one does not look. ### Added +- **Statmon files open in GemDB Stats, inside the editor.** Click a `.out.gz` or a `statmon….out` + file, right-click any `.out` and choose **Open in GemDB Stats**, or run + **GemDB: Open Statistics File…**, and the file opens in a tab of GemTalk's statmon viewer: browse + processes, plot statistics, compare them on one chart. Clicking the file name opens another file + in its place, through VS Code's own dialog, so under code-server it browses the server where the + files are. Large files load in the background with a progress bar, and the tabs come back when + the window reloads. It needs no database, so it works before GemDB is set up. - **The database reserves the free license's full 10 GB, with room to collect its garbage.** GemDB Code caps the database at the license's 10 GB (until now it stopped at 8 GB), reserves all of it on disk when the database first starts, and keeps 500 MB of it free, so the database protects diff --git a/CLAUDE.md b/CLAUDE.md index 2dcbfae..2e6fb3a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -19,6 +19,7 @@ npm run test:integration # a real database in a temp root path; seconds npm run bundle # esbuild -> out/extension.js npm run bundle:grail # assemble the Grail payload (needs a C toolchain) npm run bundle:mcp # assemble the MCP server payload (needs nothing) +npm run bundle:stats # fetch the pinned GemDB Stats web build (or STATS_WEB=) npm run test:extent # build the extent the integration suite starts from npm run package # .vsix npm run dev:fresh # this checkout in a throwaway profile + empty root path (also :cached-engine, :keep-root) diff --git a/docs/README.md b/docs/README.md index 7540fb5..4fa52af 100644 --- a/docs/README.md +++ b/docs/README.md @@ -11,5 +11,6 @@ changing it. | [`grail.md`](grail.md) | how the Grail payload is built and staged, the Python↔GCI bridge (exit codes, `input()`, `print()`, encoding), and the gotchas found building on it | touching the bridge or building an application on Grail | | [`mcp-server.md`](mcp-server.md) | the bundled MCP server: payload, session cost and leak, read-only mode, stop order, client registration | changing how the MCP server is built, started, stopped or exposed | | [`repository-space.md`](repository-space.md) | the 10 GB cap, reserved on disk, and the free-space threshold; what the stone does below it (stops reclaiming garbage, refuses and ends `gemdb` sessions); the `gemdb` account and what needs DataCurator; why idle sessions are aborted; what a garbage collection has to wait for | changing the stone's space settings, garbage collection, database accounts, or anything that holds a session open | +| [`statistics.md`](statistics.md) | GemDB Stats in a webview: why a webview, the message protocol with GemDB Stats, where the web build comes from and what `bundle-stats.mjs` checks and changes, and what is still unmeasured | changing how statmon files are opened or how the GemDB Stats build is bundled | | [`reaching-windows.md`](reaching-windows.md) | the options for a Windows story (WSL, remote server, Docker) and what still needs a Windows machine | starting any Windows work | | [`demo/`](demo/) | one self-contained directory per demo, with measured findings | building something a demo already explored, or adding a demo | diff --git a/docs/statistics.md b/docs/statistics.md new file mode 100644 index 0000000..2ff205c --- /dev/null +++ b/docs/statistics.md @@ -0,0 +1,116 @@ +# GemDB Stats + +GemDB Code shows [GemDB Stats](https://github.com/GemTalk/GemDB_Stats), +GemTalk's statmon viewer, in an editor tab. A Flutter web build of it ships in +the `.vsix` as `stats/`. `src/statistics.ts` hosts it, and +`scripts/bundle-stats.mjs` prepares it. GemDB Stats' side of the arrangement is +its `docs/embedding.md`. + +## Why a webview + +The editor is not always on the machine that holds the statmon files. Under +code-server the only screen is a browser tab, so a desktop app cannot appear +there. A webview is an iframe in that tab whose files the remote machine +serves, so one build works in desktop VS Code and in hosted VS Code. It needs +no port, no server process and no login of its own. + +Viewing statistics needs no database. The commands are registered before the +platform gate, so they work on machines that cannot host a database. + +## Opening a file + +GemDB Stats is a read-only custom editor, `gemdb.statistics`, and the default +editor for `*.out.gz` and `statmon*.out`, so a click on either opens it. A +`.out.gz` is unreadable in the text editor anyway, and statmon names its files +`statmon.out` or `statmonitor__.out`. Any other `.out` (such +as `a.out`, a compiled program) still opens as text. It was `option` for every +`.out` at first, and a click on a `.out.gz` then showed the text editor's +"binary or unsupported encoding" page. + +- **GemDB: Open Statistics File…** shows VS Code's open dialog, then opens a + tab on the file. +- **Open in GemDB Stats** opens a `.out` or `.out.gz` from the explorer's + right-click menu: every selected file, or, run without one (from a + keybinding), a file picked in the dialog. +- **Open With… → GemDB Stats** is offered for the same files. +- Each file gets one tab (`supportsMultipleEditorsPerDocument: false`), so a + file already open is brought forward, not parsed a second time. Tabs open + out of preview mode, or opening several from the explorer would leave only + the last. +- `retainContextWhenHidden` keeps a hidden tab's page alive. Bringing a + discarded one back would mean starting Flutter and parsing the file again. +- VS Code brings the tabs back after a window reload by resolving the editor + again: the page is rebuilt, says `ready`, and is sent its file. + +## The protocol + +Three messages, as GemDB Stats' `docs/embedding.md` specifies: + +| Message | Direction | GemDB Code's part | +| --- | --- | --- | +| `{type: 'ready'}` | app → GemDB Code | Answer with `open` for the tab's file, every time: a reloaded page says `ready` again. | +| `{type: 'open', url, name}` | GemDB Code → app | `url` is `asWebviewUri(file)`. Sent only in answer to `ready`. | +| `{type: 'pickFile'}` | app → GemDB Code | The user clicked the file name. Show VS Code's open dialog, then show the picked file in a new tab in the same place, and close this one. | + +`pickFile` uses VS Code's dialog rather than the browser's because under +code-server the browser's dialog browses the user's own machine, not the +server where the statmon files are. + +Each tab shows one file for as long as it is open, and its page can fetch +only under the panel's `localResourceRoots`: the build and that file's +folder. The folder is as narrow as it can be: VS Code refuses a request for a +resource that is itself a root (read in VS Code 1.140's loader). A picked file +gets a new tab rather than widening this one's roots, because widening them +reloads the page, leaves it able to read every folder it was ever shown, and +an `open` posted across the reload could make the app parse a large file +twice. A picked file already open in another tab brings that tab forward. + +## The build + +A GemDB Stats release publishes `GemDB-Stats--web.tar.gz` beside its +desktop builds, from v1.1.0 on. `vendor-pins.sh` pins its URL and SHA-256, and +`npm run bundle:stats` downloads it, checks it and assembles `stats/`. GemDB +Code's CI does not carry the Flutter SDK, which is why it uses a release asset +rather than building a pinned commit, as Grail and the MCP server do. For a +local build, set `STATS_WEB` to `GemDB_Stats/app/build/web`. + +`bundle-stats.mjs` refuses a build a webview cannot run: + +- one that loads CanvasKit from Google's CDN, which the CSP refuses: build with + `--no-web-resources-cdn`; +- one that offers a Wasm build, whose renderers the payload leaves out: build + without `--wasm`; +- one whose bootstrap registers a service worker, which a webview cannot host. + GemDB Stats' own bootstrap template registers none, from PR #10 on. + +Its changes to the page are few: a CSP first in ``, `` +pointing at the webview, and a nonce on every script. They are made at bundle +time, into a `host.html` template, so a page GemDB Code cannot host fails in +front of whoever is packaging it. `fillHostPage` in `statistics.ts` fills the +three `{{GEMDB_*}}` placeholders for each panel. + +It also leaves out what a webview never loads: the Wasm renderers, debug +symbols, the service worker, and the PWA manifest and icons. That cuts the +47 MB build to about 23 MB. `check-vsix.sh` asserts the payload is present and +those files are not. + +## Measured + +- **2026-10-04, headless Chrome 154.** The generated page, with the full CSP. + Flutter drew its first frame, there were no CSP violations and no service + worker, and the page could fetch a statmon file. +- **2026-10-05, VS Code.** The app showed its loading screen, then grey. + Chrome reproduced it once the page and the build were on different origins, + as they are in a webview: Flutter's first `history.replaceState` resolved its + URL against `` and threw a SecurityError. The first test used one + origin and could not see it. GemDB Stats fixed it at the source in PR #10: + the app sets no URL strategy, so Flutter never touches the history. + +## Open + +- **Not yet seen in code-server.** +- **Progress on a large file in VS Code.** A 258 MB `.out.gz` showed progress + and loaded in under 40 s in Chrome. VS Code 1.140's webview resource loader + streams files in chunks rather than reading them whole first (read in its + source, 2026-10-06), so the progress bar should move there too. Not yet + watched in VS Code. diff --git a/eslint.config.mjs b/eslint.config.mjs index 3521349..e00f4a0 100644 --- a/eslint.config.mjs +++ b/eslint.config.mjs @@ -10,10 +10,19 @@ export default tseslint.config( // it. The rest are build artifacts that ESLint would otherwise walk on // its own, since (unlike Prettier 3) it does not read .gitignore: out/** // is this project's own bundle, grail/** and mcp/** are third-party - // payloads staged by bundle-grail.sh and bundle-mcp.sh, dist/** is + // payloads staged by bundle-grail.sh and bundle-mcp.sh, stats/** is the + // GemDB Stats web build staged by bundle-stats.mjs, dist/** is // packages fetched from CI by fetch-vsix.sh, and .test-extent/** is the // database extent build-test-extent.sh builds for the integration suite. - ignores: ['out/**', 'grail/**', 'mcp/**', 'dist/**', '.test-extent/**', 'src/gci/**'], + ignores: [ + 'out/**', + 'grail/**', + 'mcp/**', + 'stats/**', + 'dist/**', + '.test-extent/**', + 'src/gci/**', + ], }, js.configs.recommended, ...tseslint.configs.recommended, diff --git a/package.json b/package.json index 884a49f..a2d2e6a 100644 --- a/package.json +++ b/package.json @@ -216,6 +216,21 @@ "when": "gemdb.running && gemdb.hasSavedStacks" } ], + "customEditors": [ + { + "viewType": "gemdb.statistics", + "displayName": "GemDB Stats", + "selector": [ + { + "filenamePattern": "*.out.gz" + }, + { + "filenamePattern": "statmon*.out" + } + ], + "priority": "default" + } + ], "commands": [ { "command": "gemdb.install", @@ -323,6 +338,18 @@ "category": "GemDB", "icon": "$(output)" }, + { + "command": "gemdb.openStatistics", + "title": "Open Statistics File…", + "category": "GemDB", + "icon": "$(graph-line)" + }, + { + "command": "gemdb.openInStats", + "title": "Open in GemDB Stats", + "category": "GemDB", + "icon": "$(graph-line)" + }, { "command": "gemdb.copyTelemetryId", "title": "Copy Telemetry ID", @@ -571,6 +598,10 @@ { "command": "gemdb.restoreStack", "when": "gemdb.running && gemdb.hasSavedStacks" + }, + { + "command": "gemdb.openInStats", + "when": "false" } ], "debug/variables/context": [ @@ -623,6 +654,13 @@ "when": "debugType == gemdb && !gemdb.savedStackOpen", "group": "z_gemdb@1" } + ], + "explorer/context": [ + { + "command": "gemdb.openInStats", + "when": "resourceFilename =~ /\\.out(\\.gz)?$/", + "group": "navigation@9" + } ] }, "walkthroughs": [ @@ -719,6 +757,7 @@ "test:integration": "vitest run --config vitest.integration.config.mts", "bundle:grail": "./scripts/bundle-grail.sh", "bundle:mcp": "./scripts/bundle-mcp.sh", + "bundle:stats": "node scripts/bundle-stats.mjs", "test:extent": "./scripts/build-test-extent.sh", "//dev:fresh": "Launch this working copy in a separate editor on a throwaway profile (empty global storage) and an empty gemdb.rootPath: a brand-new user's first run, engine download included. Machine-wide shared memory is not reset; see the script's header.", "dev:fresh": "bash scripts/dev-fresh.sh", diff --git a/scripts/bundle-stats.mjs b/scripts/bundle-stats.mjs new file mode 100644 index 0000000..96614dd --- /dev/null +++ b/scripts/bundle-stats.mjs @@ -0,0 +1,337 @@ +#!/usr/bin/env node +// +// Assemble the GemDB Stats payload that ships inside the .vsix. +// +// GemDB Stats is GemTalk's statmon viewer, a Flutter app with a web build. +// GemDB Code shows that build in a webview (src/statistics.ts), so a statmon +// file can be charted wherever the editor is -- including code-server, where +// the only screen is a browser tab and a desktop app cannot appear. +// +// Like Grail and the MCP server, the payload is another repository's code, +// pinned in vendor-pins.sh. Unlike them it is not built here: compiling it +// needs the Flutter SDK, so GemDB Stats publishes the web build as a release +// asset, and this script downloads the pinned one and checks its SHA-256. +// +// Usage: +// npm run bundle:stats # the pinned release asset +// STATS_WEB=/path/to/GemDB_Stats/app/build/web npm run bundle:stats # a local build +// +// Environment: +// STATS_WEB a local `flutter build web` output to bundle instead of downloading +// STATS_URL a web-build .tar.gz to download instead of the pinned one +// STATS_SHA256 its expected SHA-256; unchecked when STATS_URL is given without it +// STATS_OUT where to assemble the payload (default: stats/, which is what +// the .vsix ships; the unit tests point it elsewhere) +// +// What it checks and changes, and why HERE rather than when the panel opens: +// every one of these can fail, and a failure here is in front of whoever is +// packaging, while the same failure at run time is a blank panel on a user's +// screen. GemDB Stats' docs/embedding.md is the other side of this. +// +// - The build. It must load CanvasKit from its own files (the page's CSP +// refuses Google's CDN), be dart2js only (the Wasm renderers are left out), +// and start Flutter without a service worker (a webview runs its own). +// - The page. A webview needs a Content-Security-Policy, a that +// points at the webview's resource URL, and a nonce on every script. The +// build's index.html becomes `host.html`, a template whose three +// `{{GEMDB_*}}` placeholders statistics.ts fills for each panel; everything +// else on it, the loading screen included, is GemDB Stats' to design. +// - The files. The Wasm renderers, debug symbols, the service worker and the +// PWA manifest never load in a webview: more than half of the build's 47 MB. + +import { execFileSync } from 'node:child_process'; +import crypto from 'node:crypto'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; + +const REPO = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..'); + +/** What the hosted page loads. A build missing any of these is not one this can host. */ +const REQUIRED = [ + 'index.html', + 'flutter.js', + 'flutter_bootstrap.js', + 'main.dart.js', + 'version.json', + 'assets/AssetManifest.bin.json', + 'assets/FontManifest.json', + // Flutter's loader picks the Chromium variant where the browser supports it + // (VS Code's own window, Chrome, Edge) and the full one elsewhere (Firefox + // or Safari pointed at code-server), so both stay. + 'canvaskit/canvaskit.js', + 'canvaskit/canvaskit.wasm', + 'canvaskit/chromium/canvaskit.js', + 'canvaskit/chromium/canvaskit.wasm', +]; + +/** + * Paths, relative to the build, that never ship. Everything else does: GemDB + * Stats decides what its build contains, and a list of what to KEEP would + * silently drop the next asset it adds. + */ +function isLeftOut(relative) { + const name = path.posix.basename(relative); + return ( + name.endsWith('.symbols') || + // The Wasm renderers. Only a `--wasm` build loads them, and checkBuild + // refuses one. + /^canvaskit\/(skwasm|wimp)/.test(relative) || + // Loaded only when the build config sets `preferWebParagraph`, which + // checkBuild refuses. + relative.startsWith('canvaskit/webparagraph/') || + // host.html takes its place. + relative === 'index.html' || + // Generated by every build, and registered by none that checkBuild accepts. + relative === 'flutter_service_worker.js' || + // A PWA's install metadata. A webview cannot install anything. + relative === 'manifest.json' || + relative.startsWith('icons/') || + relative === '.last_build_id' + ); +} + +/** + * Refuse a build a webview cannot run, judged from flutter_bootstrap.js: the + * `_flutter.buildConfig` that `flutter build web` writes into it, and the + * `_flutter.loader.load(...)` call after it. + */ +function checkBuild(bootstrap) { + const match = /^_flutter\.buildConfig = (\{.*\});$/m.exec(bootstrap); + if (!match) { + throw new Error('flutter_bootstrap.js has no `_flutter.buildConfig = {...};` line.'); + } + const config = JSON.parse(match[1]); + // Without this the loader fetches CanvasKit from www.gstatic.com, which the + // page's CSP refuses -- and which a server with no internet access could not + // reach anyway. + if (config.useLocalCanvasKit !== true) { + throw new Error( + 'The build loads CanvasKit from a CDN. Build it with `--no-web-resources-cdn`.', + ); + } + if (config.canvasKitVariant !== undefined || config.preferWebParagraph !== undefined) { + throw new Error( + 'The build chooses its own CanvasKit variant, which may need files this payload leaves out.', + ); + } + const builds = config.builds ?? []; + if (!builds.some((b) => b.compileTarget === 'dart2js' && b.mainJsPath === 'main.dart.js')) { + throw new Error('The build has no dart2js entry point at main.dart.js.'); + } + // The loader prefers Wasm where the browser supports it, and its renderers + // are left out of this payload, so a page that offered it would fail there. + if (builds.some((b) => b.compileTarget === 'dart2wasm')) { + throw new Error('The build includes a Wasm build. Build without `--wasm`.'); + } + // Only the code after the config: the flutter.js inlined above it mentions + // service workers in its own implementation. + const start = bootstrap.slice(match.index + match[0].length); + if (/serviceWorker/.test(start)) { + throw new Error( + 'flutter_bootstrap.js registers a service worker, which a webview cannot host. ' + + 'GemDB Stats from PR #10 on starts Flutter without one.', + ); + } + return config; +} + +/** + * The policy the page runs under. `{{GEMDB_CSP_SOURCE}}` becomes the + * webview's own resource origin, which serves both the build and the statmon + * file, so nothing else is reachable. + */ +const CSP = [ + "default-src 'none'", + // The bootstrap and what it loads come from the resource origin; the page's + // own inline script carries the nonce. CanvasKit is WebAssembly, which a CSP + // blocks unless 'wasm-unsafe-eval' allows it. + "script-src 'nonce-{{GEMDB_NONCE}}' {{GEMDB_CSP_SOURCE}} 'wasm-unsafe-eval'", + // Flutter writes its own + + +
Loading…
+ + + + +`; + +const BUILD_CONFIG = { + engineRevision: 'af7e796e', + builds: [{ compileTarget: 'dart2js', renderer: 'canvaskit', mainJsPath: 'main.dart.js' }, {}], + useLocalCanvasKit: true, +}; + +/** + * flutter_bootstrap.js: flutter.js inlined, then the config, then the load + * call — GemDB Stats' own template, which registers no service worker, unless + * `load` says otherwise. + */ +function bootstrap(config: unknown, load = '_flutter.loader.load();'): string { + return [ + // The inlined flutter.js mentions service workers in its own code. + 'async load({serviceWorkerSettings:e}={}){ /* flutter.js */ }', + `_flutter.buildConfig = ${JSON.stringify(config)};`, + '', + load, + ].join('\n'); +} + +let dir: string; +let web: string; +let out: string; + +function write(relative: string, content = 'x'): void { + const file = path.join(web, relative); + fs.mkdirSync(path.dirname(file), { recursive: true }); + fs.writeFileSync(file, content); +} + +function bundle(env: Record = { STATS_WEB: web }): { + status: number | null; + stderr: string; +} { + const result = spawnSync(process.execPath, [SCRIPT], { + env: { ...process.env, STATS_OUT: out, ...env }, + encoding: 'utf8', + }); + return { status: result.status, stderr: result.stderr }; +} + +/** The same, without blocking this process, which may be serving the download. */ +function bundleAsync( + env: Record, +): Promise<{ status: number | null; stderr: string }> { + return new Promise((resolve) => { + const child = spawn(process.execPath, [SCRIPT], { + env: { ...process.env, STATS_OUT: out, ...env }, + }); + let stderr = ''; + child.stderr.on('data', (chunk: Buffer) => (stderr += chunk.toString())); + child.on('close', (status) => resolve({ status, stderr })); + }); +} + +const shipped = (relative: string) => fs.existsSync(path.join(out, relative)); +const hostHtml = () => fs.readFileSync(path.join(out, 'host.html'), 'utf8'); + +beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'gemdb-stats-')); + web = path.join(dir, 'web'); + out = path.join(dir, 'out'); + write('index.html', INDEX_HTML); + write('flutter_bootstrap.js', bootstrap(BUILD_CONFIG)); + write('version.json', JSON.stringify({ app_name: 'vsd', version: '1.0.0' })); + for (const file of [ + 'flutter.js', + 'main.dart.js', + 'favicon.png', + 'logo.svg', + 'manifest.json', + 'flutter_service_worker.js', + '.last_build_id', + 'icons/Icon-192.png', + 'assets/AssetManifest.bin.json', + 'assets/FontManifest.json', + 'assets/fonts/MaterialIcons-Regular.otf', + 'canvaskit/canvaskit.js', + 'canvaskit/canvaskit.js.symbols', + 'canvaskit/canvaskit.wasm', + 'canvaskit/chromium/canvaskit.js', + 'canvaskit/chromium/canvaskit.wasm', + 'canvaskit/skwasm.wasm', + 'canvaskit/wimp.wasm', + 'canvaskit/webparagraph/canvaskit.wasm', + ]) { + write(file); + } +}); + +afterEach(() => { + fs.rmSync(dir, { recursive: true, force: true }); +}); + +describe('bundle-stats.mjs', () => { + it('ships what the dart2js build loads, and none of what it never does', () => { + expect(bundle().status).toBe(0); + + for (const file of [ + 'host.html', + 'flutter.js', + 'flutter_bootstrap.js', + 'main.dart.js', + 'logo.svg', + 'assets/fonts/MaterialIcons-Regular.otf', + 'canvaskit/canvaskit.wasm', + 'canvaskit/chromium/canvaskit.wasm', + ]) { + expect(shipped(file), file).toBe(true); + } + for (const file of [ + 'index.html', + 'flutter_service_worker.js', + 'manifest.json', + 'icons/Icon-192.png', + '.last_build_id', + 'canvaskit/canvaskit.js.symbols', + 'canvaskit/skwasm.wasm', + 'canvaskit/wimp.wasm', + 'canvaskit/webparagraph/canvaskit.wasm', + ]) { + expect(shipped(file), file).toBe(false); + } + }); + + it('puts the policy first in , before any script', () => { + bundle(); + const html = hostHtml(); + + const head = html.indexOf(''); + expect(html.indexOf('Content-Security-Policy')).toBeGreaterThan(head); + expect(html.indexOf('Content-Security-Policy')).toBeLessThan(html.indexOf(' at the webview, and drops the manifest the policy would refuse', () => { + bundle(); + const html = hostHtml(); + + expect(html).toContain(''); + expect(html).not.toContain(''); + expect(html).not.toContain('manifest.json'); + }); + + it('starts the app with its own bootstrap, under the nonce like every script', () => { + bundle(); + const scripts = hostHtml().match(/]*>/g) ?? []; + + // The loading screen's, and the bootstrap. + expect(scripts).toHaveLength(2); + expect(scripts.every((tag) => tag.includes('nonce="{{GEMDB_NONCE}}"'))).toBe(true); + expect(scripts[1]).toContain('src="flutter_bootstrap.js"'); + }); + + it('refuses a bootstrap that registers a service worker', () => { + // Flutter's stock template; GemDB Stats replaced it in PR #10. + write( + 'flutter_bootstrap.js', + bootstrap( + BUILD_CONFIG, + '_flutter.loader.load({ serviceWorkerSettings: { serviceWorkerVersion: "1" } });', + ), + ); + + const result = bundle(); + + expect(result.status).toBe(1); + expect(result.stderr).toContain('service worker'); + expect(shipped('host.html')).toBe(false); + }); + + it('refuses a build that loads CanvasKit from a CDN', () => { + write('flutter_bootstrap.js', bootstrap({ ...BUILD_CONFIG, useLocalCanvasKit: false })); + + const result = bundle(); + + expect(result.status).toBe(1); + expect(result.stderr).toContain('--no-web-resources-cdn'); + }); + + it('refuses a build that offers Wasm, whose renderers it leaves out', () => { + write( + 'flutter_bootstrap.js', + bootstrap({ + ...BUILD_CONFIG, + builds: [{ compileTarget: 'dart2wasm', renderer: 'skwasm' }, ...BUILD_CONFIG.builds], + }), + ); + + const result = bundle(); + + expect(result.status).toBe(1); + expect(result.stderr).toContain('--wasm'); + }); + + it('names what a build is missing', () => { + fs.rmSync(path.join(web, 'canvaskit/chromium/canvaskit.wasm')); + + const result = bundle(); + + expect(result.status).toBe(1); + expect(result.stderr).toContain('canvaskit/chromium/canvaskit.wasm'); + }); + + it('refuses a page it cannot find the bootstrap in', () => { + write('index.html', INDEX_HTML.replace(/'; + registerStatistics(file('/ext'), world); +}); + +describe('a GemDB Stats tab', () => { + it('tells its page its file each time the page says it is ready', async () => { + const page = showTab(file('/data/statmon.out.gz')); + + await page.say('ready'); + await page.say('ready'); + + expect(page.posted).toEqual([ + { type: 'open', url: 'https://resource.example/data/statmon.out.gz', name: 'statmon.out.gz' }, + { type: 'open', url: 'https://resource.example/data/statmon.out.gz', name: 'statmon.out.gz' }, + ]); + }); + + it('sends nothing before its page is ready', () => { + const page = showTab(file('/data/statmon.out')); + + expect(page.posted).toEqual([]); + }); + + it('lets its page read the build and the file’s folder, and run scripts', () => { + const page = showTab(file('/data/statmon.out')); + + expect(page.options?.enableScripts).toBe(true); + expect(page.options?.localResourceRoots?.map((root) => root.path)).toEqual([ + '/ext/stats', + '/data', + ]); + }); + + it('points its page at the bundled build', () => { + const page = showTab(file('/data/statmon.out')); + + expect(page.html).toContain(''); + expect(page.html).not.toContain('{{GEMDB_'); + }); + + it('says so when this build of GemDB Code has no GemDB Stats', () => { + template = undefined; + + const page = showTab(file('/data/statmon.out')); + + expect(page.html).toContain('does not include GemDB Stats'); + }); +}); + +describe('opening files', () => { + it('opens every file selected in the explorer, each in a tab of its own', async () => { + const selected = [file('/data/a.out'), file('/data/b.out.gz')]; + + await run('gemdb.openInStats', selected[0], selected); + + expect(opened.map((open) => open.file)).toEqual(['/data/a.out', '/data/b.out.gz']); + }); + + it('opens the right-clicked file when nothing else is selected', async () => { + await run('gemdb.openInStats', file('/data/a.out')); + + expect(opened.map((open) => open.file)).toEqual(['/data/a.out']); + }); + + it('asks for a file when given none, as from a keybinding', async () => { + picks = [file('/data/a.out')]; + + await run('gemdb.openInStats'); + + expect(opened.map((open) => open.file)).toEqual(['/data/a.out']); + }); + + it('opens nothing when the open dialog is cancelled', async () => { + picks = [undefined]; + + await run('gemdb.openStatistics'); + + expect(opened).toEqual([]); + }); +}); + +describe('a file picked from inside a tab', () => { + it('opens in its place: a tab where this one is, and this one closes', async () => { + const page = showTab(file('/data/a.out'), 3); + picks = [file('/elsewhere/b.out')]; + + await page.say('pickFile'); + + expect(opened).toEqual([{ file: '/elsewhere/b.out', column: 3 }]); + expect(closed).toEqual(['/data/a.out']); + }); + + it('opens the dialog beside the file the tab is showing', async () => { + const page = showTab(file('/data/a.out')); + + await page.say('pickFile'); + + expect(pickedNear).toEqual(['/data/a.out']); + }); + + it('brings forward another tab already showing it, and leaves this tab alone', async () => { + const page = showTab(file('/data/a.out')); + showTab(file('/data/b.out')); + picks = [file('/data/b.out')]; + + await page.say('pickFile'); + + expect(opened).toEqual([{ file: '/data/b.out', column: undefined }]); + expect(closed).toEqual([]); + }); + + it('changes nothing when it is the file the tab already shows', async () => { + const page = showTab(file('/data/a.out')); + picks = [file('/data/a.out')]; + + await page.say('pickFile'); + + expect(opened).toEqual([]); + expect(closed).toEqual([]); + }); + + it('changes nothing when the dialog is cancelled', async () => { + const page = showTab(file('/data/a.out')); + picks = [undefined]; + + await page.say('pickFile'); + + expect(opened).toEqual([]); + expect(closed).toEqual([]); + }); +}); diff --git a/src/extension.ts b/src/extension.ts index 1a75c0d..9848649 100644 --- a/src/extension.ts +++ b/src/extension.ts @@ -54,6 +54,7 @@ import { debugFile } from './debugFile'; import { openRepl, runFile } from './repl'; import { closeSessionFor, logoutAll, setInputHandler } from './session'; import { GemDbStatusBar } from './statusBar'; +import { registerStatistics } from './statistics'; import { StatusViewProvider } from './statusView'; import { SETUP_OUTCOME, @@ -127,6 +128,11 @@ export function activate(context: vscode.ExtensionContext): void { registerMcpProvider(extensionPath, () => ensureMcpRunning(extensionPath)), ); + // GemDB Stats charts a statmon file in an editor tab. Before the platform + // gate: it reads a file and needs no database, so it works on a machine + // that cannot host one. + context.subscriptions.push(registerStatistics(context.extensionUri)); + // A closed notebook gives its session back. Sessions are scarce — the // database allows ten at once, and its own gems spend some of that — so a // window that opened and closed notebooks all morning must not still be diff --git a/src/statistics.ts b/src/statistics.ts new file mode 100644 index 0000000..725a94f --- /dev/null +++ b/src/statistics.ts @@ -0,0 +1,264 @@ +import * as crypto from 'crypto'; +import * as fs from 'fs'; +import * as path from 'path'; +import * as vscode from 'vscode'; +import { errorMessage, log } from './log'; + +/** + * GemDB Stats in an editor tab: a statmon file charted by GemTalk's statmon + * viewer, whose Flutter web build ships in the .vsix as `stats/`. + * + * It is a webview rather than the desktop app because the editor is not always + * on the machine with the files. Under code-server the only screen is a + * browser tab, and a webview is an iframe in that tab whose files are served + * by the remote machine — so the same panel works wherever the editor does, + * with no port, no server process and no login of its own. The page fetches + * the statmon file from the webview's resource origin, the same way it + * fetches its own code. + * + * Viewing statistics needs no database, so none of this waits for one, and it + * works on platforms GemDB cannot host a database on. + * + * The page and GemDB Code talk in three messages, specified on GemDB Stats' + * side in its docs/embedding.md: the app says `ready`, GemDB Code answers + * `open` with the file's URL, and a click on the file name in the app asks for + * another with `pickFile`. That last one is answered with VS Code's own open + * dialog, because under code-server the browser's would browse the user's + * laptop rather than the server the statmon files are on. + * + * `scripts/bundle-stats.mjs` turns the build's index.html into `host.html`, a + * template carrying the page's CSP. What is left for here is what differs per + * panel: its three `{{GEMDB_*}}` placeholders. + */ + +/** The custom editor's view type, as package.json declares it. */ +export const STATISTICS_VIEW_TYPE = 'gemdb.statistics'; + +/** What GemDB Code sends the page: show this file, replacing any before it. */ +export interface OpenMessage { + type: 'open'; + url: string; + name: string; +} + +export interface HostPageValues { + /** The build's own resource URL, ending in `/`; relative URLs resolve against it. */ + base: string; + /** The webview's resource origin, which serves the build and the file. */ + cspSource: string; + /** Fresh for every page, so nothing injected into the page can guess it. */ + nonce: string; +} + +/** `host.html` with this panel's values in place of its placeholders. */ +export function fillHostPage(template: string, values: HostPageValues): string { + const fills: Record = { + '{{GEMDB_BASE}}': values.base, + '{{GEMDB_CSP_SOURCE}}': values.cspSource, + '{{GEMDB_NONCE}}': values.nonce, + }; + // One pass, so a value that happens to contain a placeholder is not filled + // in turn. + return template.replace( + /\{\{GEMDB_[A-Z_]+\}\}/g, + (placeholder) => fills[placeholder] ?? placeholder, + ); +} + +/** A page that says why there is no chart, in the editor's own colours. */ +export function unavailablePage(nonce: string, message: string): string { + const text = message.replace(/[&<>"]/g, (c) => `&#${c.charCodeAt(0)};`); + return ` + + + + + + +

${text}

+`; +} + +/** + * What GemDB Stats needs from VS Code, passed in so the unit tests can stand + * in for it. + */ +export interface StatisticsWorld { + /** The bundled build: `stats/` in the extension. */ + build(): vscode.Uri; + /** `host.html` from the build; throws when the build is not there. */ + hostTemplate(): string; + /** What a page showing `file` may fetch: the build, and that file's folder. */ + roots(file: vscode.Uri): vscode.Uri[]; + /** VS Code's open dialog, for a statmon file. */ + pickFile(near?: vscode.Uri): Promise; + /** Open `file` in GemDB Stats, or bring its tab forward if it has one. */ + openEditor(file: vscode.Uri, column?: vscode.ViewColumn): Promise; + /** Whether `file` has a GemDB Stats tab. */ + isOpen(file: vscode.Uri): boolean; + /** Close `file`'s GemDB Stats tabs. */ + closeEditor(file: vscode.Uri): Promise; +} + +const same = (a: vscode.Uri, b: vscode.Uri): boolean => a.toString() === b.toString(); + +function vscodeWorld(extensionUri: vscode.Uri): StatisticsWorld { + const build = () => vscode.Uri.joinPath(extensionUri, 'stats'); + const tabsOf = (file: vscode.Uri) => + vscode.window.tabGroups.all + .flatMap((group) => group.tabs) + .filter( + (tab) => + tab.input instanceof vscode.TabInputCustom && + tab.input.viewType === STATISTICS_VIEW_TYPE && + same(tab.input.uri, file), + ); + return { + build, + hostTemplate: () => fs.readFileSync(path.join(build().fsPath, 'host.html'), 'utf8'), + // The file's folder, not the file: VS Code refuses a request for a + // resource that is itself a root, so a root has to be a folder above it + // (read in VS Code 1.140's loader, 2026-10-06). The folder is as narrow as + // the page's access can be. + roots: (file) => [build(), vscode.Uri.joinPath(file, '..')], + pickFile: async (near) => + ( + await vscode.window.showOpenDialog({ + title: 'Open Statistics File', + openLabel: 'Open', + canSelectMany: false, + defaultUri: near, + filters: { 'statmon files': ['out', 'gz'], 'All files': ['*'] }, + }) + )?.[0], + // Not a preview: opening several files from the explorer would otherwise + // leave only the last, each replacing the one before it. + openEditor: async (file, column) => { + await vscode.commands.executeCommand('vscode.openWith', file, STATISTICS_VIEW_TYPE, { + viewColumn: column, + preview: false, + }); + }, + isOpen: (file) => tabsOf(file).length > 0, + closeEditor: async (file) => { + await vscode.window.tabGroups.close(tabsOf(file)); + }, + }; +} + +/** + * GemDB Stats as a read-only custom editor: one tab per file, showing that + * file for as long as it is open. VS Code keeps one tab per file and brings + * the tabs back after a window reload, which calls `resolveCustomEditor` + * again; the page is rebuilt, says `ready`, and is sent its file. + * + * A file picked from inside a tab opens in its own tab where this one is, + * and this one closes. Widening this page's resource roots to the new file's + * folder instead would reload the page anyway, would leave it able to read + * every folder it had ever been shown, and whether VS Code delivered a + * message posted across that reload would decide whether a large file was + * parsed once or twice. + */ +export class StatisticsEditorProvider implements vscode.CustomReadonlyEditorProvider { + constructor(private readonly world: StatisticsWorld) {} + + openCustomDocument(uri: vscode.Uri): vscode.CustomDocument { + return { uri, dispose: () => {} }; + } + + resolveCustomEditor(document: vscode.CustomDocument, panel: vscode.WebviewPanel): void { + const file = document.uri; + const webview = panel.webview; + webview.options = { enableScripts: true, localResourceRoots: this.world.roots(file) }; + webview.onDidReceiveMessage((message: { type?: unknown }) => { + if (message.type === 'ready') { + // Every time, not just the first: a page that reloads says `ready` + // again and has to be told its file again. + const open: OpenMessage = { + type: 'open', + url: webview.asWebviewUri(file).toString(), + name: path.posix.basename(file.path), + }; + void webview.postMessage(open); + } else if (message.type === 'pickFile') { + void this.picked(file, panel.viewColumn); + } + }); + webview.html = this.page(webview); + } + + /** The user picked another file from inside `file`'s tab. */ + private async picked(file: vscode.Uri, column: vscode.ViewColumn | undefined): Promise { + const picked = await this.world.pickFile(file); + if (!picked || same(picked, file)) return; + if (this.world.isOpen(picked)) { + // Open in another tab already: that one comes forward, and this one + // keeps its file. + await this.world.openEditor(picked); + return; + } + await this.world.openEditor(picked, column); + await this.world.closeEditor(file); + } + + private page(webview: vscode.Webview): string { + const nonce = crypto.randomBytes(16).toString('base64'); + let template: string; + try { + template = this.world.hostTemplate(); + } catch (error) { + log(`GemDB Stats is not in this build: ${errorMessage(error)}`); + return unavailablePage( + nonce, + 'This build of GemDB Code does not include GemDB Stats. ' + + 'A development build gets it from `npm run bundle:stats`.', + ); + } + return fillHostPage(template, { + base: `${webview.asWebviewUri(this.world.build()).toString()}/`, + cspSource: webview.cspSource, + nonce, + }); + } +} + +export function registerStatistics( + extensionUri: vscode.Uri, + world: StatisticsWorld = vscodeWorld(extensionUri), +): vscode.Disposable { + return vscode.Disposable.from( + vscode.window.registerCustomEditorProvider( + STATISTICS_VIEW_TYPE, + new StatisticsEditorProvider(world), + { + // A hidden webview is discarded by default, and bringing it back would + // start Flutter and parse the file again — tens of seconds for a large + // one. Kept, at the cost of its memory. + webviewOptions: { retainContextWhenHidden: true }, + // One tab per file: a second would parse it again and hold a second + // copy in memory. + supportsMultipleEditorsPerDocument: false, + }, + ), + // From the Command Palette: pick a file first. + vscode.commands.registerCommand('gemdb.openStatistics', async () => { + const file = await world.pickFile(); + if (file) await world.openEditor(file); + }), + // From the explorer: every selected file, or, with none (a keybinding), + // one the user picks. + vscode.commands.registerCommand( + 'gemdb.openInStats', + async (file?: vscode.Uri, selected?: vscode.Uri[]) => { + const files = selected?.length ? selected : file ? [file] : []; + if (files.length === 0) { + const picked = await world.pickFile(); + if (picked) files.push(picked); + } + for (const each of files) await world.openEditor(each); + }, + ), + ); +} diff --git a/vendor-pins.sh b/vendor-pins.sh index 4ec0ce8..c4ef9dd 100644 --- a/vendor-pins.sh +++ b/vendor-pins.sh @@ -1,8 +1,8 @@ #!/bin/sh # # Pinned upstream commits for the payloads GemDB bundles into the .vsix. -# Sourced by scripts/bundle-grail.sh and scripts/bundle-mcp.sh; not meant to -# be run on its own. +# Sourced by scripts/bundle-grail.sh and scripts/bundle-mcp.sh, and read as +# text by scripts/bundle-stats.mjs; not meant to be run on its own. # # Full 40-character shas, not tags for now: neither upstream publishes tags worth # tracking yet. Override per-build with GRAIL_REF/MCP_REF -- these are read only @@ -61,3 +61,16 @@ PINNED_GRAIL_REF=9f8b1161101608cb57d696887c672a3a5c5ce9f2 # with its own line (#46, #51) instead of one `line ?` per method, and # eval_python runs as `__main__` (#52). PINNED_MCP_REF=d836520816d47bc5b8bb43386293f1b9dffbefce + +# GemDB Stats: a release asset, not a commit. Its web build needs the Flutter +# SDK to compile, which GemDB Code's CI does not carry, so GemDB Stats' +# release.yml publishes it as GemDB-Stats--web.tar.gz beside the +# desktop builds, and bundle-stats.mjs downloads this one and refuses it unless +# the SHA-256 matches (the release's SHA256SUMS.txt carries the value). Override +# per-build with STATS_URL (and STATS_SHA256), or STATS_WEB for a local build. +# +# v1.1.0 (3cc86a7), released 2026-10-06: the first release with the web build. +# It includes GemDB_Stats PR #10, which lets GemDB Code choose the file +# (ready / open / pickFile) and starts Flutter without a service worker. +PINNED_STATS_URL=https://github.com/GemTalk/GemDB_Stats/releases/download/v1.1.0/GemDB-Stats-1.1.0-web.tar.gz +PINNED_STATS_SHA256=1b748c36c0db9ef12fce563feb64ab8254fa8c8bd780d1d45bb514752d16f538