ci(e2e): run core in the stack and load pinned drivers from the build farm

.github/workflows/e2e-advisory.yml

@@ -170,7 +170,8 @@ jobs:
             # Brings up an isolated PlaceOS stack (its own compose project, own
             # volumes, ports 9443/9080) and seeds it. Needs network access to
             # GitHub once, to clone PlaceOS/www-core into the www volume — that is
-            # where the platform /login page comes from.
+            # where the platform /login page comes from — and to the build farm,
+            # which core fetches the suite's driver binaries from.
             - name: Bring up the local PlaceOS stack
               run: e2e/stack/up.sh
 

e2e/README.md

@@ -52,10 +52,19 @@ e2e/stack/down.sh            # stop        (--volumes to wipe)
 ```
 
 It is trimmed to what the e2e path exercises: postgres, elasticsearch, redis,
-search-ingest, frontend-loader, auth, rest-api, staff-api, nginx, init. Dropped
-from PlaceOS/local: core, edge, triggers, dispatch, source, influx, chronograf,
+search-ingest, frontend-loader, auth, rest-api, staff-api, core, nginx, init.
+Dropped from PlaceOS/local: edge, triggers, dispatch, source, influx, chronograf,
 mosquitto, minio and the loki/grafana profile — roughly half the containers.
 
+core runs real drivers from PlaceOS/drivers, pinned to a commit in
+`e2e/support/drivers/drivers.env.ts`. It compiles nothing itself: binaries come
+from the PlaceOS build farm (build.placeos.run), which builds a commit it has not
+seen for the CPU architecture on first request and serves a download after that.
+`seed.ts` creates the driver rows and waits for core to hold the binaries, so a
+spec never waits on the farm. If the drivers do not load, the seed logs a warning
+and continues: only the specs that need a room status fail. To test a drivers branch against the suite, set
+`E2E_DRIVERS_URI`, `E2E_DRIVERS_BRANCH` and `E2E_DRIVERS_COMMIT` before `up.sh`.
+
 Two things a cold start taught us that a long-lived stack hides:
 
 - **`frontend-loader` is not optional**, even though the SPA is served by the dev
@@ -240,6 +249,7 @@ CI-specific choices worth knowing:
 | Reclaim step | A self-hosted machine is not a fresh VM; a previous aborted run can leave the stack up or port 4214 held. |
 | No `vm.max_map_count` bump | Needed on GitHub-hosted Linux, meaningless on macOS — the value lives inside Colima's VM, which already sets it to 1048576. Restore it if reverting to `ubuntu-latest`. |
 | Network access to GitHub | `up.sh` clones `PlaceOS/www-core` into the `www` volume once — that is where the platform `/login` page comes from. |
+| Network access to the build farm | core fetches driver binaries from build.placeos.run and the S3 bucket it hands out links to. A commit the farm has not built for the runner's architecture is compiled there on the first run that asks. |
 | Network access to Docker Hub, every run | `up.sh --pull-only` runs first as its own step, bounded to 20 minutes and advisory, so `latest` tracks the registry rather than the day the runner was set up. Each image pulls on its own; one that fails or stalls leaves the stack on the image already present, and "Record backend inputs" shows which. `SELF_HOSTED_RUNNER.md` lists the hosts. |
 
 ## Booking specs: what the backend actually does

e2e/stack/SELF_HOSTED_RUNNER.md

@@ -28,7 +28,8 @@ What it *does* need is outbound 443. Check that first:
 for host in github.com api.github.com codeload.github.com \
             objects.githubusercontent.com release-assets.githubusercontent.com \
             pipelines.actions.githubusercontent.com \
-            registry-1.docker.io auth.docker.io production.cloudflare.docker.com; do
+            registry-1.docker.io auth.docker.io production.cloudflare.docker.com \
+            build.placeos.run placeos-drivers.s3-ap-southeast-2.amazonaws.com; do
     printf '%-48s ' "$host"
     curl -sS -o /dev/null -m 10 -w '%{http_code}\n' "https://$host" 2>&1 | tail -1
 done
@@ -42,6 +43,10 @@ timeouts, not 200s.
 - **registry-1.docker.io / auth.docker.io / production.cloudflare.docker.com** —
   pulling the PlaceOS images. The one people forget; without it the stack cannot start.
   Every run pulls (`up.sh --pull-only`, its own bounded step), so this needs to stay open, not just work once.
+- **build.placeos.run / placeos-drivers.s3-ap-southeast-2.amazonaws.com** — the
+  build farm core fetches driver binaries from, and the bucket its download links
+  point at. The stack is torn down with its volumes after every run, so every run
+  downloads them again.
 
 > A slow link looks like a block. Colima's first VM image download failed here with a
 > timeout that read like a firewall; the host was fine, just slow. Retry before

e2e/stack/docker-compose.yml

@@ -7,11 +7,10 @@
 #     with a running one. Everything here is project-scoped (no container_name,
 #     no fixed network name), so it can run alongside a developer's own stack
 #     without touching it.
-#  2. It is what CI needs. This is a trimmed stack — only what the workplace e2e
-#     path actually exercises. Dropped from PlaceOS/local: core, edge, triggers,
-#     dispatch, source, frontend-loader, influx, chronograf, mosquitto, minio and
-#     the loki/promtail/grafana profile. That is roughly half the containers and
-#     most of the RAM.
+#  2. It is what CI needs. This is a trimmed stack — only what the e2e path
+#     actually exercises. Dropped from PlaceOS/local: edge, triggers, dispatch,
+#     source, influx, chronograf, mosquitto, minio and the loki/promtail/grafana
+#     profile. That is roughly half the containers and most of the RAM.
 #
 # frontend-loader IS required, despite the SPA being served by the Angular dev
 # server. It does not only deploy app builds: on startup it clones
@@ -20,6 +19,14 @@
 # 404 at /login and the real-login spec dead-ends. (Consequence for CI: bringing
 # this stack up needs network access to GitHub once, to populate the volume.)
 #
+# core runs the drivers. It compiles nothing itself: a driver binary comes from
+# the PlaceOS build farm (`BUILD_URL`, build.placeos.run unless overridden with
+# E2E_BUILD_URL), keyed by repository, branch, commit, file and CPU architecture,
+# and lands in the `core-drivers` volume. The farm builds a key it has not seen
+# on first request, which is why `e2e/support/drivers/drivers.env.ts` pins the
+# commit. Bring-up needs the farm reachable, and the S3 bucket the farm hands
+# out presigned links to (placeos-drivers.s3-ap-southeast-2.amazonaws.com).
+#
 # Bring it up with ./up.sh — it generates secrets and seeds before returning.
 
 volumes:
@@ -28,6 +35,7 @@ volumes:
   redis-data:
   nginx-data:
   www:
+  core-drivers:
 
 x-deployment-env: &deployment-env
   PLACE_LOG_FORMAT: JSON
@@ -246,6 +254,28 @@ services:
           *search-ingest-client-env,
         ]
 
+  core:
+    image: placeos/core:${PLACEOS_TAG:-latest}
+    hostname: core # load-bearing: rest-api reaches it by the name it registers in redis
+    restart: always
+    # Reaps driver processes, which are children of core.
+    init: true
+    <<: *std-logging
+    depends_on:
+      postgres: { condition: service_healthy }
+      redis: { condition: service_started }
+    volumes:
+      - core-drivers:/app/bin/drivers
+    env_file:
+      - .secrets/.env.secret_key
+    ulimits:
+      nofile: 40000
+    environment:
+      <<: [*deployment-env, *postgresdb-client-env, *redis-client-env]
+      BUILD_URL: ${E2E_BUILD_URL:-https://build.placeos.run}
+      # The User-Agent core sends the farm, so its logs can tell this stack apart.
+      CLUSTER_NAME: ${E2E_STACK_PROJECT:-placeos-e2e}
+
   staff-api:
     image: placeos/staff-api:${PLACEOS_TAG:-latest}
     hostname: staff # load-bearing: nginx upstream is `staff`

e2e/stack/up.sh

@@ -136,6 +136,15 @@ echo "  /login -> 200"
 step "seeding platform entities (init)"
 dc run --rm init start
 
+# After init: core subscribes to the driver and module tables as it starts, and
+# a core that comes up before the migrations have created them stops managing
+# drivers for good while still answering HTTP.
+step "starting core"
+dc up -d core
+
+# The last step of the seed waits for core to hold the suite's driver binaries.
+# A commit the build farm has already built for this CPU architecture is a
+# download; one it has not takes a few minutes.
 step "seeding e2e fixtures"
 cd ../..
 E2E_BACKEND_URL="https://localhost:${HTTPS_PORT}" bunx tsx e2e/support/seed.ts

e2e/support/api.ts

@@ -10,6 +10,30 @@ import { BACKEND_URL, RoleName, roleFor } from './env';
 export const ENGINE_API = '/api/engine/v2';
 export const STAFF_API = '/api/staff/v1';
 
+/** The rows of a listing, which is either a bare array or `{ results }`. */
+export function asList<T>(body: unknown): T[] {
+    if (Array.isArray(body)) return body;
+    return (body as { results?: T[] } | null)?.results ?? [];
+}
+
+/** Does this error body mean "the row is already there"? */
+export function alreadyExists(body: string): boolean {
+    return /already (exists|taken)|has already been taken|must be unique|should be unique|duplicate/i.test(
+        body,
+    );
+}
+
+/** GET a JSON body, or throw with the status and body. */
+export async function getJson(
+    api: APIRequestContext,
+    path: string,
+    params?: Record<string, string>,
+) {
+    const res = await api.get(path, { params });
+    if (!res.ok()) throw new Error(`GET ${path} failed: HTTP ${res.status()} ${await res.text()}`);
+    return res.json();
+}
+
 /** Read the raw bearer a worker's auth fixture wrote for a role. */
 export function readToken(role: RoleName, workerIndex = 0): string {
     const { tokenPath } = roleFor(role, workerIndex);