Repository navigation
feat(stream): a leased simulator's or emulator's screen, streamed to whoever is looking at it - #576
Merged
Merged
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Contributor
Author
HandoffStopping here at the operator's request. This PR is untouched by what follows — the work below Done
Both commits are through the real commit hook (format, lint, comment-lint, typecheck). I used Also verified live, and this caught a real bug: the iOS stream still works end-to-end over HTTP Not done
Findings
Blocked onNothing for the vendoring work — it is ready to review. Android is blocked on a decision, not a person: is a ~21 fps software-encoded Android stream |
… a pipe Co-Authored-By: Apex <noreply@callstack.com>
Proof of feasibility plus measured performance, not a finished feature.
A leased iOS simulator's screen is captured by a Swift CLI, encoded H.264, and
handed to a browser as the body of one GET /v1/stream/leases/{leaseId}. The
request is the only on-command and closing it the only off-command, so nothing
runs until someone asks to look and nothing runs when they stop; a lease that
ends by any route stops the capture too, including under a connected viewer.
No contract change and no protocol bump: the stream rides the relay an
ADR 0026/0028 exchange already uses, as the second named relay target, and the
envelope crosses the link unreframed. A gateway forwards a stream without
parsing one byte of it.
Measured on an Apple silicon Mac, 330x720 at 1500 kbps:
- an untouched screen sits at the capture's 5 fps idle floor, 9-76 kbit/s
- real swipes reach 18.6-23.2 fps and 880-1097 kbit/s
- a relayed read costs nothing measurable over reading the tool directly
- the capture process itself is 66-78 MB
The capture program is serve-sim's pipeline, vendored and patched: its
FrameCapture finds a device in the device set named by SIMLOCK_IOS_DEVICE_SET,
where serve-sim searches only CoreSimulator's default set and simlock boots
every simulator in its own `simctl --set` root.
Known gaps, all deliberate at this stage: the tool is not shipped or installed,
simulators only (a physical device is refused STREAM_NOT_SUPPORTED), and one
unit test on the silence bound fails in roughly one run in five -- it drains the
body to observe a timeout whose step the stream's own prefetch decides.
…enance Everything third-party this repo carries lives under vendor/, one directory per upstream, each holding its artifacts, its license texts and a NOTICE.md. A CI check in source-hygiene enforces the shape; a rules clause states it. serve-sim (Apache-2.0, commit 6d00968): the capture tool moves from tools/ to vendor/screen-capture/ with license text, per-file origin table, the two deviations, and the one command that re-diffs against pristine upstream — the early warning when an Xcode major moves a private API. Three provably dead files (242 lines) are gone; H264Encoder, VideoEncoder, PixelBufferUtils and Xcode are rewritten as ours against public API only, live-verified: the avcC description payload is byte-identical before and after on a real capture. scrcpy (Apache-2.0, v5.0.1): the server artifact is vendored byte-identical at a pinned SHA-256, with the nested Kotlin-stdlib and JetBrains-annotation attributions. No code consumes it yet — its notice says so.
A ScreenCapture on the Driver interface opens a device's screen and hands back the capture envelope as bytes plus a bounded close; DriverCatalog routes it by platform and refuses a platform whose driver offers none, so the stream module never names a platform. The envelope's tag table and encoder live in core, once, for every producer and reader. Two ports for what a capture needs that ProcessRunner and TcpProbe cannot give: a child with binary stdout and stdin as its liveness handle (BytePipeLauncher), and a long-lived loopback connection (TcpStreamConnector).
This was referenced Oct 15, 2026
…ptures its own screen The worker link opens a lease's screen through DriverCatalog.openScreen and the driver's ScreenCapture, so the `platform !== "ios"` gate, the device-blind merge of every driver's lease environment and the iOS tool's flags all leave the module and main.ts. The iOS driver now runs the capture program itself (stdin as liveness, 2 s grace, then SIGKILL); `stream.command` is gone and `drivers.ios.screenCaptureCommand` overrides the program pnpm build now puts in dist/bin. Fixes from the review of #576: - A capture is registered before it starts and its lease is re-read after its first frame, so a lease that ends while a capture starts can no longer leave the next holder's screen streaming to the old viewer. The lease-end participant waits for each capture's close. - The first-frame bound and the between-frames bound are separate and run on the Clock port, which removes the flaky silence test's race. - A gateway reads a worker's refusal to 16 KiB, forwards its code only when it is shaped like ours, and refuses a codec or size it cannot describe instead of calling it h264. - main.ts builds the module from two factories; the gateway path has tests. The module has its manual, and dependency-cruiser keeps it off fs, child process, the HTTP front, the gateway and the drivers.
…e capture tool carry The scrcpy server bundles Kotlin stdlib 2.2.10, whose kotlin.time is derived from ThreeTen-BP (BSD-3-Clause, notice required in binaries) and whose MathJVM.kt is derived from Boost (BSL-1.0, object code exempt). Ship the ThreeTen-BP text, correct the attribution table, record how to check upstream's signed SHA256SUMS.txt, and name the Android driver as the artifact's consumer, with its client ours in src/drivers/android. The four Swift files rewritten from serve-sim stay serve-sim's files, modified: each gets the MODIFIED FILE header and the notice says Apache-2.0. Xcode.swift is no longer listed as both unchanged and rewritten. The hygiene check recognises a BSD-3-Clause text by its third clause, since the verbatim text never names the license.
…deadline GatewayRelay applied one 330 s deadline to the whole exchange, response body included, so every screen stream relayed through a gateway was cut with WORKER_UNREACHABLE at about 330 s, though a stream does not end on its own. The deadline now bounds only the channel wait and the wait for the response head. Once the head is in, the body is bounded by inactivity: 330 s with no response bytes (DEFAULT_IDLE_MS, GatewayRelayOptions.idleMs) ends the exchange as a dead worker does — WORKER_UNREACHABLE, relay.cancel, channel aborted. The same rule applies to every relay target; the gateway still names none. Caller cancel/abort and a worker disconnect end the exchange as before. agent-device is unchanged: its proxy (node_modules/@agent-device/proxy, upstreamSignal) aborts every exchange, body included, 300 s after it starts on the worker, and simlock sets no upstreamTimeoutMs, so no exchange the worker finishes can keep a body open, or silent, for 330 s after the gateway sees its head. Its /rpc body may sit silent between NDJSON progress lines (agent-device src/daemon/server/http-rpc-envelope.ts at the vendored commit), which is why the idle bound is not shorter than 300 s. The existing relay unit tests, the gateway relay e2e file (in the unit lane) and the agent-device gateway-front tests pass unchanged except the two that asserted the old cut of a stalled body.
…e capture programs come from Android emulators now stream as iOS simulators do; physical phones are refused. stream.command is gone: drivers.ios.screenCaptureCommand overrides the program pnpm build puts in dist/bin. The browser lane's fake driver offers a screen that runs its stand-in program, named in the driver script. Both EVENTS.md files say why a stream emits no event.
… and reads the tags every capture writes A test pins the console's tag table to core's, the one the iOS program, the Android driver and the stream module all use.
The Android driver offers a ScreenCapture when the host passes the scrcpy server jar and a TcpStreamConnector. Each open pushes the jar under a fresh 31-bit scid, runs scrcpy-server 5.0.1 on the device, and reaches its abstract socket by speaking adb's host protocol to Simlock's own adb server, so no adb forward is left on the host. The video socket is parsed by a pure push parser (codec id, session and media packets); H.264 goes from Annex-B to AVCC with a 4-byte-length avcC record; every session packet becomes a fresh meta frame, so a rotation re-describes the stream. The body is pull-based: the socket is read only when the viewer pulls, and nothing is dropped. close is idempotent and bounded: socket, then the server's shell, then the jar. adb-registrar now shares the host-protocol framing.
…es Android's capture scrcpy's encoder repeats a still picture once and then sends nothing until the screen changes, which every silence bound downstream -- the stream module's 30 s, a gateway's 330 s -- would read as a dead capture. While its socket is open the Android session now sends an empty `alive` frame (tag 0x06, in core's table; the console skips it) after 5 s of quiet. main.ts hands the Android driver the scrcpy server pnpm build copies into dist/vendor/scrcpy and a TcpStreamConnector, so Android emulators stream.
…th stops A viewer leaving and a lease ending under a connected viewer each leave no simlock-screen-capture process running.
… closed, not left running Found live on an emulator: a viewer that gave up while the Android server was still starting released its slot before the capture attached, so nothing ever closed it -- the host adb shell and the device's scrcpy server kept running. A released slot now closes whatever attaches to it, and the link refuses a picture whose request is already aborted. The slow-lane stream tests print the lease's stdout when a lease fails.
…s, not the stream's lifetime Found live on an emulator: scrcpy sends only when the screen changes, so a lifetime average showed a smooth swipe as about 2 fps. Both figures now cover the last two seconds and refresh every second, so they fall to zero on a still screen.
Unused exports, re-exports and fake members go; the stream index exports only its two factories. The console's frame reader splits its envelope parsing and its per-tag handling out of readStream, the Screen pane's facts become their own component, and the Android session's connection attempt and the driver's capture wiring become their own methods. The scrcpy parser, the frame mapper and the body's pull keep one branch per wire case and say so.
…omplexity bound The reader hands frames to a table of handlers by tag and consumes each read in its own step; the pane's stream state is a hook and its picture a component.
Indexed reads of the canvas's pixel data are possibly undefined under noUncheckedIndexedAccess; the PoC was pushed without the hook that checks it.
V3RON
force-pushed
the
feature/stream-poc
branch
from
October 15, 2026 11:22
660e117 to
e52d7b6
Compare
V3RON
changed the title
V3RON
added a commit
that referenced
this pull request
Oct 15, 2026
The module is wired by the composition root, not ADR 0030's host. iOS's capture tool is built by pnpm build on the host; the scrcpy jar is copied after its digest check. Android sends an alive frame on a still screen, drops nothing and needs no reclaim sweep. The tag table gains seed and alive; the gateway's header bounds, 502 and relay-failure answers match checkedMeta and refusal.ts. No dispose hook; the log records opened and refused.
…stream, with 503 and why The relay fails with the fleet's codes; the stream's gateway link let them fall through to a 500 INTERNAL. WORKER_UNREACHABLE now reaches the viewer as 503 with its own code, and RELAY_TARGET_MISSING as 503 STREAM_UNAVAILABLE naming the worker.
…ft alone, and what the spec check found - A driver's open takes an AbortSignal; the worker link aborts it when the viewer leaves or the lease ends. Android starts no server once stopped after its push and cleans up; the lease-end participant waits for an open in progress to settle before reclaim, bounded by leasing's signal. - Android says a picture is alive only once a first picture exists, so an encoder that never paints meets the 30 s silence bound. - A contract shape changed (configSchema.stream): protocol 36 -> 37. - stream.bitrateKbps starts at 100, the iOS encoder's own floor. - The stream capability carries no reason when available; the wire drops it. - Tests: two viewers of one lease, a lease ending mid-open, build-capture's pin check; the every-offset parser test runs on small payloads so it proves the framing in milliseconds; the Android slow-lane test holds a still screen past 30 s and releases the lease under a viewer.
… step Keeps start under Fallow's complexity bound after the stop checks.
V3RON
added a commit
that referenced
this pull request
Oct 15, 2026
…, protocol bump A driver's open takes a stop signal and the lease end waits for an open in progress. Android's alive frames start after the first key frame. The gateway maps relay failures to 503 and defines its 502. bitrateKbps starts at 100; available carries no reason; the config contract bumps the protocol. Captures per lease are unbounded in v1.
…ing the screen server
…am's off switch and still screen are pinned - GatewayRelay takes a Clock (the daemon's) for the body's idle timer, as architecture rule 9 asks of a new timer. - The console lane checks the worker reports the stream capability when streaming is on and none when it is off. - The Android slow-lane test holds a stream 45 s and requires an alive frame, so it proves a still screen was kept open, not only that one kept painting.
…pen settles; a stream through a real gateway is tested - A released slot stays registered until its open settles and what it attached is closed, so a lease that ends after the viewer left still waits for the device to be left alone; a failed open abandons its slot. - The relay's deadline timer also runs on the Clock port. - src/daemon/stream-gateway.e2e.test.ts runs a gateway and a worker in process: a stream relayed through the gateway, the lease released on the gateway stopping the capture on the worker, and a worker that does not stream answering 503 STREAM_UNAVAILABLE naming it. - The iOS slow-lane test holds a still simulator 35 s past nothing touching it.
…l fits its budget src/drivers/android/screen/AGENTS.md holds the scrcpy client's path, timers, rules and gaps, so the Android driver's manual is back under 150 lines; the gateway manual folds its relay timers into one row. New known gaps: an iOS stream can freeze after a reboot under its lease, and a daemon killed outright can leave a screen server on a still emulator until the screen changes or the device is reclaimed.
…n where they end - stream-gateway.e2e.test.ts: a viewer that leaves the gateway stops the capture on the worker. - The Android slow-lane test checks the emulator holds no scrcpy server and no jar after a viewer leaves a still screen, then releases a lease under a fresh viewer; the iOS one checks the stream is still open after 35 s. - The screen manual drops a gap the live check contradicts: the device server ended when its host adb shell died with the daemon.
…gs, not by a partial strip
… review round 1's tests and claims made true
…tests and claims made true
V3RON
marked this pull request as ready for review
October 15, 2026 13:24
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #593
A leased iOS simulator's or Android emulator's screen, live in the console and at
GET /v1/stream/leases/{leaseId}, on a worker and through a gateway. Design: ADR 0031 (#594). One PR, one review.src/stream/): no platform names; opens a screen through the driver'sscreenCapture, registers before it starts, re-reads the lease after the first frame, stops every capture when its lease ends.pnpm buildbuilds it on macOS.adb forward; analiveframe keeps a still screen open.capabilities.stream; the worker context gains a registry read and the drivers' screens. No protocol change.Status
Implement: done Review: 2 rounds done; the move onto ADR 0030 is not reviewed yet Hardware: live check passed on both platforms; slow lane not green yet Gate: not run
Next: review round 3 on the move to the module contract, slow lane, the gateway check on real devices, then merge after #594
Done when
src/stream/gateway-link.test.ts,src/stream/create.test.ts; not run live)src/stream/*.test.ts)pgrep -xandadb shell)worker-link.test.ts,captures.test.ts,session.test.ts; slow lane pending)session.test.ts; slow lane holds 45 s and requiresalive, pending)capabilities.stream.enabled: false: no route, capabilityoff; on:available(e2e/console/screen.spec.ts, 5/5 Chromium)pnpm buildchecks the scrcpy SHA-256 (a mismatch fails it) and builds the iOS program on macOS (src/build-capture.test.ts)NOTICE.md(src/source-hygiene.test.ts)scripts/slow-e2e.sh e2e/slow-ios-stream.test.ts e2e/slow-android-stream.test.tspasses (first run failed at lease with the disk nearly full)Assumptions
STREAM_NOT_SUPPORTEDon both platforms in v1.Review
Spec review: 7 blocking, 3 fixed, 3 notes. Code review: 15 blocking, 15 fixed, 3 notes. Claims review: 8 blocking, 8 fixed, 9 notes.
Rounds: 2. Tokens: 0.11M.
Fixed in round 1 (
12943e48): the console gave up on a stream's headers after 10 s, before a worker's own refusal; it now waits 90 s (ADR 0031 records the exception to ADR 0013). The rest were tests that could not fail and wording. Round 2 (3d387864): tests for the 30 s defaults and wording; Stop now also cancels a starting stream.Rejected:
Written by an agent.