Repository navigation
feat(layers): composition layer support behind enableLayers() - #516
Merged
Merged
Conversation
Layers are optional in two independent places, and a session existing says nothing about either: the session may not have been granted the layers feature, and the binding may not offer the layer type wanted. The two backings are worth telling apart rather than reducing to a boolean. XRMediaBinding lets the compositor drive a video element with the app never drawing a frame for it, which is the cheap path, but Quest is the only browser that ships it. Chrome has quad layers only through XRWebGLBinding, where the app uploads each frame itself. Anything that wants to run in both places has to know which one it is on. preferWebGL exists because Quest has both and would otherwise always take the media path, leaving the WebGL path with no hardware to run on.
three.js sets layers to a single projection layer once when the session starts and never touches that array again. Its later updateRenderState calls only carry depthNear and depthFar, which merge, so adding to the array is safe, but the array has to be rebuilt in full every time and leaving the projection layer out of it blanks the scene. That is a sharp edge to hand to callers, so the manager owns it: nothing can be added before setBaseLayer has been told what three.js is rendering into, and every submission puts that layer first. Ordering follows the spec, earlier entries composite behind later ones, so app layers sit in front of the scene. The session, binding and context are held together because the WebGL path needs all three to upload a frame, and dropped together when the session ends so nothing outlives it.
Quad layers need the layers feature named when the session is requested. It cannot be turned on afterwards, so a session that did not ask for it can never have one, and createQuadLayer throws NotSupportedError rather than degrading. It goes in as an optional feature, not a required one, so asking for layers on a platform without them still gives a working session that falls back to drawing into the scene. Off by default, since it costs a feature request that most apps have no use for.
Video is the case where going through the eye buffer costs the most. A 1.2m quad two metres away covers roughly a third of the field of view, so on a Quest 3 panel it lands on about 620 pixels across. Hand it a 2048 wide source and around 70 percent of the detail is gone before the compositor ever sees the frame, and what survives gets resampled a second time when it warps to the head pose. As a composition layer it is sampled once, at its own resolution. This is the XRMediaBinding path, where the compositor drives the video element and the app never draws a frame for it. VideoLayer reports whether it got a layer rather than throwing when it cannot, so an app can ask for one everywhere and draw the video into the scene on platforms that have none. Each way it can fail falls back separately: no session, no base layer to compose against, no media binding, and a platform that advertises the binding but still refuses a particular element, which happens with one whose metadata has not loaded yet. Quad extents need halving here and it looks like a bug without the reason. The spec means width and height as full metres and Chromium passes them to OpenXR unchanged, but the Quest compositor treats them as half extents and returns a quad at twice the size. It is a property of the compositor rather than of this binding, so it is decided by the platform. See immersive-web/layers#324, still open. Height comes from the video's aspect ratio unless given, defaulting to 16:9 while metadata is still loading rather than dividing by a zero videoWidth.
Chrome does not implement XRMediaBinding and is not going to, so every Android XR device falls back to drawing the video into the scene, which is the case this whole thing exists to avoid. This adds the path Chrome does have: create the quad through XRWebGLBinding and upload each frame into its texture with texSubImage2D. The layer texture is allocated once, at the size given when it is created, and texSubImage2D refuses a source that does not fit it. Guessing a size and later uploading a frame of a different one raises a GL error on every frame, which on a Quest took the whole session down, so it waits for the video to report real dimensions instead. Binding a texture directly also has to be undone. three.js caches GL state and skips calls it believes are redundant, so leaving the layer texture bound left its cache describing something untrue and it went on to render with the wrong texture. The active unit, the 2D binding and the flip flag are all put back. Uploads happen only for frames the video has actually advanced past. The source runs at 30fps and the headset renders at 90, so two in three were pushing an identical 2048x1152 image, which is about 850MB/s of texture traffic for nothing. needsRedraw still forces one, since that means the compositor lost the layer contents. Layers are created with isStatic false, because getSubImage raises InvalidStateError on a static layer once needsRedraw has gone false and a video needs a new frame every time. Layout is named explicitly since 'default' is a TypeError for quad layers. getSubImage also throws until updateRenderState has taken effect, a frame or two after the layer is added, so a failed upload drops that frame rather than tearing the layer down.
The first attempt at this demo used a sample video off a CDN. Both copies looked identical, which was not the comparison working, it was the source being lower resolution than the eye buffer already: there was nothing left for the layer path to preserve. So the card is generated instead, at 2048x1152 onto a canvas and fed to both sides through captureStream. Text runs from 96px down to 12px and there are one, two and three pixel line pairs, which is where resampling shows first and where the eye buffer path visibly loses. Being generated also means no asset to host, and the resolution can be changed to find where the difference stops mattering.
Puts the same generated card up twice, once as a composition layer and once as an ordinary textured plane in the scene, so the only variable is the path the frame takes to the display. The two sides swap on a timer, which is load bearing rather than a gimmick. Lenses are sharpest in the middle and you read one panel at a time, so a fixed left and right layout cannot separate "the layer is sharper" from "that side happened to be in the sweet spot". Swapping does separate them: a difference that follows the path is real, one that stays on a side is the optics. Select swaps early and deliberately does not stop the timer. Squeeze is bound to select end, so a grip press while holding the controller ended the comparison for good, with nothing on screen to say why. The readout says which binding was asked for, which one is actually running and how many frames have been uploaded, because a line that only says "media" cannot distinguish a request that was ignored from one that was never made. It renders to the page and into the session both, since a DOM overlay disappears the moment you enter XR. ?webglLayer=1 forces the WebGL path on a platform that has both, which is what lets a Quest stand in for Android XR.
Wrapped returns and argument lists the formatter wanted broken differently. No behaviour change.
dli7319
self-requested a review
August 13, 2026 00:47
Collaborator
|
Both sides are looking the same to me on Galaxy XR. Hmm. |
Contributor
Author
ah, thanks for testing. it might be falling back to a normal scene texture on both sides, which would explain why they look the same. does the in-headset readout show |
Collaborator
|
I plan to submit this but I probably won't have time to figure out why it doesn't work on Galaxy XR. |
Contributor
Author
cool thanks! once Aura releases later this year I'll probably see if I can figure it out |
salmanmkc
marked this pull request as draft
September 15, 2026 22:58
salmanmkc
marked this pull request as ready for review
September 15, 2026 23:13
SpatialPanel was removed in e3a53dc on 2026-08-04, but the demo still constructed it. Initialization threw, leaving the script in failedScripts without its composition-layer callbacks. Use UICard and UIText for the readout and add only their required import-map mappings. Keep the existing manager, layer lifecycle and startup order.
Collaborator
|
I loaded it up on quest. The layers part is working but the page crashes after a minute or two. 
|
Collaborator
|
Longer trace: 
[Trace-20260919T214539.json.gz](https://github.com/user-attachments/files/32429417/Trace-20260919T214539.json.gz)
|
dli7319
reviewed
Sep 20, 2026
Collaborator
|
Also my agy says we shouldn't keep repeatedly calling |
Share one drawing loop across both video consumers and update quad transforms instead of recreating native layers. Bound ready-source attachment attempts, preserve placement across sessions, and release resources on teardown and rejected submissions. Add regressions for producer lifetime, stable layer identity, fallback, retries, session re-entry, and cleanup errors.
Use the XRMediaBinding and XRMediaQuadLayerInit declarations already provided by @types/webxr instead of maintaining a handwritten constructor shape. Keep runtime availability guards for browsers without media layers.
Load three.js 0.186.0 to match the SDK and avoid disposal failures from the older Object3D implementation. Check all three import-map entries against the SDK peer requirement.
salmanmkc
force-pushed
the
feat-layers-split
branch
from
September 20, 2026 10:40
d480bee to
084ee36
Compare
Contributor
Author
|
thanks for the Quest report and trace. fixed the duplicate drawing loop and changed the swaps to update the existing quad transform instead of calling 084ee36 also aligns the demo with three.js 0.186.0. the old pin caused a reproducible cleanup error in the browser. i have been testing the updated preview on my Quest 3 for a couple of minutes without a crash so far. encouraging, but not claiming a confirmed long-run fix or that both the media and forced-WebGL paths have been retested yet. - |
dli7319
reviewed
Sep 20, 2026
| "@pmndrs/msdfonts": "https://cdn.jsdelivr.net/npm/@pmndrs/msdfonts@1.0.64/dist/index.min.js", | ||
| "@preact/signals-core": "https://cdn.jsdelivr.net/npm/@preact/signals-core@1.14.0/dist/signals-core.mjs", | ||
| "yoga-layout/load": "https://cdn.jsdelivr.net/npm/yoga-layout@3.2.1/dist/src/load.js", | ||
| "troika-three-text": "https://cdn.jsdelivr.net/gh/protectwise/troika@028b81cf308f0f22e5aa8e78196be56ec1997af5/packages/troika-three-text/src/index.js", |
Collaborator
There was a problem hiding this comment.
Can you remove the troika imports including bidi-js and webgl-sdf-generator?
Collaborator
There was a problem hiding this comment.
I'll just push it myself since it's trivial
Collaborator
|
Just retested and it works fine now after 2 minutes. Thank you Still the same interesting memory graph on Quest tho. |
Contributor
Author
I'll try look into the graph a bit more, thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The device recording below was made on a Quest 3 against an earlier SDK build, with both the media and forced WebGL paths tested. The current demo has not been re-validated on a headset. Android XR hardware remains untested, including the dimensions and full behaviour of its WebGL path.
layer.demo.mp4
Anything drawn into the eye buffer is resampled at least twice before you see it: once into the buffer, which is already coarser than the panel across the area a given object covers, and again when the compositor warps it to head pose. A quad composition layer skips both and is sampled once, at its own resolution, straight by the compositor. That is a property of the render path rather than of any one headset.
The cost is easiest to see with numbers. A 1.2m quad two metres away covers about a third of the field of view, so on a Quest 3 it lands on roughly 620 eye buffer pixels across. Feed that a 2048 wide source and around 70% of the detail is gone before the compositor has even seen the frame. Fine text and single pixel lines are where it shows.
Adds
LayerManager,LayerCapabilityandVideoLayerundersrc/layers/, plus a demo indemos/layers/. All behindoptions.enableLayers(), off by default.After the spatial UI migration, the demo still called
SpatialPanel, which no longer exists. That throws during script init, so the script never activates or reaches layer attachment. Replaces only the readout withUICardandUIText, plus the importmap entries that UI needs. The demo still creates its own manager; layer creation, video uploads and startup order are unchanged.The same real Chromium check fails on
b0e15e19withSpatialPanel is not a constructorand the script infailedScripts. After the readout fix, the script is active with no init exception. The card is constructed and mounted in the scene graph with the expectedUITextvalues, both video meshes play, swaps update positions and text values, and the originalObject3D.layersmask stays intact. This browser takes the unsupported-layer fallback path, not native composition layers.Visible glyph rasterization could not be confirmed under headless SwiftShader. A separate control reproduces this pre-existing rasterizer limitation with an unmodified minimal public
UICard: the same heading rectangle measures 35 bright pixels under SwiftShader versus 854 under real Metal. Headed SwiftShader also gives 35. The regression evidence here confirms initialization, scene-graph mounting and text values, not that the text was visibly rasterized.Both backings are implemented, because neither one covers the devices we care about on its own.
XRMediaBindingis the cheap path, where the compositor drives the video element and the app never draws a frame, but Quest is the only browser that ships it. Chrome has quad layers only throughXRWebGLBinding, where the app uploads each frame itself, so that is the path Android XR needs.LayerManagerexists because three.js setslayers: [projectionLayer]once at session start and never touches that array again. Anything extra has to be composed back in together with the projection layer, and dropping it blanks the scene, sosetBaseLayeris required before anything can be added.VideoLayerreports whether it got a layer instead of throwing, so an app can ask for one everywhere and just draw into the scene where there isn't one. It falls back separately for no session, no base layer, no binding, and a platform that advertises a binding but refuses a particular element.The quad sizing is worth calling out, because the two paths deliberately differ and it looks like a bug otherwise. The spec means
widthandheightas full metres, and Chromium passes them to OpenXR unchanged, but the Quest compositor treats them as half extents and returns a quad at twice the size. It is a property of the compositor rather than of one binding, so the correction is decided by the platform, not the path. This is immersive-web/layers#324, still open. Worth knowing that three.js and A-Frame both hardcode a/2for Quest, which means copying either of them renders the quad at half size on Android XR.A few other things the spec is explicit about and the official samples get wrong or omit:
layouthas to be named because'default'is aTypeErrorfor quad layers,isStatichas to be false orgetSubImageraisesInvalidStateErroronceneedsRedrawclears, andgetSubImagethrows untilupdateRenderStatehas taken effect a frame or two after the layer is added, so a failed upload drops that frame rather than tearing the layer down.Two faults only showed up on device. The layer texture is allocated once at the size given at creation, and the test card is a canvas stream that reports no dimensions for its first frames, so guessing a size and then uploading a differently sized frame raised a GL error every frame and took the whole session down. And uploading through raw
gl.bindTextureleft three.js's state cache describing something untrue, so it carried on rendering with the compositor's texture bound. Uploads are also skipped for frames the video has not advanced past, since the source runs at 30fps against a 90fps render loop.The demo puts the same generated card up twice, once as a layer and once as a plane in the scene, so the only variable is the path the frame takes. It is generated rather than a video file because the sample clips I tried are lower resolution than the eye buffer, so both sides looked identical and it showed nothing. The two sides swap on a timer, which matters: lenses are sharpest in the middle and you read one panel at a time, so a fixed left/right layout cannot separate "the layer is sharper" from "that side was in the sweet spot". Swapping does separate them, and on device the difference follows the layer across the swap.
?webglLayer=1forces the WebGL path on a platform that has both, which is what lets a Quest stand in for Android XR.Type of Change
Media / Screen Recordings & Screenshots (If Applicable)
Checklist