Repository navigation
EXL3: use branch-free trellis window extraction - #718
Merged
Merged
Conversation
mrv777
marked this pull request as ready for review
October 12, 2026 12:35
satindergrewal
added a commit
to satindergrewal/TensorFold
that referenced
this pull request
Oct 13, 2026
…3 experts Checked ash hart PRs open+closed (read-only). Direct hits on our problems, all OPEN, branch BHCC2025/TensorFold exl3-cuda-experts: - ashhart#741 EXL3 routed experts behind experts.Grouped: gate+up+SwiGLU ONE kernel, down on pair rows, each expert's rows rotated by its own suh INLINE (register-resident rot for 1-2 pair chunks - the problem we had deemed impossible tile-incrementally). Rides on grouped.zig's Plan (in our tree). 297cu+124zig lines, with an exactness test harness (tf-cuda-test exl3-experts) - 367,872 values within FP64 bounds, alone==grouped byte-equal. - ashhart#740 EXL3 projections behind qlinear (the stacked base). - ashhart#718 branch-free trellis window extraction (perf). - ashhart#729 share decoded weights across prefill rows (perf). - ashhart#728 MiMo-V2.6 CUDA EXL3 two-GPU (another full CUDA EXL3 family). - ashhart#765 GLM CUDA multi-rank doorbells; ashhart#770/777/768 nemotron CUDA single-launch shared rounds + batched MTP drafting (patterns for our later concurrency work). PLAN: vendored exl3_experts.cu/.zig + exl3.cuh into our tree (this commit, unwired). Next: build wiring (fatbin + mod), port moeBlock to Grouped.decode with the router->plan plumbing from grouped.zig, run their exl3-experts test on our checkpoint, parity, bench. Our own grouped kernels (stage-1 parity) stay as fallback.
jschmied
pushed a commit
to jschmied/TensorFold
that referenced
this pull request
Oct 13, 2026
Owner
|
Thank you, @mrv777. A shift and a masked carry in place of the two-way window is a neat change, and the reasoning holds at every offset: the carry sits below bit 16 from 1 to 16 and is masked at 0, and above 16 both forms keep the same bits of both words. Your commit landed in 1.0.9 under your name ( What we ran, on an M3 Ultra and an M5 Ultra, against the build without your change:
|
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The hoisted EXL3 decoder branches on each window’s bit offset. Replace that branch with unsigned shifts and a masked carry, preserving zero-offset behavior and all floating-point operations.
A GPU regression checks 35,040 windows across every offset and rejects a broken zero-offset mask.
M1 Max validation:
Broader 1-bit inst3 prefill timings remain inconclusive. Other chips and end-to-end throughput are unmeasured.
#681 has merged; this PR contains only the branch-free follow-up. The follow-up commit contains this change. Raw logs remain local.