Skip to content

EXL3: use branch-free trellis window extraction - #718

Merged
ashhart merged 2 commits into
ashhart:mainfrom
mrv777:perf/exl3-branch-free-windows
Oct 13, 2026
Merged

ashhart merged 2 commits into
ashhart:mainfrom
mrv777:perf/exl3-branch-free-windows

Conversation

@mrv777

@mrv777 mrv777 commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

The hoisted EXL3 decoder branches on each window’s bit offset. Replace that branch with unsigned shifts and a masked carry, preserving zero-offset behavior and all floating-point operations.

A GPU regression checks 35,040 windows across every offset and rejects a broken zero-offset mask.

M1 Max validation:

  • 19 GPU tests pass; all 82 original fixtures match output bits. Another 87 large cases pass full scalar-reference checks, and 33 cases preserve buffer guards and row padding. Scalar errors remain unchanged.
  • Five batches of 20 alternating pairs give median GPU-time ratios of 1.08–1.47× across the original 12 mul1/3-bit cases. Longer repeats confirm the weakest case; controls stay near 1.00×. Memory pressure stays normal.
  • Full build: 1,313 pass, two platform skips, only the existing GLM failure (GLM: fix sparse attention thread limit on M1 #704). Native build and golden checks have no unexpected differences.

Broader 1-bit inst3 prefill timings remain inconclusive. Other chips and end-to-end throughput are unmeasured.

#681 has merged; this PR contains only the branch-free follow-up. The follow-up commit contains this change. Raw logs remain local.

@mrv777
mrv777 marked this pull request as ready for review October 12, 2026 12:35
satindergrewal added a commit to satindergrewal/TensorFold that referenced this pull request Oct 13, 2026
…3 experts

Checked ash hart PRs open+closed (read-only). Direct hits on our
problems, all OPEN, branch BHCC2025/TensorFold exl3-cuda-experts:
- ashhart#741 EXL3 routed experts behind experts.Grouped: gate+up+SwiGLU
  ONE kernel, down on pair rows, each expert's rows rotated by its
  own suh INLINE (register-resident rot for 1-2 pair chunks - the
  problem we had deemed impossible tile-incrementally). Rides on
  grouped.zig's Plan (in our tree). 297cu+124zig lines, with an
  exactness test harness (tf-cuda-test exl3-experts) - 367,872
  values within FP64 bounds, alone==grouped byte-equal.
- ashhart#740 EXL3 projections behind qlinear (the stacked base).
- ashhart#718 branch-free trellis window extraction (perf).
- ashhart#729 share decoded weights across prefill rows (perf).
- ashhart#728 MiMo-V2.6 CUDA EXL3 two-GPU (another full CUDA EXL3 family).
- ashhart#765 GLM CUDA multi-rank doorbells; ashhart#770/777/768 nemotron CUDA
  single-launch shared rounds + batched MTP drafting (patterns for
  our later concurrency work).
PLAN: vendored exl3_experts.cu/.zig + exl3.cuh into our tree (this
commit, unwired). Next: build wiring (fatbin + mod), port moeBlock
to Grouped.decode with the router->plan plumbing from grouped.zig,
run their exl3-experts test on our checkpoint, parity, bench. Our
own grouped kernels (stage-1 parity) stay as fallback.
@ashhart
ashhart merged commit 5d7d05a into ashhart:main Oct 13, 2026
jschmied pushed a commit to jschmied/TensorFold that referenced this pull request Oct 13, 2026
@ashhart

ashhart commented Oct 13, 2026

Copy link
Copy Markdown
Owner

Thank you, @mrv777. A shift and a masked carry in place of the two-way window is a neat change, and the reasoning holds at every offset: the carry sits below bit 16 from 1 to 16 and is masked at 0, and above 16 both forms keep the same bits of both words. Your commit landed in 1.0.9 under your name (2e8749c61), and the PR's head is merged into the history with no change to the tree (5d7d05a25), so GitHub shows this PR as merged.

What we ran, on an M3 Ultra and an M5 Ultra, against the build without your change:

  • Tests. test-exl3 and test-exl3-gpu: 32 of 32 on both chips, your window check included (31 of 31 before it).
  • Bits. The bench's parity check between the hoisted kernel and the baseline finds no differing bit on either chip.
  • Speed, k = 6144, n = 25600, mul1, the hoisted kernel's mean GPU time: one row, 1.209 to 0.997 ms on the M3 Ultra (17.5% less) and 0.945 to 0.836 ms on the M5 Ultra (11.5% less); 16 rows, 3.755 to 3.404 ms and 3.035 to 2.608 ms. The unchanged baseline kernel moved by under 1%.

@ashhart ashhart mentioned this pull request Oct 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants