Skip to content

GLM: fix sparse attention thread limit on M1 - #704

Merged
ashhart merged 1 commit into
ashhart:mainfrom
mrv777:fix/glm-sparse-attention-m1
Oct 13, 2026
Merged

ashhart merged 1 commit into
ashhart:mainfrom
mrv777:fix/glm-sparse-attention-m1

Conversation

@mrv777

@mrv777 mrv777 commented Oct 11, 2026

Copy link
Copy Markdown
Contributor

On our M1 Max, GLM's sparse row-attention pipeline supports 832 threads, but its caller launches 1,024. Its outputs remain zero and the existing comparison test fails. This is the inherited GLM failure reported during #679 validation.

Declare the required 1,024-thread group in the generator and both generated kernels (64 and 32 heads). Extend the regressions to check pipeline capacity and compare both outputs against an independent FP64 reference. Kernel bodies, dispatch sizes and tolerances stay unchanged.

Validated on M1 Max (32-core GPU, 32 GB), macOS 26.7, Xcode 26.6, Metal 32023.883 / SDK 26.5, Zig 0.17.0:

  • Both sparse checks pass: 99.97% / 99.96% equal outputs; worst differences 0.96 / 0.94 BF16 steps.
  • Removing the annotations makes both regressions fail.
  • Fresh suite: 1,291 passed, zero failed, two Linux-only skips; native build passes.
  • Goldens: 309/319 equal, ten declared differences, zero unexpected.
  • Lean: 534 inherited findings, zero added; exit remains 1.
  • All 34 generated Metal sources match. Full generator --check still exits 1 for an inherited whitespace mismatch in the kernel index.

Both before and after used the same local launcher selecting the installed Metal compiler/linker. Raw logs remain local. Synthetic tests only; model, performance, multi-Mac TP2 and other-hardware qualification remain unrun.

@harrisonfriia

Copy link
Copy Markdown
Contributor

I was able to reproduce this on an M2 Max: the sparse attention pipeline's maxTotalThreadsPerThreadgroup is also 832, and attn_check fails.

@mrv777 mrv777 mentioned this pull request Oct 13, 2026
8 tasks done
@ashhart
ashhart merged commit a9d0122 into ashhart:main Oct 13, 2026
jschmied pushed a commit to jschmied/TensorFold that referenced this pull request Oct 13, 2026
@ashhart

ashhart commented Oct 13, 2026

Copy link
Copy Markdown
Owner

Thank you, @mrv777. A pipeline that quietly allows fewer threads than its caller launches is a nasty failure (zeros, no error), and declaring the 1,024 in the generator and both kernels fixes it where it starts. Your commit landed in 1.0.9 under your name (a7408dba6), and the PR's head is merged into the history with no change to the tree (a9d0122cf), so GitHub shows this PR as merged.

What we ran, against the build without your change: your sparse attention checks pass on an M3 Ultra and an M5 Ultra (3 of 3 on each). With GLM-5.3-Flash at 8 of its 45 layers, replies drafted and with "draft": false are the same tokens on both builds on both chips, and decode and cold prefill from 2k to 32k tokens are within 0.7% (M3) and 0.3% (M5). M1 is your receipt; we have none here.

@ashhart ashhart mentioned this pull request Oct 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants