Repository navigation
GLM-5.3-Flash support for oQ in Zig engine - #773
Merged
Merged
Conversation
7 tasks
Contributor
|
Posted by Claude (Anthropic's AI assistant) on behalf of @sethforprivacy, as part of our local-inference tuning work on a Mac Studio M5 Ultra. @grantoverton the M5 checks for this PR at Live.
|
8 tasks done
Contributor
|
M1 Max check at This is the M1 KDA issue covered by #708 (which depends on #704). In a separate local copy, adding its That PASS is for the compatibility experiment, not the unmodified PR. The original snapshots were unchanged. Live drafted/plain, cached/fresh and concurrent/solo checks were not run: the roughly 115 GB GLM oQ2 checkpoint exceeds this Mac's 32 GB capacity.
|
oQ checkpoints write per-module widths as language_model.model.layers.N..., while loaders ask for model.layers.N... or the bare path, so an exact-key lookup missed every override and read the tensor at the global width. The table keeps each entry under the config's own key, as before, and also indexes it under its canonical module key (no .weight suffix, no known prefix, the rule the 27B and Flash Next families use); `width` takes the key as given, then its module. Two spellings of one module that declare different widths refuse the table as unsupported, the error `detect` already returns, so its callers' error sets are unchanged; the same width twice is fine. Test that fails before: the fixture reads a 5-bit down_proj declared under the oQ prefix by the loader spelling; on main it fails the packed shape check (read at the global width). Also: every prefix form and a trailing .weight name one module; aliases must agree; an override in any spelling answers every spelling and stays under its own key. Gate on main 0efbd44 (M3 Ultra, Zig 0.17.0): `zig build test test-golden -Dcpu=apple_m1` has 309/319 goldens equal and 0 unexpected; every step passes but one that fails on main too (core.checkpoint_metal's split-checkpoint test: `get` logs an error on the DeferredTensor path the test expects). lean_check: 539 problems (main 539), no new findings.
… tables repeated for a smaller group A projection of mixed-width members (the Flash Next hyper-connection down and inject pair, where 0.6.6's QWeights.of widens) reads as one matrix at their common width, the most bits and the smallest group. Narrowing and a group that does not divide the source group are refused. Used only where 0.6.6 widens; no caller on this commit. Tests (widen.zig, new): every source width and every wider target dequantize to the same fp64 values, bit for bit; a stack of experts widens row for row; the same width comes back as it is; narrowing bits, growing groups and widths the kernels do not run are refused; the common width takes the most bits and the smallest group; fused members keep each one's values in order. Gate on main 0efbd44 (M3 Ultra, Zig 0.17.0): `zig build test test-golden -Dcpu=apple_m1` has 309/319 goldens equal and 0 unexpected; every step passes but one that fails on main too (core.checkpoint_metal's split-checkpoint test: `get` logs an error on the DeferredTensor path the test expects). lean_check: 539 problems (main 539), no new findings.
…wider output Chips without tensor units now decode every MLX affine width (2, 3, 4, 5, 6, 8 bits) in the row kernel: 3- and 5-bit rows unpack eight codes from three or five bytes, low code first, as MLX packs them. The 2-, 4-, 6- and 8-bit paths are unchanged. A 3- or 5-bit row must end on a word, as the checkpoint stores it. callStrided launches the row kernel with an output row stride, so one member of a fused stack (one run of equal width) writes its columns straight into the stack output. The default call, its dims and its kernels are unchanged. tf-row-widths-check (zig/build/core_checks.zig) runs it on the GPU: each width at groups of 32, 64 and 128 against an fp64 product from the MLX reference dequantization, within the fp32-sum bound; one row alone against the same row in 2-, 3-, 8- and 16-row windows, relu2, indexed slots and a strided output, bit for bit; a fused stack of 48-column members at mixed widths (4-bit among them, group 64) written through the stride at windows up to 48 rows, each member as its own call; and, given the unchanged kernel source, the widths it already served byte for byte against it. Tests that fail before: "shapes the row kernels serve" now validates 3 and 5 bits (main refuses them), "3- and 5-bit rows take whole words of codes", "every width has its own pipeline slot" and "a strided call writes rows at least a matrix wide" (no callStrided on main). tf-row-widths-check on an M3 Ultra (no tensor units), given main's row_projection.metal: PASS 64512 values within the fp32-sum bound (worst err/bound 0.9400), 408192 bit-equal pairs, 301056 outputs byte-equal to main's kernels at the widths main serves (3 and 5 bits have no main kernel to compare). Gate on main 0efbd44 (M3 Ultra, Zig 0.17.0): `zig build test test-golden -Dcpu=apple_m1` has 309/319 goldens equal and 0 unexpected; every step passes but one that fails on main too (core.checkpoint_metal's split-checkpoint test: `get` logs an error on the DeferredTensor path the test expects). lean_check: 536 problems (main 539), no new findings.
…d 16-row members Format accepts 2-, 3- and 5-bit codes beside 4, 6 and 8, in groups of 32, 64 or 128. The non-4-bit path widens 2- and 3-bit codes to nibbles for the bf16 x uint4 op and 5-, 6- and 8-bit codes to bytes for the uint8 op, the staging the Python lane matmul uses, so each width keeps that arithmetic. 3- and 5-bit groups of 32 are an odd number of words a lane; they are fetched a word at a time instead of in pairs, which would drop the last word. Layout.y_stride (default N) sets the elements between output rows in every kernel (tf_lane, tf_lane_pair, tf_lane_reg), so a member of a fused stack writes its columns of the fused rows; the output bound checks follow it. The store of the simdgroup-matrix layout (Macs without tensor units, ashhart#679) takes the same stride and keeps its column guard. N may be a multiple of 16 on the plain kernel: the last 32-column tile is packed with zero codes past N, its metadata reads and stores are already guarded by N, and weightWords counts the padded tiles (sourceWords is MLX's W). Cooperative and reg layouts still need N % 32 == 0. encodeSums checks its sums buffer by the group count, as ashhart#698 sizes the dispatch. Without tensor units the admitted layout is unchanged: 6-bit group 32 on the plain kernel (ashhart#679); every other width still answers TensorUnitsRequired there, now checked at 2, 3 and 5 bits too. With the defaults the 4-, 6- and 8-bit kernels preprocess (metal -E) to the same text as main's for every group, schedule, output, prepared-sums/reg layout and both fragment layouts tried (148 variants), so the uniform paths keep their kernels. Host tests that fail before: 2, 3 and 5 bits validate; y_stride reaches the source and below N is refused; a 48-row member validates (cooperative and reg refuse it); every width x group x {64, 48} rows packs MLX codes bit for bit with zero padding. The device program covers every width, 48-row members and strided output on tensor units; without them (M3 Ultra): PASS 14 layouts, 1904 bit-equal row/window pairs, 208 windows equal to fn_lane's recorded bytes, 7424 CPU values (max roundoff-bound fraction 0.0144), 10 refusals, and a 48-row member and two strided layouts at 6/32 equal the unstrided rows with the gaps untouched. fn_lane's recorded windows are compared only on the layouts they were recorded at. Gate on main 0efbd44 (M3 Ultra, Zig 0.17.0): `zig build test test-golden -Dcpu=apple_m1` has 309/319 goldens equal and 0 unexpected; every step passes but one that fails on main too (core.checkpoint_metal's split-checkpoint test: `get` logs an error on the DeferredTensor path the test expects). lean_check: 536 problems (main 539), no new findings.
affine_mm took 4- and 8-bit weights only, so a 2-bit matrix (oQ2's routed experts) had no tiled prompt product. At 2 bits a thread loads its 32 codes of a step as 8 bytes and writes each code into the bf16 tile as itself, with no offset (as at 8 bits; 0..3 is exact in bf16), so the group sums are the 4- and 8-bit path's. A first try with the 4-bit "+128" offset stayed within the float64 bound, but a reference with scales 1% off no longer exceeded it at K = 4096, so the codes enter exactly. The 4- and 8-bit sources preprocess as before. Test that fails before: "dense and gathered, each format, against exact sums on the host" adds 2/64 and 2/32, dense and gathered; source() refused 2 bits. The 4- and 8-bit sources preprocess (metal -E) to main's text at both groups, both outputs and both fragment layouts (16 variants). tf-affine-ref (next commit) holds the 2-bit products to float64. Gate on main 0efbd44 (M3 Ultra, Zig 0.17.0): `zig build test test-golden -Dcpu=apple_m1` has 309/319 goldens equal and 0 unexpected; every step passes but one that fails on main too (core.checkpoint_metal's split-checkpoint test: `get` logs an error on the DeferredTensor path the test expects). lean_check: 536 problems (main 539), no new findings.
…y row's bits in every window The device side of ashhart#633's quant-ref for the shared prompt products, item 5 of ashhart#716. tf-affine-ref takes no arguments and needs no model: seeded synthetic codes, scales, biases and inputs, packed as MLX affine. The engine module now names core/affine_mm so a check program can reach it. - affine_mm at 2, 4 and 8 bits, groups 32 and 64, bf16 and fp32 out, dense and as a one-expert gather, at [192, 1024] and [256, 4096]. - simd_qmm (4-bit, group 64) as tiles and as the scalar twin at 1 and 2 rows, at [40, 512], [256, 4096] and [6400, 256] (K chunks of 32, 16 and 8). The reference dequantizes the same codes in float64 and multiplies in float64. Each product prints its worst column (max abs error, and that error over the column's largest |y|) and its largest error as a fraction of the bound: (group + K / group + 8) fp32 roundings of the terms' magnitudes as the kernels sum them (4-bit affine_mm codes plus 128), plus one bf16 ulp for bf16 outputs. A reference with its scales 1% off must exceed the bound. Then every row of a 129-row call must equal, byte for byte, the same row computed in windows of 1, 2, 3, 8, 16, 17, 33, 64 and 65 rows (every full window over the 129 rows, so each row is seen at every position), which crosses the 16-row fragments, the 32- and 64-row tiles and the gather's switch between them. The twin must equal the tiles wherever twinExact says it does, since only then do callers take it. Receipts, M3 Ultra (no tensor units: nax.h's simdgroup-matrix layout), on main 0efbd44: - every product within 0.4935 of its bound; every row's bits equal at every window, both kernels, every format; the twin equals the tiles at 1 and 2 rows on all three shapes, as twinExact says. - worst columns, bf16 out (dense and gather equal), at [192, 1024] and [256, 4096]: affine_mm q2 g32 2.64e-2 (rel 3.23e-3) and 6.22e-2 (3.78e-3), q2 g64 2.87e-2 (3.51e-3) and 3.12e-2 (2.47e-3), q4 g32 2.45e-2 (2.65e-3) and 3.12e-2 (3.01e-3), q4 g64 1.56e-2 (2.89e-3) and 3.12e-2 (2.53e-3), q8 g32 0.496 (3.41e-3) and 0.968 (3.63e-3), q8 g64 0.453 (2.87e-3) and 0.956 (3.38e-3); simd_qmm 1.19e-2 (2.57e-3), 3.13e-2 (2.87e-3) and 1.54e-2 (3.74e-3). All under 2^-8 of the column's largest |y|; 0.20 to 0.49 of the bound. - fp32 out: at most 0.0081 of the bound (q2 1.5e-6 to 7.7e-6, rel 1.9e-7 to 6.0e-7). - scales 1% off: 2.3 to 29 times the bound for bf16 outputs, 2.7 to 391 for fp32, 44 to 273 for simd_qmm. Not run: an M5, where affine_mm takes the tensor units; `zig build tf-affine-ref && ./zig-out/bin/tf-affine-ref`. The fp32 bound is the worst case for any summation order, so a changed order still passes; the measured errors sit two orders below it, and the bit checks are what pin the order. Gate on main 0efbd44 (M3 Ultra, Zig 0.17.0): `zig build test test-golden -Dcpu=apple_m1` has 309/319 goldens equal and 0 unexpected; every step passes but one that fails on main too (core.checkpoint_metal's split-checkpoint test: `get` logs an error on the DeferredTensor path the test expects). lean_check: 536 problems (main 539), no new findings.
…t; a width-carrying matrix replaces Q4
Every GLM projection was typed 4-bit/group-64 (weights.zig's Q4), so an oQ checkpoint could not even be planned.
- Widths (quant.zig): each module's MLX affine width from config.json through core/quant (keys.canonical, then
GLM's own spellings: the MTP block is layer 45, forget_gate folds into the KDA gate). Aliases that disagree,
a missing table and other modes are refused; a tensor stored at another width than declared is refused by name,
never read at a guessed width.
- Affine{bits, group} in place of Q4 (affine.zig): stored shapes, byte spans and column cuts follow each tensor's
width; a fused matrix whose members differ in format plans one run a member at its output rows, as 0.6.6's QSplit
calls each part (glm5_next/linear.py); a one-format stack stays one matrix. The load plan's layer and expert bytes
are counted at those widths.
- mlx-lm's layout (names.zig), which the public oQ checkpoints use: the MTP block, hyper-connections, stacked
experts, the absorbed embed_q/unembed_out pair and a bf16 eh_proj; a fused conv1d.weight is cut into q/k/v; fp32
hyper-connection mixes and router narrow to bf16 only when exact (else refused naming fix, shard, offset, shape).
- The dry-run plan prints its matrices by width. A real load still refuses any matrix the generated 4/64 kernels
cannot read (the next commit gives the device every width); TP2 and expert parallel refuse any width but 4/64.
The uniform 4/64 checkpoint plans the same matrices, offsets and bytes as the 4-bit-only loader (a digest pinned
before the change).
Tests that fail before (no width reading, no mlx-lm names on main): weights_test.zig, oQ-shaped fake checkpoints in
both layouts, each tensor found at its own width; overrides under either prefix; a tensor stored at another width,
two spellings or the MTP block and layer 45 at two widths, an odd member and a routed gate/up pair at two widths
refused by name; oQ4's KDA input and MLA x_proj planned as one run a member; mlx-lm's conv1d and fp32 mixes planned
as the original layout's tensors. quant.zig, affine.zig, names.zig and weights_fix.zig have their own unit tests.
Gate on main 0efbd44 + the core commits (M3 Ultra, Zig 0.17.0): `zig build test test-golden -Dcpu=apple_m1` has
309/319 goldens equal and 0 unexpected; every step passes but one that fails on main too (core.checkpoint_metal's
split-checkpoint test logs an error on the path it expects). lean_check: 533 problems (main 539), no new findings.
…t stacks a product a member GLM has no tensor-unit-only kernel (its *_nax kernels take the simdgroup-matrix fragments without tensor units), so the pre-M5 path only lacked products at other widths. glm_rows.metal (wide.zig) runs them over core row_projection's fp32 group sums, 2/3/4/5/6/8 bits in groups of 32/64/128, every row its own sums (one row alone == the same row in any window): - dense with input and output row strides and per-head batching: each split-stack member at its own output rows (oQ4's KDA input q, v at 5 bits and k, f_a, g_a, b at 8 is six products), mlx-lm's embed_q and unembed_out a head; - by expert id (decode's routed experts, gate and up fused with SwiGLU, glm_glue's sigmoid_fast and bfloat overloads compiled in, not copied), and by sorted pairs (prompt's routed experts); then the existing combine; - bf16 (the MTP head's unquantized eh_proj); the embedding at 6 and 8 bits. KDA f_b and g_b at 4/64 keep the generated kernel's fused dots; at any other width they are row products first and one instance of the generated kda_rows reads them in place of its dots (0.6.6's unfused shape); the products borrow the dense MLP scratch, so state refuses a config whose KDA width outgrows it. A shared expert not at 4/64 runs its own products (no shared-in-routed launch); routed experts not at 4/64 run by id and by sorted pair. Prompt chunks: a one-width 8/64 matrix runs on core affine_mm, split stacks and other widths on the row kernels. The new pipelines load only when a checkpoint has a matrix the 4/64 kernels cannot read, and their source hash joins the kernels' identity; a failed compile releases every pipeline built before it. The uniform 4/64 checkpoint compiles and launches what it did. The previous commit's refusals stay: a member whose rows are not a multiple of 4 (the row kernels read four rows), routed gate and up at two widths, any width but 4/64 with TP2 or expert parallel. Tests that fail before (no wide.zig on the previous commit): 4/64 of one width alone takes the generated kernels; f_b and g_b at any other width (8/64 too) take the products made apart; a failed compile releases every pipeline built before it and a load counts each one (GPU); glm_rows's SwiGLU compiles glm_glue's helpers, no copy of its own. The previous commit's load-time refusal of every other width is gone. Gate on main 0efbd44 + the core commits (M3 Ultra, Zig 0.17.0): `zig build test test-golden -Dcpu=apple_m1` has 309/319 goldens equal and 0 unexpected; every step passes but one that fails on main too (core.checkpoint_metal's split-checkpoint test logs an error on the path it expects). lean_check: 533 problems (main 539), no new findings.
…float64, each row bits in every launch
No model files: seeded synthetic codes, scales, biases and inputs, packed as MLX affine.
- Every bits {2,3,4,5,6,8} x group {32,64,128} on three shapes, and every GLM projection class at the public oQ
checkpoints' widths at real shapes (KDA in, f_b/g_b, KDA o, MLA x/qr/o, embed_q/unembed_out, dense, shared,
routed, lm_head), decode rows and the prompt's product separately, against float64 on every column of two rows.
The bound is the fp32 sums' depth (k / 8 + 64 additions) plus a bf16 rounding; each class prints its worst column
(max |err| and its share of max |y|). A reference with scales 1% off must exceed the bound in every class and width.
- Bit-equal pairs: one row alone == the same row in 2-, 3- and 16-row windows; strided == contiguous; per head ==
each head dense; by id == dense; GLU by id == glm_swiglu(dense gate | up); sorted == by id (experts of 8, 5, 2 and 1
pairs); the row kernels == core row_projection; a split stack in one call == each member alone (oQ4's KDA input
5/8/5/8/8/8 and x_proj 6/6/5/5, at the decode pitch and the 64-padded prompt pitch); experts [2048, 4096] and
[4096, 2048] at 2 and 4 bits by pick and sorted == dense.
- KDA: the separate f_b/g_b products then kda_rows == the generated 4/64 kda_rows byte for byte (output, state, conv
window) on every 4-bit code, random per-group power-of-two scales and nonzero biases.
- The 6- and 8-bit embedding against float64 within two bf16 roundings; a neighbouring group's bias is caught.
The engine module exports core/quant (`tensorfold.quant`) for the check's MLX reader.
tf-glm-rows-check (M3 Ultra, no tensor units): PASS 1,328,072 values within the fp32-sum bound of float64 (worst
0.9839 of it, bf16 output rounding; worst-column |err| / max|y| 1.7e-3 to 3.6e-3 a class), 1,780,096 bit-equal
pairs; scales 1% off exceed the bound in every class and width.
Gate on main 0efbd44 + the core commits (M3 Ultra, Zig 0.17.0): `zig build test test-golden -Dcpu=apple_m1` has
309/319 goldens equal and 0 unexpected; every step passes but one that fails on main too (core.checkpoint_metal's
split-checkpoint test logs an error on the path it expects). lean_check: 533 problems (main 539), no new findings.
…e_mm oQ2's routed experts (2-bit, group 64) ran a prompt chunk on the sorted row kernels: 81% of a 2048-row chunk's GPU time. With core affine_mm reading 2-bit codes, wide.routed(gate, up, down) picks affine_mm's 4/64 pipelines, its 2/64 ones (wide.Pipes.mm2) or the sorted row kernels, and 2/64 experts take gatherGlu/gatherTo as 4/64 ones do; other widths stay on the row kernels. Decode is unchanged. Test that fails before: "prompt chunks gather routed experts on affine_mm at 4/64 and 2/64 only; other widths take the sorted row products" (no wide.routed). tf-glm-rows-check adds gatherCheck: 2/64 experts gathered on affine_mm == the dense product (GLU == glm_swiglu(dense gate | up), down == dense) at expert shapes, and the routed q2 prompt classes now run on affine_mm against float64. tf-glm-rows-check (M3 Ultra, no tensor units): PASS 1,328,072 values within the fp32-sum bound of float64 (worst 0.9839 of it, bf16 output rounding; worst-column |err| / max|y| 1.7e-3 to 3.6e-3 a class), 1,976,704 bit-equal pairs; scales 1% off exceed the bound in every class and width. TensorFold/GLM-5.3-Flash-MLX-oQ2-MTP @dc55627 on an M3 Ultra (plan 109.01 GB: 2-bit/64 x126, 4-bit/64 x3, 8-bit/64 x290; loads in 21-24 s): drafted == `"draft": false` 4/4 at 160 tokens (MTP acceptance 74-88%; plain 39.5-40.1 tok/s, drafted 50.9-54.4); resumed == fresh (a 2427-token prompt at 0 and 2425 cached tokens; the second turn resumed at 2425 == a server with `--prompt-cache-gib 0`); concurrent == solo, 2 streams. Cache off: prefill 2427 tokens 5.09-5.11 s (475-477 tok/s; 163 tok/s with the experts on the sorted row kernels), decode 256 tokens 40.8-41.0 tok/s. Gate on main 0efbd44 + the core commits (M3 Ultra, Zig 0.17.0): `zig build test test-golden -Dcpu=apple_m1` has 309/319 goldens equal and 0 unexpected; every step passes but one that fails on main too (core.checkpoint_metal's split-checkpoint test logs an error on the path it expects). lean_check: 533 problems (main 539), no new findings.
grantoverton
force-pushed
the
oqxe/glm
branch
from
October 13, 2026 20:21
9777aeb to
5fe7e78
Compare
ThinkOffApp
pushed a commit
to ThinkOffApp/TensorFold
that referenced
this pull request
Oct 14, 2026
ThinkOffApp
pushed a commit
to ThinkOffApp/TensorFold
that referenced
this pull request
Oct 14, 2026
The contributor's four commits landed above with the same code; the quant export one of them adds was already in from ashhart#756. This keeps the PR's head in the history with no change to the tree.
Owner
|
Thank you, @grantoverton. GLM-5.3-Flash on a Mac serves mixed-width checkpoints such as TensorFold/GLM-5.3-Flash-MLX-oQ2-MTP. A uniform 4-bit checkpoint compiles none of the new pipelines. Two Macs and expert parallel still take 4-bit group-64 checkpoints only, and refuse others by name at load. Your commits are in 1.0.10 under your name ( The oQ checkpoints rest on your runs on an M3 Ultra and @sethforprivacy's on an M5 Ultra, both at |
This was referenced Oct 14, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebased onto main c36b536 (1.0.9) on 2026-10-13: no conflicts in this change's code; every commit passes the gate again on the new base.
Part of #716; stacks on #755. This branch carries #755's six commits until that one merges, so please review the last four commits here (from "GLM-5.3-Flash reads each module's MLX affine width and mlx-lm's layout" on). GLM-5.3-Flash types every projection as 4-bit/group-64 on main (
weights.zig'sQ4), so an oQ checkpoint cannot even be planned. With this, TensorFold/GLM-5.3-Flash-MLX-oQ2-MTP @dc55627 and TensorFold/GLM-5.3-Flash-MLX-oQ4-MTP @9872546 load, draft and serve exactly on an M3 Ultra. Nothing here waits for an M5: GLM has no tensor-unit-only kernel (its*_naxkernels take the simdgroup-matrix fragments without tensor units), and every width now has a product before tensor units. The uniform 4/64 checkpoint keeps its load plan (a digest pinned before the change), its kernels and its launches.What this changes
Four commits on the shared change:
Q4.quant.zig): each module's width fromconfig.jsonthroughcore/quant, then GLM's own spellings (the MTP block is layer 45;forget_gatefolds into the KDA gate). Aliases that disagree, a missing table and other modes are refused; a tensor stored at another width than declared is refused by name, never read at a guessed width.Affine{bits, group}in place ofQ4(affine.zig): stored shapes, byte spans and column cuts follow each tensor's width. A fused matrix whose members differ in format plans one run a member at its output rows, as 0.6.6'sQSplit(glm5_next/linear.py); a one-format stack stays one matrix. The load plan counts layer and expert bytes at those widths.names.zig), which the public oQ checkpoints use: the MTP block, hyper-connections, stacked experts, the absorbedembed_q/unembed_outpair and a bf16eh_proj. A fusedconv1d.weightis cut into q/k/v; fp32 hyper-connection mixes and router narrow to bf16 only when exact, else the load is refused naming the tensor, shard, offset and shape.glm_rows.metal(wide.zig) runs over corerow_projection's fp32 group sums, 2/3/4/5/6/8 bits in groups of 32/64/128, every row its own sums:embed_qandunembed_outa head at a time;glm_glue's helpers rather than a copy) and by sorted pairs (the prompt's routed experts), then the existing combine;eh_proj), and the embedding at 6 and 8 bits.kda_rowsreads them in place of its dots (0.6.6's unfused shape). They borrow the dense MLP scratch, so a config whose KDA width outgrows it is refused.affine_mm, split stacks and other widths on the row kernels.tf-glm-rows-check: GLM's products at every MLX width against float64, and each row's bits in every launch.affine_mm. oQ2's 2/64 routed experts spent 81% of a 2048-row chunk's GPU time on the sorted row kernels; they now takegatherGlu/gatherToonaffine_mm's 2/64 pipelines, as 4/64 experts do. Decode is unchanged.Exactness
TensorFold/GLM-5.3-Flash-MLX-oQ2-MTP @dc55627, TensorFold/GLM-5.3-Flash-MLX-oQ4-MTP @9872546, and the uniform 4/64 (below), greedy, MTP drafts on unless
"draft": false. Same checkpoint, binary and settings within each comparison.Q4).bench_openaiandbench_concurrenttoken hash is identical across the four runs, and the earlier exactness runs gave all 21 contract hashes identical (four prompts plain and drafted, resumed equals fresh at 0 and 2,425 cached tokens and on a cache-off server, the second turn resumed equals a cache-off server, concurrent equals solo with 2 streams)."draft": false4/4 at 160 tokens (MTP acceptance 74 to 88%); resumed equals fresh (a 2,427-token prompt at 0 and 2,425 cached tokens); the second turn resumed at 2,425 equals a server with--prompt-cache-gib 0; concurrent equals solo (bench_concurrent's alone and serial checks at 1 and at 4 requests over 2 streams, both prompts: 34/34 equal); greedy text sane ("3:15 PM + 2:30 = 5:45 PM").x_proj/qr_projof layer 7 and the MTP layer.--parallel 2, on oQ2 and oQ4): a fresh"draft": falseprobe (each bench prompt, greedy, 128 tokens) gives the reference hash and the drafted probe equals it, with partial acceptance seen (accepted < drafted) on one prompt (oQ2) and both (oQ4); then after a stream dropped mid-decode, a long prompt dropped mid-prompt, a stream dropped beside a running probe (whose hash is checked too) and a request past the window (refused, the reset path), the drafted and plain probes equal the fresh hashes. 25 steps, pass, on each checkpoint (hashes oQ289faa950b278/fd4c707155c9, oQ47152f0a8505e/0821b2106b69).tf-glm-seams, a test program on this branch, not in the PR): the exact 2,048-token public prefill array, in the engine's one 16,384-row chunk against chunks of 128 and of 37, comparing every KDA layer's recurrent and conv state, the MLA rows and the last row's logits at seams 256, 512, …, 1,792, the prompt end, and after 64 one-row decode windows.forward.zig) rather than the prompt path (prompt.zig). Main does the same on the uniform 4/64: I ran the same test program on main 0efbd44 (the program only, no engine change) and it gives the same result, part for part. This is Proposal: restore single-Mac GLM-5.3-Flash on the M3 Ultra #743's item 3 (prompt chunks and 16-row windows give different bits); the 256-token token streams of prompt chunks and ofGLM_PROMPT=0windows also differ on main and here, with identical hashes on both.glm_swiglu(dense gate | up); sorted equals by id (experts of 8, 5, 2 and 1 pairs); the row kernels equal corerow_projection; a split stack in one call equals each member alone (oQ4's KDA input 5/8/5/8/8/8 andx_proj6/6/5/5, at the decode pitch and the 64-padded prompt pitch); 2/64 experts gathered onaffine_mmequal the dense product. 1,976,704 bit-equal pairs.kda_rowsequal the generated 4/64kda_rowsbyte for byte in the output, the recurrent state and the conv window, on every 4-bit code with random per-group power-of-two scales and nonzero biases.FP64 per class
tf-glm-rows-checkon the M3 Ultra (no tensor units), seeded synthetic MLX affine weights. The bound is the fp32 sums' depth (k / 8 + 64 additions) plus one bf16 rounding; every column of two rows is compared against float64, decode rows and the prompt's product separately on identical input rows. A reference with scales 1% off must exceed the bound in every class and width.Every bits {2,3,4,5,6,8} × group {32,64,128} on three shapes ([132, 384], [64, 2688], [40, 1536]), worst error over the bound:
Every GLM projection class at the public oQ checkpoints' widths, at its real shape:
Per class: decode and prompt worst column against the float64 bound
1,328,072 values within the bound, worst 0.9839 (KDA f_b/g_b at 8 bits: bf16 output rounding); worst-column |err| over the column's largest |y| 1.7e-3 to 3.6e-3 in every class. The 6- and 8-bit embedding gathers are within two bf16 roundings of float64, and a neighbouring group's bias is caught. Core
tf-affine-refat 2 bits (groups 32 and 64, dense and gathered) is in #755.Performance
Mac Studio, M3 Ultra (80-core GPU), 256 GB, macOS 26.4.1 (25E253), Zig 0.17.0. Fresh builds from clean detached worktrees,
zig build native -Dcpu=apple_m1;tensorfold-nativesha256d508376cf1fdc1f3…(this branch) and1c8c96881568…(main 0efbd44). Every run starts from a fresh server, with other checkpoints' file cache evicted first.Decode and concurrency.
serve <checkpoint> --name glm --parallel 2 --context 16384 --prompt-cache-gib 0, MTP drafts on;bench_openai.py … --tokens 256 --reps 3 --temperatures 0(medians;fibonacci-raw, code, andgpu-chat-no-think, chat) andbench_concurrent.py --levels 1,4 --alone --serial(aggregate tok/s; 4 requests share the 2 streams):bench_openaicode/v1/completionsThe uniform path is unchanged: the branch's two runs average 0.2% below main's on
bench_openaicode (102.56 against 102.73) and 5.1% above on chat (63.82 against 60.72; the chat prompt's single reps range from 60.0 to 65.5 across the four runs, and main's reach 63.2), nobench_concurrentcell is more than 0.2% below main's average, and every token hash is identical. The oQ checkpoints have no before-run: main refuses them at load. So their speed is reported as served decode beside oMLX and mlx-vlm, the standard servers for GLM-5.3-Flash (an mlx-vlm model type). Against this branch's uniform runs, oQ4 decodes code at 0.61× and chat at 0.87×, while oQ2's code is level (0.99×; MTP acceptance is high on that prompt) and its chat is 0.85×. Their non-4/64 products run on the generic row kernels, and a split stack is one launch per member; a follow-up PR series works on that speed (#716).Cold prefill (uniform 4/64; main and this branch). Exact token arrays: upstream's public prefill paragraph ("A lighthouse keeper writes in the log as the boats cross the bay.",
tools/zig/qwen27_gate.py) repeated and tokenized once without special tokens (tokenizer.jsonsha25619e77364…);/v1/completionswith the ids,max_tokens 1,--parallel 1 --context 66560 --prompt-cache-gib 0, every request cold (cached_tokens0), a 2k warmup, three reps of each size, medians:--context 132096)Native-window prefill not run; happy to add it if wanted.
oQ prefill (a 2,427-token prompt, cache off, the earlier exactness servers): oQ2 475 to 477 tok/s and oQ4 380, against the uniform 4/64's 506.7 to 507.7 (0.94× and 0.75×). Before the last commit oQ2 prefilled the same prompt at 163 tok/s.
Memory
Load plans as logged: uniform 4/64 180.58 GB, oQ4 184.66 GB, oQ2 109.01 GB in buffers. Physical footprint (
darwin_rusage_info_v4, the server process) at--parallel 2 --context 16384after load and at the end: uniform 185.29 / 184.92 GB (peak 185.34 GB, main and this branch equal to 0.02 GB), oQ4 189.38 / 189.01 GB (peak 189.43 GB), oQ2 113.70 / 113.33 GB (peak 113.78 GB). At--parallel 1 --context 66560, uniform peaks at 186.06 GB (main) and 186.03 GB (branch); at--context 132096, 187.33 GB on both. During the full oQ4 load free memory never went under 44.9 GB.Fidelity
Teacher-forced against oMLX 0.7.1.dev1 on the same checkpoint: oMLX's own GLM-5.3 load path (its vendored
glm5_nextfrom mlx-vlm PR #2030 on mlx-vlm 0.7.7, with each per-module width's runtime aliases), which keeps every projection at its own width and drops the MTP tensors. Stock mlx-vlm 0.7.6 fuses q/k/v into oneLinear, so it cannot load oQ4 (layer 6 stores q/k/v at 5 and 8 bits). oMLX renders upstream's four public teacher prompts (copy, arithmetic, story, code; chat template, thinking off), prefills each in one pass and decodes 256 greedy tokens with EOS ignored, no drafts, keeping its top-20 logits at every step; that continuation is the fixed history. This branch'stf-glm-teacher(a test program, not in the PR) then feeds every history token and records its top two at all 1,024 positions. Logits are bf16.On the uniform checkpoint main and this branch give the same 28 differences at the same positions, so the gap to oMLX is TensorFold's GLM against oMLX's, and it is there before this change. The oQ checkpoints sit at or under it: oQ4 has fewer differences than the uniform checkpoint, oQ2 one more with fewer beyond the tie policy. In every difference this branch's pick is oMLX's second (or third) token, never a token oMLX ranks lower, and no position has a pick below another token's logit.
Every difference (oMLX's pick and its margin over the runner-up; this branch's pick and its margin; this branch's pick's rank in oMLX's top 20):
uniform 4/64 (main and this branch, identical): 28 differences, 26 beyond the tie policy
oQ4: 19 differences, 12 beyond the tie policy
oQ2: 29 differences, 22 beyond the tie policy
Host suite and lean check
zig build test test-golden -Dcpu=apple_m1at each of the four commits, from a clean detached worktree: 158 of 160 steps; goldens 309/319 equal, the 10 declared differences, 0 unexpected;tf-glm-rows-checkpasses from the commit that adds it on. The one failing step is main's own:core.checkpoint_metal's "a split checkpoint keeps later tensors in the file, packs the rest, and stages the same bytes" logs an error (getcallsstd.log.errbefore returning theerror.DeferredTensorthe test expects, and the runner counts logged errors). It fails the same way on main0efbd447.weights_test.zig's oQ-shaped fake checkpoints in both layouts find every tensor at its own width; overrides under either prefix; a tensor stored at another width, two spellings or the MTP block and layer 45 at two widths, an odd member and a routed gate/up pair at two widths are refused by name; oQ4's KDA input and MLAx_projplan as one run a member; mlx-lm's conv1d and fp32 mixes plan as the original layout's tensors; 4/64 alone takes the generated kernels and any other f_b/g_b width the separate products; a failed compile releases every pipeline built before it (GPU); prompt chunks route routed experts toaffine_mmat 4/64 and 2/64 only. The uniform plan's digest is unchanged.python3 tools/zig/lean_check.py: 533 problems against main's 539 at every commit, no new findings. Gone: three over-long comments inweights.zigand Core support for oQ in Zig engine #755's three. Inherited findings left in files this branch touches: over-long comments inchecks.zig(5),engine.zig(6),forward.zig(8),kernels.zig(1),load_plan.zig(2),moe.zig(2),mtp.zig(2),prompt.zig(11),weights.zig(1), a multi-line comment inkernels/metal/sources.zig, and Core support for oQ in Zig engine #755's. Every touched file is at or under 600 lines;zig fmtclean.Platforms
tf-glm-rows-checkandtf-affine-refPASS, and oQ2 serves exactly (drafted, cached and concurrent).tf-affine-refPASS;tf-glm-rows-checkneeds GLM: fix KDA and router thread limits on M1 #708's KDA thread limit on M1 and passes with it.affine_mm(8/64 and 2/64 prompts) compiles its tensor-unit variant, and the row kernels' one- and two-row instances have shown row invariance only under the M3's compiler. Commands:then, greedy, per prompt: a request and the same with
"draft": false(same tokens); a ~2.4k-token prompt sent twice (cached 0, then cached) and a second turn compared with a server started with--prompt-cache-gib 0; two requests at once compared with each alone. I have no M5; @sethforprivacy has since run these (above).Reproduction settings
GLM_LOAD_LIMIT_GB=208, which I set as the machine's owner. The default limit is 70% of RAM, 179.2 GB here, and the plan is 184.7 GB for oQ4 and 180.6 GB even for the official uniform 4-bit, so neither loads on a 256 GB Mac without it (oQ2 at 109 GB does). The limit only admits the load and changes no arithmetic. The engine warns that past 70% a load can stall macOS until its watchdog restarts the Mac: run nothing else large beside it.iogpu.wired_limit_mbis 240000, also set by me.Where the arithmetic differs from 0.6.6
Bit equality with 0.6.6's kernels is not the bar (#716); the float64 bound per product, row invariance and the exactness contract are.
qmv_rows_brounds group sums to bf16 at 5 and 6 bits.affine_mmin prompts, with fp32 sums, not MLX's 2-bitquantized_matmul. One-width 8/64 prompt products run onaffine_mm.PM.matmul.InexactBf16), where 0.6.6'sbf16_if_exactkeeps it in fp32.The open question in #716
GLM follows 0.6.6's
QSplit: once a fused matrix's members differ in format, every member is its own product at its own output rows, members that share a width included. Nothing is grouped across members, so the per-member column map #716 asks about isn't needed here (Flash Next is the family that uses one).Follow-ups
embed_qrun the decode's per-head launch serially over the chunk's rows.Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo (oQ4, oQ2 and the uniform 4/64, full models, served).Receipts: https://gist.github.com/grantoverton/57663bbdd9b1ebcd4d5524e310923403