perf(qwen27): an accepted DeltaNet chain scans its state in place, no state publish copy

[jkuepker]

In a chain round, the qwen27 DeltaNet pool scans each layer's recurrent state into the next-state buffer, then copies it back into the slot. Each of the 27B's 48 DeltaNet layers holds a 3 MiB fp32 state per slot (48 value heads × 128 × 128), so every decoded token read and wrote about 288 MiB just for that copy, in one extra dependent dispatch per layer.

Each scan thread reads and writes only its own state words. So when a round's segments keep their slots, the chain scan can run on the state where it is, and the publish step copies only the conv history. Plain decode steps, verified draft chains and prompt chunks are all such rounds: round_plan builds every runner plan with next_slot == slot, including the multi-slot runners from #578. A plan whose slots move still takes the old scan-and-copy path, so forwardChain stays correct for any plan validatePlan accepts.

The output bits don't change: the scan does the same arithmetic on the same inputs, and only its destination changes. #568 removed a similar copy in the qwen3_5 port before the 27B moved to this engine. This removes the copy in the qwen27 engine.

One file changes: zig/src/families/qwen27/gdn_pool_gpu.zig, +19/−7. The round's alias check (Buffers.check) runs before next_state is pointed at state; the comment at that line gives the reason.

Exactness

All on one Apple M5 Max, against origin/main e7c523eb (1.0.7) built the same way.

check	main	this branch
qwen27_gate.py check (4 prompts × 256 greedy tokens, --chunk 128 --compare-chunk 16)	copy and arithmetic equal the Python 0.6.6 oracle 256/256; story and code leave it at BF16 near-ties (exit 1)	tokens, margins and positions equal to main for all 4 prompts; prompt_state_byte_equal 4/4
drafted == plain (tf-qwen27-dflash-run --prototype q4 --compare-plain, DFlash2)	4/4	4/4; rounds 89/30/80/32 and verified rows equal to main
qwen27_http_gate.py (raw/chat × plain/draft)	16/16	16/16, token SHAs equal to main
concurrent (bench_concurrent.py --levels 1,4 --alone --serial, code and chat)	queued replies equal solo	12/12 at 4 streams, serial 1/1, equal to main
tf-qwen27-streams (real weights) at 4 and 8 streams	pass	pass; output identical to main's
model-free device checks: tf-qwen27-synthetic, -lanes-check, -served-check, -dflash-served-check, -dflash-synthetic	exit 0	exit 0; every log byte-identical to main's

The gate exits 1 on main too. Story (token 68) and code (token 21) leave the oracle at a BF16 near-tie: same two contenders, both margins within one BF16 step.

Speed

Same Mac and settings throughout, with no other model process running. Builds of origin/main e7c523eb and of this branch, run alternately (ABBA).

Plain greedy decode (tf-qwen27-run, 256 tokens, 5 interleaved pairs per context). A local patch adds per-step timing to both builds; it isn't part of this PR. Prompt, decode and token hashes are equal in every pair.

prompt	main	this PR	tokens/s	pairs faster
2048 tokens	35.03 ms/token	33.69	+4.0% (paired median +3.7%)	5/5
16384 tokens	37.44	36.50	+2.6% (paired median +3.1%)	5/5

At 16K both builds' spreads are about 1% and their ranges don't overlap.

Served (native server, 2 sessions per build, alternated main, e1, attn, attn, e1, main for plain and main, e1, attn, drafted, drafted, attn, e1, main for drafted). These show no change I can claim: every difference is inside main's own session-to-session spread.

• Plain (tools/bench_openai.py --tokens 256 --reps 3 --temperatures 0, --no-drafts): fibonacci 29.72 → 29.74 tok/s, chat 29.20 → 29.80 (main's two sessions differ by 2.3% on chat).
• Drafted (tools/zig/qwen27_served_bench.py, DFlash2 4-bit, 3 repeats): copy −0.1%, arithmetic +0.2%, story +0.6%, code −0.5%. Draft rounds and tokens per round are identical on every build (2.88, 8.53, 3.20 and 8.00).
• Cold prefill (fresh KV, --prompt-cache-gib 0): 2k +0.3%, 8k −1.1%, 32k +2.5%; main's two 32k sessions are 0.5% apart but its within-session repeats span 4.5%, so no prefill change is claimed. 64k wasn't run.

mlx_lm 0.32.0 (mlx 0.32.3), plain, on the same two prompts: 30.86 and 30.82 tok/s.

Host checks

• zig build test test-golden -Dcpu=apple_m1: 143/143 steps, 1290/1292 tests passed, 0 failed, 2 skipped. test-golden: 309/319 equal, the 10 known differences, 0 unexpected. All the same as main.
• python3 tools/zig/lean_check.py: 534 problems on main and 534 on this branch, 0 new.
• zig fmt --check is clean on the touched file.

Setup: Apple M5 Max (40-core GPU, 128 GB), macOS 27.0 (26A428), Xcode 27.0, Zig 0.17.0. Checkpoint TensorFold/Qwen3.8-27B-MLX-4bit @ 22d8d538; drafter z-lab/Qwen3.8-27B-DFlash2 @ 50307d4c, 4-bit. Not run: CUDA and ROCm (Metal-only change), and Apple M1–M4.

[ashhart]

Thank you, @jkuepker. Three careful speed changes to the 27B on Metal, each with its exactness argument written down, and all three hold on our machines: the in-place DeltaNet scan (#711), the decode attention without the tail kernel (#712), and the draft-side changes (#713). Your ten commits landed in 1.0.9 under your name, rebased onto 1.0.8 (only #713's first commit changed: three cb.failure() != null checks became the gpuFailed(...) helper that 1.0.8 added). Each PR's head is merged into the history with no change to the tree (7098195be, fed0e24d1, 4a8002d8d), so GitHub shows all three PRs as merged.

What we ran, with all three together against the tree they started from:

• Exactness. On an M3 Ultra and an M5 Ultra, 17 cases (9 greedy, thinking included, and 8 seeded sampled, 96 tokens each): drafted equal to "draft": false on both builds, and the same tokens across builds. On the M5 Ultra, 9 prompts' full raw token arrays were equal across builds, and each drafted reply's final state equalled its plain run's. On the M3 Ultra, a conversation with a regenerated, an edited and a deleted turn gave the same replies with the prompt cache on and off, and 2 and 4 concurrent streams gave each stream's solo reply.
• Speed, one stream, 256 tokens. M5 Ultra (four runs a build): greedy code 406.1 to 417.0 tok/s, prose 100.6 to 103.6; seeded code 294.0 to 316.9, prose 86.2 to 89.0. M3 Ultra (six runs): greedy code 150.6 to 154.7, prose 53.3 to 54.8. Prompt processing at 16k, 32k and 64k tokens on the M3 Ultra was 0.2 to 0.4% faster.
• On the release. On an M5 Ultra, the 17 cases gave the same tokens on the release candidate as on 1.0.8. In the 1.0.9 release gate, against 1.0.8, an M5 Max gave 1.0.8's tokens in all 17 cases and 12 chat turns, resumed equal to fresh, and an M3 Ultra did too while decoding prose 2.2 to 2.8% faster and code level to 1.5% faster (a fresh server's code 0.8 to 2.0% faster), with prompts 0.4 to 0.7% faster.