Flash Next: shared rounds by default on native Metal

[cwschroeder]

Flash Next currently queues requests behind its single-reply host with the default flags. This branch routes default --parallel auto to shared lane rounds with drafting, choosing up to 16 sessions from the available memory. A lone greedy request uses GPU-side rounds and yields when another request arrives. If fewer than two sessions fit, the warmed engine transfers to the single-reply host without reloading or warming again.

Explicit --parallel 1, learning and paired serving retain their supported paths. Shared sessions remain single-Mac.

Candidate ef85c8b0b8402fc5186a8123fcf2a4ef87042acd, upstream base 46a08c18608153bfc566703a88acc883bc6140f4:

• Shared sampled windows use the upstream GPU sampler with each stream's seed, settings and absolute positions. The former host sampler remains a historical reference.
• Residency includes runner buffers, all session-owned buffers, hidden rows and later snapshots. The validated host load limit now also bounds initial residency and later joins, with a 2 GiB margin inside it. The default remains 70% of RAM; an explicit load limit can raise or lower it. GLM's opt-in residency likewise honors its explicit validated limit. Cache overrides do not bypass this bound.
• Cached replies retain background warm turns. Residency callbacks detach before resources are freed; terminal streams are swept before their jobs are freed. Failed opens stop and join a spawned follower before releasing its engine.
• The history retains all 35 prior contributor commits and nine merge nodes, including Seth's and Bob's work.

Local validation on M5 Max/macOS 26.6.2 (25G83), Zig 0.17.0:

• Complete No GPU workflow commands pass, including eight served-helper regressions; frozen answers have zero unexpected differences.
• Whole-tree format and lean checks pass: 517 inherited findings, zero new.
• zig build native test-native tf-flashnext-batch -Dcpu=apple_m1 --summary all: 51/51 steps, 29 tests passed, one skipped. Restoring the old residency rule makes both new CPU regressions fail.
• Astra reviewed the source changes without a remaining material blocker.

Native SHA256: 568500c244ee1cbfb21e1bee4841177cc325eaa746a49e1355718be9bd04e6bb.

External hardware receipt on the preceding d0f61385 head: Seth reports model parity, two streams outgrowing their sessions, eight concurrent streams and lone-driver handoffs on an M5 Ultra/256 GB, using the 6-bit checkpoint at 45032a1. That run exposed the residency-limit mismatch fixed here. The new residency policy still needs a hardware retest.

This PR remains a draft. All four upstream checks and all eight fork checks pass on this head. The compiled synthetic --sampling-only and seeded FZ_SAMPLE=1 model gates have not run. Fresh idle recovery, cold prefill and performance receipts remain open; the external results above apply to their pinned earlier head. No deployment is claimed.

[ashhart]

Thank you, @cwschroeder. Running Flash Next's streams through the shared lane core is the right shape, and the receipts are careful (180 paired comparisons, the chained-MTP draft waits from 256 to 84).

It isn't in 1.0.6 for two reasons. Concurrency on our engine is the default path for every family, not a flag, so --parallel N on Flash Next should go through the shared rounds with sampling included, not only greedy. And it needs our gate on a 256 GB-class Mac with the release checkpoint, which we couldn't schedule in tonight's landing window. We'd like to work it into the default path with you after 1.0.6: the per-stream state, admission between rounds and slot release you built carry over directly.

[cwschroeder]

Thanks, @ashhart. With 1.0.7 out I'd like to take you up on working this into the default path. A proposal for how to cut it, so each piece lands on its own:

1. Flash Next on the shared lane host, as the default path. Every --parallel value, 1 included, goes through api.LaneHost and the shared rounds. greedy_only goes away and sampling runs through the lane core's keyed draws, as for Nemotron. The single-reply driver stays only as long as you want it as a reference for the gate. Speed-up mode stays out of this PR: a pair keeps refusing --parallel above 1 until the one-Mac path has passed.
2. The batch MTP chain (FZ_BATCH_MTP_CHAIN) as a separate PR on top, since it is an experiment with its own receipt.
3. Receipts in the PR description, not under docs/receipts, and files kept under the 600-line line.

Structured output and tool_choice: "required" would then come from the lane core (structures = true). Should that go in the same PR or follow it?

Three questions before I start:

• Is anyone on your side already moving Flash Next onto LaneHost? I don't want to build it twice.
• For the gate: will you run it on your M5 Ultra with the release checkpoint? Would a 256 GB M3 Ultra receipt with TensorFold/Qwen3.8-Flash-Next-MLX-6bit-MTP help alongside it?
• Separately, a small PR is coming that lets Flash Next's lane projection load on Macs without tensor units (M1 to M4), which today stop at TensorUnitsRequired. It's independent of this one.

[BobClawblaw]

A 256 GB M5 Ultra receipt with the release checkpoint, since the gate asks for one.

Build. 77ebc3fc merged onto main e7c523eb (v1.0.7). Three conflicts, all mechanical: README.md (both paragraphs kept), zig/src/native/cuda.zig (main's startup line), zig/src/tensorfold.zig (both import lists). zig build native -Dcpu=apple_m1, Zig 0.17.0, macOS 27.0.1, Mac Studio M5 Ultra, 256 GB. Checkpoint TensorFold/Qwen3.8-Flash-Next-MLX-6bit-MTP at 45032a19.

Serve. The same binary twice, one process at a time, greedy, thinking off, prompt cache off:

tensorfold-native serve $M --parallel 1 --context 65536 --temperature 0 --no-thinking --max-tokens 4096
tensorfold-native serve $M --parallel 4 --context 16384 --prompt-cache-gib 0 --temperature 0 --no-thinking --max-tokens 4096

--parallel 4 --context 65536 refuses with the native Flash Next batching engine cannot open (OverMemoryLimit): the engine logs 173.3 GiB in use once loaded, and the 70% share of 256 GB leaves about 4 GiB for slots, under four sessions' state at 65k. At 16k it opens.

Decode, bench/ruler.py from the Spark recipe (greedy, 256-token replies, a distinct prompt per stream, median of 3 runs, tokens a second):

Build	Streams	prose	code	structured	list	first token
--parallel 1, 1 request	1	144.6	205.9	172.3	194.4	0.09-0.16 s
--parallel 4, 1 request	1	131.0	180.0	152.7	183.3	0.12-0.18 s
--parallel 4, 4 requests, each	4	52.8	80.7	64.4	78.6	0.33-0.36 s
--parallel 4, 4 requests, aggregate	4	213.8	321.8	254.2	314.8

The --parallel 1 row equals the 1.0.7 release build on the same Mac (144.5 / 206.2 / 172.3 / 194.3). Four shared streams give 1.5-1.6x the lone stream's tokens, with a third of a second to the first token where the serial driver queues them 2-3 s; the shared path costs a lone request 6-12%.

Exactness on the --parallel 4 server, the Spark recipe's quality/suite.py: concurrent (four streams together against each alone) 12/12; drafts (drafted against "draft": false) 12/12, 137 against 81 tok/s.

Receipts (ruler JSON, server logs, suite output): qwen38-flashnext-mlx6bit-m5-ultra/evidence/2026-10-11-pr606-parallel. The same repository has the release build's full ruler, prompt passes to 261k and the suite with thinking off, for the baseline.

Happy to rerun on the default-path cut when it exists, and at larger windows if the slot budget moves.

[BobClawblaw]

Follow-up: four sessions at the full 262k window, with one change to the slot budget.

Why 65k refused. The batching host sizes slots from 70% of RAM - footprint once loaded - margin. On this Mac the engine logs 173.3 GiB in use once loaded, so 179.2 - 173.3 - 2 leaves about 4 GiB for sessions, and a 262,112-token session holds 7.5 GiB of attention state (Session.bytes). Four of them need 30 GiB, which the Mac has (256 GB) and the share does not.

The change, one commit on your branch: a5e409e3 on BobClawblaw:flashnext-batch-load-limit, 48 lines, based on 77ebc3fc. FLASHNEXT_LOAD_LIMIT_GB is read as the GLM loader reads GLM_LOAD_LIMIT_GB: a load limit in GB, capped at the GPU's recommended working set, the same warn past the 70% default and the same refusal for text that is not a number above 0. cache_fit.ceiling() holds the arithmetic with a unit test; the host uses it for both checks (before Backend.init and after); the OverMemoryLimit message now names the three ways out (fewer streams, a smaller --context, or the limit). Unset, nothing changes. zig fmt --check clean, zig build test passes, built and run at both 77ebc3fc and a merge onto main e7c523eb. Cherry-pick it, or I can open it as a PR against your branch, whichever suits.

With it, FLASHNEXT_LOAD_LIMIT_GB=225 --parallel 4 --context 262144 --prompt-cache-gib 0, same Mac and checkpoint as above, four 262,112-token slots open (Flash Next: 4 session slots), and the ruler is the 16k result again:

Streams	prose	code	structured	list	first token
1	129.6	177.3	150.2	180.0	0.18-0.23 s
4, each	51.9	76.9	60.9	76.5	0.38-0.41 s
4, aggregate	212.1	313.2	247.9	306.5

concurrent 12/12 and drafts 12/12 (132 against 77 tok/s) on that server. Four 217k-token prompts sent at once all answered their needle, first tokens at 59, 118, 180 and 241 s (prompt passes stay serial across streams, about 3,700 tok/s in total); a 261,300-token prompt alone read in 71.5 s, as on the single-reply driver. Free memory sampled every 5 s never went under 32 GiB of 256.

Receipts: evidence/2026-10-11-pr606-parallel, the -256k files.

[BobClawblaw]

Sampling for the shared sessions, if you want it on this branch: 1874ffe0 on BobClawblaw:flashnext-batch-sampling, one commit on 77ebc3fc, 26 lines plus the sampler. It's the shared-session half of #719, which samples on the single-reply driver.

What it does. The batching host stops reporting greedy_only. A stream with a temperature draws each token on the host from the head's bf16 logits with the lane core's keyed sampler, the same draw the Qwen3.5 Metal backend makes, so temperature, top_k, top_p, min_p and seed work:

• First token: drawn from the prompt pass's last row, right after the last chunk (Backend.prefill).
• Each round: each stream's window rows are drawn from t.logits at that stream's positions, overwriting the GPU's argmax picks, so a draft is kept only when it equals the drawn token. Greedy streams keep the GPU picks untouched, so greedy and sampled streams can share a round.
• families/flashnext/sample.zig: the same file as in Flash Next on Metal: sampled decoding (temperature, top_k, top_p, min_p, seed) on the host #719. It does a one-pass top-k over bf16 keys, then lanes.sampling.choose, and has a host test against qwen3_5's draw. Whichever lands first, the other merges it unchanged.

Receipt, Mac Studio M5 Ultra (256 GB), macOS 27.0.1, TensorFold/Qwen3.8-Flash-Next-MLX-6bit-MTP at 45032a19, --parallel 4 --context 16384 --prompt-cache-gib 0 --temperature 0 --no-thinking:

• Greedy token_sha on four prompts equals the release build's.
• At temperature 0.7: the same seed gives the same reply, another seed differs, and drafted equals "draft": false (124/129 against 90 tok/s).
• Four sampled streams sent together each equal their solo replies. Two greedy and two sampled streams mixed in one round each equal their solo replies.
• A 300-token reply: greedy 123.1 tok/s, sampled at 0.7 122.6 tok/s, alone on the shared path.
• zig build test-flashnext-host 53/53, zig fmt --check clean.
• The one check that "fails" is mine: 20 seeds at temperature 1 asking for a random number from 1 to 10 all answer 7. The model puts nearly all its weight there; my fork build gave the same.

My fork serves this at --parallel 4 with a 262k window. Cherry-pick it, or I can open it as a PR against your branch.

[cwschroeder]

Thanks, @BobClawblaw. The release-checkpoint receipt on your 256 GB Mac, plus the sampling and load-limit changes, give us concrete additions for #606.

Please open two separate PRs against cwschroeder/TensorFold:feat/flashnext-continuous-batching: sampling 1874ffe0 and the load-limit override a5e409e3, preserving your commits. Sharing the same sample.zig with #719 makes sense. I will rebase the integration onto current upstream before building or qualifying it. I can review the additions against your recorded greedy, seeded, mixed-stream and long-context cases.

@ashhart, the default shared-lane path remains the target you set. I will keep the batch-MTP-chain experiment separate from that integration. The complementary hardware target here is M3 Ultra / 256 GB with the SIMD fallback in #679; I have not qualified these two additions on that hardware yet.

[sethforprivacy]

Posted by Claude (Anthropic's AI assistant) on behalf of @sethforprivacy, as part of our local-inference tuning work on a Mac Studio M5 Ultra.

A data point for making the shared path the default: as of today our production lane serves this branch's shared sessions. Before, it ran the Python build (0.6.5 plus our patches, --parallel 16). We needed a few additions to match what that build served, and one of them bears on BobClawblaw's single-stream row (--parallel 4 with one request: 131 / 180 tok/s prose / code against 145 / 206).

• Machine: Mac Studio M5 Ultra, 256 GB, macOS 27.0, TensorFold/Qwen3.8-Flash-Next-MLX-6bit-MTP @ 45032a1.
• Build: BobClawblaw's flashnext-video (ddc17bb: this branch, the load limit, images and video), Flash Next on Metal: sampled decoding (temperature, top_k, top_p, min_p, seed) on the host #719 and 1874ffe0 for sampling, and one commit of ours: add3da3 on sethforprivacy:flashnext-lane-parity. Its message covers each part and how we checked it.

One request alone at the single-reply driver's speed. We gave the batch host a lone driver, LaneHost's api.Lone, the way the 27B's host has one.

• Prompt: a lone text request runs the shared sessions' own prompt pass into the engine's session, so the states the prompt cache keeps are the shared path's.
• Decode: it then decodes on the engine's GPU-side rounds, the single-reply driver's loop, split out as Engine.decode.
• Handoff: when a second request arrives, the stream's state is left as a shared round leaves it. For greedy, the round already in flight has copied the kept DeltaNet states and n-gram tail and absorbed the head; a sampled reply runs that round start itself. The stream then moves to the smallest free session that holds it, and the lane core takes it on.
• What stays on the shared rounds: images, logprobs, grammars and thinking budgets.

Decode alone, 512 tokens (code / prose / edit)	tok/s
1.0.7, single-reply driver	215 / 136 / 197
Shared rounds, one stream	189 / 124 / 181
Shared rounds, a lone stream's depth priced from timed forwards	209 / 127 / 203
Lone driver, handing off to the shared rounds	213 / 134 / 195

The replies are bit-identical across paths:

• The decode replies' shas equal the shared rounds' replies.
• Concurrent equals solo at 3/3 and 8/8, with solo now on the lone driver.
• We handed seven replies over mid-reply, sending the second request about 400 characters in: three greedy, and three sampled at 0.7 with a seed. Each equals its solo reply.
• An 18,893-token greedy reply was handed over at 3,041 tokens, then outgrew its 16K session and moved to a 32K one. It equals a run that never left the shared rounds.

So --parallel 1 going through LaneHost need not cost single-request decode.

The rest, measured as a sampled ladder. Temperature 0.7, ~200-token prompts, 400-token replies, aggregate tok/s and mean first token:

Streams	Branch as we found it (3 × 262K sessions)	With our commit	Python lane (0.6.5 + patches)
1	110, 0.21 s	113, 0.17 s	109, 0.19 s
2	73, 2.35 s	153, 0.31 s	133, 0.44 s
4	102, 4.22 s	196, 0.50 s	177, 0.68 s
8	116, 8.64 s	232, 0.90 s	213, 1.35 s
16	(3 sessions fit)	255, 1.73 s	259, 2.46 s

• Sessions sized to the traffic. FLASHNEXT_SLOT_CONTEXTS=65536,32768x3,16384x11 puts the sessions past the engine's own 262K one in 11.6 GiB, 16 streams in all; three whole-window sessions took 15.4 GiB. A stream's session holds its prompt plus its reply's next FLASHNEXT_SLOT_GROW tokens. One that outgrows it moves at a round's end to a free larger session (Session.moveTo, a GPU copy, the same bits), or waits for one. This sidesteps the OverMemoryLimit BobClawblaw hit at 65K: our production prompts run p50 8.3K, p95 52K.
• Priced rounds. The batch backend gave the lane core no costs, so every shared round filled 16 rows with drafts however few landed. Shared forwards are now timed at load (9.9 ms at 1 row, 32.2 at 16), each draft past a stream's first is priced (a lane-core draft_step_ms), and every stream's drafts run in one command buffer.
• The first token after idle. Most of the old first-token times were macOS unwiring idle GPU memory. That also hits 1.0.8's single-reply host, so it went up as its own PR against main: Flash Next keeps its buffers wired between requests: a residency set, and a keepalive tick the GPU runs #732.
• Prompt cache. The cache is 16 GiB, and with --learn-dir an evicted state that its conversation has not moved past is spilled to disk and read back on lookup (64 GiB). That matches what the Python lane's --spill-gib did.

Happy to open any of these as PRs against feat/flashnext-continuous-batching, or wherever the default-path work lands. The lone driver and the priced rounds matter most for a default path.

[BobClawblaw]

@cwschroeder opened as asked, each with its one commit unchanged against feat/flashnext-continuous-batching:

• Flash Next shared sessions sample: a request's temperature draws each token on the host from the head's logits cwschroeder/TensorFold#7: sampling, 1874ffe0
• Flash Next shared sessions: FLASHNEXT_LOAD_LIMIT_GB lets the slots take the server past 70% of RAM cwschroeder/TensorFold#8: the load-limit override, a5e409e3

They don't depend on each other and merge cleanly in either order. #7's description has the greedy, seeded and mixed-stream receipts; #8's has the long-context ones. Neither has been run on M1 to M4.

[BobClawblaw]

@sethforprivacy thanks for add3da3. I tried it on another M5 Ultra (256 GB, macOS 27.0.1, same checkpoint at 45032a19) with my recipe's settings: --parallel 4 --context 262144 (four full 262K sessions, FLASHNEXT_LOAD_LIMIT_GB=240), --prompt-cache-gib 24 --prompt-cache-over-cap, vision and video on.

There, one request alone was erratic and mostly slow, 30 to 207 tok/s on identical requests. It traces to the residency set, not the lone driver.

One request alone, 256 tokens, tok/s. Code and prose alternate, eight requests each run:

add3da3, these settings	code	prose
as is	202, 64, 206, 71	37, 40, 59, 61
FLASHNEXT_NO_RESIDENCY=1	188, 207, 206, 206	136, 136, 136, 136
FZ_LONE_UNPRICED=1	73, 207, 75, 207	137, 60, 62, 65
--parallel 1 (the single-reply host, no lone driver)	207, 207, 207, 206	63, 77, 54, 136
--context 65536 or 16384, everything else as is	207	136 to 137

• The work doesn't change. Fast and slow runs of the same request have the same rounds and accepted drafts, for example 54 rounds and 204/304 drafts for the code reply. sample on a slow reply puts the decode thread in waitUntilCompleted at engine.zig:783 for 1,429 of 1,465 samples. The GPU-side rounds themselves run about three times slower.
• Both hosts' GPU-side rounds are hit. The lone driver and --parallel 1 slow down. With FLASHNEXT_NO_LONE=1 the shared rounds stayed fast with the same set (212 / 131 tok/s).
• It follows the set's size. The log reads 5140 buffers (195.8 GiB) held resident at 262K, and those runs are slow. The set is 178.8 GiB at 65K and 172.8 GiB at 16K, and those runs are fast. My guess is a macOS limit near 75% of RAM (192 GiB) for memory a set keeps wired. I haven't confirmed it. Your sized sessions keep the whole engine near 200 GiB, which is probably why you didn't see this.

With the set off, the rest of the commit is a clear win here. One stream on the lone driver gives 207 / 136 tok/s against 175 / 128 on my current build. Its greedy and sampled replies equal the shared rounds' (4/4 greedy, 8/8 sampled across three settings before I stopped to debug).

A fix might be to hold only the engine's own buffers and the tower, or to skip the set past a size. I haven't tried either. I can rerun any variant you want on this configuration.

[cwschroeder]

@BobClawblaw, #7 and #8 are received. I am reviewing them separately, including the seeded sampler and the memory-limit arithmetic. Your original commits and hardware receipts will stay attributed to you; M3 qualification is still outstanding.

@sethforprivacy, please open separate PRs against cwschroeder/TensorFold:feat/flashnext-continuous-batching for the lone-driver handoff, the measured round costs, and the sized/growing sessions. If one needs another, make that dependency explicit. Those are useful pieces for moving #606 toward the default shared-lane path Ash requested. Keeping the residency change in #732, and vision/cache-spill work separate, will let us review and qualify each change without bringing the full bundle along.

Bob's large-window counterexample should be part of the residency gate: both first-token latency after idle and sustained decode at the larger working set. His measurements isolate the residency setting in that configuration; the suggested 75% threshold is still a hypothesis. Please link that receipt from #732 when addressing it. For the handoff/session PRs, please retain the greedy and seeded solo/concurrent comparisons, the mid-reply handoff, and the growing-session cases in their descriptions.

[ashhart]

Hello @cwschroeder and @sethforprivacy,

Thank you for pricing shared Flash Next rounds from forwards timed at load, and for running every stream's drafts in one command buffer.

The default server still serves one reply at a time. openFlashNext opens the batch host only when --parallel is fixed and above 1 (metal.zig:317). auto and --parallel 1 still take flashnext.open (metal.zig:331). Concurrent requests need to share lane rounds on that default path, with drafts on. FZ_BATCH_MTP_SERIAL=1 can stay the opt-out on the batch host (backend.zig:310).

A push that serves shared rounds from the default flags is the update we need.

Thank you again.

[sethforprivacy]

Posted by Claude (Anthropic's AI assistant) on behalf of @sethforprivacy, as part of our local-inference tuning work on a Mac Studio M5 Ultra.

@ashhart thanks, understood: shared rounds from the default flags, drafts on, with FZ_BATCH_MTP_SERIAL=1 kept as the opt-out. Some notes for the default switch, from running the batch host as our production server on an M5 Ultra:

• A lone request has to keep the single-reply speed. On the shared rounds alone, a request decodes at 187 / 125 / 179 tok/s (code / prose / edit, 512 tokens). The single-reply host does 215 / 136 / 197. Flash Next: a lone request on the engine's GPU-side rounds, handed to the shared rounds when a second arrives cwschroeder/TensorFold#11's lone driver runs a lone request on the engine's GPU-side rounds and hands it to the shared rounds when a second request arrives: 213 / 136 / 196, with the same replies. So the default path should include TensorFold 0.3.1: say plainly when a checkpoint has no recipe #11.
• auto needs a session count from memory. The batch host wants at least two sessions, so auto would take as many as flashnext_batch_meta.fit allows, and fall back to the single-reply host when fewer than two fit. Flash Next shared sessions sized to the traffic: FLASHNEXT_SLOT_CONTEXTS, and a stream that outgrows its session moves cwschroeder/TensorFold#10's sized sessions make that count larger at the same memory: on our 256 GB Mac, 16 streams in 11.6 GiB, where three whole-window sessions took 15.4 GiB.
• What stays on the single-reply host: the batch host refuses --speed-up (two Macs) and doesn't serve --learn. With either set, the default would keep flashnext.open.

One question: should an explicit --parallel 1 stay on the single-reply host? It asks for one reply at a time. Or do you want it on the batch host too, with a single session?

@cwschroeder, #9 is merged, and #11 and #10 are mergeable on your branch. Do you want to make the routing change in #606 after they land, or should we open it against your branch as a fourth PR? We can do it either way.

[cwschroeder]

@ashhart @sethforprivacy I will take the default-routing change and its routing tests in #606. I am checking #10's new migration-failure and lane-cleanup changes and #11's merge with #9, then integrating against current main before pushing the combined update.

For auto, the batch host should size sessions from available memory, with drafts on, and keep the single-reply path when two sessions cannot fit. The lone driver from #11 needs to be included to protect single-request speed. I will keep the existing learning and two-Mac paths on their supported host.

For explicit --parallel 1, I will preserve one-reply-at-a-time behaviour pending your answer to Seth's question. Default auto is the routing gap this patch will close. FZ_BATCH_MTP_SERIAL=1 remains the serial-draft escape hatch.

The new fork heads have green Linux host/format checks. Growing-session model parity on the corrected head is still outstanding, and the Mac split-checkpoint test remains an open validation issue, even though it reproduces on the previous base. I will keep those gates explicit.

[ashhart]

Thank you, @cwschroeder, for pinning down the explicit --parallel 1 case, and @sethforprivacy for the shared-round work.

Explicit --parallel 1 will keep one reply at a time, with later requests waiting and drafts still on. auto will use the batch host sized from available memory, with drafts on and the lone driver for a single active session; it falls back to the single-reply host when fewer than two sessions fit. --speed-up and --learn stay on the single-reply host, and FZ_BATCH_MTP_SERIAL=1 remains the opt-out.

The split-checkpoint test passes in the 1.0.12 candidate's Mac suite. “checkpoint tensor text.weight stays in its file” is expected: the test deliberately checks DeferredTensor, then reads the weight through a Stage and compares it with the fully loaded tensor.

Posted by Ash's AI Assistant Synthena

[cwschroeder]

@ashhart @sethforprivacy the combined update is pushed at 67803da4, on current main c7137bb3, and follows the routing you confirmed. Fork #10 and #11 are integrated with their original commits preserved.

The low-memory fallback transfers the already loaded engine to the single-reply host and retains its footprint limit. It does not load another model copy. I also added host-level regressions for a cancelled stream failing a paused peer, prefill cancellation, cleanup with a failing trace allocator, and a shared keep error after one stream has finished. The unfixed cancellation cases time out; the unfixed keep case aborts. The corrected cases pass and leave no live streams or duplicate terminal events.

On an M5 Max, macOS 26.6.2, Zig 0.17.0: native build, whole-tree formatting, host test build, server goldens and Cortheon fuzz checks pass. The lean comparison is 527 inherited findings on each side, zero new. The expected DeferredTensor checkpoint test passes on this current-main integration too. Current-head remote CI is running; formatting has passed.

I kept #606 as a draft for the remaining gates. For your next available lane window, please pin this combined head for default-flags solo/concurrent token parity, lone/shared handoff and growing-session migration. Earlier speed receipts remain attached to their older commits; we still need the new head's model and performance receipts.

[cwschroeder]

@ashhart @sethforprivacy I added the served-gate coverage for the default route at 15d9bec6, still on main c7137bb3. The native engine sources and binary are unchanged from 67803da4, whose four GitHub checks passed.

The recipe now runs 1,default,auto,4 at the same configurable context. default omits --parallel; auto passes it explicitly. With --require-shared-rounds, each shared block needs at least one logged multi-stream GPU forward during its requests. Running-request counters alone cannot pass that check, and startup or late log lines are excluded. That option enables batch logging, so use a separate run without it for timing.

Eight model-free regressions pass, including repeated auto blocks, retained failure receipts and compatibility with the MTP-chain helper. Astra caught a log-interval boundary issue; the new negative test failed before the fix and passes after it. The regressions now run in the No GPU workflow. Local whole-tree Zig tests, native build, formatting, server goldens and fuzz checks pass; lean has zero new findings.

For the next hardware window, please use this head and the updated recipe. No model or speed receipt is claimed for it yet; #606 remains a draft.

[cwschroeder]

@ashhart @sethforprivacy Draft update at 8cee8efd, rebased onto main 46a08c18 with the contributor authors and nine merge nodes retained.

The new main samples single-request replies on the GPU. Shared windows now use that same tf_sample_full kernel instead of the historical host sampler, with disjoint absolute-position ranges and per-window seeds/settings. The serial encoder orders sampling after the target head, before picks are read. This aligns the sampler arithmetic; full route parity still needs the model gate.

Shared residency includes all fixed session buffers and the hidden plane, plus later snapshots. Its 70-percent limit remains separate from the configured load cap. The integration also retains the new legacy Learn/disk/sampling paths and ports sweep-before-free cleanup to the extracted host end paths. Removing that sweep breaks the existing failed-keep CPU regression on this tree.

Local Zig 0.17.0 host/native/kernel/golden/fuzz validation passes 219/219 build steps; the exact-commit -Dcpu=apple_m1 build passes, as do eight served-helper CPU regressions, formatting and lean (zero new findings). Astra reviewed the integration and window offsets. All four upstream GitHub checks now pass on 8cee8efd: No GPU, whole-tree formatting, desktop and mobile. All six fork checks also pass on that same head.

For the next exclusive GPU window, the compiled harness adds:

zig-out/bin/tf-flashnext-batch --sampling-only
FZ_SAMPLE=1 FZ_N=128 zig-out/bin/tf-flashnext-batch MODEL_DIR /tmp/flashnext-prompts/*.json

The first uses synthetic logits and independent expected positions, including untouched greedy/spare rows. The second checks fixed seeds across legacy/shared, drafts, cancellation, slot reuse and prefix restoration. Neither has run yet. Served parity, migration/lone handoff, idle recovery and performance remain open; the PR stays a draft.

[cwschroeder]

@ashhart @sethforprivacy small follow-up at d0f61385: when auto fits only one session, the single-reply host now takes the already warmed engine. Previously that fallback ran the engine warmup twice. Legacy opening still refuses learning-on-pair before warming, then warms once.

The full local host/kernel/golden/fuzz checks, eight served-helper regressions, whole-tree formatting and the pinned native build pass. Lean has zero new findings. Astra found no material issue in the transfer or error ownership. Start-time savings remain unmeasured. All four upstream checks and all six fork checks now pass on d0f61385. The Linux run also passes the eight served-helper regressions and lean with zero new findings. #606 remains a draft for the GPU/model gates.

[cwschroeder]

@ashhart @sethforprivacy M3 Ultra receipt for ef85c8b0 (1.0.12 + #9/#10/#11, the default route and the residency fix): all pass.

Mac Studio M3 Ultra, 256 GB, macOS 27.0.1, Zig 0.17.0, zig build native -Dcpu=apple_m1 (native SHA256 2ad09c0c21dd3d2b3561ad129a4f1bf76329b971ee77924ef6c05824e4a7cd1e). TensorFold/Qwen3.8-Flash-Next-MLX-6bit-MTP, exclusive GPU, one server at a time.

• Suites: zig build test 204/204 steps, 3017/3027 tests (10 skipped); the split-checkpoint test passes here as well. test-core-lane 41/41 (14 layouts, 1904 bit-equal row/window pairs). Formatting passes.
• tf-flashnext-batch --sampling-only: PASS.
• Checkpoint gate, FZ_N=128, the recipe's fixtures (23, 66, 2,764 and 11,237 prompt tokens): greedy and FZ_SAMPLE=1 both PASS. Draft chains are exact at every depth; 4 streams give 512 exact tokens with drafts off and on; cancellation, slot reuse, prompt admission and a restored prefix in a dynamic-capacity slot pass. Four streams against legacy serial: drafts off 14.6 s vs 17.5 s (sampled 15.0 vs 18.0), drafts on 14.7 to 16.4 s vs 15.8 s (sampled 14.7 vs 15.6).
• Served gate (--parallel 1,default,auto,4 --context 16384 --require-shared-rounds): exact, shared rounds verified (61 logged multi-stream rounds per iteration in each shared block). default and auto open 9 × 16K sessions with a 176.4 GiB residency set; 4 holds 173.5 GiB.
• Residency and idle latency at our production settings (--context 262144 --prompt-cache-gib 8 --prompt-cache-over-cap):

Route	Startup	First token after 2 / 5 / 10 s idle (median of 3)
explicit --parallel 1	171.7 GiB kept wired, inside 70% of 256 GiB	0.141 / 0.138 / 0.143 s
auto (no flag)	fewer than two 262K sessions fit, so the single-reply host takes the warmed engine; 171.7 GiB wired inside the 179.2 GiB limit	0.137 / 0.134 / 0.139 s
auto with FLASHNEXT_SLOT_CONTEXTS=16384x6, FLASHNEXT_SLOT_GROW=2048	7 sessions (262144 + 6 × 16416), lone driver on; 175.2 GiB wired inside the 179.2 GiB limit	0.257 / 0.248 / 0.250 s

32768x3,16384x8 was refused at load with a clear message (7.9 GiB listed, 4.8 GiB fits beside the prompt cache).

• Parity across those three routes: a greedy probe with drafts off and on gives the same fingerprint as our current production build. The idle probe and the four fixtures (256 reply tokens) give identical fingerprints on all three routes, alone and started one second apart. On the sized host the staggered run logged one handoff (a lone stream at 89 tokens handed to the shared rounds). No GPU errors in any log.

One observation: on the shared host the lone path's first token after idle is about 0.11 s slower than the single-reply host on the same Mac (0.25 s vs 0.14 s). Not a blocker, but worth a look before auto serves single users by default.

Not covered here: session growth under load and long-reply handoffs. Seth's M5 Ultra receipt covers those on d0f61385, and ef85c8b0 changes only the residency policy on top of it.

[cwschroeder]

@ashhart @sethforprivacy two questions from a review pass on ef85c8b0, with the M3 Ultra numbers above.

What auto serves on a 256 GB Mac. At the native window, auto takes the single-reply host there: a second 262K session costs 7.68 GiB (Session.bytes), and with the model and the engine's own session at 171.7 GiB, less than that is left under 70% of RAM. The shared route starts at 128K or less (3.90 GiB a session), with sized sessions as in the third row above, or with a raised FLASHNEXT_LOAD_LIMIT_GB, as in Seth's runs at 214.7 GB. If that is the intended fallback, the README should say so for 256 GB Macs. If auto should share rounds there too, it could keep the engine's whole-window session and size the others, as FLASHNEXT_SLOT_CONTEXTS does by hand.

Prompt cache under auto. With --prompt-cache-gib unset, the batch host reserves nothing for the cache (reserved is 0 in flashnext_batch_host.open), so the sessions take the room first. The 16K served gate opened 9 sessions, which puts the room at 4.6 to 5.2 GiB on that Mac. The shared host's cache then gets what 8 extra sessions leave, under 1 GiB, where the single-reply host would give it about 5 GiB. The recipe says auto sizes sessions after "reserving the prompt cache", which holds only for an explicit --prompt-cache-gib. Should auto reserve the single-reply default before it sizes sessions, or should the recipe say the cache gets the remainder?

tf-flashnext-batch --sampling-only also passes on an M5 Max (macOS 26.6.2, Zig 0.17.0, -Dcpu=apple_m1), with zig fmt --check, zig build test and test-core-lane green on the same head.

Two small fixes follow on the branch as 5164380e; the engine and the receipts above are unchanged by it:

• --parallel 4 --learn served one reply at a time without a word, while --parallel 4 --speed-up refused with a message. Both refuse now, with a test.
• The OverMemoryLimit message gets its remedies back (fewer streams, a smaller --context or --prompt-cache-gib, a higher FLASHNEXT_LOAD_LIMIT_GB).

[ashhart]

Thank you, @cwschroeder, the two 15:31 questions and M3 Ultra receipt make the default-route tradeoff concrete.

We will reserve the single-host default prompt cache first when --prompt-cache-gib is unset, then size sessions from the remaining room and print both budgets at startup. The full-window fallback remains the warmed single host when fewer than two sessions fit.

We checked 5164380: flashnext_batch_host.zig:63 currently reserves zero for the unset case. We can fold in the decided reservation/logging fix, default/auto and low-memory controls, then run the final growing-session, long-handoff and seeded cache/draft comparisons. The cache choice is settled; your existing served and idle receipts remain useful for that integration.

Posted by Ash's AI Assistant Synthena