Flash Next shared sessions: FLASHNEXT_LOAD_LIMIT_GB lets the slots take the server past 70% of RAM

[BobClawblaw]

What this changes

FLASHNEXT_LOAD_LIMIT_GB lets the batching host's session slots take the server past the 70%-of-RAM share, as GLM_LOAD_LIMIT_GB does for the GLM loader. Unset, nothing changes.

• Why: the host sizes slots from 70% of RAM - footprint once loaded - margin. On a 256 GB Mac the engine is at 173.3 GiB once loaded, so 179.2 - 173.3 - 2 leaves about 4 GiB for sessions. One 262,112-token session holds 7.5 GiB (Session.bytes), so four need 30 GiB, which the Mac has but the share doesn't.
• The setting: a load limit in GB, capped at the GPU's recommended working set. Past the 70% default it gives the same warn the GLM loader does, and it refuses text that isn't a number above 0.
• cache_fit.ceiling() holds the arithmetic, with a unit test. The host uses it for both checks, before Backend.init and after.
• The OverMemoryLimit message now names the three ways out: fewer streams, a smaller --context, or the limit.

One commit, a5e409e3, on 77ebc3fc; it merges cleanly with the sampling PR in either order.

Receipt

Mac Studio M5 Ultra (256 GB), macOS 27.0.1, TensorFold/Qwen3.8-Flash-Next-MLX-6bit-MTP at 45032a19, FLASHNEXT_LOAD_LIMIT_GB=225 --parallel 4 --context 262144 --prompt-cache-gib 0:

• Four 262,112-token slots open (Flash Next: 4 session slots).
• Ruler (tok/s, prose / code / structured / list): 1 stream 129.6 / 177.3 / 150.2 / 180.0; 4 streams 212.1 / 313.2 / 247.9 / 306.5 in all.
• The Spark recipe's quality suite: concurrent 12/12, drafts 12/12.
• Four 217k-token prompts sent at once all found their needle, with first tokens at 59, 118, 180 and 241 s. A 261,300-token prompt alone read in 71.5 s.
• Free memory, sampled every 5 s, never went under 32 GiB of 256.
• Since then the same change has served a 262k window with four streams and a 24 GiB prompt cache (FLASHNEXT_LOAD_LIMIT_GB=240, about 219 GiB in use). Eight 262k slots are refused at that limit with the new message, as they should be.

zig fmt --check is clean and zig build test passes, built at 77ebc3fc and on a merge onto main e7c523eb. Receipts: evidence/2026-10-11-pr606-parallel, the -256k files.

Not run: M1 to M4 (including your M3 Ultra) and CUDA. Unset, the host's arithmetic is unchanged.

[cwschroeder]

The integration onto upstream 0efbd447 builds its Flash Next host tests: 58/58 pass, with sampling #7 included. I have one memory-budget contract question before integrating #8.

The slot checks use the new cap and reserve an explicit prompt-cache budget, but the later fit.budget(...) still plans the automatic cache against the independent 70%-of-RAM ceiling. For example, with 100 GiB RAM, a 60 GiB load limit and 50 GiB footprint after slot allocation, the slot check accepts 50 + 2 GiB margin. The automatic cache can then receive 16 GiB, taking the potential footprint to 66 GiB. An explicit 16 GiB cache is instead reserved against the load limit and rejected (50 + 2 + 16 > 60).

Is the limit intended to cover only model/session allocation or the eventual total including prompt cache? If the former, please document this distinction and add a test for automatic versus explicit cache. If the latter, the selected cap needs to reach cache planning as well. A larger load limit also does not by itself raise the explicit cache's separate 70% gate. This is a contract clarification, not a claim of an observed macOS stall.

The pure arithmetic is reviewable locally; M3 model and long-context qualification remain outstanding.

[BobClawblaw]

@cwschroeder good catch, and a slightly embarrassing one. I built a load limit, then left the prompt cache planner reading the old 70% ceiling, like a bouncer who checks IDs at the front door while the side door stays propped open. Sorry about that. The intent was always the whole footprint, so it's the latter.

The contract: FLASHNEXT_LOAD_LIMIT_GB (or 70% of RAM when unset) covers the server's eventual total: the model, the sessions and the prompt cache.

The fix: commit 03cc8741.

• One ceiling: cache_fit.budgetUnder(limit, ...) plans the cache under the same ceiling the slot checks use. fitUnder holds the arithmetic. budget and fit keep 70% of RAM for the other hosts, so they're unchanged.
• Automatic cache (--prompt-cache-gib left out): it gets what the sessions leave under the limit.
• Explicit cache: it's held to the same limit. A raised limit therefore raises its gate too. Past the limit only --prompt-cache-over-cap lets it through, as before.
• Docs and messages: the recipe doc and the limit's comment state the contract. The OverMemoryLimit message now names the cache and the limit instead of blaming "70% of RAM".

Test: a unit test in cache_fit.zig (under a load limit the automatic cache gets what the limit leaves, ...) uses your case: 100 GiB of RAM, a 60 GiB limit and 50 GiB in use after slot allocation.

• Automatic cache: now 8 GiB (60 − 50 − 2). The old 70% plan gave 18 GiB, not 16 as I first wrote it in the test, which only adds to the embarrassment.
• Explicit cache: 16 GiB is refused, 8 GiB is kept, and --prompt-cache-over-cap still passes 16.
• Raised limit: at 80 GiB, an explicit 16 GiB fits with no override.

zig build test: 1,146 passed, 2 skipped. zig fmt --check is clean.

Live, on the 256 GB M5 Ultra at --parallel 4 --context 262144, about 194.6 GiB in use once loaded:

FLASHNEXT_LOAD_LIMIT_GB	--prompt-cache-gib	Result
240	24, no override	serves; 24.0 GiB of 26.1 GiB free under the limit. Before this commit it needed --prompt-cache-over-cap.
240	unset (automatic)	serves; the cache takes 26.1 GiB, exactly what the limit leaves.
225	24	refused at startup with the new message.

The commit merges cleanly with #7 in either order. I moved the doc paragraph so it doesn't touch the sentence #7 rewrites. As you say, M1 to M4 and your M3 Ultra are still unqualified.

[BobClawblaw]

@cwschroeder one more thing for your integration onto upstream 0efbd447. Since ashhart#606's base, upstream rewrote cache_fit.zig: the automatic cache now stops at DEFAULT_CEILING (16 GiB), and budgetBeside is new. So 03cc8741 will conflict when you rebase. (That 16 GiB ceiling also explains your example's 16 GiB, where the old base gave 18. So you were right, and my "correction" was only right for the old base. One more for the apology tally.)

I've done the rebase so you don't have to: BobClawblaw:flashnext-batch-load-limit-0efbd447. The two #8 commits to take are:

1. acf45bb8: the load limit, as a5e409e3. One change on this base: budgetBeside's local ceiling (the default-ceiling note) is renamed by_default, because this commit adds a cache_fit.ceiling function and Zig refuses the shadow.
2. 0acefcf3: the cache contract, on upstream's planner. budgetUnder(limit, gib, over, spare, later, ...) sits beside budgetBeside, and fitUnder(cap, ...) beside fit. The automatic budget is the smaller of what the limit leaves and DEFAULT_CEILING. A given budget is held to the limit, and --prompt-cache-over-cap still passes it. Your example in the test: the automatic cache is 8 GiB under the 60 GiB limit, against DEFAULT_CEILING under 70% of RAM. Under a raised 80 GiB limit, a given 24 GiB fits with no override, and the automatic cache still stops at 16 GiB.

The branch's merge commit (016c98c8, ashhart#606's head merged onto 0efbd447) is only there so the two commits build. Use your own integration. Mine keeps upstream's cuda.zig line, unions the tensorfold.zig imports, and takes ashhart#606's README lines plus upstream's 27B drafter sentence.

Checks on the branch:

• Each commit builds on its own.
• cache_fit.zig tests: 5 passed on the first commit, 6 on the second.
• zig build test-flashnext-host passes, and zig fmt --check is clean.
• zig build test: 1,452 of 1,460 passed, 8 skipped. One step fails: core.checkpoint_metal's "a split checkpoint keeps later tensors in the file, …" logs checkpoint tensor text.weight stays in its file. It fails the same way on plain 0efbd447 (1,505 of 1,513 passed), so it's upstream's, not ours.

I haven't served this base: the arithmetic is unit-tested, and the same code path is what ran live on the old base. Happy to run it if you want a receipt on 0efbd447.