Repository navigation
CI receipt: default Flash Next shared rounds on current upstream - #2
Draft
cwschroeder wants to merge 40 commits into
Draft
cwschroeder wants to merge 40 commits into
cwschroeder wants to merge 40 commits into
Conversation
Pack session windows into one target layer loop behind the shared lane core. Isolate recurrent and attention caches, normalize kept states with a fenced GPU copy, preserve draft and snapshot semantics, and keep the single-reply and TP drivers as defaults. Include CPU metadata contracts, checkpoint exactness/lifecycle gates and served concurrency comparisons.
cwschroeder
force-pushed
the
feat/flashnext-continuous-batching
branch
from
October 10, 2026 17:47
5153d76 to
77ebc3f
Compare
cwschroeder
changed the base branch from
upstream/main-20261010-2f8284e
to
upstream/main-20261010-bec00ae
October 10, 2026 17:47
…ke the server past 70% of RAM, as GLM_LOAD_LIMIT_GB does for GLM
… token on the host from the head's logits The first token from the prompt pass's last row, each verify row from the window's logits, with the lane core's keyed sampler (top-k in one pass over bf16 keys, equal to qwen3_5's draw; families/flashnext/sample.zig, the same file as ashhart#719). Sampled replies are the same drafted and undrafted, alone and beside other streams; greedy replies are unchanged.
…ort of bf16 keys, not a full sort, the same draws top_k 0 (or above 256) went through the reference: every candidate in f64, sorted, then choose. Now a counting sort of the 65,536 bf16 keys gives choose's order (ids ascending within a key), top-p and min-p cut it as choose does with libm's exp and the same pairwise sum, and the keyed Gumbel draw runs in that order, passing over candidates whose noise is provably a nat short of the best score (one hash, no logs) and stopping once none can reach it. 248k vocabulary: 13.7-16.8 ms a row to 0.33-0.65 ms; the host test now also compares top_k 0, 300 and the whole row, top-p 1, min-p and wide logits.
…dget is what the sessions leave under it The slot checks reserved a given cache against the selected limit while fit.budget still planned the cache against 70% of RAM, so an automatic cache could take the server past the limit and a raised limit did not raise a given cache's own gate. cache_fit.budgetUnder plans the cache under the same ceiling (fitUnder; budget and fit keep 70% for the other hosts), with a test for the automatic and the given budget under a limit. The recipe doc, the limit's comment and the OverMemoryLimit message name the contract.
…nf noise wins as in the reference lanes.sampling.uniform's largest value, (2^53 - 1) 2^-53 + 2^-54, rounds to exactly 1.0 in f64, so the reference's Gumbel score is +inf there; the early stop assumed noise of at most 37.43 and could pass that candidate over (a seed that hits it: expected 4, found 0). Past a 38-nat shortfall the draw now still asks each candidate whether its uniform is 1 (one hash, no logs) and stops only once the best score is +inf; the pruning bound reads 1 - u from the f64 uniform itself (exact), so a u of 1 is never passed over. The reviewer's case is a host test, with the same candidate inside the bound and behind another leader. Speed unchanged (0.33-0.63 ms a row at 248k).
…_LIMIT_GB lets the slots take the server past 70% of RAM
…st's temperature draws each token on the host from the head's logits
…ching Conflicts: README.md keeps both the batching note and main's 27B wording; cache_fit.zig keeps FLASHNEXT_LOAD_LIMIT_GB's budget under main's 16 GiB automatic cap and deferred stream accounting; cuda.zig is main's; tensorfold.zig exports both sides.
…ream's drafts in one command buffer The batch backend gave the lane core no costs, so each shared round filled up to 16 rows with drafts however few landed, and each stream's head ran in its own command buffers (a wait per draft). - batch_costs.zig times, on the last session before any stream holds it, a shared forward at 1..16 rows (the fastest of four, after six warm-up windows) and a chained head step (absorb plus 8 drafts against absorb plus 1, in one command buffer). Backend.facts reports them: shared_costs for the allocator, and a lone stream's window_costs and mtp_step_ms for its depth (FZ_LONE_UNPRICED=1: the old heuristic). - The lane core prices each draft past a stream's first at that step (Model.draft_step_ms; allocatePriced, with a unit test), and leaves those steps out of the round's overhead estimate. - Backend.draftTogether encodes every request's absorb and chained drafts into one command buffer (Session encodeDrafts / heldDrafts) and waits once; FZ_BATCH_MTP_SERIAL=1 keeps the per-stream path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…XTS, and a stream that outgrows its session moves Every session held the whole window, so few fit beside the model (three 262K sessions in 15.4 GiB on a 256 GB Mac). - FLASHNEXT_SLOT_CONTEXTS lists the sessions past the engine's own (token counts, CONTEXTxN for N alike, e.g. "65536,32768x3,16384x11": 16 streams in 11.6 GiB); unset, nothing changes. - A new stream takes the smallest free session that holds its prompt and its reply's next FLASHNEXT_SLOT_GROW tokens (default 8192); LaneHost's admit hook (as on main since ashhart#639) holds a request until one is free. - Before a round, a stream left too little room moves to the smallest free larger session that holds its next FLASHNEXT_SLOT_GROW tokens (Session.moveTo: its cache rows, DeltaNet states, n-gram tail, head rows and pooled blocks copied on the GPU at a round's end, states settled first), or pauses (Stream.paused) until one frees. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the shared rounds when a second arrives One stream through the shared rounds decodes slower than the single-reply driver: a host-driven round each window. The batch host now gives LaneHost a lone driver (api.Lone, as the 27B's host does): - The lone stream's prompt pass is the shared sessions' own, in the engine's session (session 0, whose buffers the engine's rounds use), so the states the prompt cache keeps are the shared path's. - It then decodes on the engine's GPU-side rounds: generateFrom's decode, split out as Engine.decode, two rounds in flight. - When LaneHost's yield hook asks (a request waits, or a cancel), the loop stops after a landed round; the round ahead has already started as rounds do (the kept DeltaNet states, the n-gram tail, the head's absorb of the last window), so the session's host fields are set to match (position, history, head position, pooled counts clamped) and the lane core adopts the stream: its next window starts at the cache's end with nothing held. - FLASHNEXT_NO_LONE=1 serves every request through the shared rounds. The lone driver takes greedy streams only (api.Lone.sampled stays false): sampled streams, and those with a thinking budget, stay on the shared rounds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…wait for a session that doesn't exist; old path when unset Review fixes (cwschroeder on #10): - slots.room decides before each round from the reply's end (prompt + max_new + a window): a stream whose session holds it is never moved or paused, so one in the largest session (admission's guarantee) runs to its end however many drafts are rejected near the limit. A stream that does need more room moves while a larger session is free, else pauses on a round that still fits; it only waits when a larger session exists, and the waits go up in size, so a paused stream resumes once that session frees. - slots.parse validates each repetition count against the sessions still allowed before appending, and adds the margin with checked arithmetic (an overflowing count or repeat is refused, nothing allocated for it). - FLASHNEXT_SLOT_CONTEXTS unset keeps the old path: the first free session (ContextFull when it can't hold the reply), no admission hook, no growth checks. - Tests (slots.zig, in the core test root): the list's parsing, overflow and session-count refusals; session choice unsized (first free) and sized (smallest that holds the first grow); near-limit rounds with every draft but one rejected in a session that holds the reply (never pauses); a smaller session moving while a larger one is free and pausing on a round that fits otherwise; and no wait when no session is larger.
… run, FZ_BATCH_UNPRICED=1 skips timing, fuller pricing tests Review fixes (cwschroeder on #9): - batch_costs.time prefills a 192-token scratch prompt and runs windows of up to MAXR rows past it, so a session needs batch_costs.ROWS (209) rows. timeCosts now checks fits(s.m.cap) first and serves unpriced (the allocator as before) when the session is smaller, as at --context 128; time itself refuses with SessionTooSmall. Boundary test at ROWS - 1 and ROWS. - FZ_BATCH_UNPRICED=1 skips the timing entirely: no costs, so the allocator, draft budgets and the lone depth rule are the unpriced ones from before this PR. - allocate.zig: tests for the first-draft exemption (a 100 ms step still takes a stream's first draft), asymmetric streams (priced [3, 1] against unpriced [3, 3]) and a zero step (priced equals allocate). - Doc comments in batch_costs.zig, session.zig and backend.zig cut to one line. - batch_costs.zig joins the core test root (tensorfold.zig), so its boundary test runs.
…t, which can move the context The prompt slice was taken before the first token's commit and handed to Engine.decode after it; a commit that grows the context frees the old buffer, so decode copied freed memory into the rounds' history. - Backend.commitFirst commits the first token and returns stream.prompt() from the context as it is then. decode copies the prompt to the rounds' history before any round commits, so later commits leave it alone. - A test fills the context to capacity, refuses growth in place (FailingAllocator resize_fail_index 0), and checks the returned prompt is the moved context's, with the prompt's tokens. - backend.zig joins the core test root (tensorfold.zig), so this test runs.
…rwards timed at load; every stream's drafts in one command buffer
Conflicts: backend.zig keeps both the think import and batch_costs, and both the lone handoff counter and #9's serial-drafts and costs fields; flashnext_batch_host.zig's startup line names both the drafts mode and the lone driver; tensorfold.zig's test root takes both backend.zig and batch_costs.zig.
…rst had taken it), doc comments on one line
…ssions Conflicts: backend.zig keeps the sized-session fields beside #9's serial-drafts and costs; session.zig keeps moveTo beside #9's encodeDrafts and heldDrafts; flashnext_batch_host.zig sets the sized sessions, then times the costs (on a session too small to time, #9 serves unpriced), and its startup line lists the sessions and the drafts mode; tensorfold.zig's test root takes both slots.zig and batch_costs.zig.
…rge left shadowing the lone driver (Backend.lone)
… an error; the lane core sweeps streams a release ends; harness on Backend.init's capacities Review fixes (cwschroeder on #10): - slots.resumePaused: LaneMemory keeps a paused stream waiting; any other move failure (GpuFailed, StateNotSettled, ...) ends it through Backend.failed: slots.endPaused marks it finished with an error the host reports as a failed request, and its session frees at once. Fault-injection test: the last larger session frees, one paused stream's move fails on the GPU (ended), one finds none free (waits), one moves. - The lane core: a release can now end another stream, so Engine.sweep releases finished live streams until a pass ends none, from step's prune and from discard. Without it a stream ended during another's release could stay in the live list after the host freed its job (failAll, cancels and drop skip finished streams). Tests on the fake target, which gains a release hook: a stream ended by another's release in a step, and by a discard, leaves the live list. - zig/tests/flashnext_batch.zig: Backend.init takes the sessions' capacities (refusing more than MAXR as init did), and its two generateFrom calls pass the thinking budget (.{}) that 1.0.8 added, which already broke the harness on 04c8571.
cwschroeder
changed the base branch from
upstream/main-20261010-bec00ae
to
upstream/main-20261016-c7137bb
October 16, 2026 07:07
cwschroeder
changed the title
cwschroeder
changed the base branch from
upstream/main-20261016-c7137bb
to
upstream/main-20261016-46a08c18
October 16, 2026 09:44
cwschroeder
force-pushed
the
feat/flashnext-continuous-batching
branch
from
October 16, 2026 09:44
15d9bec to
8cee8ef
Compare
…the memory limit remedies An explicit --parallel N above one with --learn served one reply at a time without a word, while the same request with --speed-up refused. Both refuse now, from one tested predicate. The OverMemoryLimit message names its remedies again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft CI receipt for upstream PR #606. Keep this fork receipt open.
Flash Next currently queues requests behind its single-reply host with the default flags. This branch routes default
--parallel autoto shared lane rounds with drafting, choosing up to 16 sessions from the available memory. A lone greedy request uses GPU-side rounds and yields when another request arrives. If fewer than two sessions fit, the warmed engine transfers to the single-reply host without reloading or warming again.Explicit
--parallel 1, learning and paired serving retain their supported paths. Shared sessions remain single-Mac.Candidate
ef85c8b0b8402fc5186a8123fcf2a4ef87042acd, upstream base46a08c18608153bfc566703a88acc883bc6140f4:Local validation on M5 Max/macOS 26.6.2 (25G83), Zig 0.17.0:
zig build native test-native tf-flashnext-batch -Dcpu=apple_m1 --summary all: 51/51 steps, 29 tests passed, one skipped. Restoring the old residency rule makes both new CPU regressions fail.Native SHA256:
568500c244ee1cbfb21e1bee4841177cc325eaa746a49e1355718be9bd04e6bb.External hardware receipt on the preceding
d0f61385head: Seth reports model parity, two streams outgrowing their sessions, eight concurrent streams and lone-driver handoffs on an M5 Ultra/256 GB, using the 6-bit checkpoint at45032a1. That run exposed the residency-limit mismatch fixed here. The new residency policy still needs a hardware retest.This PR remains a draft. All four upstream checks and all eight fork checks pass on this head. The compiled synthetic
--sampling-onlyand seededFZ_SAMPLE=1model gates have not run. Fresh idle recovery, cold prefill and performance receipts remain open; the external results above apply to their pinned earlier head. No deployment is claimed.