Repository navigation
Conversation
Draft-only implementation and independent diagnostics. Preserve the unchanged failed whole-model numerical gate and disclose unresolved source-CUDA unary/full-head qualification limits. Include pinned inputs, public provenance, bounded actual serving receipts, admission and error-path hardening. No release-qualified model claim.
|
The gate question from your review is now answered with a measurement, and it lands on your side of the fork: the 1% whole-model gate is measuring the model's sensitivity. FP32 oracle vs FP64 oracle, same 8 tokens, same checkpoint bytes, no GPU: NRMS 2.90% (0.028957653279577746), max absolute logit error 1.08, greedy agreement 8/8. The independent reference now runs both widths from one compile-time parameter; the fp32 instance is regression-proven bit-identical to the one that produced the retained 4.0961% receipt, and 4,208 per-stage finiteness checks cover both chains. So the model's own fp32 sensitivity — the hyperconnection mixing amplifying fp32-level differences across 40 layers — is the dominant term in the failed gate, exactly the mechanism you named. Two supporting measurements point the same way:
Given the number, the gate we can defend has two parts:
Also landed since your review, on the lane core:
The evidence docs, receipts and the goldens file stay out of the tree per your note; everything above is re-runnable from the branch the next push carries. |
|
A second point on the sensitivity reading, from the 2.8bpw pack now that it loads end to end (kept-dense forms included, dense path on both sides of the comparison):
The whole-model NRMS rises as the quantization gets coarser, at the same operator code. A fixed kernel bug would contribute roughly constant error at both widths; an error dominated by the model amplifying fp32-level differences grows with the weights' coarseness. That is the signature the 2.90% sensitivity floor predicted, now measured at two quantizations. The per-stage attribution continues in the same harness and will name any stage above its own rounding level; nothing so far has been. So the concrete proposal stands: per-operator gates at each stage's own fp32-vs-fp64 bound, and a whole-model bound stated relative to the sensitivity floor rather than as an absolute 1%. Happy to run whatever form you'd rather gate on. |
|
The layer-by-layer pass is done, and your guess was the right place to look: sparse/compressed attention got the full treatment — every stage of the operator now runs against an FP64 reference, kernel vs FP32-contract vs FP64-ideal on identical decoded rows, production shapes (512 width, 32 heads, window-128 ring, 512 compressed rows, both RoPE spectra, positions to 16383). The verdict table:
So CSA2's arithmetic does not explain the 4.096%: the kernel is bit-identical to the FP32 source contract everywhere except two named residuals — a tie-break at an exact cancellation, and the sink cases at Two harness bugs surfaced while building it, both worth knowing for the other operators: the FP64 ideal originally omitted the documented BF16 einsum/score-storage boundaries (making 0.37% look like a kernel bug when the ideal was wrong), and the top-k check read score words as positions, which made What remains as a transmission path is support, not rounding: when the k-th indexer score is a genuine tie (0 quanta of margin), 2 of 512 picks flip under an FP64-vs-FP32 score change; at a 13-quantum margin, none. A flipped pick adds or removes a key from the attention support — that is how a one-bit difference becomes a percent of logits without any stage being "wrong". The margin metric is in the harness and prints per case; the next measurement is the selection-margin distribution on real layer-21 indexer scores. The real-data snapshot agrees: in the layer-21 same-input capture, That completes the per-operator pass: hyperconnection mixing and sparse/compressed attention both sit at rounding level against their FP64 references, so the whole-model 4.096% stays as measured — the model amplifying FP32-level differences (2.90% floor, growing with quant coarseness) plus support flips at tied selections, not a kernel error. The per-operator gates and the relative whole-model bound from my earlier comment are the proposal; happy to fold the margin measurement into the served gate if you want it there. |
|
The selection-margin measurement from my last comment is done, on real weights at depth. Setup: a 2600-token prompt prefilled through the resident model, then one diagnostic token whose indexer selection scratch (score tiles and top-k heaps, all eight indexer layers) is captured verbatim and decoded offline. Result: the 512-pick selection boundary sits inside an exact tie of the BF16-quantized scores at every layer measured — layers 2, 8, 14 and 20, with tie-runs of 15, 3, 2 and 8 keys at the boundary. The mechanism is the quantization itself: the pinned source contract rounds each key's score to BF16, so 2600 scores collapse to a few hundred distinct values, and the boundary of a 512-pick selection lands inside a tie-run as the norm, not the exception. Four more layers (24–36) split their scoring into tile iterations; the capture holds only the last tile for those, and a per-tile hook is needed to finish them — the four measured layers are unambiguous. What this means for the gate: a FP32-vs-FP64 score difference (~1e-7 relative) is far below the BF16 score quantum (~2^-8 relative), so inside a tie-run the pick order is decided by exactly that noise — support flips at tied boundaries are live on production weights, at every indexed layer, for every token past the window fill. The output effect of one flip is bounded: tied keys carry equal scores, so the change is the swapped keys' value difference times the tied-key weight — small per flip, and the whole-model amplifier is the same one the sensitivity floor already measures. So the gate proposal stands as posted — per-operator rounding-level gates, plus a whole-model bound relative to the measured FP32 sensitivity floor — with the support-flip term folded into the sensitivity accounting rather than treated as a kernel defect. If you want it measured directly, the flip-count metric is already in the operator harness and the margin capture is a runner mode; both are in the tree. One more reproducibility note: the branch's own oracle now reproduces the retained 4.0960584769% whole-model number digit-for-digit (8 teacher tokens, greedy agreement 8/8), so the gate receipt no longer depends on my earlier scratch runs. |
|
One-line follow-up to the margin measurement: the four tile-split layers are settled without a per-tile hook — the top-k heaps themselves carry the boundary. The entries equal to the 512th's score number 7 (layer 24), 13 (28), 4 (32) and 9 (36); the 512-pick cut falls inside a tie-run at every one of the eight indexed layers, matching the tile-complete four. Distinct BF16 scores per 512 picks: 71–288, so the pile-up is heavy everywhere. Nothing changes in the gate proposal; the support-flip term stays folded into the sensitivity accounting. |
|
The split is up as four reviewable pieces, rebased onto current main and CI-green: #824 (the Engram token map), #825 (the family — configuration, weights, staging and the operator kernels), #826 (the fp64 reference tests), #827 (the lane backend and the family guide). The served-gate piece stays pending the gate-form discussion; its evidence ships in those bodies. This draft is superseded — closing it, and the gate question stands in the pieces' text: per-operator rounding-level gates plus a whole-model bound relative to the measured FP32 sensitivity floor, with the selection-boundary ties folded into the sensitivity accounting. |
1 participant
What this changes
Draft-only implementation and qualification work for DeepSeek V4.1 Flash on resident Metal, proposed in #685. Includes EXL3 projections, original Engram tables and offline token mapping, hyperconnections, sparse/compressed attention, routed/shared experts, resident RoPE, serialized text serving, official template/tool decoding, hardware stage profiling, and independent operator/whole-model diagnostics.
Not ready to merge. Keep this PR a draft. The qualified-model table is unchanged. Unsupported serving features are rejected rather than approximated or silently enabled. This is an executable experimental backend, not a scaffold, but numerical qualification remains failed.
Numerical gate: explicitly failed
Latest retained eight-token independent whole-model receipt:
TeacherReferenceMismatchremains a blocking failure. Greedy agreement, operator checks, partition exactness and diagnostic reuse of native inputs/coefficients do not clear it. No tolerance was relaxed and no source coefficient was injected into qualification.A retained HC exponent differs by one FP32 ULP from actual source CUDA, changing a BF16 expansion cell; six alternative Metal exponent implementations did not fix it and were rejected. Source-conformant reduction differences also remain distinguishable from errors against the independent wide oracle. The canonical 64-head CUDA sparse kernel compiles but cannot launch on the available GB10: 141,312 shared-memory bytes required versus a 101,376-byte device limit. Executed 16-head checks are not full-head or whole-model qualification.
Documentation and provenance
THIRD_PARTY_NOTICES.mdand the fixtures.d959d804d5101b79a49b6ff1bf3c54cd5affa9a5a78c503d925f3090040b31c3.Historical source receipts came from
bec00ae6961a43a9ab73c4fa9338c167b3f82851plus local candidate changes, not a clean release commit. This draft integrates them onto maine7c523ebfa16b8094cf9e35e3773a7d25befcc5b. Historical 61 Metal + 11 reference checks, bit-exact 257-token chunk replay, API timing and zero-I/O claims are not presented as newly exercised checks of this integration. A separate retained trace recorded 20,480 read bytes and zero page-ins and remains a failed zero-I/O receipt.Review boundary
This deliberately broad draft preserves the complete local DS4.1 implementation and diagnostic work for engineering review; it is not presented as a small merge-ready feature. Shared contributions overlap #648, #649, #650, #680, #681 and #684, which must be reconciled before merge. The branch is based on main rather than an unmerged PR head.
Current serving is single-active-sequence, with queued clients and cancellation between bounded chunks. Vision, draft heads, thinking budgets, Sliding/Living Weights, learned states and retained prompt caches are unsupported. Full million-token execution and full source-CUDA whole-model parity are unqualified. No before/after served-speed claim is made.
The PR must remain a draft until the unchanged numerical gate, affected checks, actual runtime/history/admission paths and documented residency/long-context gates pass. Further numerical investigation continues locally. No Hugging Face contribution requiring authentication is part of this publication.
Current-branch receipt
Published implementation commit:
d334ad7fc92d3af53b0ec31677bad597bdae6c4a.Environment: Apple M3 Ultra, 512 GiB RAM, macOS 26.6.2, Zig 0.17.0, Xcode Metal.
Both public checkpoint revisions are pinned in the guide. No new whole-model oracle run
or full million-token/source-CUDA qualification is claimed.
DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer \ TENSORFOLD_V41_TOKENIZER_DIR=/models/dsv41-original \ zig build native tf-dsv41-run tf-exl3-bench test-deepseek-v41 \ test-deepseek-v41-reference test-server test-golden test \ -Dcpu=apple_m1 -Doptimize=fast -j2 --summary all zig-out/bin/tf-exl3-bench --shape 4 --warmup 1 --repeats 1 --prefill 2Paths above are portable examples for the actual pinned local snapshots; no private home path is published.
remaining successful test steps reused from cache. During integration the dedicated
DS suite passed 75 device/host cases, the independent reference suite 13 cases,
and server consumer suite 155/155. These counts are not whole-model comparisons.
No golden outputs were rewritten.
mismatches on the selected one-row, routed-six and two-row prefill paths;
sampled independent FP64 oracle gates passed. One timing repeat is not a speedup claim.
lookup and metadata matched the retained pack. Only the refreshed input-config hash differed.
context 4,096 admitted 413,735,409,352 device bytes, explicit
EXPERIMENTAL / UNQUALIFIEDwarning, startup 113.9 s.READYreplies: matching content and token counts after an interveningarithmetic request; 12 prompt / 3 completion / 0 cached tokens.
lookup({"q":"Paris"}), exactly one call/index 0,275 prompt / 38 completion / 0 cached tokens,
finish_reason:"tool_calls".The prompt crossed the real 256-row prefill boundary.
image input, and a full-context prompt with no reply room.
reason=client-left; next request succeeded withREADY.Graceful SIGTERM shutdown exited 0.
disconnect/recovery and tool-request intervals. This does not erase the historical
failed full trace or qualify every resident/long-context workload.
Engineering review fixes were exercised: invalid route IDs can no longer consume stale
expert slots; malformed/unfinished legacy DSML remains content under the one-call policy;
namespaced tool markup is withheld from SSE prose; tokenizer errors propagate as actual
server/stream failures rather than replacement text or missed stops; released Engram
bucket count is validated separately from token vocabulary. Invalid-route and legacy
content regressions were observed failing before their fixes and passing afterward.
Publication hygiene and unrun scope
Only bounded attributed numeric fixtures are included.
1,169 / 534; zero added findings. The checker is nonzero, not clean.
readiness remain unresolved. No HF-token-dependent contribution was attempted.
Draft readiness is an explicit future reviewer decision; this publication does not
mark the PR ready, merge it, or add DS4.1 to the qualified-model list.