cuda: out_prod dequantizes a 2D weight in row blocks — the output head's backward holds one block (dd109d3c, second cut)

[joelteply]

What

Second cut on the quantized-src0 out_prod (after #3): for a 2D weight (ne02 == ne03 == 1, 2D dst) the dequantized transient is a row block of at most 256 MiB of F32, not the whole W. out_prod sums over src0's rows (ne01), so each block is dequantized through the same F32 converter and its GEMM accumulates into dst (beta = 0 for the first block, 1 after). Batched and broadcast shapes keep the whole-copy path from #3.

Why now (Cormac on #3): the largest frozen matmul in a LoRA backward is the output head, out_prod(W_out, grad^T) with W_out = n_embd × n_vocab — ~3.1 GB of F32 for a 27B beside the resident model. Blocked, it is one 256 MiB buffer.

Measured (RTX 5090, CUDA 13.2, Windows, this branch)

test-backend-ops test -b CUDA0 -o OUT_PROD against the CPU backend:

passed	234/234 (Backend CUDA0: OK)
new cases	k = 600000 rows at m = 256 (three blocks): Q4_K trans_b 0 and 1, Q8_0 — all OK
FAIL / refused by CUDA	0 / 0

🤖 Generated with Claude Code

https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc

[joelteply]

APPROVED at 8bed10816 (base feat/props-weight-residency). The row-blocked path reads right against the whole-copy one it replaces for 2D weights:

• out_prod sums over ne01 (W's rows), so dequantizing rows rows from r0 * nb01 and GEMMing ne0 × ne1 × rows with beta 0 then 1 is the same sum in blocks. Quantized rows are whole blocks, so each slice is block-aligned for to_fp32.
• The src1 k-offset matches the existing op convention in both orientations: r0 * ldb when op(B) = Bᵀ reads B as n × k, and r0 when B is already transposed.
• The peak transient is rows_per_block × ne00 × 4 ≤ 256 MiB (at least one row), so the output head's backward holds one block, not ~3 GB.
• Batched/broadcast shapes (attention/GQA) keep the dense copy; those are small.

The three new cases pin exactly the multi-block accumulation (k = 600000, three blocks at m = 256), in both src1 orientations at Q4_K plus Q8_0, against CPU: the only way a wrong offset or a missing beta = 1 shows up. 234/234 on the 5090 is the gate met. Speed stays the S3 gate's question; the fused dequant-in-GEMM route is the follow-on if the per-block to_fp32 shows in the step time.

Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE