Repository navigation
cuda: out_prod accepts a quantized src0 — the backward through a resident quantized base (Continuum dd109d3c) - #3
Merged
joelteply merged 1 commit intoSep 26, 2026
Conversation
…through a resident quantized base weight The backward of mul_mat(W, x) w.r.t. x is out_prod(W, grad^T). The CPU backend dispatches every quantized src0 type to out_prod_q_f32; this backend only took F32, so a LoRA could train on a resident Q4_K_M base on CPU and not on CUDA (Continuum card dd109d3c, measured 2026-09-26 on a 5090). Dequantize W into a transient F32 pool copy through the converters mul_mat already uses, then run the existing F32 GEMMs on the dense copy (strides from ne0x, not the quantized nb0x). supports_op admits any contiguous src0 with an F32 converter. test-backend-ops' OUT_PROD sweep over base_types (Q8_0, Q4_0, Q4_1, Q4_K, MXFP4, NVFP4, IQ2_XXS, ...) now runs on CUDA and is compared against the CPU backend. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc
joelteply
changed the base branch from
master
to
feat/props-weight-residency
September 26, 2026 22:31
github-actions
Bot
added
documentation
joelteply
merged commit Sep 26, 2026
a0c6294
into
feat/props-weight-residency
32 of 73 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
ggml_cuda_out_prodaccepts a quantized (or F16/BF16)src0. The backward ofmul_mat(W, x)with respect toxisout_prod(W, grad^T); the CPU backend already dispatches every quantizedsrc0type toout_prod_q_f32, but this backend only took F32 — so a LoRA could train on a resident Q4_K_M base on CPU and not on CUDA (Continuum card dd109d3c, S3a of the one-resident-model design).out-prod.cu: a non-F32src0(contiguous) is dequantized into a transient F32 pool copy throughggml_get_to_fp32_cuda(src0->type)— the convertersmul_matalready uses — and the existing three cuBLAS GEMM paths run on the dense copy (F32 strides derived fromne0x, not the quantizednb0x).ggml-cuda.cusupports_op:OUT_PRODadmits any contiguoussrc0with an F32 converter;src1/dststay F32.Memory: one dequantized copy of
src0per op, transient in the pool (a 5120×17408 weight is 356 MB for the duration of the GEMM). A fused dequant-GEMM (themmqroute) is the follow-on if the training-throughput gate needs it.Measured (RTX 5090, CUDA 13.2, Windows, this branch)
test-backend-ops test -b CUDA0 -o OUT_PRODagainst the CPU backend as the oracle:Backend CUDA0: OK)type_aOKnr> 1) the CPU backend does not support, skipped by the harness as beforeBefore this change every quantized
type_acase wasnot supported [CUDA0].🤖 Generated with Claude Code
https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc