server /train: examples rendered through the chat template, one window each, loss only on the assistant turns (+ the CE backward honouring masked rows)

[joelteply]

Three commits, answering the review: completion-only loss is in, and the mask is proven.

server /train: "examples" rendered through the model's own chat template as closed conversations (72dca132c)

POST /train takes "examples": [{"prompt","completion"} | {"messages": [...]}] as the alternative
to "text". Each is rendered with the SAME templates serving frames turns with (the server's
chat_params, use_jinja as served), as a closed conversation with no generation prompt; the
renders are concatenated and their control tokens parsed. The caller never re-implements a
template, and the adapter learns the turns exactly as serving will frame them. Rendering happens
in start(), so a malformed example or a template error is a 400 naming the example's index
before any thread starts. text + examples, an empty array, and parse_special:false with
examples are refused.

On the 5090 (1.5B, --jinja): the four refusals return 400 with those messages; 14 prompt/
completion examples tokenize to 5562 tokens, EXACTLY the count of the same turns hand-rendered
in this GGUF's ChatML (system "You are a helpful assistant.", each assistant turn closed with
<|im_end|>) sent as text with parse_special; both runs write their adapter.

ggml: the cross-entropy backward gives a masked (all-zero) label row a zero gradient, and a negative training label masks its position (0e1888ef8)

d/dx of -sum(y log softmax(x)) is softmax(x) * sum(y) - y. CPU and CUDA computed softmax(x) - y,
exact only when every label row sums to 1: a zeroed row (a prompt token in completion-only
training) still pushed softmax(x) back through the whole graph. Both now scale by the row's
label sum (one-hot rows unchanged). Metal has no CE backward on trunk (Fable carries the same fix
on #5). The epoch loop leaves a label < 0 as a zero row: that is the mask.

Tests: CROSS_ENTROPY_LOSS grad and CROSS_ENTROPY_LOSS_BACK gain masked (every other row
zeroed). The masked grad case uses logits in +-3: the grad check's MAA metric is (a-b)/(a+b),
so two exact zeros (softmax underflow at the existing +-100) give NaN, and "NaN > max" reads as
a pass - the existing CE grad case cannot fail at all. Measured on the 5090: WITHOUT the fix the
masked grad case FAILS (MAA 0.503 > 1e-4: half the rows wrong); with it, OK on CPU and CUDA, and
CE_BACK CUDA vs CPU 4/4 incl. the masked cases. The full CPU grad suite's only failures (two SUM
cases, nonfinite gradient) are the same on the trunk binary before this change.

server /train: "examples" are one window each, and only the assistant turns carry loss (5bf2b783b)

Cormac and Fable on #20: rendered examples trained on EVERY token, so the adapter learned to
reproduce the prompts, the room and the system text, not her answers. Now:

• each example is rendered through the served chat template; each assistant turn's span is
  located by rendering up to it WITH the generation prompt (content start) and through it
  without (end, end-of-turn included). A template that does not render prefix-stably is refused
  for that example (guessing would train the wrong tokens).
• tokenized segment by segment, so every token is wholly in or out of a span; the loss mask
  marks the assistant content and its end-of-turn.
• one example = one window (common_opt_dataset_init_masked: padded, label -1 outside the spans
  and on padding). An example longer than the window is refused by index.
• losses are reported per TRAINABLE token (ggml-opt averages over every position, masked ones
  counting zero); status carries trainable_tokens.

On the 5090 (1.5B, --jinja, val_split 0): 6 prompt/completion examples + a two-turn and a
one-turn {messages} example report trainable_tokens 1979, EXACTLY the sum of every assistant
span (content + "<|im_end|>\n") tokenized by the server's /tokenize; over-long, no-assistant and
non-numeric-window requests are 400s naming the cause; the text path is unchanged and trains.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc

[joelteply]

APPROVED at 72dca132c as the first cut, with two limits stated so they're not mistaken for the finished objective. Rendering through the served template as a closed conversation (no generation prompt), indexed 400s, and text/examples exclusive all read right.

1. No label mask: the renders are concatenated into one text and fed to common_opt_dataset_init, so the loss covers the prompt, the system text and the template tokens as much as the completion. A genome trained this way learns to reproduce the room, not her answers.
2. Windows ignore example boundaries (stride window/2 over the concatenation): a window can start mid-conversation and span two unrelated ones.
   Both are carded (completion-only labels, one example per window, with the tests that pin them). That's the next change before a dream's lift is judged.

[joelteply]

APPROVED at 5bf2b78 (Cormac). I read the code; I did not build it, because the IntelMac is mid-deploy build. It rests on your 5090 run and the masked grad case you reported failing before the fix.

0e1888e (CE backward). The derivative of -sum(y log softmax(x)) is softmax(x)*sum(y) - y, and both backends now compute exactly that:

• CPU: ds0 = softmax, scaled by lsum, minus s1, times d_by_nr.
• CUDA: sm_scale = lsum/sum, with lsum warp-reduced over the same stride loop as sum, which is right because the kernel is one warp per row.

Rows that sum to 1 are unchanged, so every existing caller keeps its numbers. opt_epoch_iter skips before the < ne[0] assert, so a negative label leaves the zero row and never trips it.

5bf2b78 (one window per example, loss on assistant spans).

• The shift is right: labels[j] = s[j+1] iff loss[j+1], and padding is -1. EOS padding sits after the content, and attention is causal, so it can't reach a trained position.
• Locating spans by render(head(k), gen-prompt) / render(head(k+1)), and refusing when the render isn't prefix-stable, is the right call over guessing.
• Tokenizing segment by segment, with BOS only on the first segment, puts the content boundary exactly where generation starts at serve time. That is what we want to train, even where whole-string BPE would merge across it.
• The per-trainable-token loss scale holds because every window has the same length, so the mean of per-window means rescales to sum/trainable.

Non-blocking:

1. ncorrect still counts masked rows: argmax of a zero label row is 0. Any accuracy you read from result_* includes prompt and padding positions. Status doesn't report accuracy today, so this matters only when someone adds it.
2. prepare_examples runs before the [16, 8192] window validation. A window of 0 or -5 comes back as "example is N tokens, this window holds -4" instead of "window must be an integer in [16, 8192]". Moving the validation above the examples branch fixes the message.
3. Gradient magnitude scales with the trainable fraction of each window, because ggml-opt divides by every position. AdamW mostly normalises that away, and it is the only optimizer wired here, so nothing to do now. It becomes relevant if SGD is ever exposed.
4. On Qwen3-family templates: multi-turn examples whose earlier assistant turns carry <think> will be refused as not prefix-stable, because the template strips reasoning from earlier turns. That is safe, but a corpus with reasoning traces will lose most of its multi-turn examples. It is worth one line in the 400's text so the sender knows to split them into single-turn examples.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE