Skip to content

Nemotron: fix SSM backward thread limit on M1 - #715

Merged
ashhart merged 1 commit into
ashhart:mainfrom
mrv777:fix/nemotron-training-m1
Oct 13, 2026
Merged

ashhart merged 1 commit into
ashhart:mainfrom
mrv777:fix/nemotron-training-m1

Conversation

@mrv777

@mrv777 mrv777 commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

On M1 Max, tf_train_ssm_back reports a 768-thread limit, but training dispatches 1,024 threads per head. The launch can leave gradients unwritten without a command-buffer error.

Declare a 1,024-thread limit and add regressions for gradients, seeded checkpoints, shared groups and checkpoint boundaries, including the production 64-head/8-group shape. Arithmetic and dispatch sizes are unchanged.

Validated on M1 Max with Zig 0.17.0:

This PR is independent of the GLM fixes. Raw logs remain local. Full-model training and other GPUs remain untested.

@mrv777
mrv777 marked this pull request as ready for review October 12, 2026 03:09
@ashhart
ashhart merged commit c41fdf1 into ashhart:main Oct 13, 2026
jschmied pushed a commit to jschmied/TensorFold that referenced this pull request Oct 13, 2026
@ashhart

ashhart commented Oct 13, 2026

Copy link
Copy Markdown
Owner

Thank you, @mrv777. Gradients left unwritten with no command-buffer error would have been hard to find from a training run, and the new tests pin every shape that matters. Your commit landed in 1.0.9 under your name (5d5e0bda0), and the PR's head is merged into the history with no change to the tree (c41fdf1ac), so GitHub shows this PR as merged.

Your gradient checks pass on an M3 Ultra and an M5 Ultra (2 of 2 on each, with the same errors on both chips), and the macOS host suite passes. We haven't timed a Living Weights training step with it; the arithmetic and dispatch are unchanged, so we expect no difference. M1 is your receipt.

@ashhart ashhart mentioned this pull request Oct 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants