fix(ssd): Qwen3.6 --stream-experts works again (b769 crash), first-run streaming, README re-measure - #192
Draft
solderzzc wants to merge 1 commit into
Draft
fix(ssd): Qwen3.6 --stream-experts works again (b769 crash), first-run streaming, README re-measure#192solderzzc wants to merge 1 commit into
solderzzc wants to merge 1 commit into
Conversation
- Bump mlx-swift-lm to include SharpAI/mlx-swift-lm#69 (SSD path crashes and eager weight load after the upstream sync). #71 follows before merge. - --stream-experts on a model that isn't downloaded yet now fetches it before planning, so streaming is activated for that load instead of the loader loading every expert. - README: re-measure the Qwen3.6-35B-A3B table on the fixed build and flag that b769 --stream-experts crashes on Qwen3.5/3.6. The earlier SSD numbers were from before the sync. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Release b769 crashes on the first request with
--stream-expertson Qwen3.5/3.6 MoE:The mlx-swift-lm upstream sync that came in with #167 broke the SSD path in three ways: a double unsort, evals inside compiled decode traces, and a concurrent loader that loaded every expert into memory. The README's Qwen3.6 SSD numbers came from before that sync and were never re-measured afterwards.
Changes
--mtp/draft verify, and scoping streaming to the main model) follows before this leaves draft.--stream-experts(Server.swift): if the model isn't on disk yet,modelDirectorywas nil, soExpertStreamingConfigwas never activated and the load went eager. The model is now fetched first, with the sameHubApiroot and patterns the loader uses, so streaming activates on the first run too.docs/profiling/m6: re-measured the Qwen3.6-35B-A3B table (GPU and SSD, 548 to 40.8K tokens) on the fixed build, and added a warning that b769--stream-expertscrashes on Qwen3.5/3.6.Verification (Mac mini M6, 32 GB, median of 3, 1 run at 40.8K, needle checked in every run)
Before the sync (old README): SSD 321 / 402 / 403 / 340 prefill, 13.2 / 13.1 / 12.9 / 11.9 decode, 6.0–7.7 GB. The only remaining gap is prefill at ~550 tokens. On b769: crash. Swap growth was 0 in every run.
The first-run path wasn't exercised end to end (it needs a model that isn't cached yet). It builds, and the cached path is unchanged:
modelDirectoryresolves and the new block is skipped.AI usage: written by Claude Code (Claude Opus 5.5) in the M6 benchmarking session, with the repo owner's approval to open this PR. The #71 bugs and the first-run gap were found by the SwiftLM review session.
🤖 Generated with Claude Code