Skip to content

fix(ssd): stream the directory we load, resume partial first-run downloads, keep exiting events on their own line - #196

Merged
solderzzc merged 2 commits into
mainfrom
fix/ssd-streaming-followups-2
Sep 26, 2026
Merged

solderzzc merged 2 commits into
mainfrom
fix/ssd-streaming-followups-2

Conversation

@solderzzc

Copy link
Copy Markdown
Member

Follow-ups from the SwiftLM review session's review of #192 (the owner OK'd them as post-merge work), plus one gap found while testing them.

Changes

  • Streaming uses the directory that's actually loaded. Since Fix SSD streaming buffer overflow with MTP, and scope streaming to its model mlx-swift-lm#71, loadWeights streams only when the loaded directory matches the one streaming was activated for. modelConfig and the streaming directory came from different lookups, so a model found only by resolveModelDirectory (for example ~/.cache) got a second copy downloaded into Application Support and loaded without streaming. When --stream-experts is set, modelConfig now points at modelDirectory.
  • Found while testing: resolveModelDirectory doesn't check that the weights are present. An empty or partial ~/.cache snapshot (for example from an interrupted download) resolved as the model directory, so the prefetch was skipped and streaming was activated for a directory the loader would never read. With --stream-experts, a resolved directory that fails ModelStorage.validateLocalModelDirectory is now dropped.
  • Partial first-run download: the prefetch reuse check used to look only for config.json. The architecture probe or an interrupted download can leave that behind, and the session then planned with a partial size. It now validates every shard, and otherwise resumes through snapshot.
  • Progress bars: ProgressTracker.finish() stops redrawing and ends the \r line. It runs when the prefetch or the main load returns or throws. emitEvent also starts a fresh line if a bar is still open. Before, a mid-download failure glued the exiting JSON onto the progress line, where the Aegis daemon (serde_json per line) and the Electron manager (startsWith('{')) couldn't see it. This problem already existed for the loader's bar.
  • README: the b769/b773 crash warning now covers quantized MoE models generally. It reproduced on M6 with Gemma 4 26B-A4B too: broadcast_shapes (260,8,8,2816) vs (260,8,1) on b769. The Gemma 4 8-bit --stream-experts row predated fix: propagate throws from LanguageModel.newCache and adapt to mlx-swift-lm PR #62 #167, so I re-measured it: 9.1 tok/s (was 8.8), 9.5K tokens ok, 32K swaps, 7.3 GB. The data is appended to docs/profiling/m6/gemma4_26b_a4b_8bit.*.

Verification (Mac mini M6, 32 GB)

  • Interrupted download: run 1 was killed mid-download, leaving config.json plus the tokenizer files. Before the directory-validation fix, run 2 skipped the prefetch, because an empty ~/.cache snapshot resolved as the model directory. With it, the next start re-downloads, finishes the bar with a newline, prints the next log line on a fresh line, and reaches ready. Run 3 reuses the Application Support copy without downloading.
  • Regression, Qwen3.6-35B-A3B --stream-experts, 548 tokens: 248.6 prefill / 13.89 decode tok/s at 5.69 GB, with no "loading … without streaming" line. Streaming still activates for a cached model.
  • Gemma 4 26B-A4B 4-bit --stream-experts on the fixed build: 16.75 / 17.01 tok/s decode at 548 / 2.3K tokens, 4.4–5.1 GB.
  • The emitEvent newline after a mid-download network failure wasn't reproduced; it's based on code reading.

AI usage: written by Claude Code (Claude Opus 5.5) in the M6 benchmarking session, with the repo owner's approval to open this PR. The review findings came from the SwiftLM review session.

🤖 Generated with Claude Code

solderzzc and others added 2 commits September 26, 2026 08:36
…scope

Follow-ups from the #192 review:
- --stream-experts drops a resolved model directory that fails validation (an
  empty or partial snapshot), and the prefetch reuse check validates every
  shard instead of just config.json, so an interrupted download is resumed
  rather than planned with a partial size.
- Load from the directory streaming was activated for, so a different lookup
  can't pick another copy that silently loads without streaming.
- ProgressTracker.finish() ends the bar line when a download returns or throws,
  and emitEvent starts a fresh line if a bar is open, so the exiting event is
  never glued onto a progress line the daemon can't parse.
- README: the b769/b773 crash covers quantized MoE models generally
  (reproduced with Gemma 4 26B-A4B too); re-measure the Gemma 4 8-bit
  --stream-experts row (9.1 tok/s, 9.5K ok, 32K swaps, 7.3 GB).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Real-time recording on the official b782 binary: 13.6 tok/s decode, 5.1 GB
peak, no swap. Shown in the README's Qwen3.6 GPU vs SSD section.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@solderzzc
solderzzc merged commit 1735cd7 into main Sep 26, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant