From 305b16962eb86a11b3f68e87e8513c474aad6020 Mon Sep 17 00:00:00 2001 From: simba Date: Fri, 25 Sep 2026 14:22:04 -0700 Subject: [PATCH 1/3] fix(ssd): Qwen3.6 --stream-experts works again; first-run streaming - Bump mlx-swift-lm to include SharpAI/mlx-swift-lm#69 (SSD path crashes and eager weight load after the upstream sync). #71 follows before merge. - --stream-experts on a model that isn't downloaded yet now fetches it before planning, so streaming is activated for that load instead of the loader loading every expert. - README: re-measure the Qwen3.6-35B-A3B table on the fixed build and flag that b769 --stream-experts crashes on Qwen3.5/3.6. The earlier SSD numbers were from before the sync. Co-Authored-By: Claude Opus 5.5 --- README.md | 20 +++++----- Sources/SwiftLM/Server.swift | 16 +++++++- docs/profiling/m6/qwen36_35b_a3b_4bit.jsonl | 44 ++++++++++----------- docs/profiling/m6/qwen36_35b_a3b_4bit.md | 16 ++++---- mlx-swift-lm | 2 +- 5 files changed, 57 insertions(+), 41 deletions(-) diff --git a/README.md b/README.md index 7ee3d1c..f85d453 100644 --- a/README.md +++ b/README.md @@ -65,14 +65,14 @@ The first SwiftLM numbers from a **32 GB** Mac. Every other table in this README | Model (4-bit unless noted) | Weights | Mode | Decode, short prompt | Longest prompt that passed | Peak GPU | |---|---|---|---|---|---| | **`gemma-4-26b-a4b-it-4bit`** (MoE, ~4B active) | 15.3 GB | GPU | **52.2 tok/s** | 80.7K tokens | 19.5 GB | -| **`Qwen3.6-35B-A3B-UD-MLX-4bit`** (MoE, ~3B active) | 21.6 GB | GPU | **46.7 tok/s** | 40.8K tokens | 22.4 GB | -| `Qwen3.6-35B-A3B-UD-MLX-4bit` | 21.6 GB | `--stream-experts` | 13.2 tok/s | 40.8K tokens | 7.7 GB | +| **`Qwen3.6-35B-A3B-UD-MLX-4bit`** (MoE, ~3B active) | 21.6 GB | GPU | **47.0 tok/s** | 40.8K tokens | 21.5 GB | +| `Qwen3.6-35B-A3B-UD-MLX-4bit` | 21.6 GB | `--stream-experts` | 13.2 tok/s | 40.8K tokens | 5.8 GB | | `Qwen3.8-27B-4bit` (dense) | 11.3 GB | GPU | 9.3 tok/s | 40.8K tokens | 18.4 GB | | `gemma-4-26b-a4b-it-8bit` | ~26 GB | GPU | swaps (+3.1 GB on the first prompt) | — | — | | `gemma-4-26b-a4b-it-8bit` | ~26 GB | `--stream-experts` | 8.8 tok/s | 9.5K tokens (32K swapped) | 7.6 GB | - **MoE models are the sweet spot at 32 GB.** Only the active experts are read for each token, so they decode 5–6× faster than a dense 27B. A 4-bit MoE with up to about 22 GB of weights runs entirely on the GPU. -- **Qwen3.6-35B-A3B on a base M6 reaches 76%** of the M1 Ultra 64 GB decode speed below (46.7 vs 61.7 tok/s). +- **Qwen3.6-35B-A3B on a base M6 reaches 76%** of the M1 Ultra 64 GB decode speed below (47.0 vs 61.7 tok/s). - **Dense 27B decode is bandwidth-bound.** 9.3 tok/s × 11.3 GB is about 105 GB/s, roughly 60% of the M6's rated 170 GB/s. - **An 8-bit 26 GB model needs SSD streaming** and tops out at about 10K tokens of context. @@ -92,10 +92,12 @@ Every needle check passed. `--mtp` with the bf16 assistant (`gemma-4-26B-A4B-it- | Prompt tokens | GPU prefill / decode (tok/s) | GPU peak | `--stream-experts` prefill / decode (tok/s) | SSD peak | |---|---|---|---|---| -| ~550 | 808 / 46.7 | 20.4 GB | 321 / 13.2 | 6.0 GB | -| ~2.3K | 969 / 45.5 | 21.0 GB | 402 / 13.1 | 6.2 GB | -| ~9.8K | 849 / 43.4 | 21.1 GB | 403 / 12.9 | 6.6 GB | -| 40.8K | 635 / 35.1 | 22.4 GB | 340 / 11.9 | 7.7 GB | +| ~550 | 714 / 47.0 | 19.8 GB | 256 / 13.2 | 5.6 GB | +| ~2.3K | 968 / 45.7 | 20.1 GB | 403 / 13.0 | 5.6 GB | +| ~9.8K | 858 / 43.4 | 20.4 GB | 401 / 12.7 | 5.6 GB | +| 40.8K | 615 / 36.1 | 21.5 GB | 336 / 12.0 | 5.8 GB | + +> ⚠️ **`--stream-experts` crashes on Qwen3.5/3.6 in release b769** (`broadcast_shapes … (N,8,8,2048)` on the first request). The mlx-swift-lm upstream sync in #167 broke the SSD path. Earlier versions of this table were measured before that sync and were never re-checked afterwards. Fixed in SharpAI/mlx-swift-lm#69 and #71; the table above was re-measured with those fixes. ### Qwen3.8-27B-4bit (dense) @@ -119,13 +121,13 @@ Every needle check passed. `--mtp` with the bf16 assistant (`gemma-4-26B-A4B-it- 1. **The MLX buffer cache was unbounded on full-GPU loads.** It could grow to the whole 26.8 GB working set. It is now sized from the RAM left after weights and KV. 2. **The KV-cache estimate counted every layer as full attention.** Gemma 4 (25 of 30 layers use a 1,024-token sliding window) was overestimated 10×, and Qwen3.5/3.8 (48 of 64 layers are linear attention) 4×. On 32 GB that pushed Gemma into CPU/GPU layer partitioning, which crashed with a Metal GPU timeout. 3. **An auto-detected VLM that failed to load exited the server.** `Qwen3.6-35B-A3B-UD-MLX-4bit` ships a `preprocessor_config.json` without `image_mean`. SwiftLM now falls back to text-only unless you pass `--vision`. -4. **Vision-capable models skipped chunked prefill.** On the older mlx-swift-lm pin, a text-only prompt on the VLM path ran through the model in a single pass. It's fixed by the mlx-swift-lm bump in #167. Every number in this section was measured on `main` with that bump. +4. **Vision-capable models skipped chunked prefill.** On the older mlx-swift-lm pin, a text-only prompt on the VLM path ran through the model in a single pass. It's fixed by the mlx-swift-lm bump in #167. Every number in this section was measured on `main` after that bump; the Qwen3.6 table was re-measured with SharpAI/mlx-swift-lm#69 and #71. > ℹ️ **`--turbo-kv` long-range recall is fixed** ([#175](https://github.com/SharpAI/SwiftLM/issues/175), SharpAI/mlx-swift-lm#65). Before the fix, once a prompt passed the 2,048-token compression threshold, attention only saw the recent hot window and positions restarted, so Qwen3.8-27B-4bit got exact lookups wrong. Attention now covers the compressed history too, which makes `--turbo-kv` slower than before (97 s vs 72 s on an 11.8K-token prompt on the M6). `--turbo-kv` still has no effect when `--ctx-size` is set (the attention layers use `RotatingKVCache`). > > ℹ️ QAT-quantized Gemma 4 MTP assistants (`…-qat-assistant-4bit`) now load and run (SharpAI/mlx-swift-lm#66). > -> ℹ️ **`--gpu-layers N` with MoE models:** the Metal GPU timeout on the first request ([#176](https://github.com/SharpAI/SwiftLM/issues/176)) is fixed in #177. CPU-resident MoE layers still run on a single core and are very slow (Gemma 4 26B-A4B with `--gpu-layers 23`: ~0.4 tok/s prefill, ~3 s per decoded token on both M5 and M6), and SwiftLM now warns about this at startup. Treat `--gpu-layers` as a way to avoid running out of memory, not as a speed trade-off. On a 32 GB Mac try `--stream-experts` first (Qwen3.6-35B-A3B: 13.2 tok/s decode, 7.7 GB GPU). +> ℹ️ **`--gpu-layers N` with MoE models:** the Metal GPU timeout on the first request ([#176](https://github.com/SharpAI/SwiftLM/issues/176)) is fixed in #177. CPU-resident MoE layers still run on a single core and are very slow (Gemma 4 26B-A4B with `--gpu-layers 23`: ~0.4 tok/s prefill, ~3 s per decoded token on both M5 and M6), and SwiftLM now warns about this at startup. Treat `--gpu-layers` as a way to avoid running out of memory, not as a speed trade-off. On a 32 GB Mac try `--stream-experts` first (Qwen3.6-35B-A3B: 13.2 tok/s decode, 5.8 GB GPU). Reproduce: diff --git a/Sources/SwiftLM/Server.swift b/Sources/SwiftLM/Server.swift index 121134b..3d6a792 100644 --- a/Sources/SwiftLM/Server.swift +++ b/Sources/SwiftLM/Server.swift @@ -641,9 +641,23 @@ struct MLXServer: AsyncParsableCommand { // hand-copied or huggingface-cli model it returns nil — which skipped the MoE // guard *and* the ExpertStreamingConfig activation while still setting lazyLoad, // i.e. lazy weights with no streamer (the #112 memory shape) and no diagnostic. - let modelDirectory = + var modelDirectory = ModelStorage.validatedContentDirectory(for: modelId) ?? resolveModelDirectory(modelId: modelId) + if self.streamExperts, modelDirectory == nil, + !FileManager.default.fileExists(atPath: modelId) + { + // First run: fetch now so streaming is activated for this load. Otherwise the + // loader downloads it later and loads every expert into memory. Same hub root + // as the loader below, so it reuses these files. + print("[SwiftLM] --stream-experts: downloading \(modelId) before loading...") + let hub = HubApi( + downloadBase: URL.applicationSupportDirectory + .appendingPathComponent("MLX", isDirectory: true) + .appendingPathComponent("HuggingFace", isDirectory: true)) + modelDirectory = try await hub.snapshot( + from: modelId, matching: ["*.safetensors", "*.json", "*.jinja"]) + } var mainModelProfile: ModelProfile? = nil if self.streamExperts, let dir = modelDirectory { mainModelProfile = ModelProfiler.profile(modelDirectory: dir, modelId: modelId) diff --git a/docs/profiling/m6/qwen36_35b_a3b_4bit.jsonl b/docs/profiling/m6/qwen36_35b_a3b_4bit.jsonl index 7a4d69b..69675ae 100644 --- a/docs/profiling/m6/qwen36_35b_a3b_4bit.jsonl +++ b/docs/profiling/m6/qwen36_35b_a3b_4bit.jsonl @@ -1,22 +1,22 @@ -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 512, "run": -1, "warmup": true, "peak_gpu_gb": 19.93, "swap_delta_gb": 0.0, "min_free_pct": 26, "status": "OK", "prompt_tokens": 527, "prefill_tps": 129.9, "ttft_s": 4.06, "decode_tps": 46.7, "gen_tokens": 128, "needle_ok": true, "degenerate": false} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 512, "run": 0, "warmup": false, "peak_gpu_gb": 20.38, "swap_delta_gb": 0.0, "min_free_pct": 27, "status": "OK", "prompt_tokens": 548, "prefill_tps": 807.6, "ttft_s": 0.69, "decode_tps": 46.83, "gen_tokens": 128, "needle_ok": true, "degenerate": false} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 512, "run": 1, "warmup": false, "peak_gpu_gb": 20.38, "swap_delta_gb": 0.0, "min_free_pct": 26, "status": "OK", "prompt_tokens": 560, "prefill_tps": 834.8, "ttft_s": 0.69, "decode_tps": 46.7, "gen_tokens": 128, "needle_ok": true, "degenerate": false} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 512, "run": 2, "warmup": false, "peak_gpu_gb": 20.34, "swap_delta_gb": 0.0, "min_free_pct": 27, "status": "OK", "prompt_tokens": 541, "prefill_tps": 799.5, "ttft_s": 0.69, "decode_tps": 46.38, "gen_tokens": 128, "needle_ok": true, "degenerate": false} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 2048, "run": 0, "warmup": false, "peak_gpu_gb": 20.96, "swap_delta_gb": 0.0, "min_free_pct": 27, "status": "OK", "prompt_tokens": 2336, "prefill_tps": 962.0, "ttft_s": 2.46, "decode_tps": 45.46, "gen_tokens": 128, "needle_ok": true, "degenerate": false} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 2048, "run": 1, "warmup": false, "peak_gpu_gb": 20.74, "swap_delta_gb": 0.0, "min_free_pct": 27, "status": "OK", "prompt_tokens": 2350, "prefill_tps": 968.9, "ttft_s": 2.46, "decode_tps": 45.02, "gen_tokens": 128, "needle_ok": true, "degenerate": false} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 2048, "run": 2, "warmup": false, "peak_gpu_gb": 20.86, "swap_delta_gb": 0.0, "min_free_pct": 28, "status": "OK", "prompt_tokens": 2346, "prefill_tps": 970.1, "ttft_s": 2.45, "decode_tps": 45.48, "gen_tokens": 128, "needle_ok": true, "degenerate": false} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 8192, "run": 0, "warmup": false, "peak_gpu_gb": 20.8, "swap_delta_gb": 0.0, "min_free_pct": 27, "status": "OK", "prompt_tokens": 9798, "prefill_tps": 875.2, "ttft_s": 11.28, "decode_tps": 43.42, "gen_tokens": 128, "needle_ok": true, "degenerate": false} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 8192, "run": 1, "warmup": false, "peak_gpu_gb": 21.06, "swap_delta_gb": 0.0, "min_free_pct": 27, "status": "OK", "prompt_tokens": 9809, "prefill_tps": 849.2, "ttft_s": 11.63, "decode_tps": 43.36, "gen_tokens": 128, "needle_ok": true, "degenerate": false} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 8192, "run": 2, "warmup": false, "peak_gpu_gb": 20.82, "swap_delta_gb": 0.0, "min_free_pct": 26, "status": "OK", "prompt_tokens": 9829, "prefill_tps": 838.2, "ttft_s": 11.81, "decode_tps": 43.62, "gen_tokens": 128, "needle_ok": true, "degenerate": false} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 32768, "run": 0, "warmup": false, "peak_gpu_gb": 22.38, "swap_delta_gb": 0.0, "min_free_pct": 20, "status": "OK", "prompt_tokens": 40829, "prefill_tps": 634.9, "ttft_s": 64.6, "decode_tps": 35.08, "gen_tokens": 128, "needle_ok": true, "degenerate": false} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 512, "run": -1, "warmup": true, "peak_gpu_gb": 5.59, "swap_delta_gb": 0.0, "min_free_pct": 75, "status": "OK", "prompt_tokens": 542, "prefill_tps": 115.5, "ttft_s": 4.7, "decode_tps": 12.9, "gen_tokens": 128, "needle_ok": true, "degenerate": false} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 512, "run": 0, "warmup": false, "peak_gpu_gb": 5.99, "swap_delta_gb": 0.0, "min_free_pct": 72, "status": "OK", "prompt_tokens": 548, "prefill_tps": 318.2, "ttft_s": 1.74, "decode_tps": 13.25, "gen_tokens": 128, "needle_ok": true, "degenerate": false} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 512, "run": 1, "warmup": false, "peak_gpu_gb": 5.95, "swap_delta_gb": 0.0, "min_free_pct": 72, "status": "OK", "prompt_tokens": 534, "prefill_tps": 320.6, "ttft_s": 1.68, "decode_tps": 13.22, "gen_tokens": 128, "needle_ok": true, "degenerate": false} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 512, "run": 2, "warmup": false, "peak_gpu_gb": 5.97, "swap_delta_gb": 0.0, "min_free_pct": 72, "status": "OK", "prompt_tokens": 549, "prefill_tps": 325.7, "ttft_s": 1.7, "decode_tps": 13.22, "gen_tokens": 128, "needle_ok": true, "degenerate": false} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 2048, "run": 0, "warmup": false, "peak_gpu_gb": 6.24, "swap_delta_gb": 0.0, "min_free_pct": 71, "status": "OK", "prompt_tokens": 2366, "prefill_tps": 401.6, "ttft_s": 5.92, "decode_tps": 13.12, "gen_tokens": 128, "needle_ok": true, "degenerate": false} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 2048, "run": 1, "warmup": false, "peak_gpu_gb": 6.22, "swap_delta_gb": 0.0, "min_free_pct": 72, "status": "OK", "prompt_tokens": 2324, "prefill_tps": 403.2, "ttft_s": 5.79, "decode_tps": 13.06, "gen_tokens": 128, "needle_ok": true, "degenerate": false} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 2048, "run": 2, "warmup": false, "peak_gpu_gb": 6.19, "swap_delta_gb": 0.0, "min_free_pct": 71, "status": "OK", "prompt_tokens": 2356, "prefill_tps": 399.0, "ttft_s": 5.94, "decode_tps": 13.12, "gen_tokens": 128, "needle_ok": true, "degenerate": false} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 8192, "run": 0, "warmup": false, "peak_gpu_gb": 6.26, "swap_delta_gb": 0.0, "min_free_pct": 72, "status": "OK", "prompt_tokens": 9811, "prefill_tps": 399.9, "ttft_s": 24.61, "decode_tps": 12.89, "gen_tokens": 128, "needle_ok": true, "degenerate": false} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 8192, "run": 1, "warmup": false, "peak_gpu_gb": 6.6, "swap_delta_gb": 0.0, "min_free_pct": 74, "status": "OK", "prompt_tokens": 9810, "prefill_tps": 402.6, "ttft_s": 24.45, "decode_tps": 12.78, "gen_tokens": 128, "needle_ok": true, "degenerate": false} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 8192, "run": 2, "warmup": false, "peak_gpu_gb": 6.22, "swap_delta_gb": 0.0, "min_free_pct": 72, "status": "OK", "prompt_tokens": 9872, "prefill_tps": 404.1, "ttft_s": 24.51, "decode_tps": 12.85, "gen_tokens": 128, "needle_ok": true, "degenerate": false} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 32768, "run": 0, "warmup": false, "peak_gpu_gb": 7.73, "swap_delta_gb": 0.0, "min_free_pct": 67, "status": "OK", "prompt_tokens": 40806, "prefill_tps": 340.2, "ttft_s": 120.24, "decode_tps": 11.91, "gen_tokens": 128, "needle_ok": true, "degenerate": false} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 512, "run": -1, "warmup": true, "peak_gpu_gb": 19.78, "swap_delta_gb": 0.0, "min_free_pct": 28, "status": "OK", "prompt_tokens": 527, "prefill_tps": 331.2, "ttft_s": 1.6, "decode_tps": 46.94, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-60", "answer_head": "The secret code word is MARLIN-60.\n\nElias Thorne had spent forty years watching the sea from the top of the Blackwood Point Lighthouse. To the casual observer, his life was a monotonous cycle of polishing brass, trimming wicks, and logging "} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 512, "run": 0, "warmup": false, "peak_gpu_gb": 19.46, "swap_delta_gb": 0.0, "min_free_pct": 27, "status": "OK", "prompt_tokens": 548, "prefill_tps": 713.8, "ttft_s": 0.78, "decode_tps": 46.97, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-23", "answer_head": "The secret code word is MARLIN-23.\n\nElias Thorne had lived with the sea for forty years, but the lighthouse on Blackwood Point was his true home. It was not merely a structure of stone and glass; it was a living entity that breathed with th"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 512, "run": 1, "warmup": false, "peak_gpu_gb": 19.46, "swap_delta_gb": 0.0, "min_free_pct": 27, "status": "OK", "prompt_tokens": 560, "prefill_tps": 726.0, "ttft_s": 0.78, "decode_tps": 47.03, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "PELICAN-81", "answer_head": "The secret code word is PELICAN-81.\n\nElias Thorne had lived with the sea for forty years, his life measured not in hours, but in the rhythmic sweep of the Great Point Lighthouse beam. The structure itself was a sentinel of granite and iron,"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 512, "run": 2, "warmup": false, "peak_gpu_gb": 19.78, "swap_delta_gb": 0.0, "min_free_pct": 28, "status": "OK", "prompt_tokens": 541, "prefill_tps": 697.8, "ttft_s": 0.79, "decode_tps": 46.87, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "HERON-20", "answer_head": "The secret code word is HERON-20.\n\nElias Thorne had lived with the sea\u2019s rhythm for forty years, a man carved from the same salt-weathered granite as the lighthouse he tended. His world was not one of grand adventures or bustling harbors, b"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 2048, "run": 0, "warmup": false, "peak_gpu_gb": 20.02, "swap_delta_gb": 0.0, "min_free_pct": 28, "status": "OK", "prompt_tokens": 2336, "prefill_tps": 966.3, "ttft_s": 2.45, "decode_tps": 45.64, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "PELICAN-68", "answer_head": "The secret code word is PELICAN-68.\n\nElias Thorne had lived with the sea\u2019s rhythm for forty years, a man carved from salt spray and stubbornness. His home was the old granite lighthouse perched on the jagged cliffs of Blackwood Point, a sol"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 2048, "run": 1, "warmup": false, "peak_gpu_gb": 20.06, "swap_delta_gb": 0.0, "min_free_pct": 28, "status": "OK", "prompt_tokens": 2350, "prefill_tps": 969.7, "ttft_s": 2.45, "decode_tps": 45.72, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-77", "answer_head": "The secret code word is **MARLIN-77**.\n\n***\n\nElias Thorne had not seen the sun rise in three days, but he knew it was coming. As the keeper of the Blackwood Point Lighthouse, his existence was measured not in hours, but in the rhythmic swee"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 2048, "run": 2, "warmup": false, "peak_gpu_gb": 20.06, "swap_delta_gb": 0.0, "min_free_pct": 28, "status": "OK", "prompt_tokens": 2346, "prefill_tps": 967.5, "ttft_s": 2.46, "decode_tps": 45.82, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-75", "answer_head": "The secret code word is MARLIN-75.\n\nElias Thorne had lived with the sea\u2019s rhythm for forty years, his life measured not in hours, but in the sweeping arc of the lighthouse beam cutting through the coastal fog. The structure, a stoic sentine"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 8192, "run": 0, "warmup": false, "peak_gpu_gb": 20.06, "swap_delta_gb": 0.0, "min_free_pct": 27, "status": "OK", "prompt_tokens": 9798, "prefill_tps": 898.5, "ttft_s": 10.98, "decode_tps": 43.63, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "PELICAN-70", "answer_head": "The secret code word is PELICAN-70.\n\nElias Thorne had lived in the solitary stone tower of Blackwood Point for thirty years, a man carved from the same granite that formed the cliff beneath his feet. To the outside world, he was merely a fu"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 8192, "run": 1, "warmup": false, "peak_gpu_gb": 20.41, "swap_delta_gb": 0.0, "min_free_pct": 27, "status": "OK", "prompt_tokens": 9809, "prefill_tps": 857.8, "ttft_s": 11.52, "decode_tps": 43.38, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "OSPREY-30", "answer_head": "The secret code word is OSPREY-30.\n\nElias Thorne had lived within the walls of the Blackwood Point Lighthouse for thirty years, a duration that felt both like a blink of an eye and an eternity of solitude. His world was defined by the rhyth"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 8192, "run": 2, "warmup": false, "peak_gpu_gb": 20.22, "swap_delta_gb": 0.0, "min_free_pct": 27, "status": "OK", "prompt_tokens": 9829, "prefill_tps": 845.5, "ttft_s": 11.71, "decode_tps": 43.43, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "PELICAN-57", "answer_head": "The secret code word is PELICAN-57.\n\nElias Thorne had lived in the solitary stone tower of Blackwood Point for thirty years, a man carved from the same granite that formed the cliff beneath his feet. To the outside world, his life was a mon"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 32768, "run": 0, "warmup": false, "peak_gpu_gb": 21.54, "swap_delta_gb": 0.0, "min_free_pct": 22, "status": "OK", "prompt_tokens": 40829, "prefill_tps": 615.3, "ttft_s": 66.64, "decode_tps": 36.05, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "OSPREY-28", "answer_head": "The secret code word is OSPREY-28.\n\nElias Thorne had lived in the solitary stone tower of Blackwood Point for thirty years, a duration that felt both like a blink and an eternity. The lighthouse was not merely a structure; it was his compan"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 512, "run": -1, "warmup": true, "peak_gpu_gb": 4.91, "swap_delta_gb": 0.0, "min_free_pct": 75, "status": "OK", "prompt_tokens": 542, "prefill_tps": 115.1, "ttft_s": 4.72, "decode_tps": 12.98, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "OSPREY-54", "answer_head": "The secret code word is OSPREY-54.\n\nElias Thorne had lived with the sea for forty years, his life measured not in hours, but in the rhythmic pulse of the lighthouse beam cutting through the Atlantic fog. The structure, a towering sentinel o"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 512, "run": 0, "warmup": false, "peak_gpu_gb": 5.59, "swap_delta_gb": 0.0, "min_free_pct": 73, "status": "OK", "prompt_tokens": 548, "prefill_tps": 256.1, "ttft_s": 2.15, "decode_tps": 13.23, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "PELICAN-32", "answer_head": "The secret code word from the notes is **PELICAN-32**.\n\n***\n\nElias Thorne had not seen the mainland in three years, not since the storm that took his brother and left him as the sole guardian of the Blackwood Point Lighthouse. The isolation"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 512, "run": 1, "warmup": false, "peak_gpu_gb": 4.93, "swap_delta_gb": 0.0, "min_free_pct": 74, "status": "OK", "prompt_tokens": 534, "prefill_tps": 256.1, "ttft_s": 2.09, "decode_tps": 13.23, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "OSPREY-26", "answer_head": "The secret code word is OSPREY-26.\n\nElias Thorne had lived within the walls of the Blackwood Point Lighthouse for thirty years, a duration that felt less like time passing and more like the slow, rhythmic turning of the great Fresnel lens a"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 512, "run": 2, "warmup": false, "peak_gpu_gb": 5.33, "swap_delta_gb": 0.0, "min_free_pct": 74, "status": "OK", "prompt_tokens": 549, "prefill_tps": 260.6, "ttft_s": 2.12, "decode_tps": 13.26, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "HERON-40", "answer_head": "The secret code word is HERON-40.\n\n***\n\nElias Thorne had not seen the sun in three days, a fact he noted with the same detached precision he applied to the rotation of the great Fresnel lens above him. Perched on the jagged spine of Blackro"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 2048, "run": 0, "warmup": false, "peak_gpu_gb": 5.57, "swap_delta_gb": 0.0, "min_free_pct": 75, "status": "OK", "prompt_tokens": 2366, "prefill_tps": 404.9, "ttft_s": 5.87, "decode_tps": 13.12, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "OSPREY-75", "answer_head": "The secret code word is OSPREY-75.\n\nElias Thorne had lived with the sea\u2019s rhythm for forty years, his life measured not in hours, but in the sweeping arc of the lighthouse beam cutting through the Atlantic fog. The tower, a stoic sentinel o"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 2048, "run": 1, "warmup": false, "peak_gpu_gb": 5.15, "swap_delta_gb": 0.0, "min_free_pct": 74, "status": "OK", "prompt_tokens": 2324, "prefill_tps": 400.9, "ttft_s": 5.83, "decode_tps": 13.02, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-96", "answer_head": "The secret code word is MARLIN-96.\n\nElias Thorne had lived with the sea\u2019s rhythm for forty years, his life measured not in hours, but in the turning of the great Fresnel lens at the Blackwood Point Lighthouse. The structure stood like a sol"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 2048, "run": 2, "warmup": false, "peak_gpu_gb": 5.57, "swap_delta_gb": 0.0, "min_free_pct": 75, "status": "OK", "prompt_tokens": 2356, "prefill_tps": 403.3, "ttft_s": 5.87, "decode_tps": 13.03, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "HERON-55", "answer_head": "The secret code word is HERON-55.\n\nElias Thorne had lived in the solitary stone tower of Blackwood Point for thirty years, a man carved from the same granite that formed the cliff beneath his feet. To the outside world, his life was a monot"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 8192, "run": 0, "warmup": false, "peak_gpu_gb": 5.09, "swap_delta_gb": 0.0, "min_free_pct": 74, "status": "OK", "prompt_tokens": 9811, "prefill_tps": 400.6, "ttft_s": 24.57, "decode_tps": 12.81, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "HERON-51", "answer_head": "The secret code word is HERON-51.\n\nElias Thorne had not seen the sun rise in three days, a fact that weighed heavier on his shoulders than the salt-crusted wool of his coat. He was the keeper of the Needle, a jagged spire of granite that ju"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 8192, "run": 1, "warmup": false, "peak_gpu_gb": 5.09, "swap_delta_gb": 0.0, "min_free_pct": 74, "status": "OK", "prompt_tokens": 9810, "prefill_tps": 399.8, "ttft_s": 24.62, "decode_tps": 12.68, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-58", "answer_head": "The secret code word is MARLIN-58.\n\nElias Thorne had lived in the solitary stone tower of Blackwood Point for thirty years, a man carved from the same granite that formed the cliff beneath his feet. To the outside world, his life was a mono"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 8192, "run": 2, "warmup": false, "peak_gpu_gb": 5.56, "swap_delta_gb": 0.0, "min_free_pct": 74, "status": "OK", "prompt_tokens": 9872, "prefill_tps": 403.6, "ttft_s": 24.54, "decode_tps": 12.72, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "PELICAN-41", "answer_head": "The secret code word is PELICAN-41.\n\nElias Thorne had lived in the solitary stone tower of Blackwood Point for thirty years, a man carved from the same granite that formed the cliff beneath his feet. His days were not measured by the tickin"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 32768, "run": 0, "warmup": false, "peak_gpu_gb": 5.8, "swap_delta_gb": 0.0, "min_free_pct": 73, "status": "OK", "prompt_tokens": 40806, "prefill_tps": 335.9, "ttft_s": 121.74, "decode_tps": 12.03, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "HERON-53", "answer_head": "The secret code word is HERON-53.\n\nThe salt spray had long since bleached Elias\u2019s skin to the color of old parchment, and the wind in his ears was a constant, roaring companion that no human voice could ever quite match. He was the keeper o"} diff --git a/docs/profiling/m6/qwen36_35b_a3b_4bit.md b/docs/profiling/m6/qwen36_35b_a3b_4bit.md index f8daed2..1691838 100644 --- a/docs/profiling/m6/qwen36_35b_a3b_4bit.md +++ b/docs/profiling/m6/qwen36_35b_a3b_4bit.md @@ -4,11 +4,11 @@ Apple M6 · 32 GB · runs=3 (long=1) · warmup=1 · gen=128 · temperature 0 · | Config | Context (prompt tok) | Prefill tok/s | TTFT s | Decode tok/s | Peak GPU GB | Swap Δ GB | Min free % | Checks | |---|---|---|---|---|---|---|---|---| -| Vanilla | 512 (548) | 807.6 | 0.69 | 46.7 | 20.38 | 0.0 | 26 | ok | -| Vanilla | 2048 (2346) | 968.9 | 2.46 | 45.46 | 20.96 | 0.0 | 27 | ok | -| Vanilla | 8192 (9809) | 849.2 | 11.63 | 43.42 | 21.06 | 0.0 | 26 | ok | -| Vanilla | 32768 (40829) | 634.9 | 64.6 | 35.08 | 22.38 | 0.0 | 20 | ok | -| SSD | 512 (548) | 320.6 | 1.7 | 13.22 | 5.99 | 0.0 | 72 | ok | -| SSD | 2048 (2356) | 401.6 | 5.92 | 13.12 | 6.24 | 0.0 | 71 | ok | -| SSD | 8192 (9811) | 402.6 | 24.51 | 12.85 | 6.6 | 0.0 | 72 | ok | -| SSD | 32768 (40806) | 340.2 | 120.24 | 11.91 | 7.73 | 0.0 | 67 | ok | +| Vanilla | 512 (548) | 713.8 | 0.78 | 46.97 | 19.78 | 0.0 | 27 | ok | +| Vanilla | 2048 (2346) | 967.5 | 2.45 | 45.72 | 20.06 | 0.0 | 28 | ok | +| Vanilla | 8192 (9809) | 857.8 | 11.52 | 43.43 | 20.41 | 0.0 | 27 | ok | +| Vanilla | 32768 (40829) | 615.3 | 66.64 | 36.05 | 21.54 | 0.0 | 22 | ok | +| SSD | 512 (548) | 256.1 | 2.12 | 13.23 | 5.59 | 0.0 | 73 | ok | +| SSD | 2048 (2356) | 403.3 | 5.87 | 13.03 | 5.57 | 0.0 | 74 | ok | +| SSD | 8192 (9811) | 400.6 | 24.57 | 12.72 | 5.56 | 0.0 | 74 | ok | +| SSD | 32768 (40806) | 335.9 | 121.74 | 12.03 | 5.8 | 0.0 | 73 | ok | diff --git a/mlx-swift-lm b/mlx-swift-lm index 7cc37a0..e7df2ae 160000 --- a/mlx-swift-lm +++ b/mlx-swift-lm @@ -1 +1 @@ -Subproject commit 7cc37a0bc3d4379a5ea60f2a2d2041834fc43b95 +Subproject commit e7df2aefb1871d5cbca9bbdb99f6659e95d53581 From 9c52e257d2b2ec441f05e17837158823732910ef Mon Sep 17 00:00:00 2001 From: simba Date: Sat, 26 Sep 2026 01:59:27 -0700 Subject: [PATCH 2/3] chore: bump mlx-swift-lm to include SharpAI/mlx-swift-lm#71 Picks up the SSD persistent-buffer regrow (--stream-experts with --mtp or a draft model), streaming scoped to the main model's directory, and the end-to-end SSD tests (#70, #71). Co-Authored-By: Claude Opus 5.5 --- mlx-swift-lm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mlx-swift-lm b/mlx-swift-lm index e7df2ae..5596071 160000 --- a/mlx-swift-lm +++ b/mlx-swift-lm @@ -1 +1 @@ -Subproject commit e7df2aefb1871d5cbca9bbdb99f6659e95d53581 +Subproject commit 5596071bfe613abd35c2478fbe71ae03f10cb04f From 43794f5e40e011d49930b1b95f85c25051297b29 Mon Sep 17 00:00:00 2001 From: simba Date: Sat, 26 Sep 2026 05:55:37 -0700 Subject: [PATCH 3/3] fix(ssd): tighten the --stream-experts first-run prefetch Review fixes for the prefetch added in this PR: - Skip it for --info, which would otherwise download the whole model. - Reuse an existing copy under the loader's hub root (localRepoLocation), so later starts don't re-list and re-HEAD every file (or re-hash them offline). - Show download progress with ProgressTracker. - Set the load phase first, so a bad or gated id reports model_load_failed instead of binary_error. - README: the crash warning covers b773 too. Co-Authored-By: Claude Opus 5.5 --- README.md | 2 +- Sources/SwiftLM/Server.swift | 26 +++++++++++++++++++------- 2 files changed, 20 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index f85d453..a3c385d 100644 --- a/README.md +++ b/README.md @@ -97,7 +97,7 @@ Every needle check passed. `--mtp` with the bf16 assistant (`gemma-4-26B-A4B-it- | ~9.8K | 858 / 43.4 | 20.4 GB | 401 / 12.7 | 5.6 GB | | 40.8K | 615 / 36.1 | 21.5 GB | 336 / 12.0 | 5.8 GB | -> ⚠️ **`--stream-experts` crashes on Qwen3.5/3.6 in release b769** (`broadcast_shapes … (N,8,8,2048)` on the first request). The mlx-swift-lm upstream sync in #167 broke the SSD path. Earlier versions of this table were measured before that sync and were never re-checked afterwards. Fixed in SharpAI/mlx-swift-lm#69 and #71; the table above was re-measured with those fixes. +> ⚠️ **`--stream-experts` crashes on Qwen3.5/3.6 in releases b769 and b773** (`broadcast_shapes … (N,8,8,2048)` on the first request). The mlx-swift-lm upstream sync in #167 broke the SSD path. Earlier versions of this table were measured before that sync and were never re-checked afterwards. Fixed in SharpAI/mlx-swift-lm#69 and #71; the table above was re-measured with those fixes. ### Qwen3.8-27B-4bit (dense) diff --git a/Sources/SwiftLM/Server.swift b/Sources/SwiftLM/Server.swift index 4f1a2b0..a117434 100644 --- a/Sources/SwiftLM/Server.swift +++ b/Sources/SwiftLM/Server.swift @@ -644,19 +644,31 @@ struct MLXServer: AsyncParsableCommand { var modelDirectory = ModelStorage.validatedContentDirectory(for: modelId) ?? resolveModelDirectory(modelId: modelId) - if self.streamExperts, modelDirectory == nil, + if self.streamExperts, !self.info, modelDirectory == nil, !FileManager.default.fileExists(atPath: modelId) { - // First run: fetch now so streaming is activated for this load. Otherwise the - // loader downloads it later and loads every expert into memory. Same hub root - // as the loader below, so it reuses these files. - print("[SwiftLM] --stream-experts: downloading \(modelId) before loading...") + // Streaming must be activated for the directory the loader reads, so resolve it + // before loading. Same hub root as the loader below, which reuses these files. let hub = HubApi( downloadBase: URL.applicationSupportDirectory .appendingPathComponent("MLX", isDirectory: true) .appendingPathComponent("HuggingFace", isDirectory: true)) - modelDirectory = try await hub.snapshot( - from: modelId, matching: ["*.safetensors", "*.json", "*.jinja"]) + let localRepo = hub.localRepoLocation(Hub.Repo(id: modelId)) + if FileManager.default.fileExists( + atPath: localRepo.appendingPathComponent("config.json").path) + { + modelDirectory = localRepo + } else { + // First run. A failed download is a model problem, not a binary one. + phase = .architectureProbe + print("[SwiftLM] --stream-experts: downloading \(modelId) before loading...") + let prefetchTracker = ProgressTracker(modelId: modelId) + modelDirectory = try await hub.snapshot( + from: modelId, matching: ["*.safetensors", "*.json", "*.jinja"] + ) { progress in + prefetchTracker.printProgress(progress) + } + } } var mainModelProfile: ModelProfile? = nil if self.streamExperts, let dir = modelDirectory {