From fe76e47d99a5a236304318edcfcf78347025a703 Mon Sep 17 00:00:00 2001 From: simba Date: Sun, 27 Sep 2026 20:30:36 -0700 Subject: [PATCH 1/2] docs(m6): raw benchmark data from release b782 Re-measured on the official b782 binary with scripts/profiling/m6_bench.py: - Qwen3.6-35B-A3B: full GPU and --stream-experts table at 548 to 40.8K tokens, replacing the pre-b782 run, with a note that the 40.8K needle check is unreliable in both modes (OSPREY-NN code words), so those rows are throughput only. - Gemma 4 26B-A4B 4-bit (short prompt), Gemma 4 8-bit --stream-experts (533 and 9.5K) and Qwen3.8-27B (548 and 9.8K): a b782 section on top; the existing rows stay, labelled as an earlier build. - The MTP file is labelled as an earlier build. New JSONL rows carry "build": "b782"; existing rows are unchanged. Co-Authored-By: Claude Opus 5.5 --- docs/profiling/m6/gemma4_26b_a4b_4bit.jsonl | 4 ++ docs/profiling/m6/gemma4_26b_a4b_4bit.md | 14 +++++- .../m6/gemma4_26b_a4b_4bit_mtp_bf16.md | 2 + docs/profiling/m6/gemma4_26b_a4b_8bit.jsonl | 7 +++ docs/profiling/m6/gemma4_26b_a4b_8bit.md | 15 +++++- docs/profiling/m6/qwen36_35b_a3b_4bit.jsonl | 50 +++++++++++-------- docs/profiling/m6/qwen36_35b_a3b_4bit.md | 24 +++++---- docs/profiling/m6/qwen38_27b_4bit.jsonl | 7 +++ docs/profiling/m6/qwen38_27b_4bit.md | 15 +++++- 9 files changed, 103 insertions(+), 35 deletions(-) diff --git a/docs/profiling/m6/gemma4_26b_a4b_4bit.jsonl b/docs/profiling/m6/gemma4_26b_a4b_4bit.jsonl index c6c2b24..ec793cb 100644 --- a/docs/profiling/m6/gemma4_26b_a4b_4bit.jsonl +++ b/docs/profiling/m6/gemma4_26b_a4b_4bit.jsonl @@ -22,3 +22,7 @@ {"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "TurboKV", "context": 8192, "run": 2, "warmup": false, "peak_gpu_gb": 15.74, "swap_delta_gb": 0.0, "min_free_pct": 39, "status": "OK", "prompt_tokens": 9602, "prefill_tps": 970.0, "ttft_s": 10.1, "decode_tps": 45.62, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "HERON-76", "answer_head": "The secret code word is HERON-76.\n\nElias Thorne had lived within the salt-sprayed stone walls of the Blackwood Lighthouse for thirty-two years, a duration that felt less like a career and more like a slow merging of man and masonry. Every m"} {"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "TurboKV", "context": 32768, "run": 0, "warmup": false, "peak_gpu_gb": 17.55, "swap_delta_gb": 0.0, "min_free_pct": 32, "status": "OK", "prompt_tokens": 39672, "prefill_tps": 786.6, "ttft_s": 51.13, "decode_tps": 31.47, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-59", "answer_head": "The secret code word is MARLIN-59.\n\nElias Thorne had lived within the salt-crusted stone walls of the Blackwood Rock Lighthouse for twenty-two years, a duration that felt less like a measurement of time and more like a slow, rhythmic breath"} {"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "TurboKV", "context": 65536, "run": 0, "warmup": false, "peak_gpu_gb": 19.28, "swap_delta_gb": 0.0, "min_free_pct": 26, "status": "OK", "prompt_tokens": 80922, "prefill_tps": 630.3, "ttft_s": 129.75, "decode_tps": 24.51, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "HERON-75", "answer_head": "The secret code word is **HERON-75**.\n\n***\n\nThe brass lantern of the Great North Beacon did not merely shine; it breathed. For Elias, a man whose skin had become as weathered and salt-crusted as the driftwood on the shores of Oakhaven, the "} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "Vanilla", "context": 512, "run": -1, "warmup": true, "peak_gpu_gb": 13.8, "swap_delta_gb": 0.0, "min_free_pct": 46, "status": "OK", "prompt_tokens": 520, "prefill_tps": 529.4, "ttft_s": 1.01, "decode_tps": 53.93, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-60", "answer_head": "The secret code word is MARLIN-60.\n\nElias Thorne lived in the rhythm of the tides and the mechanical pulse of the Great North Light. For thirty years, the lighthouse had been his only constant companion, a towering sentinel of granite and g", "build": "b782"} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "Vanilla", "context": 512, "run": 0, "warmup": false, "peak_gpu_gb": 14.16, "swap_delta_gb": 0.0, "min_free_pct": 46, "status": "OK", "prompt_tokens": 533, "prefill_tps": 774.9, "ttft_s": 0.74, "decode_tps": 53.86, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-23", "answer_head": "The secret code word is MARLIN-23.\n\nElias Thorne had lived within the salt-crusted stone walls of the Blackwood Rock Lighthouse for twenty-two years, and in that time, he had learned that silence was not the absence of sound, but the presen", "build": "b782"} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "Vanilla", "context": 512, "run": 1, "warmup": false, "peak_gpu_gb": 14.29, "swap_delta_gb": 0.0, "min_free_pct": 46, "status": "OK", "prompt_tokens": 542, "prefill_tps": 785.7, "ttft_s": 0.73, "decode_tps": 53.85, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "PELICAN-81", "answer_head": "The secret code word is PELICAN-81.\n\nElias Thorne was a man of salt, iron, and silence. For thirty-two years, he had lived within the whitewashed stone walls of the Blackwood Rock Lighthouse, a lonely sentinel rising from the churning Atlan", "build": "b782"} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "Vanilla", "context": 512, "run": 2, "warmup": false, "peak_gpu_gb": 14.01, "swap_delta_gb": 0.0, "min_free_pct": 46, "status": "OK", "prompt_tokens": 527, "prefill_tps": 780.9, "ttft_s": 0.72, "decode_tps": 53.98, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "HERON-20", "answer_head": "The secret code word is HERON-20.\n\nThe salt spray was a constant companion to Elias, a fine mist that coated the brass fittings of the lantern room and turned his beard into a coarse, white thicket. For thirty years, he had climbed the wind", "build": "b782"} diff --git a/docs/profiling/m6/gemma4_26b_a4b_4bit.md b/docs/profiling/m6/gemma4_26b_a4b_4bit.md index d845afa..d63e603 100644 --- a/docs/profiling/m6/gemma4_26b_a4b_4bit.md +++ b/docs/profiling/m6/gemma4_26b_a4b_4bit.md @@ -1,4 +1,16 @@ -### `mlx-community/gemma-4-26b-a4b-it-4bit` +### `mlx-community/gemma-4-26b-a4b-it-4bit` — release b782 + +Apple M6 · 32 GB · macOS 27.0 · **release b782** (official `SwiftLM-b782-macos-arm64.tar.gz`) · `scripts/profiling/m6_bench.py` · runs=3 (1 at 32K+) · warmup=1 · gen=128 · temperature 0 · medians + +Only the short prompt was re-run on b782. + +| Config | Context (prompt tok) | Prefill tok/s | TTFT s | Decode tok/s | Peak GPU GB | Swap Δ GB | Min free % | Checks | +|---|---|---|---|---|---|---|---|---| +| Vanilla | 512 (533) | 780.9 | 0.73 | 53.86 | 14.29 | 0.0 | 46 | ok | + +--- + +#### Earlier build (before b782) > **Correction (Sep 24):** the `TurboKV` rows below were run with `--ctx-size`, which on mlx-swift-lm 460ff81 gives the attention layers a `RotatingKVCache`, and `--turbo-kv` only applies to `KVCacheSimple`. So these rows are effectively vanilla, not TurboKV. See #175 for real TurboKV behaviour. diff --git a/docs/profiling/m6/gemma4_26b_a4b_4bit_mtp_bf16.md b/docs/profiling/m6/gemma4_26b_a4b_4bit_mtp_bf16.md index a3d73f0..037d94b 100644 --- a/docs/profiling/m6/gemma4_26b_a4b_4bit_mtp_bf16.md +++ b/docs/profiling/m6/gemma4_26b_a4b_4bit_mtp_bf16.md @@ -1,5 +1,7 @@ ### `mlx-community/gemma-4-26b-a4b-it-4bit` +> Measured on an earlier build (before b782). + Apple M6 · 32 GB · runs=3 (long=1) · warmup=1 · gen=128 · temperature 0 · medians | Config | Context (prompt tok) | Prefill tok/s | TTFT s | Decode tok/s | Peak GPU GB | Swap Δ GB | Min free % | Checks | diff --git a/docs/profiling/m6/gemma4_26b_a4b_8bit.jsonl b/docs/profiling/m6/gemma4_26b_a4b_8bit.jsonl index 08f826e..043a9be 100644 --- a/docs/profiling/m6/gemma4_26b_a4b_8bit.jsonl +++ b/docs/profiling/m6/gemma4_26b_a4b_8bit.jsonl @@ -15,3 +15,10 @@ {"model": "mlx-community/gemma-4-26b-a4b-it-8bit", "config": "SSD", "context": 8192, "run": 0, "warmup": false, "peak_gpu_gb": 6.46, "swap_delta_gb": 0.0, "min_free_pct": 66, "status": "OK", "prompt_tokens": 9546, "prefill_tps": 148.4, "ttft_s": 64.52, "decode_tps": 8.22, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "PELICAN-68", "answer_head": "The secret code word is PELICAN-68.\n\nElias Thorne had lived within the salt-crusted walls of the Blackwood Reach Lighthouse for twenty-two years, a duration that felt less like a career and more like a slow merging of man and stone. His wor"} {"model": "mlx-community/gemma-4-26b-a4b-it-8bit", "config": "SSD", "context": 8192, "run": 1, "warmup": false, "peak_gpu_gb": 5.92, "swap_delta_gb": 0.0, "min_free_pct": 66, "status": "OK", "prompt_tokens": 9511, "prefill_tps": 145.4, "ttft_s": 65.61, "decode_tps": 8.18, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "HERON-85", "answer_head": "The secret code word is HERON-85.\n\nElias Thorne had lived within the salt-crusted walls of the Blackwood Reach lighthouse for twenty-two years, a duration that felt less like a career and more like a slow merging of man and stone. His world"} {"model": "mlx-community/gemma-4-26b-a4b-it-8bit", "config": "SSD", "context": 8192, "run": 2, "warmup": false, "peak_gpu_gb": 7.3, "swap_delta_gb": 0.0, "min_free_pct": 59, "status": "OK", "prompt_tokens": 9549, "prefill_tps": 145.7, "ttft_s": 65.73, "decode_tps": 8.16, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-30", "answer_head": "The secret code word is MARLIN-30.\n\nElias Thorne had lived within the salt-crusted walls of the Blackwood Rock Lighthouse for twenty-two years, a duration long enough for the rhythmic pulse of the Fresnel lens to become his own heartbeat. T"} +{"model": "mlx-community/gemma-4-26b-a4b-it-8bit", "config": "SSD", "context": 512, "run": -1, "warmup": true, "peak_gpu_gb": 5.59, "swap_delta_gb": 0.0, "min_free_pct": 73, "status": "OK", "prompt_tokens": 520, "prefill_tps": 90.2, "ttft_s": 5.8, "decode_tps": 9.57, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-60", "answer_head": "The secret code word is MARLIN-60.\n\n***\n\nElias Thorne had lived within the salt-crusted walls of the Blackwood Reach lighthouse for twenty-two years, a duration long enough for the rhythmic pulse of the Fresnel lens to become his own heartb", "build": "b782"} +{"model": "mlx-community/gemma-4-26b-a4b-it-8bit", "config": "SSD", "context": 512, "run": 0, "warmup": false, "peak_gpu_gb": 6.53, "swap_delta_gb": 0.0, "min_free_pct": 73, "status": "OK", "prompt_tokens": 533, "prefill_tps": 196.9, "ttft_s": 2.76, "decode_tps": 9.75, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-23", "answer_head": "The secret code word is MARLIN-23.\n\n***\n\nElias Thorne had lived within the salt-crusted walls of the Blackwood Reach lighthouse for twenty-two years, a span of time measured not by calendars, but by the rhythmic pulse of the great Fresnel l", "build": "b782"} +{"model": "mlx-community/gemma-4-26b-a4b-it-8bit", "config": "SSD", "context": 512, "run": 1, "warmup": false, "peak_gpu_gb": 5.59, "swap_delta_gb": 0.0, "min_free_pct": 73, "status": "OK", "prompt_tokens": 542, "prefill_tps": 209.5, "ttft_s": 2.64, "decode_tps": 9.67, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "PELICAN-81", "answer_head": "The secret code word is PELICAN-81.\n\n***\n\nElias Thorne had lived within the salt-crusted walls of the Blackwood Reach Lighthouse for twenty-two years, a span of time that had turned his skin into something resembling cured leather and his e", "build": "b782"} +{"model": "mlx-community/gemma-4-26b-a4b-it-8bit", "config": "SSD", "context": 512, "run": 2, "warmup": false, "peak_gpu_gb": 5.6, "swap_delta_gb": 0.0, "min_free_pct": 73, "status": "OK", "prompt_tokens": 527, "prefill_tps": 202.4, "ttft_s": 2.65, "decode_tps": 9.6, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "HERON-20", "answer_head": "The secret code word is HERON-20.\n\n***\n\nElias Thorne had lived within the salt-crusted walls of the Blackwood Reach Lighthouse for twenty-two years, a span of time that had turned his skin to leather and his eyes to the color of a storm-tos", "build": "b782"} +{"model": "mlx-community/gemma-4-26b-a4b-it-8bit", "config": "SSD", "context": 8192, "run": 0, "warmup": false, "peak_gpu_gb": 6.21, "swap_delta_gb": 0.0, "min_free_pct": 71, "status": "OK", "prompt_tokens": 9546, "prefill_tps": 304.9, "ttft_s": 31.53, "decode_tps": 8.59, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "PELICAN-68", "answer_head": "The secret code word is PELICAN-68.\n\n***\n\nElias Thorne had lived within the salt-crusted walls of the Blackwood Rock Lighthouse for thirty-two years, a duration that felt less like a career and more like a slow merging of man and stone. His", "build": "b782"} +{"model": "mlx-community/gemma-4-26b-a4b-it-8bit", "config": "SSD", "context": 8192, "run": 1, "warmup": false, "peak_gpu_gb": 7.11, "swap_delta_gb": 0.0, "min_free_pct": 71, "status": "OK", "prompt_tokens": 9511, "prefill_tps": 285.7, "ttft_s": 33.5, "decode_tps": 8.49, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "HERON-85", "answer_head": "The secret code word is HERON-85.\n\nElias Thorne had lived within the salt-crusted walls of the Blackwood Reach lighthouse for thirty-two years, a duration that felt less like a career and more like a slow merging of man and stone. Every mor", "build": "b782"} +{"model": "mlx-community/gemma-4-26b-a4b-it-8bit", "config": "SSD", "context": 8192, "run": 2, "warmup": false, "peak_gpu_gb": 6.0, "swap_delta_gb": 0.0, "min_free_pct": 72, "status": "OK", "prompt_tokens": 9549, "prefill_tps": 246.6, "ttft_s": 38.93, "decode_tps": 8.39, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-30", "answer_head": "The secret code word is MARLIN-30.\n\nElias Thorne had lived within the salt-crusted walls of the Blackwood Rock Lighthouse for twenty-two years, a duration that felt less like a career and more like a slow merging of man and stone. His world", "build": "b782"} diff --git a/docs/profiling/m6/gemma4_26b_a4b_8bit.md b/docs/profiling/m6/gemma4_26b_a4b_8bit.md index d997446..fe99a4d 100644 --- a/docs/profiling/m6/gemma4_26b_a4b_8bit.md +++ b/docs/profiling/m6/gemma4_26b_a4b_8bit.md @@ -1,4 +1,17 @@ -### `mlx-community/gemma-4-26b-a4b-it-8bit` +### `mlx-community/gemma-4-26b-a4b-it-8bit` — release b782 + +Apple M6 · 32 GB · macOS 27.0 · **release b782** (official `SwiftLM-b782-macos-arm64.tar.gz`) · `scripts/profiling/m6_bench.py` · runs=3 (1 at 32K+) · warmup=1 · gen=128 · temperature 0 · medians + +`SSD` = `--stream-experts` (b769 and b773 crash in this mode). 32K was not re-run; on the fix build it swapped (see below). + +| Config | Context (prompt tok) | Prefill tok/s | TTFT s | Decode tok/s | Peak GPU GB | Swap Δ GB | Min free % | Checks | +|---|---|---|---|---|---|---|---|---| +| SSD | 512 (533) | 202.4 | 2.65 | 9.67 | 6.53 | 0.0 | 73 | ok | +| SSD | 8192 (9546) | 285.7 | 33.5 | 8.49 | 7.11 | 0.0 | 71 | ok | + +--- + +#### Earlier build (before b782) Apple M6 · 32 GB · runs=3 (long=1) · warmup=1 · gen=128 · temperature 0 · medians diff --git a/docs/profiling/m6/qwen36_35b_a3b_4bit.jsonl b/docs/profiling/m6/qwen36_35b_a3b_4bit.jsonl index 69675ae..1dd4df4 100644 --- a/docs/profiling/m6/qwen36_35b_a3b_4bit.jsonl +++ b/docs/profiling/m6/qwen36_35b_a3b_4bit.jsonl @@ -1,22 +1,28 @@ -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 512, "run": -1, "warmup": true, "peak_gpu_gb": 19.78, "swap_delta_gb": 0.0, "min_free_pct": 28, "status": "OK", "prompt_tokens": 527, "prefill_tps": 331.2, "ttft_s": 1.6, "decode_tps": 46.94, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-60", "answer_head": "The secret code word is MARLIN-60.\n\nElias Thorne had spent forty years watching the sea from the top of the Blackwood Point Lighthouse. To the casual observer, his life was a monotonous cycle of polishing brass, trimming wicks, and logging "} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 512, "run": 0, "warmup": false, "peak_gpu_gb": 19.46, "swap_delta_gb": 0.0, "min_free_pct": 27, "status": "OK", "prompt_tokens": 548, "prefill_tps": 713.8, "ttft_s": 0.78, "decode_tps": 46.97, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-23", "answer_head": "The secret code word is MARLIN-23.\n\nElias Thorne had lived with the sea for forty years, but the lighthouse on Blackwood Point was his true home. It was not merely a structure of stone and glass; it was a living entity that breathed with th"} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 512, "run": 1, "warmup": false, "peak_gpu_gb": 19.46, "swap_delta_gb": 0.0, "min_free_pct": 27, "status": "OK", "prompt_tokens": 560, "prefill_tps": 726.0, "ttft_s": 0.78, "decode_tps": 47.03, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "PELICAN-81", "answer_head": "The secret code word is PELICAN-81.\n\nElias Thorne had lived with the sea for forty years, his life measured not in hours, but in the rhythmic sweep of the Great Point Lighthouse beam. The structure itself was a sentinel of granite and iron,"} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 512, "run": 2, "warmup": false, "peak_gpu_gb": 19.78, "swap_delta_gb": 0.0, "min_free_pct": 28, "status": "OK", "prompt_tokens": 541, "prefill_tps": 697.8, "ttft_s": 0.79, "decode_tps": 46.87, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "HERON-20", "answer_head": "The secret code word is HERON-20.\n\nElias Thorne had lived with the sea\u2019s rhythm for forty years, a man carved from the same salt-weathered granite as the lighthouse he tended. His world was not one of grand adventures or bustling harbors, b"} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 2048, "run": 0, "warmup": false, "peak_gpu_gb": 20.02, "swap_delta_gb": 0.0, "min_free_pct": 28, "status": "OK", "prompt_tokens": 2336, "prefill_tps": 966.3, "ttft_s": 2.45, "decode_tps": 45.64, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "PELICAN-68", "answer_head": "The secret code word is PELICAN-68.\n\nElias Thorne had lived with the sea\u2019s rhythm for forty years, a man carved from salt spray and stubbornness. His home was the old granite lighthouse perched on the jagged cliffs of Blackwood Point, a sol"} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 2048, "run": 1, "warmup": false, "peak_gpu_gb": 20.06, "swap_delta_gb": 0.0, "min_free_pct": 28, "status": "OK", "prompt_tokens": 2350, "prefill_tps": 969.7, "ttft_s": 2.45, "decode_tps": 45.72, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-77", "answer_head": "The secret code word is **MARLIN-77**.\n\n***\n\nElias Thorne had not seen the sun rise in three days, but he knew it was coming. As the keeper of the Blackwood Point Lighthouse, his existence was measured not in hours, but in the rhythmic swee"} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 2048, "run": 2, "warmup": false, "peak_gpu_gb": 20.06, "swap_delta_gb": 0.0, "min_free_pct": 28, "status": "OK", "prompt_tokens": 2346, "prefill_tps": 967.5, "ttft_s": 2.46, "decode_tps": 45.82, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-75", "answer_head": "The secret code word is MARLIN-75.\n\nElias Thorne had lived with the sea\u2019s rhythm for forty years, his life measured not in hours, but in the sweeping arc of the lighthouse beam cutting through the coastal fog. The structure, a stoic sentine"} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 8192, "run": 0, "warmup": false, "peak_gpu_gb": 20.06, "swap_delta_gb": 0.0, "min_free_pct": 27, "status": "OK", "prompt_tokens": 9798, "prefill_tps": 898.5, "ttft_s": 10.98, "decode_tps": 43.63, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "PELICAN-70", "answer_head": "The secret code word is PELICAN-70.\n\nElias Thorne had lived in the solitary stone tower of Blackwood Point for thirty years, a man carved from the same granite that formed the cliff beneath his feet. To the outside world, he was merely a fu"} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 8192, "run": 1, "warmup": false, "peak_gpu_gb": 20.41, "swap_delta_gb": 0.0, "min_free_pct": 27, "status": "OK", "prompt_tokens": 9809, "prefill_tps": 857.8, "ttft_s": 11.52, "decode_tps": 43.38, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "OSPREY-30", "answer_head": "The secret code word is OSPREY-30.\n\nElias Thorne had lived within the walls of the Blackwood Point Lighthouse for thirty years, a duration that felt both like a blink of an eye and an eternity of solitude. His world was defined by the rhyth"} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 8192, "run": 2, "warmup": false, "peak_gpu_gb": 20.22, "swap_delta_gb": 0.0, "min_free_pct": 27, "status": "OK", "prompt_tokens": 9829, "prefill_tps": 845.5, "ttft_s": 11.71, "decode_tps": 43.43, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "PELICAN-57", "answer_head": "The secret code word is PELICAN-57.\n\nElias Thorne had lived in the solitary stone tower of Blackwood Point for thirty years, a man carved from the same granite that formed the cliff beneath his feet. To the outside world, his life was a mon"} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 32768, "run": 0, "warmup": false, "peak_gpu_gb": 21.54, "swap_delta_gb": 0.0, "min_free_pct": 22, "status": "OK", "prompt_tokens": 40829, "prefill_tps": 615.3, "ttft_s": 66.64, "decode_tps": 36.05, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "OSPREY-28", "answer_head": "The secret code word is OSPREY-28.\n\nElias Thorne had lived in the solitary stone tower of Blackwood Point for thirty years, a duration that felt both like a blink and an eternity. The lighthouse was not merely a structure; it was his compan"} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 512, "run": -1, "warmup": true, "peak_gpu_gb": 4.91, "swap_delta_gb": 0.0, "min_free_pct": 75, "status": "OK", "prompt_tokens": 542, "prefill_tps": 115.1, "ttft_s": 4.72, "decode_tps": 12.98, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "OSPREY-54", "answer_head": "The secret code word is OSPREY-54.\n\nElias Thorne had lived with the sea for forty years, his life measured not in hours, but in the rhythmic pulse of the lighthouse beam cutting through the Atlantic fog. The structure, a towering sentinel o"} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 512, "run": 0, "warmup": false, "peak_gpu_gb": 5.59, "swap_delta_gb": 0.0, "min_free_pct": 73, "status": "OK", "prompt_tokens": 548, "prefill_tps": 256.1, "ttft_s": 2.15, "decode_tps": 13.23, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "PELICAN-32", "answer_head": "The secret code word from the notes is **PELICAN-32**.\n\n***\n\nElias Thorne had not seen the mainland in three years, not since the storm that took his brother and left him as the sole guardian of the Blackwood Point Lighthouse. The isolation"} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 512, "run": 1, "warmup": false, "peak_gpu_gb": 4.93, "swap_delta_gb": 0.0, "min_free_pct": 74, "status": "OK", "prompt_tokens": 534, "prefill_tps": 256.1, "ttft_s": 2.09, "decode_tps": 13.23, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "OSPREY-26", "answer_head": "The secret code word is OSPREY-26.\n\nElias Thorne had lived within the walls of the Blackwood Point Lighthouse for thirty years, a duration that felt less like time passing and more like the slow, rhythmic turning of the great Fresnel lens a"} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 512, "run": 2, "warmup": false, "peak_gpu_gb": 5.33, "swap_delta_gb": 0.0, "min_free_pct": 74, "status": "OK", "prompt_tokens": 549, "prefill_tps": 260.6, "ttft_s": 2.12, "decode_tps": 13.26, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "HERON-40", "answer_head": "The secret code word is HERON-40.\n\n***\n\nElias Thorne had not seen the sun in three days, a fact he noted with the same detached precision he applied to the rotation of the great Fresnel lens above him. Perched on the jagged spine of Blackro"} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 2048, "run": 0, "warmup": false, "peak_gpu_gb": 5.57, "swap_delta_gb": 0.0, "min_free_pct": 75, "status": "OK", "prompt_tokens": 2366, "prefill_tps": 404.9, "ttft_s": 5.87, "decode_tps": 13.12, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "OSPREY-75", "answer_head": "The secret code word is OSPREY-75.\n\nElias Thorne had lived with the sea\u2019s rhythm for forty years, his life measured not in hours, but in the sweeping arc of the lighthouse beam cutting through the Atlantic fog. The tower, a stoic sentinel o"} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 2048, "run": 1, "warmup": false, "peak_gpu_gb": 5.15, "swap_delta_gb": 0.0, "min_free_pct": 74, "status": "OK", "prompt_tokens": 2324, "prefill_tps": 400.9, "ttft_s": 5.83, "decode_tps": 13.02, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-96", "answer_head": "The secret code word is MARLIN-96.\n\nElias Thorne had lived with the sea\u2019s rhythm for forty years, his life measured not in hours, but in the turning of the great Fresnel lens at the Blackwood Point Lighthouse. The structure stood like a sol"} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 2048, "run": 2, "warmup": false, "peak_gpu_gb": 5.57, "swap_delta_gb": 0.0, "min_free_pct": 75, "status": "OK", "prompt_tokens": 2356, "prefill_tps": 403.3, "ttft_s": 5.87, "decode_tps": 13.03, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "HERON-55", "answer_head": "The secret code word is HERON-55.\n\nElias Thorne had lived in the solitary stone tower of Blackwood Point for thirty years, a man carved from the same granite that formed the cliff beneath his feet. To the outside world, his life was a monot"} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 8192, "run": 0, "warmup": false, "peak_gpu_gb": 5.09, "swap_delta_gb": 0.0, "min_free_pct": 74, "status": "OK", "prompt_tokens": 9811, "prefill_tps": 400.6, "ttft_s": 24.57, "decode_tps": 12.81, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "HERON-51", "answer_head": "The secret code word is HERON-51.\n\nElias Thorne had not seen the sun rise in three days, a fact that weighed heavier on his shoulders than the salt-crusted wool of his coat. He was the keeper of the Needle, a jagged spire of granite that ju"} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 8192, "run": 1, "warmup": false, "peak_gpu_gb": 5.09, "swap_delta_gb": 0.0, "min_free_pct": 74, "status": "OK", "prompt_tokens": 9810, "prefill_tps": 399.8, "ttft_s": 24.62, "decode_tps": 12.68, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-58", "answer_head": "The secret code word is MARLIN-58.\n\nElias Thorne had lived in the solitary stone tower of Blackwood Point for thirty years, a man carved from the same granite that formed the cliff beneath his feet. To the outside world, his life was a mono"} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 8192, "run": 2, "warmup": false, "peak_gpu_gb": 5.56, "swap_delta_gb": 0.0, "min_free_pct": 74, "status": "OK", "prompt_tokens": 9872, "prefill_tps": 403.6, "ttft_s": 24.54, "decode_tps": 12.72, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "PELICAN-41", "answer_head": "The secret code word is PELICAN-41.\n\nElias Thorne had lived in the solitary stone tower of Blackwood Point for thirty years, a man carved from the same granite that formed the cliff beneath his feet. His days were not measured by the tickin"} -{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 32768, "run": 0, "warmup": false, "peak_gpu_gb": 5.8, "swap_delta_gb": 0.0, "min_free_pct": 73, "status": "OK", "prompt_tokens": 40806, "prefill_tps": 335.9, "ttft_s": 121.74, "decode_tps": 12.03, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "HERON-53", "answer_head": "The secret code word is HERON-53.\n\nThe salt spray had long since bleached Elias\u2019s skin to the color of old parchment, and the wind in his ears was a constant, roaring companion that no human voice could ever quite match. He was the keeper o"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 512, "run": -1, "warmup": true, "peak_gpu_gb": 19.8, "swap_delta_gb": 0.0, "min_free_pct": 26, "status": "OK", "prompt_tokens": 527, "prefill_tps": 253.4, "ttft_s": 2.09, "decode_tps": 48.36, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-60", "answer_head": "The secret code word is MARLIN-60.\n\nElias Thorne had spent forty years watching the sea from the top of the Blackwood Point Lighthouse. To the casual observer, his life was a monotonous cycle of polishing brass, trimming wicks, and logging ", "build": "b782"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 512, "run": 0, "warmup": false, "peak_gpu_gb": 20.18, "swap_delta_gb": 0.0, "min_free_pct": 25, "status": "OK", "prompt_tokens": 548, "prefill_tps": 662.4, "ttft_s": 0.84, "decode_tps": 48.42, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-23", "answer_head": "The secret code word is MARLIN-23.\n\nElias Thorne had lived with the sea for forty years, but the lighthouse on Blackwood Point was his true home. It was not merely a structure of stone and glass; it was a living entity that breathed with th", "build": "b782"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 512, "run": 1, "warmup": false, "peak_gpu_gb": 20.2, "swap_delta_gb": 0.0, "min_free_pct": 25, "status": "OK", "prompt_tokens": 560, "prefill_tps": 712.3, "ttft_s": 0.8, "decode_tps": 48.35, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "PELICAN-81", "answer_head": "The secret code word is PELICAN-81.\n\nElias Thorne had lived with the sea for forty years, his life measured not in hours, but in the rhythmic sweep of the Great Point Lighthouse beam. The structure itself was a sentinel of granite and iron,", "build": "b782"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 512, "run": 2, "warmup": false, "peak_gpu_gb": 20.11, "swap_delta_gb": 0.0, "min_free_pct": 25, "status": "OK", "prompt_tokens": 541, "prefill_tps": 686.9, "ttft_s": 0.8, "decode_tps": 48.05, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "HERON-20", "answer_head": "The secret code word is HERON-20.\n\nElias Thorne had lived with the sea’s rhythm for forty years, a man carved from the same salt-weathered granite as the lighthouse he tended. His world was not one of grand adventures or bustling harbors, b", "build": "b782"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 2048, "run": 0, "warmup": false, "peak_gpu_gb": 19.84, "swap_delta_gb": 0.0, "min_free_pct": 26, "status": "OK", "prompt_tokens": 2336, "prefill_tps": 964.9, "ttft_s": 2.46, "decode_tps": 46.92, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "PELICAN-68", "answer_head": "The secret code word is PELICAN-68.\n\nElias Thorne had lived with the sea’s rhythm for forty years, a man carved from salt spray and stubbornness. His home was the old granite lighthouse perched on the jagged cliffs of Blackwood Point, a sol", "build": "b782"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 2048, "run": 1, "warmup": false, "peak_gpu_gb": 20.39, "swap_delta_gb": 0.0, "min_free_pct": 26, "status": "OK", "prompt_tokens": 2350, "prefill_tps": 960.5, "ttft_s": 2.48, "decode_tps": 46.93, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-77", "answer_head": "The secret code word is **MARLIN-77**.\n\n***\n\nElias Thorne had not seen the sun rise in three days, but he knew it was coming. As the keeper of the Blackwood Point Lighthouse, his existence was measured not in hours, but in the rhythmic swee", "build": "b782"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 2048, "run": 2, "warmup": false, "peak_gpu_gb": 20.4, "swap_delta_gb": 0.0, "min_free_pct": 26, "status": "OK", "prompt_tokens": 2346, "prefill_tps": 958.7, "ttft_s": 2.48, "decode_tps": 46.83, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-75", "answer_head": "The secret code word is MARLIN-75.\n\nElias Thorne had lived with the sea’s rhythm for forty years, his life measured not in hours, but in the sweeping arc of the lighthouse beam cutting through the coastal fog. The structure, a stoic sentine", "build": "b782"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 512, "run": -1, "warmup": true, "peak_gpu_gb": 5.25, "swap_delta_gb": 0.0, "min_free_pct": 72, "status": "OK", "prompt_tokens": 547, "prefill_tps": 112.6, "ttft_s": 4.87, "decode_tps": 13.81, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "PELICAN-70", "answer_head": "The secret code word from the notes is **PELICAN-70**.\n\n***\n\nElias Thorne had not seen the sun rise in three days, a fact he noted with the same detached precision he applied to the rest of his duties at the Blackwood Point Lighthouse. The ", "build": "b782"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 512, "run": 0, "warmup": false, "peak_gpu_gb": 5.7, "swap_delta_gb": 0.0, "min_free_pct": 71, "status": "OK", "prompt_tokens": 562, "prefill_tps": 263.1, "ttft_s": 2.15, "decode_tps": 14.1, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "HERON-54", "answer_head": "The secret code word is HERON-54.\n\nElias Thorne had lived with the sea for forty years, his life measured not in hours, but in the rhythmic sweep of the Great Light at Cape Solitude. The lighthouse stood like a solitary finger pointing accu", "build": "b782"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 512, "run": 1, "warmup": false, "peak_gpu_gb": 5.28, "swap_delta_gb": 0.0, "min_free_pct": 71, "status": "OK", "prompt_tokens": 557, "prefill_tps": 263.5, "ttft_s": 2.13, "decode_tps": 14.1, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "OSPREY-34", "answer_head": "The secret code word from the notes is **OSPREY-34**.\n\n***\n\nElias Thorne had not seen another human face in three years, save for the occasional silhouette of a fishing boat against the gray horizon. His world was defined by the rhythmic pu", "build": "b782"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 512, "run": 2, "warmup": false, "peak_gpu_gb": 5.68, "swap_delta_gb": 0.0, "min_free_pct": 71, "status": "OK", "prompt_tokens": 551, "prefill_tps": 259.4, "ttft_s": 2.14, "decode_tps": 14.1, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "PELICAN-36", "answer_head": "The secret code word is PELICAN-36.\n\nElias Thorne had lived with the sea for forty years, but the lighthouse on Blackwood Point remained his only true companion. Each morning, he began his routine with the precision of a clockwork mechanism", "build": "b782"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 2048, "run": 0, "warmup": false, "peak_gpu_gb": 5.73, "swap_delta_gb": 0.0, "min_free_pct": 72, "status": "OK", "prompt_tokens": 2326, "prefill_tps": 407.1, "ttft_s": 5.75, "decode_tps": 14.1, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "OSPREY-26", "answer_head": "The secret code word is OSPREY-26.\n\nElias Thorne had lived with the sea for forty years, his life measured not in hours, but in the rhythmic pulse of the lighthouse beam cutting through the Atlantic fog. The structure, a towering sentinel o", "build": "b782"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 2048, "run": 1, "warmup": false, "peak_gpu_gb": 5.74, "swap_delta_gb": 0.0, "min_free_pct": 72, "status": "OK", "prompt_tokens": 2355, "prefill_tps": 410.2, "ttft_s": 5.78, "decode_tps": 14.16, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "OSPREY-65", "answer_head": "The secret code word is OSPREY-65.\n\nElias Thorne had lived with the sea’s rhythm for forty years, his life measured not in hours, but in the turning of the great Fresnel lens and the shifting of the tide. The lighthouse, a solitary sentinel", "build": "b782"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 2048, "run": 2, "warmup": false, "peak_gpu_gb": 5.28, "swap_delta_gb": 0.0, "min_free_pct": 72, "status": "OK", "prompt_tokens": 2332, "prefill_tps": 407.9, "ttft_s": 5.76, "decode_tps": 14.13, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-42", "answer_head": "The secret code word is MARLIN-42.\n\nElias Thorne had spent forty years as the keeper of the Blackwood Point Lighthouse, a solitary sentinel perched on the jagged cliffs overlooking the churning Atlantic. His life was a rhythm dictated by th", "build": "b782"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 8192, "run": -1, "warmup": true, "peak_gpu_gb": 20.52, "swap_delta_gb": 0.0, "min_free_pct": 23, "status": "OK", "prompt_tokens": 9768, "prefill_tps": 860.8, "ttft_s": 11.4, "decode_tps": 44.95, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-60", "answer_head": "The secret code word is MARLIN-60.\n\nElias Thorne had spent forty years watching the sea from the top of the Blackwood Point Lighthouse. To the untrained eye, his life was a monotonous cycle of polishing brass, trimming wicks, and logging th", "build": "b782"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 8192, "run": 0, "warmup": false, "peak_gpu_gb": 20.54, "swap_delta_gb": 0.0, "min_free_pct": 25, "status": "OK", "prompt_tokens": 9808, "prefill_tps": 924.2, "ttft_s": 10.7, "decode_tps": 44.91, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "OSPREY-24", "answer_head": "The secret code word is OSPREY-24.\n\nElias Thorne had lived in the solitary stone tower of Blackwood Point for thirty years, a man carved from the same granite that formed the cliff beneath his feet. His days were not measured by clocks, but", "build": "b782"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 8192, "run": 1, "warmup": false, "peak_gpu_gb": 20.54, "swap_delta_gb": 0.0, "min_free_pct": 25, "status": "OK", "prompt_tokens": 9808, "prefill_tps": 926.1, "ttft_s": 10.68, "decode_tps": 44.84, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "OSPREY-96", "answer_head": "The secret code word is OSPREY-96.\n\nElias Thorne had lived in the solitary stone tower of Blackwood Point for thirty years, a man carved from the same granite that formed the cliff beneath his feet. His days were not measured by clocks, but", "build": "b782"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 8192, "run": 2, "warmup": false, "peak_gpu_gb": 20.49, "swap_delta_gb": 0.0, "min_free_pct": 25, "status": "OK", "prompt_tokens": 9827, "prefill_tps": 927.4, "ttft_s": 10.68, "decode_tps": 44.68, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "HERON-83", "answer_head": "The secret code word is HERON-83.\n\nElias Thorne had been the keeper of the Blackwood Point Lighthouse for thirty years, a tenure marked not by dramatic storms, but by the relentless, monotonous rhythm of the sea. His days were a tapestry of", "build": "b782"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 32768, "run": 0, "warmup": false, "peak_gpu_gb": 22.07, "swap_delta_gb": 0.0, "min_free_pct": 20, "status": "OK", "prompt_tokens": 40830, "prefill_tps": 675.6, "ttft_s": 60.68, "decode_tps": 36.79, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "HERON-48", "answer_head": "The secret code word is HERON-48.\n\nElias Thorne had lived in the solitary stone tower of Blackwood Point for thirty years, a duration that felt both like a blink and an eternity. The lighthouse was not merely a structure; it was his compani", "build": "b782"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 8192, "run": -1, "warmup": true, "peak_gpu_gb": 5.43, "swap_delta_gb": 0.0, "min_free_pct": 73, "status": "OK", "prompt_tokens": 9772, "prefill_tps": 366.3, "ttft_s": 26.75, "decode_tps": 13.64, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "OSPREY-93", "answer_head": "The secret code word is OSPREY-93.\n\nElias Thorne had lived in the solitary stone tower of Blackwood Point for thirty years, a man carved from the same granite that formed the cliff beneath his feet. His days were not measured by clocks, but", "build": "b782"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 8192, "run": 0, "warmup": false, "peak_gpu_gb": 5.68, "swap_delta_gb": 0.0, "min_free_pct": 73, "status": "OK", "prompt_tokens": 9820, "prefill_tps": 415.3, "ttft_s": 23.73, "decode_tps": 13.89, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "PELICAN-81", "answer_head": "The secret code word is PELICAN-81.\n\nElias Thorne had not seen the sun in three days, a fact he noted with the same detached precision he applied to the rest of his duties at the Blackwood Point Lighthouse. The storm outside was a living th", "build": "b782"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 8192, "run": 1, "warmup": false, "peak_gpu_gb": 5.89, "swap_delta_gb": 0.0, "min_free_pct": 73, "status": "OK", "prompt_tokens": 9804, "prefill_tps": 415.0, "ttft_s": 23.71, "decode_tps": 13.87, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-98", "answer_head": "The secret code word is MARLIN-98.\n\nElias Thorne had lived in the solitary stone tower of Blackwood Point for thirty years, a man carved from the same granite that formed the cliff beneath his feet. His days were not measured by clocks, but", "build": "b782"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 8192, "run": 2, "warmup": false, "peak_gpu_gb": 5.56, "swap_delta_gb": 0.0, "min_free_pct": 73, "status": "OK", "prompt_tokens": 9868, "prefill_tps": 418.1, "ttft_s": 23.69, "decode_tps": 13.87, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "OSPREY-86", "answer_head": "The secret code word is OSPREY-86.\n\nElias Thorne had not seen the sun rise in three days, a fact that weighed heavier on his shoulders than the salt-crusted wool of his coat. He was the keeper of the Needle, a jagged spire of granite that j", "build": "b782"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 32768, "run": 0, "warmup": false, "peak_gpu_gb": 6.81, "swap_delta_gb": 0.0, "min_free_pct": 68, "status": "OK", "prompt_tokens": 40820, "prefill_tps": 346.5, "ttft_s": 118.07, "decode_tps": 13.02, "gen_tokens": 128, "needle_ok": false, "degenerate": false, "code": "OSPREY-56", "answer_head": "The secret code word from the notes is **OSPREY**.\n\n***\n\nThe salt spray had long since bleached Elias’s skin to the color of old parchment, and the wind on the jagged cliffs of Blackwood Point had become his only constant companion. For thi", "build": "b782"} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 32768, "run": 0, "warmup": false, "peak_gpu_gb": 6.11, "swap_delta_gb": 0.0, "min_free_pct": 71, "status": "OK", "prompt_tokens": 40852, "prefill_tps": 339.9, "ttft_s": 120.43, "decode_tps": 12.86, "gen_tokens": 64, "needle_ok": true, "degenerate": false, "code": "MARLIN-60", "answer_head": "The secret code word is MARLIN-60.\n\nThe salt spray had long since bleached Elias’s skin to the color of old parchment, a permanent reminder of his decades-long vigil atop the Needle Rock Lighthouse. Perched on a jagged outcrop miles from th", "build": "b782", "recheck": true} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "SSD", "context": 32768, "run": 1, "warmup": false, "peak_gpu_gb": 6.06, "swap_delta_gb": 0.0, "min_free_pct": 71, "status": "OK", "prompt_tokens": 40820, "prefill_tps": 347.0, "ttft_s": 117.9, "decode_tps": 13.0, "gen_tokens": 64, "needle_ok": false, "degenerate": false, "code": "OSPREY-67", "answer_head": "The secret code word from the notes is **OSPREY**.\n\n***\n\nThe salt spray had long since bleached Elias’s skin to the color of old parchment, and the wind on the jagged cliffs of Blackwood Point had become his only constant companion. For thi", "build": "b782", "recheck": true} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 32768, "run": 0, "warmup": false, "peak_gpu_gb": 21.81, "swap_delta_gb": 0.0, "min_free_pct": 18, "status": "OK", "prompt_tokens": 40852, "prefill_tps": 669.2, "ttft_s": 61.29, "decode_tps": 36.84, "gen_tokens": 64, "needle_ok": true, "degenerate": false, "code": "MARLIN-60", "answer_head": "The secret code word is MARLIN-60.\n\nElias Thorne had lived in the solitary stone tower of Blackwood Point for thirty years, a duration that felt both like a blink and an eternity. The lighthouse was not merely a structure; it was his compan", "build": "b782", "recheck": true} +{"model": "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit", "config": "Vanilla", "context": 32768, "run": 1, "warmup": false, "peak_gpu_gb": 22.01, "swap_delta_gb": 0.0, "min_free_pct": 20, "status": "OK", "prompt_tokens": 40820, "prefill_tps": 675.3, "ttft_s": 60.71, "decode_tps": 36.92, "gen_tokens": 64, "needle_ok": false, "degenerate": false, "code": "OSPREY-67", "answer_head": "The secret code word from the notes is **OSPERY**.\n\n***\n\nThe salt spray had long since bleached Elias’s skin to the color of old parchment, and the wind in the tower was a constant, howling companion that he had learned to ignore. For thirt", "build": "b782", "recheck": true} diff --git a/docs/profiling/m6/qwen36_35b_a3b_4bit.md b/docs/profiling/m6/qwen36_35b_a3b_4bit.md index 1691838..ec2e952 100644 --- a/docs/profiling/m6/qwen36_35b_a3b_4bit.md +++ b/docs/profiling/m6/qwen36_35b_a3b_4bit.md @@ -1,14 +1,18 @@ -### `unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit` +### `unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit` — release b782 -Apple M6 · 32 GB · runs=3 (long=1) · warmup=1 · gen=128 · temperature 0 · medians +Apple M6 · 32 GB · macOS 27.0 · **release b782** (official `SwiftLM-b782-macos-arm64.tar.gz`) · `scripts/profiling/m6_bench.py` · runs=3 (1 at 32K+) · warmup=1 · gen=128 · temperature 0 · medians + +`Vanilla` = all experts on the GPU. `SSD` = `--stream-experts`. | Config | Context (prompt tok) | Prefill tok/s | TTFT s | Decode tok/s | Peak GPU GB | Swap Δ GB | Min free % | Checks | |---|---|---|---|---|---|---|---|---| -| Vanilla | 512 (548) | 713.8 | 0.78 | 46.97 | 19.78 | 0.0 | 27 | ok | -| Vanilla | 2048 (2346) | 967.5 | 2.45 | 45.72 | 20.06 | 0.0 | 28 | ok | -| Vanilla | 8192 (9809) | 857.8 | 11.52 | 43.43 | 20.41 | 0.0 | 27 | ok | -| Vanilla | 32768 (40829) | 615.3 | 66.64 | 36.05 | 21.54 | 0.0 | 22 | ok | -| SSD | 512 (548) | 256.1 | 2.12 | 13.23 | 5.59 | 0.0 | 73 | ok | -| SSD | 2048 (2356) | 403.3 | 5.87 | 13.03 | 5.57 | 0.0 | 74 | ok | -| SSD | 8192 (9811) | 400.6 | 24.57 | 12.72 | 5.56 | 0.0 | 74 | ok | -| SSD | 32768 (40806) | 335.9 | 121.74 | 12.03 | 5.8 | 0.0 | 73 | ok | +| Vanilla | 512 (548) | 686.9 | 0.8 | 48.35 | 20.2 | 0.0 | 25 | ok | +| Vanilla | 2048 (2346) | 960.5 | 2.48 | 46.92 | 20.4 | 0.0 | 26 | ok | +| Vanilla | 8192 (9808) | 926.1 | 10.68 | 44.84 | 20.54 | 0.0 | 25 | ok | +| Vanilla | 32768 (40830) | 675.6 | 60.68 | 36.79 | 22.07 | 0.0 | 20 | ok | +| SSD | 512 (557) | 263.1 | 2.14 | 14.1 | 5.7 | 0.0 | 71 | ok | +| SSD | 2048 (2332) | 407.9 | 5.76 | 14.13 | 5.74 | 0.0 | 72 | ok | +| SSD | 8192 (9820) | 415.3 | 23.71 | 13.87 | 5.89 | 0.0 | 73 | ok | +| SSD | 32768 (40820) | 346.5 | 118.07 | 13.02 | 6.81 | 0.0 | 68 | needle 0/1, degen 0 | + +> **40.8K needle check:** the hidden code word is not recalled reliably at 40.8K tokens in either mode, so treat those rows as throughput only. Across 3 SSD runs and 2 GPU re-runs, every miss had an `OSPREY-NN` code word (answered `OSPREY` without the number, or misspelled `OSPERY`); `MARLIN-60` passed in both modes. On the same `OSPREY-67` prompt GPU and SSD fail alike, so it is the model, not SSD streaming. Every run at ≤9.8K passed. The re-runs are in the JSONL with `"recheck": true`. diff --git a/docs/profiling/m6/qwen38_27b_4bit.jsonl b/docs/profiling/m6/qwen38_27b_4bit.jsonl index 909062e..642b979 100644 --- a/docs/profiling/m6/qwen38_27b_4bit.jsonl +++ b/docs/profiling/m6/qwen38_27b_4bit.jsonl @@ -20,3 +20,10 @@ {"model": "mlx-community/Qwen3.8-27B-4bit", "config": "TurboKV", "context": 8192, "run": 1, "warmup": false, "peak_gpu_gb": 15.9, "swap_delta_gb": 0.0, "min_free_pct": 41, "status": "OK", "prompt_tokens": 9810, "prefill_tps": 250.4, "ttft_s": 39.25, "decode_tps": 8.89, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-58", "answer_head": "The secret code word is **MARLIN-58**.\n\nThe fog rolled in thick and heavy, swallowing the horizon until the world was nothing but a gray, humming void. Elias, the lighthouse keeper, stood on the gallery of the Ironclad Point Light, his brea"} {"model": "mlx-community/Qwen3.8-27B-4bit", "config": "TurboKV", "context": 8192, "run": 2, "warmup": false, "peak_gpu_gb": 15.34, "swap_delta_gb": 0.0, "min_free_pct": 42, "status": "OK", "prompt_tokens": 9872, "prefill_tps": 252.9, "ttft_s": 39.1, "decode_tps": 8.89, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "PELICAN-41", "answer_head": "The secret code word is **PELICAN-41**.\n\nThe fog rolled in thick and gray, swallowing the horizon until the world was nothing but a white void and the rhythmic, mournful cry of the gulls. Elias, the lighthouse keeper, stood on the gallery o"} {"model": "mlx-community/Qwen3.8-27B-4bit", "config": "TurboKV", "context": 32768, "run": 0, "warmup": false, "peak_gpu_gb": 16.86, "swap_delta_gb": 0.0, "min_free_pct": 37, "status": "OK", "prompt_tokens": 40806, "prefill_tps": 201.7, "ttft_s": 202.54, "decode_tps": 7.87, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "HERON-53", "answer_head": "The secret code word is HERON-53.\n\nElias had kept the lighthouse on the jagged tip of Cape Morrow for thirty years, a tenure that had worn the smoothness from his face and the color from his hair. The lighthouse, a stark white cylinder agai"} +{"model": "mlx-community/Qwen3.8-27B-4bit", "config": "Vanilla", "context": 512, "run": -1, "warmup": true, "peak_gpu_gb": 14.71, "swap_delta_gb": 0.0, "min_free_pct": 42, "status": "OK", "prompt_tokens": 527, "prefill_tps": 208.3, "ttft_s": 2.54, "decode_tps": 9.55, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-60", "answer_head": "The secret code word is **MARLIN-60**.\n\nThe fog rolled in thick and heavy, swallowing the horizon until the world was nothing but a gray, humming void. Elias, the lighthouse keeper, stood on the gallery of the Ironclad Point Light, his brea", "build": "b782"} +{"model": "mlx-community/Qwen3.8-27B-4bit", "config": "Vanilla", "context": 512, "run": 0, "warmup": false, "peak_gpu_gb": 15.19, "swap_delta_gb": 0.0, "min_free_pct": 42, "status": "OK", "prompt_tokens": 548, "prefill_tps": 231.9, "ttft_s": 2.38, "decode_tps": 9.5, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-23", "answer_head": "The secret code word is MARLIN-23.\n\nElias had kept the lighthouse on the jagged tip of Blackrock Point for thirty years, a tenure that had worn the salt into his very bones. The light, a massive Fresnel lens that swept the dark ocean in a r", "build": "b782"} +{"model": "mlx-community/Qwen3.8-27B-4bit", "config": "Vanilla", "context": 512, "run": 1, "warmup": false, "peak_gpu_gb": 15.19, "swap_delta_gb": 0.0, "min_free_pct": 42, "status": "OK", "prompt_tokens": 560, "prefill_tps": 236.9, "ttft_s": 2.38, "decode_tps": 9.51, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "PELICAN-81", "answer_head": "The secret code word is PELICAN-81.\n\nElias had kept the lighthouse on Blackrock Point for thirty years, a tenure that had worn the salt into his very bones. The light, a massive Fresnel lens that swept the dark ocean in a rhythmic, hypnotic", "build": "b782"} +{"model": "mlx-community/Qwen3.8-27B-4bit", "config": "Vanilla", "context": 512, "run": 2, "warmup": false, "peak_gpu_gb": 15.37, "swap_delta_gb": 0.0, "min_free_pct": 41, "status": "OK", "prompt_tokens": 541, "prefill_tps": 234.3, "ttft_s": 2.32, "decode_tps": 9.51, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "HERON-20", "answer_head": "The secret code word is HERON-20.\n\nElias had kept the lighthouse on the jagged tip of Blackwood Point for thirty years, a tenure that had worn the salt into his very bones. The tower stood like a white bone against the dark, churning sea, a", "build": "b782"} +{"model": "mlx-community/Qwen3.8-27B-4bit", "config": "Vanilla", "context": 8192, "run": 0, "warmup": false, "peak_gpu_gb": 15.42, "swap_delta_gb": 0.0, "min_free_pct": 41, "status": "OK", "prompt_tokens": 9815, "prefill_tps": 264.4, "ttft_s": 37.21, "decode_tps": 9.16, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "PELICAN-68", "answer_head": "The secret code word is PELICAN-68.\n\nThe fog rolled in thick and heavy, swallowing the horizon until the world was nothing but a gray, humming void. Elias, the lighthouse keeper, stood at the base of the spiral staircase, his boots heavy wi", "build": "b782"} +{"model": "mlx-community/Qwen3.8-27B-4bit", "config": "Vanilla", "context": 8192, "run": 1, "warmup": false, "peak_gpu_gb": 16.08, "swap_delta_gb": 0.0, "min_free_pct": 39, "status": "OK", "prompt_tokens": 9766, "prefill_tps": 263.1, "ttft_s": 37.21, "decode_tps": 9.17, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "HERON-85", "answer_head": "The secret code word is HERON-85.\n\nThe fog rolled in thick and heavy, swallowing the horizon until the world was nothing but a gray, humming void. Elias, the lighthouse keeper, stood at the base of the spiral staircase, his boots heavy with", "build": "b782"} +{"model": "mlx-community/Qwen3.8-27B-4bit", "config": "Vanilla", "context": 8192, "run": 2, "warmup": false, "peak_gpu_gb": 15.84, "swap_delta_gb": 0.0, "min_free_pct": 41, "status": "OK", "prompt_tokens": 9822, "prefill_tps": 266.6, "ttft_s": 36.93, "decode_tps": 9.16, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-30", "answer_head": "The secret code word is **MARLIN-30**.\n\nThe fog rolled in thick and heavy, swallowing the horizon until the world was nothing but a gray, humming void. Elias, the lighthouse keeper, stood at the base of the spiral staircase, his boots heavy", "build": "b782"} diff --git a/docs/profiling/m6/qwen38_27b_4bit.md b/docs/profiling/m6/qwen38_27b_4bit.md index c4e1684..8d543ee 100644 --- a/docs/profiling/m6/qwen38_27b_4bit.md +++ b/docs/profiling/m6/qwen38_27b_4bit.md @@ -1,4 +1,17 @@ -### `mlx-community/Qwen3.8-27B-4bit` +### `mlx-community/Qwen3.8-27B-4bit` — release b782 + +Apple M6 · 32 GB · macOS 27.0 · **release b782** (official `SwiftLM-b782-macos-arm64.tar.gz`) · `scripts/profiling/m6_bench.py` · runs=3 (1 at 32K+) · warmup=1 · gen=128 · temperature 0 · medians + +Only 548 and 9.8K tokens were re-run on b782. Process RAM at 9.8K was 16.1–16.7 GB (server log). + +| Config | Context (prompt tok) | Prefill tok/s | TTFT s | Decode tok/s | Peak GPU GB | Swap Δ GB | Min free % | Checks | +|---|---|---|---|---|---|---|---|---| +| Vanilla | 512 (548) | 234.3 | 2.38 | 9.51 | 15.37 | 0.0 | 41 | ok | +| Vanilla | 8192 (9815) | 264.4 | 37.21 | 9.16 | 16.08 | 0.0 | 39 | ok | + +--- + +#### Earlier build (before b782) > **Correction (Sep 24):** the `TurboKV` rows below were run with `--ctx-size`, which on mlx-swift-lm 460ff81 gives the attention layers a `RotatingKVCache`, and `--turbo-kv` only applies to `KVCacheSimple`. So these rows are effectively vanilla, not TurboKV. See #175 for real TurboKV behaviour. From 69568f4ca04dbb0a9243409737a74a863f6b8e43 Mon Sep 17 00:00:00 2001 From: simba Date: Sun, 27 Sep 2026 20:49:32 -0700 Subject: [PATCH 2/2] docs(readme): M6 tables from release b782 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Refresh the Mac mini M6 section to the b782 raw data in docs/profiling/m6: Qwen3.6 GPU/SSD at all four lengths, Gemma 4 4-bit short prompt, Gemma 4 8-bit --stream-experts, Qwen3.8 at 548 and 9.8K. Rows not re-run on b782 are marked † (earlier build). The 40.8K Qwen3.6 rows are marked throughput only (needle check unreliable in both modes). The M6/M1 Ultra line becomes 78% (48.4 vs 61.7) and notes the M1 Ultra runs used temperature 0.6 with --repeat-penalty 1.1; the M1 Ultra table is unchanged. Co-Authored-By: Claude Opus 5.5 --- README.md | 55 ++++++++++++++++++++++++++++++------------------------- 1 file changed, 30 insertions(+), 25 deletions(-) diff --git a/README.md b/README.md index bf31a2c..2666b9d 100644 --- a/README.md +++ b/README.md @@ -59,34 +59,37 @@ The first SwiftLM numbers from a **32 GB** Mac. Every other table in this README > *Hardware:* Mac mini (Mac18,5), Apple M6, 12-core GPU, 32 GB unified memory (170 GB/s), macOS 27.0. Metal working set 26.8 GB. > *Method:* [`scripts/profiling/m6_bench.py`](scripts/profiling/m6_bench.py). One warm-up, then the median of 3 runs (1 run at 32K and above), temperature 0. Every prompt starts with a unique nonce, so the prompt cache can't hit, and hides a code word that the answer must return. A memory guard aborts any case whose swap grows by more than 2 GB. Raw results: [`docs/profiling/m6/`](docs/profiling/m6/). +> *Build:* numbers are from the official **release b782** binary unless marked † (earlier build, not re-run on b782). ### What runs well on a 32 GB M6 | Model (4-bit unless noted) | Weights | Mode | Decode, short prompt | Longest prompt that passed | Peak GPU | |---|---|---|---|---|---| -| **`gemma-4-26b-a4b-it-4bit`** (MoE, ~4B active) | 15.3 GB | GPU | **52.2 tok/s** | 80.7K tokens | 19.5 GB | -| **`Qwen3.6-35B-A3B-UD-MLX-4bit`** (MoE, ~3B active) | 21.6 GB | GPU | **47.0 tok/s** | 40.8K tokens | 21.5 GB | -| `Qwen3.6-35B-A3B-UD-MLX-4bit` | 21.6 GB | `--stream-experts` | 13.2 tok/s | 40.8K tokens | 5.8 GB | -| `Qwen3.8-27B-4bit` (dense) | 11.3 GB | GPU | 9.3 tok/s | 40.8K tokens | 18.4 GB | -| `gemma-4-26b-a4b-it-8bit` | ~26 GB | GPU | swaps (+3.1 GB on the first prompt) | — | — | -| `gemma-4-26b-a4b-it-8bit` | ~26 GB | `--stream-experts` | 9.1 tok/s | 9.5K tokens (32K swapped) | 7.3 GB | +| **`gemma-4-26b-a4b-it-4bit`** (MoE, ~4B active) | 15.3 GB | GPU | **53.9 tok/s** | 80.7K tokens † | 19.5 GB † | +| **`Qwen3.6-35B-A3B-UD-MLX-4bit`** (MoE, ~3B active) | 21.6 GB | GPU | **48.4 tok/s** | 40.8K tokens ‡ | 22.1 GB | +| `Qwen3.6-35B-A3B-UD-MLX-4bit` | 21.6 GB | `--stream-experts` | 14.1 tok/s | 40.8K tokens ‡ | 6.8 GB | +| `Qwen3.8-27B-4bit` (dense) | 11.3 GB | GPU | 9.5 tok/s | 40.8K tokens † | 18.4 GB † | +| `gemma-4-26b-a4b-it-8bit` | ~26 GB | GPU | swaps (+3.1 GB on the first prompt) † | — | — | +| `gemma-4-26b-a4b-it-8bit` | ~26 GB | `--stream-experts` | 9.7 tok/s | 9.5K tokens (32K swapped †) | 7.1 GB | + +‡ Throughput only: at 40.8K the needle check isn't reliable in either mode (see the Qwen3.6 table). - **MoE models are the sweet spot at 32 GB.** Only the active experts are read for each token, so they decode 5–6× faster than a dense 27B. A 4-bit MoE with up to about 22 GB of weights runs entirely on the GPU. -- **Qwen3.6-35B-A3B on a base M6 reaches 76%** of the M1 Ultra 64 GB decode speed below (47.0 vs 61.7 tok/s). -- **Dense 27B decode is bandwidth-bound.** 9.3 tok/s × 11.3 GB is about 105 GB/s, roughly 60% of the M6's rated 170 GB/s. +- **Qwen3.6-35B-A3B on a base M6 reaches 78%** of the M1 Ultra 64 GB decode speed below (48.4 vs 61.7 tok/s). The M1 Ultra runs used temperature 0.6 and `--repeat-penalty 1.1`; the M6 runs use temperature 0. +- **Dense 27B decode is bandwidth-bound.** 9.5 tok/s × 11.3 GB is about 107 GB/s, roughly 63% of the M6's rated 170 GB/s. - **An 8-bit 26 GB model needs SSD streaming** and tops out at about 10K tokens of context. ### Gemma-4-26B-A4B 4-bit — by prompt length | Prompt tokens | Prefill / decode (tok/s) | TTFT | Peak GPU · swap growth | |---|---|---|---| -| ~530 | 733 / **52.2** | 0.8 s | 14.5 GB · 0 | -| ~2.3K | **963** / 50.2 | 2.5 s | 15.0 GB · 0 | -| ~9.5K | 959 / 45.1 | 10.1 s | 15.8 GB · 0 | -| ~39.7K | 757 / 31.0 | 53.3 s | 17.9 GB · 0 | -| ~80.7K | 622 / 24.3 | 131.3 s | 19.5 GB · 0 | +| ~530 | 781 / **53.9** | 0.7 s | 14.3 GB · 0 | +| ~2.3K † | **963** / 50.2 | 2.5 s | 15.0 GB · 0 | +| ~9.5K † | 959 / 45.1 | 10.1 s | 15.8 GB · 0 | +| ~39.7K † | 757 / 31.0 | 53.3 s | 17.9 GB · 0 | +| ~80.7K † | 622 / 24.3 | 131.3 s | 19.5 GB · 0 | -Every needle check passed. `--mtp` with the bf16 assistant (`gemma-4-26B-A4B-it-assistant-bf16`) works but is slower on the M6: 45.2 / 35.6 / 30.4 tok/s decode at ~530 / 2.3K / 9.5K tokens, against 53.0 / 50.8 / 46.0 without it. A 4-bit MoE is compute-bound, so verifying the drafted tokens costs more than it saves (the same finding as the M5 Pro tables below). ⚠️ `--mtp` also currently changes the output at temperature 0: once the context passes Gemma's 1,024-token sliding window, rejected drafts aren't rolled back ([#184](https://github.com/SharpAI/SwiftLM/issues/184)). Avoid it until that's fixed. +Every needle check passed. `--mtp` with the bf16 assistant (`gemma-4-26B-A4B-it-assistant-bf16`) works but is slower on the M6 (earlier build): 45.2 / 35.6 / 30.4 tok/s decode at ~530 / 2.3K / 9.5K tokens, against 53.0 / 50.8 / 46.0 without it. A 4-bit MoE is compute-bound, so verifying the drafted tokens costs more than it saves (the same finding as the M5 Pro tables below). ⚠️ `--mtp` also currently changes the output at temperature 0: once the context passes Gemma's 1,024-token sliding window, rejected drafts aren't rolled back ([#184](https://github.com/SharpAI/SwiftLM/issues/184)). Avoid it until that's fixed. ### Qwen3.6-35B-A3B 4-bit — GPU vs SSD streaming @@ -94,10 +97,12 @@ Every needle check passed. `--mtp` with the bf16 assistant (`gemma-4-26B-A4B-it- | Prompt tokens | GPU prefill / decode (tok/s) | GPU peak | `--stream-experts` prefill / decode (tok/s) | SSD peak | |---|---|---|---|---| -| ~550 | 714 / 47.0 | 19.8 GB | 256 / 13.2 | 5.6 GB | -| ~2.3K | 968 / 45.7 | 20.1 GB | 403 / 13.0 | 5.6 GB | -| ~9.8K | 858 / 43.4 | 20.4 GB | 401 / 12.7 | 5.6 GB | -| 40.8K | 615 / 36.1 | 21.5 GB | 336 / 12.0 | 5.8 GB | +| ~550 | 687 / 48.4 | 20.2 GB | 263 / 14.1 | 5.7 GB | +| ~2.3K | 961 / 46.9 | 20.4 GB | 408 / 14.1 | 5.7 GB | +| ~9.8K | 926 / 44.8 | 20.5 GB | 415 / 13.9 | 5.9 GB | +| 40.8K ‡ | 676 / 36.8 | 22.1 GB | 347 / 13.0 | 6.8 GB | + +Release b782 ([raw data](docs/profiling/m6/qwen36_35b_a3b_4bit.md)). ‡ At 40.8K the needle check isn't reliable in either mode: every miss had an `OSPREY-NN` code word, and GPU and SSD fail alike on the same prompt, so that row is throughput only. Every run up to 9.8K passed. > ⚠️ **`--stream-experts` crashes on quantized MoE models in releases b769 and b773** (`broadcast_shapes … (N,8,8,D)` on the first request). Reproduced on M6 with Qwen3.6-35B-A3B and Gemma 4 26B-A4B. The mlx-swift-lm upstream sync in #167 broke the SSD path. Earlier versions of this table were measured before that sync and were never re-checked afterwards. Fixed in SharpAI/mlx-swift-lm#69 and #71; the table above was re-measured with those fixes. @@ -105,10 +110,10 @@ Every needle check passed. `--mtp` with the bf16 assistant (`gemma-4-26B-A4B-it- | Prompt tokens | Prefill / decode (tok/s) | Peak GPU · swap growth | |---|---|---| -| ~550 | 233 / 9.3 | 15.2 GB · 0 | -| ~2.3K | 242 / 9.1 | 16.1 GB · 0 | -| ~9.8K | 238 / 8.9 | 16.9 GB · 0 | -| ~40.8K | 200 / 7.9 | 18.4 GB · 0 | +| ~550 | 234 / 9.5 | 15.4 GB · 0 | +| ~2.3K † | 242 / 9.1 | 16.1 GB · 0 | +| ~9.8K | 264 / 9.2 | 16.1 GB · 0 | +| ~40.8K † | 200 / 7.9 | 18.4 GB · 0 | > **Correction:** an earlier version of these tables had `--turbo-kv` columns. Those runs passed `--ctx-size`, which on this mlx-swift-lm pin gives the attention layers a `RotatingKVCache`, and `--turbo-kv` only applies to `KVCacheSimple`. They were really vanilla runs, so the columns were removed. @@ -116,8 +121,8 @@ Every needle check passed. `--mtp` with the bf16 assistant (`gemma-4-26B-A4B-it- | 8.5K-token prompt, Qwen3.8-27B-4bit | Before (old pin) | After (this release) | |---|---|---| -| Prefill | 33.4 tok/s | **~240 tok/s** (≈7×; 238 tok/s measured at 9.8K) | -| Peak memory | 38 GB process footprint | **≤19 GB** process (16.9 GB GPU peak at 9.8K) | +| Prefill | 33.4 tok/s | **~264 tok/s** (≈8×; measured at 9.8K on b782) | +| Peak memory | 38 GB process footprint | **≤19 GB** process (16.1–16.7 GB process, 16.1 GB GPU peak at 9.8K on b782) | | Swap growth | +15 GB | **0** | 1. **The MLX buffer cache was unbounded on full-GPU loads.** It could grow to the whole 26.8 GB working set. It is now sized from the RAM left after weights and KV. @@ -129,7 +134,7 @@ Every needle check passed. `--mtp` with the bf16 assistant (`gemma-4-26B-A4B-it- > > ℹ️ QAT-quantized Gemma 4 MTP assistants (`…-qat-assistant-4bit`) now load and run (SharpAI/mlx-swift-lm#66). > -> ℹ️ **`--gpu-layers N` with MoE models:** the Metal GPU timeout on the first request ([#176](https://github.com/SharpAI/SwiftLM/issues/176)) is fixed in #177. CPU-resident MoE layers still run on a single core and are very slow (Gemma 4 26B-A4B with `--gpu-layers 23`: ~0.4 tok/s prefill, ~3 s per decoded token on both M5 and M6), and SwiftLM now warns about this at startup. Treat `--gpu-layers` as a way to avoid running out of memory, not as a speed trade-off. On a 32 GB Mac try `--stream-experts` first (Qwen3.6-35B-A3B: 13.2 tok/s decode, 5.8 GB GPU). +> ℹ️ **`--gpu-layers N` with MoE models:** the Metal GPU timeout on the first request ([#176](https://github.com/SharpAI/SwiftLM/issues/176)) is fixed in #177. CPU-resident MoE layers still run on a single core and are very slow (Gemma 4 26B-A4B with `--gpu-layers 23`: ~0.4 tok/s prefill, ~3 s per decoded token on both M5 and M6), and SwiftLM now warns about this at startup. Treat `--gpu-layers` as a way to avoid running out of memory, not as a speed trade-off. On a 32 GB Mac try `--stream-experts` first (Qwen3.6-35B-A3B on b782: 14.1 tok/s decode, 5.7 GB GPU). Reproduce: