From f27085c67eb2a3c63409dbfdd5351aa9a06ae969 Mon Sep 17 00:00:00 2001 From: simba Date: Thu, 24 Sep 2026 23:12:20 -0700 Subject: [PATCH] docs(readme): flag --mtp output change at temperature 0 (#184) Co-Authored-By: Claude Opus 5.5 --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 952842d..f82b5c8 100644 --- a/README.md +++ b/README.md @@ -86,7 +86,7 @@ The first SwiftLM numbers from a **32 GB** Mac. Every other table in this README | ~39.7K | 757 / 31.0 | 53.3 s | 17.9 GB · 0 | | ~80.7K | 622 / 24.3 | 131.3 s | 19.5 GB · 0 | -Every needle check passed. `--mtp` with the bf16 assistant (`gemma-4-26B-A4B-it-assistant-bf16`) works but is slower on the M6: 45.2 / 35.6 / 30.4 tok/s decode at ~530 / 2.3K / 9.5K tokens, against 53.0 / 50.8 / 46.0 without it. A 4-bit MoE is compute-bound, so verifying the drafted tokens costs more than it saves (the same finding as the M5 Pro tables below). +Every needle check passed. `--mtp` with the bf16 assistant (`gemma-4-26B-A4B-it-assistant-bf16`) works but is slower on the M6: 45.2 / 35.6 / 30.4 tok/s decode at ~530 / 2.3K / 9.5K tokens, against 53.0 / 50.8 / 46.0 without it. A 4-bit MoE is compute-bound, so verifying the drafted tokens costs more than it saves (the same finding as the M5 Pro tables below). ⚠️ `--mtp` also currently changes the output at temperature 0: once the context passes Gemma's 1,024-token sliding window, rejected drafts aren't rolled back ([#184](https://github.com/SharpAI/SwiftLM/issues/184)). Avoid it until that's fixed. ### Qwen3.6-35B-A3B 4-bit — GPU vs SSD streaming