With --turbo-kv, Qwen3.8-27B-4bit loses exact recall once the prompt is past a few thousand tokens on an M5 Pro. The same build, model, and prompt on an M6 return correct answers.
Setup
- SwiftLM
main @ 318f712, release build
SwiftLM --model mlx-community/Qwen3.8-27B-4bit --vision [--turbo-kv]- Apple M5 Pro, 64 GB, macOS 26.6.2
temperature: 0, max_tokens: 64, one model loaded at a time
Prompt
N lines of Line i: the quick brown fox jumps over the lazy dog., joined with newlines, followed by \n\nHow many lines are there? Answer briefly.
Results (M5 Pro)
| lines |
prompt tokens |
--turbo-kv |
no --turbo-kv |
| 125 |
2,036 |
"There are 125 lines." (2/2) |
– |
| 315 |
5,266 |
"There are 1,000 lines." (2/2) |
"315 lines." |
| 500 |
8,411 |
"There are 14 lines." (2/2) |
"500" |
| 600 |
10,111 |
"There are 1,000 lines." (2/2) |
– |
| 700 |
11,811 |
"There are 1,000 lines." (3/3) |
"There are 700 lines." (3/3) |
The wrong answers are deterministic across repeats. They start somewhere between ~2K and ~5K prompt tokens.
M6
The same prompts at ~2K / 5K / 8K / 11K tokens are correct 24/24, with and without --turbo-kv.
Suspected cause
A GPU-family-dependent path in the TurboKV dequant/attention kernels, e.g. a kernel variant or threadgroup configuration selected differently on M5 than on M6. I have not bisected which kernel yet.
Workaround
On M5, don't use --turbo-kv when exact recall matters.
The README known issue is added in #174.
With
--turbo-kv, Qwen3.8-27B-4bit loses exact recall once the prompt is past a few thousand tokens on an M5 Pro. The same build, model, and prompt on an M6 return correct answers.Setup
main@ 318f712, release buildSwiftLM --model mlx-community/Qwen3.8-27B-4bit --vision [--turbo-kv]temperature: 0,max_tokens: 64, one model loaded at a timePrompt
N lines of
Line i: the quick brown fox jumps over the lazy dog., joined with newlines, followed by\n\nHow many lines are there? Answer briefly.Results (M5 Pro)
--turbo-kv--turbo-kvThe wrong answers are deterministic across repeats. They start somewhere between ~2K and ~5K prompt tokens.
M6
The same prompts at ~2K / 5K / 8K / 11K tokens are correct 24/24, with and without
--turbo-kv.Suspected cause
A GPU-family-dependent path in the TurboKV dequant/attention kernels, e.g. a kernel variant or threadgroup configuration selected differently on M5 than on M6. I have not bisected which kernel yet.
Workaround
On M5, don't use
--turbo-kvwhen exact recall matters.The README known issue is added in #174.