fix(turbokv): warn when --turbo-kv can't apply (e.g. with --ctx-size) - #181
Merged
Merged
Conversation
On mlx-swift-lm 460ff81 a bounded context (--ctx-size) gives the attention layers a RotatingKVCache, and the server only enables TurboQuant on KVCacheSimple layers. So `--turbo-kv --ctx-size N` printed turbo_kv=enabled and then silently compressed nothing. The M6 TurboKV benchmarks in #173 hit exactly this (corrected in #180). - Startup: warn when --turbo-kv and --ctx-size are both set, and say to drop --ctx-size to use --turbo-kv. - Per request: count the layers TurboQuant was enabled on, and warn once if none were (covers any model or cache setup where TurboKV can't attach). No change to cache selection or to TurboKV behaviour. Verified on a Mac mini M6 with Qwen3.8-27B-4bit, two requests per setup: --turbo-kv --ctx-size 16384 → both warnings, the per-request one once --turbo-kv → no warnings --ctx-size 16384 → no warnings Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Member
Author
|
Reviewed (code-review skill, M5 agent): no issues found. I checked that |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On mlx-swift-lm
460ff81,--ctx-size Ngives the attention layers aRotatingKVCache(e.g.Qwen35makeCache(capacity:)), andServer.swiftonly enables TurboQuant onKVCacheSimplelayers. So--turbo-kv --ctx-size Nlogsturbo_kv=enabledand then silently compresses nothing. The M6 TurboKV numbers in #173 were really vanilla because of this (corrected in #180), and it hid #175 on the M6.Fix: make it loud, don't change behaviour
--turbo-kvand--ctx-sizeare both set, print:⚠️ --turbo-kv has no effect with --ctx-size N: a bounded context gives the attention layers a RotatingKVCache, and TurboKV only compresses KVCacheSimple. Drop --ctx-size to use --turbo-kv.Cache selection and TurboKV itself are unchanged. Making TurboKV work on
RotatingKVCacheis a separate decision, and while #175 stands (TurboKV loses long-range recall), a warning is the safer change.Verified (Mac mini M6, Qwen3.8-27B-4bit, 2 requests per setup)
--turbo-kv --ctx-size 16384--turbo-kv--ctx-size 16384🤖 Generated with Claude Code