Skip to content

fix(turbokv): warn when --turbo-kv can't apply (e.g. with --ctx-size) - #181

Merged
solderzzc merged 1 commit into
mainfrom
fix/turbokv-ctx-size-warning
Sep 25, 2026
Merged

solderzzc merged 1 commit into
mainfrom
fix/turbokv-ctx-size-warning

Conversation

@solderzzc

Copy link
Copy Markdown
Member

Problem

On mlx-swift-lm 460ff81, --ctx-size N gives the attention layers a RotatingKVCache (e.g. Qwen35 makeCache(capacity:)), and Server.swift only enables TurboQuant on KVCacheSimple layers. So --turbo-kv --ctx-size N logs turbo_kv=enabled and then silently compresses nothing. The M6 TurboKV numbers in #173 were really vanilla because of this (corrected in #180), and it hid #175 on the M6.

Fix: make it loud, don't change behaviour

  • Startup: when --turbo-kv and --ctx-size are both set, print:
    ⚠️ --turbo-kv has no effect with --ctx-size N: a bounded context gives the attention layers a RotatingKVCache, and TurboKV only compresses KVCacheSimple. Drop --ctx-size to use --turbo-kv.
  • Per request: the enable loop counts the layers TurboQuant was switched on for, and warns once if there were none. That also covers other models or cache setups where TurboKV can't attach.

Cache selection and TurboKV itself are unchanged. Making TurboKV work on RotatingKVCache is a separate decision, and while #175 stands (TurboKV loses long-range recall), a warning is the safer change.

Verified (Mac mini M6, Qwen3.8-27B-4bit, 2 requests per setup)

flags warnings
--turbo-kv --ctx-size 16384 startup + per-request (printed once)
--turbo-kv none
--ctx-size 16384 none

🤖 Generated with Claude Code

On mlx-swift-lm 460ff81 a bounded context (--ctx-size) gives the attention
layers a RotatingKVCache, and the server only enables TurboQuant on
KVCacheSimple layers. So `--turbo-kv --ctx-size N` printed turbo_kv=enabled
and then silently compressed nothing. The M6 TurboKV benchmarks in #173 hit
exactly this (corrected in #180).

- Startup: warn when --turbo-kv and --ctx-size are both set, and say to drop
  --ctx-size to use --turbo-kv.
- Per request: count the layers TurboQuant was enabled on, and warn once if
  none were (covers any model or cache setup where TurboKV can't attach).

No change to cache selection or to TurboKV behaviour. Verified on a Mac mini M6
with Qwen3.8-27B-4bit, two requests per setup:
  --turbo-kv --ctx-size 16384 → both warnings, the per-request one once
  --turbo-kv                  → no warnings
  --ctx-size 16384            → no warnings

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@solderzzc

Copy link
Copy Markdown
Member Author

Reviewed (code-review skill, M5 agent): no issues found. I checked that --ctx-size goes through makeAttentionKVCache / makeHybridAttentionKVCache, which return RotatingKVCache when a capacity is set, so the startup warning holds for the models in the README. The per-request check also covers other cache types (e.g. CacheList). Merging.

@solderzzc
solderzzc merged commit d37d555 into main Sep 25, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant