fix(memory): run cleanly on 32 GB Macs + Mac mini M6 benchmarks

[solderzzc]

Stacked on #172 (the Xcode 27 build fix). Merge #172 first; this PR then retargets to main. Rebased onto #172 @ 8c14389, which already has main @ 318f712 merged in (#167: mlx-swift-lm → 460ff81).

Why

Every published SwiftLM number so far came from a 64 GB Mac. On a base Mac mini M6 with 32 GB (Mac18,5, 12-core GPU, 170 GB/s, macOS 27.0), the same workloads swapped, crashed, or wouldn't load. Each cause below was invisible on 64 GB.

Fixes

1. Cap the MLX buffer cache on full-GPU loads (ModelProfiler.swift, Server.swift). The .fullGPU branch printed the plan but never set Memory.cacheLimit, so MLX kept its default: the whole Metal working set (26.8 GB here). Prefill chunk buffers change shape as the KV offset grows, so they're rarely reused, and the cache grows until macOS swaps. The cache now gets half of the RAM left after weights, KV and an 8 GB OS reserve (floor 1 GB, capped at the old default). 64 GB machines see little change.
2. KV estimate only counts layers that actually hold a KV cache (ModelProfiler.swift). It used to count every layer as full attention. It now reads layer_types, sliding_window, num_global_key_value_heads and global_head_dim. At 64K context: Gemma-4-26B 16.1 → 1.6 GB, Qwen3.8-27B 17.2 → 4.4 GB. On 32 GB the Gemma overestimate flipped the plan to SWAP-ASSISTED with 7/30 layers on the CPU, and that path crashed with kIOGPUCommandBufferCallbackErrorTimeout, even on a 521-token prompt. Configs without layer_types keep the old estimate.
3. Auto-detected VLM that fails to load → fall back to text-only (Server.swift). unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit ships a preprocessor_config.json without image_mean. Vision was auto-enabled, the loader threw, and the server exited. It now logs the error and loads through LLMModelFactory with isVision = false. An explicit --vision still fails hard. Review follow-up: the fallback fires only when isVLMCheckpointMismatch(error) is true (DecodingError, MLXNN UpdateError, or ModelFactoryError for an unsupported type or a bad config). Cancellation, network and file-I/O errors propagate unchanged. VLMFallbackTests has 6 tests.

Tooling, docs, results

• scripts/profiling/m6_bench.py: one server per config, warm-up plus medians of N runs, a per-run nonce (so the prompt cache can't hit), a hidden code-word check, and a memory guard that kills the server and records MEM_ABORT when swap grows more than 2 GB or free memory drops below 10%.
• scripts/demo/stream_client.py: the streaming client used for the terminal recordings. It prints live prefill progress with server memory and swap.
• README.md: new "Mac mini M6 (base, 32 GB)" section (a "what runs well on 32 GB" table, per-model tables, what 32 GB exposed, known issues, reproduce commands) with a GIF.
• docs/profiling/m6/: per-model tables, raw per-run JSONL, asciinema casts and GIFs. Server logs are left out because they contain the full test prompts.

Results (M6 32 GB, main 318f712 + this PR, medians, every needle check passed)

Model	Mode	Decode, short prompt	Longest prompt that passed	Peak GPU
Gemma-4-26B-A4B 4-bit	GPU	52.2 tok/s	80.7K tokens (622 / 24.3 tok/s)	19.5 GB
Qwen3.6-35B-A3B 4-bit	GPU	46.7 tok/s	40.8K tokens	22.4 GB
Qwen3.6-35B-A3B 4-bit	--stream-experts	13.2 tok/s	40.8K tokens	7.7 GB
Qwen3.8-27B 4-bit (dense)	GPU	9.3 tok/s	40.8K tokens (200 tok/s prefill)	18.4 GB
Gemma-4-26B-A4B 8-bit	--stream-experts	8.8 tok/s	9.5K tokens	7.6 GB

A/B on the same M6: #172 alone vs this PR (Gemma-4-26B-A4B 4-bit)

Prompt	#172 only	this PR
~530 tokens	52.2 tok/s decode	53.3 tok/s decode
~2.3K tokens	508 tok/s prefill, swap +1.4 GB	998 tok/s prefill, swap 0
~9.5K tokens	MEM_ABORT, swap +3.4 GB	914 tok/s prefill, swap 0

Chunked VLM prefill: now shipped in main

Earlier versions of this PR disclosed a dependency on a local mlx-swift-lm backport. That's gone. With #167 (mlx-swift-lm 460ff81), chunked text-only prefill on the VLM path ships in main, and every number in the README section was re-measured on main + this PR without any local patch. Upstream's windowed prefill is faster than the backport, too: Qwen3.8-27B prefills at about 235 tok/s at every length (about 2.3× faster).

Rechecked on main + this PR

• ✅ --turbo-kv + PromptCache.save Metal assertion: gone. Gemma 4 TurboKV passes 13/13 runs up to 80.9K tokens.
• ✅ Gemma 4 TurboKV needle digits dropped at 2K: gone (3/3).
• ✅ Prefill heartbeat: arrives every 2 s since fix: reclaim generation slot on error and stream SSE headers before prefill #170. On the VLM path n_past/fraction stay 0 (no progress hook in the windowed prefill).
• ❌ --gpu-layers N partitioning: Metal GPU timeout on the first request. Repro: --model mlx-community/gemma-4-26b-a4b-it-4bit --gpu-layers 23, with FULL GPU memory otherwise.
• ❌ QAT Gemma 4 MTP assistants: unhandledKeys pre_projection/post_projection. The MLXLLM Gemma4AssistantModel.sanitize only takes unquantized projection weights, while the MLXVLM drafter handles quantized ones. gemma-4-26B-A4B-it-assistant-bf16 works, but MTP is slower than vanilla on the M6 (45.2 / 35.6 / 30.4 vs 53.0 / 50.8 / 46.0 tok/s).
• 🔍 Under investigation with the M5 session: Qwen3.8 + --turbo-kv gets the numbered-lines count wrong at ~11K tokens, while the needle test passes. A length sweep is running.

Checks

• Release build on Xcode 27 / Swift 6.4. No bare any X? in the Swift diff (Xcode 26.3 CI rejects it).
• All four models benchmarked on M6 32 GB through the memory-guarded harness
• A/B against fix(build): build with Xcode 27 / Swift 6.4 #172 alone (table above)
• Xcode 26.3 (the Mac M5 session): build + SwiftLMTests 185/185, including VLMFallbackTests 6/6
• CI

🤖 Generated with Claude Code