fix: propagate throws from LanguageModel.newCache and adapt to mlx-swift-lm PR #62 - #167
Merged
Merged
Conversation
…ift-lm PR #62 API changes Bump the mlx-swift-lm submodule to 348ff97 (upstream-sync PR #62) and update all call sites broken by its breaking API changes: - LanguageModel.newCache(parameters:) is now `throws`. Propagated `try` through every caller: DFlashRuntime's makeTargetCache/generateStreaming/ generateSync chain (with a documented do/catch in generate() since a plain AsyncStream cannot re-throw to its consumer), Gemma4MTPBench, InferenceEngine, and Server.swift. - GenerateCompletionInfo.totalDraftTokens was renamed/optional-ized to proposedDraftTokens/acceptedDraftTokens; updated InferenceEngine to unwrap safely instead of assuming non-optional Int. - ModelFactory is now a constrained typealias (GenericModelFactory< ModelContext, ModelContainer>) with primary associated types, which a class can no longer inherit from directly. ALMModelFactory and OmniModelFactory now conform to GenericModelFactory instead, matching upstream's own LLMModelFactory/VLMModelFactory pattern. - UserInput.audio was renamed to audios; updated the three genuine call sites (left LMInput.audio/vlmInput.audio alone, which are unrelated, unrenamed types). - Chat.Message's role cases were restructured (audios: label, ToolCall- based tool calls instead of raw dictionaries, tool(_, id:) instead of tool(_, toolCallId:)); rewrote toChatMessage() to build ToolCall values and decode JSON-string tool arguments into [String: JSONValue]. - Generation gained a new .rejectedToolCall(RejectedToolCall) case; added handling to all four previously-exhaustive switches, logging only reason/toolName/detail (never rawTextPreview, per its doc comment's privacy note). Verified both ways: `swift build -c release` and `swift build --build-tests` are clean (no errors) against the new pin (348ff97), and also clean when the submodule is temporarily rolled back to the previous pin (0e0cb47) with these same source changes in place (the added `try` keywords are harmless there). Submodule is left at 348ff97. TurboQuant C++ tests (9/9), SwiftLMTests (161/161), and SwiftBuddyTests (127 executed, 9 skipped) all pass against the final state. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Point the mlx-swift-lm submodule at the actual merged PR #62 commit (460ff81) instead of the interim pin (348ff97) this branch was originally adapted against. 460ff81 is a strict content superset of 348ff97 (confirmed via diff --stat: 81 files changed, 10594 insertions, 673 deletions, including the same Gemma4Unified fixes and extensive additional test coverage), so no regression from repointing. Package.resolved's swift-syntax pin moved to 603.0.2 as a byproduct of re-resolving against the new submodule state. Verified: `swift build -c release` and `swift test --skip-build` both pass (288 tests, 9 skipped, 0 failures). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…wn CI mlx-swift-lm's Package.swift declares swift-tools-version 6.2 (already true at the previous pin, 348ff97). SwiftLM's CI never selected a newer Xcode, so it silently depended on the default macos-15 runner image happening to ship a toolchain new enough — until this bump surfaced it: `swift package resolve` failed with "package 'mlx-swift-lm' is using Swift tools version 6.2.0 but the installed version is 6.1.0". Added the same `xcode-select -s /Applications/Xcode_26.3.app` step mlx-swift-lm's own ci.yml uses, to every job that resolves or builds the package (build_and_unit_test, speculative-decoding, dflash-speculative-decoding, speculative-decoding-eval, and ssd-draft-memory-guard's artifact-missing fallback build). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
5 tasks
DFlash's .summary never yielded .info, so streams ending on EOS or max_tokens closed without finish_reason, usage or [DONE] (test-dflash Test 3). mlx-swift-lm's concurrent loader materializes every checkpoint tensor before sanitize; dropped tensors land in MLX's buffer cache, which under --stream-experts is capped only by the SSD budget. Clear it once loading finishes (ssd-draft-memory-guard was ~0.5 GB over on the new submodule). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
solderzzc
pushed a commit
that referenced
this pull request
Sep 24, 2026
…1) + this PR main now ships chunked VLM prefill (#167), so the earlier Gemma-4-26B-A4B and Qwen3.8-27B runs, which relied on a local backport, are replaced with runs on main 318f712 + this PR. Nothing unshipped is involved any more. - Gemma-4-26B-A4B 4-bit: 52.2 tok/s decode at ~530 tokens; an 80.7K-token prompt at 622 tok/s prefill / 24.3 decode, 19.5 GB peak, 0 swap. --turbo-kv now passes all 13 runs up to 80.9K tokens (the Metal assertion and needle misses are gone). --mtp with the bf16 assistant works but is slower on the M6 (45.2 / 35.6 / 30.4 vs 53.0 / 50.8 / 46.0 tok/s). - Qwen3.8-27B 4-bit: prefill ~235 tok/s at every length (~2.3× the backport), 9.3 tok/s decode, 18.4 GB peak at 40.8K tokens. - Known issues now cover --gpu-layers partitioning (Metal GPU timeout) and QAT MTP assistants (unhandledKeys); the fixed items are removed. - m6_bench.py records the code word and the start of each answer, so a needle miss shows what the model actually said. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Member
Author
|
Merged dflash-speculative-decoding. The DFlash adapter's ssd-draft-memory-guard. Since #575, mlx-swift-lm's concurrent weight loader materializes every checkpoint tensor before
Locally on Xcode 26.3: builds clean, and 179/179 unit tests pass. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
mlx-swift-lmsubmodule to the merged PR fix(swiftbuddy): add MLXVLM and ModelArchitectureProbe to xcodeproj generator #62 commit (SharpAI/mlx-swift-lm#62,460ff81) and adapt all call sites broken by its breaking API changes:LanguageModel.newCache(parameters:)is nowthrows. Propagatedtrythrough every caller:DFlashRuntime's cache/generate chain (with a documented do/catch ingenerate()since a plainAsyncStreamcannot re-throw),Gemma4MTPBench,InferenceEngine, andServer.swift.GenerateCompletionInfo.totalDraftTokenswas renamed/optional-ized toproposedDraftTokens/acceptedDraftTokens; updatedInferenceEngineto unwrap safely.ModelFactoryis now a constrained typealias with primary associated types;ALMModelFactory/OmniModelFactorynow conform toGenericModelFactoryinstead of inheriting, matching upstream'sLLMModelFactory/VLMModelFactorypattern.UserInput.audiorenamed toaudios; updated the genuine call sites.Chat.Message's role cases restructured (audios:label,ToolCall-based tool calls,tool(_, id:)); rewrotetoChatMessage()accordingly.Generationgained.rejectedToolCall(RejectedToolCall); handled in all previously-exhaustive switches.348ff97) to the actual merged PR fix(swiftbuddy): add MLXVLM and ModelArchitectureProbe to xcodeproj generator #62 commit (460ff81) — confirmed a strict content superset viagit diff --stat(81 files, +10594/-673), so no regression.Test plan
swift build -c release— clean build, no errors (only pre-existing, unrelated warnings inServer.swift)swift test --skip-build— 288 tests passed, 9 skipped, 0 failures🤖 Generated with Claude Code