Feedback Wanted: Experimental Runtime API and GGUF LLM Support in Windows ML!

[Andrew Leader (andrewleader)]

We just published a new experimental version of Windows ML (Version 2.7.2021-experimental) that adds three new capabilities...

• Windows ML Runtime API: A Windows-native inferencing API built for performance, deep OS integration, and precise control.
• GGUF LLM support via Windows ML Text Generation APIs: A higher-level API that provides text generation over a custom language model, with both GGUF and ONNX model formats supported.
• ONNX Whisper support via Windows ML Speech Recognition APIs: A higher-level API that provides speech recognition using an ONNX Whisper model of your choice.

We want to hear your HONEST feedback about these new additions to Windows ML! Your feedback is critical to helping us design Windows ML into the best inferencing framework that meets your needs - we're building this for YOU! Are we focusing on the wrong thing? Should we tweak something slightly to be more useful for you? Are we missing any language support? Please tell us!

More information

Our full docs are available here:

• Windows ML Runtime API Overview
• Windows ML Text Generation API (includes GGUF model support)
• Windows ML Speech Recognition API

To learn more about how the Runtime API compares to ONNX Runtime, see the Runtime API vs ONNX Runtime docs

Get started

Install the release: Windows ML 2.7.2021-experimental

• Run a GGUF model: Text Generation API docs
• Transcribe audio: Speech Recognition API docs
• Explore the native runtime: Windows ML Runtime API docs
• See it in action: Runtime API samples

FAQ

• When are these APIs going GA? We don't have a planned GA date, promotion to GA will be quality-driven and determined by your feedback.
• Why isn't C# supported? We're adding C# in the next release of these experimental APIs!
• How does this compare to ONNX Runtime? See the Runtime API vs ONNX Runtime docs