diff --git a/README.md b/README.md index 308813e..809d466 100644 --- a/README.md +++ b/README.md @@ -1,58 +1,147 @@ # MagicQuant +![MagicQuant](https://raw.githubusercontent.com/magiccodingman/MagicQuant/main/assets/icon.png) + [![NuGet version](https://img.shields.io/nuget/v/MagicQuant.svg)](https://www.nuget.org/packages/MagicQuant/) [![NuGet downloads](https://img.shields.io/nuget/dt/MagicQuant.svg)](https://www.nuget.org/packages/MagicQuant/) [![Build and tests](https://github.com/magiccodingman/MagicQuant/actions/workflows/dotnet.yml/badge.svg)](https://github.com/magiccodingman/MagicQuant/actions/workflows/dotnet.yml) [![License: AGPL v3](https://img.shields.io/badge/License-AGPL_v3-blue.svg)](https://github.com/magiccodingman/MagicQuant/blob/main/LICENSE) -**Benchmark-driven GGUF quantization and mixed-precision hybrid discovery for llama.cpp.** +**Discover better GGUF size/fidelity tradeoffs. Benchmark the results. Share the tensor recipes.** + +MagicQuant is a **benchmark-driven LLM quantization, mixed-precision hybrid discovery, and tensor-configuration cloning system for llama.cpp**. It learns from existing quantization strategies, explores combinations across tensor groups, builds promising GGUFs, and measures which ones deserve a place in the final release. -MagicQuant helps answer: **which quantized versions of a model are worth keeping at each size?** It measures baseline quantizations, learns tensor-group assignments, explores hybrid combinations, and validates candidates against size and fidelity criteria. The result is a selected set of GGUF artifacts with supporting measurements, rather than an unranked collection of quantization levels. +A campaign gives you more than another quantized file: **a measured selection of useful models, an explanation of what survived and why, and manifests that let other people rebuild those configurations.** -It is a .NET command-line application that orchestrates llama.cpp and Python tooling. It does not invent a new quantization format or use evolutionary search. Hybrids must earn their place: a standard baseline can be the better result. +[Browse the models](https://huggingface.co/collections/magiccodingman/magic-quant) · [See the results](#magicquant-in-the-wild) · [Install](#install-and-run) · [Clone a release](#clone-the-recipes-onto-your-own-model) · [Read the research](https://github.com/magiccodingman/MagicQuant/blob/main/wiki/index.md) -## How it works +## Which quants are actually worth keeping? -1. **Establish baselines.** Read a local source model and measure standard quantization choices. Optionally learn tensor assignments from compatible external GGUFs. -2. **Probe tensor groups.** Measure how changes to groups such as attention, embeddings, and feed-forward tensors affect the model. -3. **Discover hybrids.** Use measured evidence and predictions to explore mixed-precision combinations with promising size/fidelity tradeoffs. -4. **Validate and select.** Measure candidates, reject poor or redundant trades, and export survivors with metadata and local provenance. +Q8, Q6, Q5, Q4: familiar names tell you roughly how a model was compressed. They do not tell you whether a particular model has a better trade hiding between those choices—or whether two downloads offer almost the same thing. -KLD and perplexity help evaluate fidelity; throughput and file size provide additional context. The results depend on the model, calibration/evaluation data, configuration, and hardware. A smaller KLD in one campaign is not a universal claim about downstream task quality. Read the [research overview](https://github.com/magiccodingman/MagicQuant/blob/main/wiki/index.md) for the selection policy and its assumptions. +MagicQuant investigates that space. It can combine one baseline's attention recipe with another's feed-forward recipe, protect groups that are expensive to damage, and compress groups where the measurements justify it. Candidates earn a place by reducing size and KLD together, delivering an unusually worthwhile fidelity improvement for a small size premium, or beating the expected trade between neighboring choices. + +**Every survivor has to earn its slot.** The final selection can contain llama.cpp baselines, configurations learned from Unsloth, and MagicQuant hybrids. The goal is a release where each download represents a meaningful choice. ## Support the project I build and maintain MagicQuant on the side, for free. Developing it and experimenting with quantizations has put a frankly ridiculous amount of terabytes written (TBW) on my drives! My Hugging Face storage is also creeping toward its cap, so there will eventually be more storage to fund. If this project helps you, [supporting the work](https://sayou.biz/support) helps with those costs. Anything helps and is always appreciated. ❤️ -## Install and run +## MagicQuant in the wild -**Linux is the tested campaign platform.** Windows has automated build, unit-test, and packaged CLI checks; full Windows quantization campaigns have not been validated. No macOS campaign support is claimed. +**[Explore the MagicQuant collection on Hugging Face →](https://huggingface.co/collections/magiccodingman/magic-quant)** -Install the [.NET 10 SDK](https://dotnet.microsoft.com/download/dotnet/10.0), then install the CLI from [NuGet](https://www.nuget.org/packages/MagicQuant/): +Published releases put the research to work on dense models, mixture-of-experts models, and related fine-tunes. Here are three examples of what that makes possible. Sizes below are decimal GB; KLD measures divergence from each experiment's reference output distributions, with lower values indicating closer agreement on that evaluation. -```bash -dotnet tool install --global MagicQuant -magicquant --version -magicquant init-config --output config.yaml -``` +### Discover useful choices between the usual quant sizes + +The [Qwen3.6-35B-A3B release](https://huggingface.co/magiccodingman/Qwen3.6-35B-A3B-MagicQuant-GGUF) contains **eight MagicQuant hybrids alongside two baseline recipes**. Selected points from its ten-entry survivor manifest show the range: + +| Recipe | Source | Size (GB) ↓ | KLD ↓ | +| --- | --- | ---: | ---: | +| LM-Q8_0 | llama.cpp | 36.90 | 0.004654 | +| **MQ-Q6_K_1** | **MagicQuant** | **31.59** | **0.005149** | +| **MQ-Q5_K_1** | **MagicQuant** | **29.19** | **0.005523** | +| **MQ-Q4_K_M_1** | **MagicQuant** | **24.82** | **0.007799** | +| **MQ-IQ3_M_1** | **MagicQuant** | **17.60** | **0.026330** | +| UD-IQ3_S | Unsloth-derived | 13.68 | 0.068376 | + +These are measured options at different storage budgets. The full release reaches down to a 9.59 GB hybrid, with a correspondingly larger divergence. The [complete table and pinned evidence](https://github.com/magiccodingman/MagicQuant/blob/main/wiki/showcase.md#discovery-a-35b-moe-release) make that trade visible. + +The original 4B worked example shows why the intermediate choices matter: **MQ-Q5_K_1 used about 5.5% more space than the UD-Q5_K_XL recipe for about 32.6% lower measured KLD**, improving on the straight-line trade between neighboring baselines. Its [full results and tensor-group recipe breakdown](https://github.com/magiccodingman/MagicQuant/blob/main/wiki/overview.md#example) remain in the research overview. + +### Take a discovered recipe set to another model + +The [Qwen3.6-35B-A3B Uncensored release](https://huggingface.co/magiccodingman/Qwen3.6-35B-A3B-Uncensored-MagicQuant-GGUF) reuses **all ten configurations** from the original release on a related model by llmfan46, including the external-derived baseline. It rebuilds the artifacts and records fresh measurements rather than repeating the full discovery search. + +For example, its cloned **MQ-Q4_K_M_1 is 24.82 GB at 0.007832 KLD**, measured against the new model's reference. The manifest makes the original work reusable; the clone benchmarks record what happened on the new weights. [See the clone evidence](https://github.com/magiccodingman/MagicQuant/blob/main/wiki/showcase.md#cloning-ten-recipes-on-a-related-model), or [clone a release yourself](#clone-the-recipes-onto-your-own-model). + +### Explore beyond a conventional starting point + +A [specialized Qwen3.8-27B MXFP4 experiment](https://huggingface.co/magiccodingman/Qwen3.8-27B-MXFP4-MagicQuant-GGUF) adapted existing MagicQuant recipes to AMD's already-quantized Quark AWQ MXFP4 model. Its downward-only conversion preserved tensors that were already smaller, producing a **14.98 GB artifact from an 18.89 GB reference—20.70% less storage—with 0.000940 measured KLD**. + +Here the reference is the **native MXFP4 GGUF**, so that KLD must not be compared with the other campaigns' numbers. This release used specialized adaptation work beyond ordinary CLI cloning; it illustrates an experiment built around reusable recipes, not an automatic promise for every input format. [Read the method, limits, and publication decisions](https://github.com/magiccodingman/MagicQuant/blob/main/wiki/showcase.md#specialized-research-downward-only-mxfp4-adaptation). + +These are published measurements, not new benchmarks run for this README. KLD is one fidelity signal, not a general capability score. External recipes in discovery are rebuilt under controlled local conditions; their results do not establish that an upstream provider's original uploads are better or worse. + +## How MagicQuant finds those trades + +### Learn the recipes inside real quantizations + +Even a baseline with a single name such as Q5_K can use different quantization types across its tensors. MagicQuant reads those actual assignments and organizes them into architecture-aware groups: embeddings, attention, feed-forward layers, output heads, and supported MoE or hybrid-architecture groups. + +Standard llama.cpp baselines supply the starting vocabulary. Compatible external GGUFs can add more recipes. **Unsloth is the maintainer's recommended starting point**, and external providers are optional. See [learning from existing quantizations](https://github.com/magiccodingman/MagicQuant/blob/main/wiki/docs/Learning-From-Existing-Quantizations.md). + +### Measure locally, search intelligently + +Trying every combination quickly becomes impractical. MagicQuant first measures isolated group changes, then uses those observations to predict which combinations deserve a real build. Its rank-safe predictor and DuckDB candidate search help focus the benchmark budget on promising dominance, near-baseline, and interior tradeoffs. + +When the main search fails to validate a winner, a bounded fallback can revisit isolated evidence for smaller improvements or carefully budgeted protection of sensitive groups. **Predictions choose what to test; actual GGUF benchmarks decide what survives.** Read the [prediction engine guide](https://github.com/magiccodingman/MagicQuant/blob/main/wiki/docs/Prediction-Engine.md). + +### Investigate interactions and surprising wins + +A group recipe that works well in a lightly compressed model may behave differently when the surrounding groups are more compressed. MagicQuant can run controlled context probes, investigate beneficial or harmful interactions, and learn scoped exceptions when a lower-bit choice performs unexpectedly well. + +Those probes are bounded so investigating interactions does not recreate an exhaustive search. Aggressive low-fidelity context probing is opt-in. See [context-aware tensor search](https://github.com/magiccodingman/MagicQuant/blob/main/wiki/docs/Regime-Aware-Search.md). + +### Build a release people can understand + +MagicQuant measures candidates, removes dominated or poor trades, and curates the remaining choices into a useful survivor list. It exports GGUFs, a model card, measurements, replacement explanations, and exact tensor maps for cloning. -The package becomes available after the first successful release publication; until then use the [source installation instructions](https://github.com/magiccodingman/MagicQuant/blob/main/docs/setup.md#build-from-source). +You can inspect **which assignments produced an artifact, what it replaced, and the measurements behind the decision**. The [manifest guide](https://github.com/magiccodingman/MagicQuant/blob/main/docs/manifests-and-cloning.md) explains what to publish and how others can use it. -Edit the generated config for your source model, architecture identity, export destination, and storage. Initialize the external toolchain, validate the config, then start the campaign: +## Clone the recipes onto your own model + +**A useful quantization recipe should travel with the release.** If someone publishes a MagicQuant model with its clone manifest, you can point the CLI at their Hugging Face repository and rebuild the selected configurations from your own compatible source weights. + +This is especially useful for fine-tunes, uncensored variants, and other related models that the original quantization provider does not host. It gives you a practical starting set without repeating the full discovery search. Clone mode rebuilds and benchmarks the manifest entries locally, including both MagicQuant hybrids and external-derived configurations. + +After [installing the CLI and preparing the toolchain](#install-and-run), create a config and edit it for your target model, architecture, storage, and any imatrix requirements: ```bash -magicquant initialize-llama-cpp -magicquant pipeline --config ./config.yaml --check-config --strict-config -magicquant pipeline --config ./config.yaml +magicquant init-config --output clone.yaml +magicquant clone-repository-quants \ + --config ./clone.yaml \ + --source-repo magiccodingman/Qwen3.6-35B-A3B-MagicQuant-GGUF ``` -Initialization can download/build llama.cpp and install Python dependencies. NuGet installs MagicQuant, not model weights or a complete GPU toolchain. Follow the [installation guide](https://github.com/magiccodingman/MagicQuant/blob/main/docs/setup.md) for native prerequisites, GPU setup, custom toolchains, and environment paths. +This example uses the published 35B recipe set above; choose a source release compatible with your target. `--source-repo` takes a Hugging Face **model repository ID**. You can also supply a local clone JSON file or a pinned raw JSON URL with `--source-json`. The source release needs `magicquant-manifest/magicquant.clone-configs.json`; your local input needs the complete compatible source model. + +Publish your own generated manifest folder alongside the release, and other people can do the same with your configurations. See the [publishing and cloning walkthrough](https://github.com/magiccodingman/MagicQuant/blob/main/docs/manifests-and-cloning.md). + +Cloning transfers tensor assignments, not proof that a recipe is optimal for different weights. It also does not automatically reproduce a provider's additional transformations, calibration recipes, or custom processing. Fresh discovery is appropriate when model changes or local measurements warrant it. + +### Give the original providers their downloads + +For the same model, the recommended pipeline default is to **link to an external provider's surviving baselines** rather than re-host copies. Leave `output.export_external_learned_baselines: false`; original creators keep the attribution and downloads they earned. + +For a variant they do not host, build those configurations locally. The clone command already rebuilds all manifest entries and does not need that pipeline flag enabled. The [best-practices guide](https://github.com/magiccodingman/MagicQuant/blob/main/docs/best-practices.md#link-upstream-for-the-same-model-build-locally-for-variants) covers the distinction. + +## Put your hardware to work + +Discovery involves many large writes and real benchmarks. MagicQuant manages both sides of that workload: + +- **Measured GPU scheduling.** It probes viable shared multi-GPU and independent per-GPU execution, then uses candidate size and available batch work to choose a suitable topology. Larger candidates can share GPUs; smaller ready-to-run candidates can use independent workers when measurements favor it. +- **Dedicated scratch storage.** Temporary GGUF work uses scratch leases with one heavy writer per configured root. Fast SSD/NVMe storage on separate physical devices can substantially help when IO is limiting throughput. +- **Reusable evidence.** Persistent model/profile/imatrix identities, cached benchmark truth, and cached hardware plans let compatible later work reuse prior results. Local run records capture settings and available toolchain versions. + +As one documented scheduling example, a two-RTX-3090 campaign measured **0.762 jobs/second with independent workers versus 0.529 with shared workers** for the tested sub-crossover workload—about 44% more aggregate throughput. That is a workload-specific result of measuring the topology, not a promised speedup for every machine. Read the [GPU scheduling study](https://github.com/magiccodingman/MagicQuant/blob/main/wiki/docs/GPU-Benchmark-Scheduling.md). + +**Plan scratch disks early.** Repeated multi-gigabyte writes can be one of a campaign's biggest bottlenecks. Two folders on the same disk still share its bandwidth; leave enough free space for concurrent artifacts and keep unrelated data outside managed scratch/export directories. Start with [storage](https://github.com/magiccodingman/MagicQuant/blob/main/docs/storage.md) and [best practices](https://github.com/magiccodingman/MagicQuant/blob/main/docs/best-practices.md). -For updates: `dotnet tool update --global MagicQuant`. For reproducible runs, install a particular release with `--version X.Y.Z` and retain your config, model revision, and run provenance. +## Install and run + +**Linux is the tested campaign platform.** Windows has automated build, unit-test, and packaged CLI checks; full Windows campaigns have not been validated. Model/architecture support depends on the installed llama.cpp converter and MagicQuant's tensor-group mappings. -## Configure a campaign +Install the [.NET 10 SDK](https://dotnet.microsoft.com/download/dotnet/10.0), then install MagicQuant from [NuGet](https://www.nuget.org/packages/MagicQuant/): + +```bash +dotnet tool install --global MagicQuant +magicquant --version +magicquant init-config --output config.yaml +``` -A minimal example (replace these paths and the architecture identity): +Edit the generated YAML for your source model and intended output. This excerpt shows the settings to start with: ```yaml paths: @@ -69,38 +158,35 @@ learning: confirm_tensor_group_profile: true ``` -Custom YAML uses typed defaults for omitted values; it does not merge with the bundled tuning profile. Start with `init-config` when you want that complete profile. See [configuration](https://github.com/magiccodingman/MagicQuant/blob/main/docs/configuration.md), [examples](https://github.com/magiccodingman/MagicQuant/tree/main/examples), and the [command reference](https://github.com/magiccodingman/MagicQuant/blob/main/docs/commands.md). +Prepare the native toolchain, check the configuration, and begin discovery: -**Plan scratch storage early.** Quantization writes and rereads large intermediate models, and storage can be a major throughput limitation. Fast SSD/NVMe scratch disks, especially separate physical devices, can materially improve throughput when IO is the bottleneck. Multiple folders on the same device still share its bandwidth. Allow space for concurrent intermediate artifacts and keep unrelated data out of managed scratch/export directories. See [storage](https://github.com/magiccodingman/MagicQuant/blob/main/docs/storage.md) and [best practices](https://github.com/magiccodingman/MagicQuant/blob/main/docs/best-practices.md). - -## Learning from external quantizations +```bash +magicquant initialize-llama-cpp +magicquant pipeline --config ./config.yaml --check-config --strict-config +magicquant pipeline --config ./config.yaml +``` -External providers are optional. MagicQuant can run using its local baseline choices alone, but compatible external tensor assignments can provide valuable additional evidence. +Initialization can download/build llama.cpp and install Python dependencies. Follow the [setup guide](https://github.com/magiccodingman/MagicQuant/blob/main/docs/setup.md) for native prerequisites, GPU setup, existing toolchains, and source installation. `init-config` gives you a commented, model-neutral profile and never overwrites an existing file. Keep separate YAML files for different campaigns; `--check-config` validates local settings without starting the work. -**Unsloth is the maintainer's recommended starting point** for external GGUF baselines. MagicQuant can learn their tensor-group patterns, rebuild a controlled equivalent from your local source model, and benchmark it in your campaign. It does not simply trust an external file's label or score. Choose the exact matching model and revision, and review its license. See the [Unsloth configuration walkthrough](https://github.com/magiccodingman/MagicQuant/blob/main/docs/best-practices.md#optional-unsloth-baselines) and [research explanation](https://github.com/magiccodingman/MagicQuant/blob/main/wiki/docs/Learning-From-Existing-Quantizations.md). +A minimal custom YAML uses typed defaults for omitted settings; it does not merge with the bundled tuning profile. The generated full profile, [configuration reference](https://github.com/magiccodingman/MagicQuant/blob/main/docs/configuration.md), and [opt-in external-provider example](https://github.com/magiccodingman/MagicQuant/blob/main/examples/pipeline-external.yaml) explain how to configure a run deliberately. -For the same model, prefer linking to the original provider's surviving baselines. For a compatible variant they do not host, cloning can rebuild the full selected set locally. Learning tensor assignments does not automatically reproduce a provider's other processing techniques. See [publishing and cloning guidance](https://github.com/magiccodingman/MagicQuant/blob/main/docs/best-practices.md#link-upstream-for-the-same-model-build-locally-for-variants). +Update with `dotnet tool update --global MagicQuant`. Install an exact release with `--version X.Y.Z` when you need to keep a campaign's application version fixed. -## Documentation +## Explore MagicQuant -| Start here | What you will find | +| I want to… | Start here | | --- | --- | -| [Installation](https://github.com/magiccodingman/MagicQuant/blob/main/docs/setup.md) | NuGet, native prerequisites, custom environments, source builds | -| [Configuration](https://github.com/magiccodingman/MagicQuant/blob/main/docs/configuration.md) | YAML, overrides, read-only validation, profiles | -| [Commands](https://github.com/magiccodingman/MagicQuant/blob/main/docs/commands.md) | Pipeline, setup, cloning, prediction validation | -| [Best practices](https://github.com/magiccodingman/MagicQuant/blob/main/docs/best-practices.md) | Scratch disks, Unsloth, reproducibility, first campaigns | -| [Manifests and cloning](https://github.com/magiccodingman/MagicQuant/blob/main/docs/manifests-and-cloning.md) | Publish a reproducible release; rebuild from another MagicQuant repository | -| [Storage](https://github.com/magiccodingman/MagicQuant/blob/main/docs/storage.md) | Persistent data, scratch leases, cache and output ownership | -| [Research](https://github.com/magiccodingman/MagicQuant/blob/main/wiki/index.md) | Measurements, prediction, pruning, hybrid selection | -| [Contributing](https://github.com/magiccodingman/MagicQuant/blob/main/CONTRIBUTING.md) | Development workflow, tests, code boundaries | -| [Releases](https://github.com/magiccodingman/MagicQuant/blob/main/docs/releases.md) | Automatic versions and NuGet trusted publishing | - -## Development and history - -Application code lives in `src/`, tests in `tests/`, operational guides in `docs/`, and research documentation in `wiki/`. Both the former MagicQuant-Wiki and MagicQuant-Pipeline histories are retained. The `evolution` command remains a compatibility alias for `pipeline`; existing database and artifact contracts are preserved. Historical research remains under `archival/` and is not current setup guidance. +| Browse published results and their evidence | [Model showcase](https://github.com/magiccodingman/MagicQuant/blob/main/wiki/showcase.md) and [Hugging Face collection](https://huggingface.co/collections/magiccodingman/magic-quant) | +| Understand the results and see the full worked example | [Research overview](https://github.com/magiccodingman/MagicQuant/blob/main/wiki/overview.md) | +| Understand how predictions become measured winners | [Prediction engine](https://github.com/magiccodingman/MagicQuant/blob/main/wiki/docs/Prediction-Engine.md) and [survivor selection](https://github.com/magiccodingman/MagicQuant/blob/main/wiki/docs/Nonlinear-Winners-And-Survivors.md) | +| Install and configure my first campaign | [Setup](https://github.com/magiccodingman/MagicQuant/blob/main/docs/setup.md), [configuration](https://github.com/magiccodingman/MagicQuant/blob/main/docs/configuration.md), and [commands](https://github.com/magiccodingman/MagicQuant/blob/main/docs/commands.md) | +| Learn from Unsloth and plan storage | [Best practices](https://github.com/magiccodingman/MagicQuant/blob/main/docs/best-practices.md) | +| Publish a release or clone someone else's | [Manifests and cloning](https://github.com/magiccodingman/MagicQuant/blob/main/docs/manifests-and-cloning.md) | +| Inspect the research and its assumptions | [Research index](https://github.com/magiccodingman/MagicQuant/blob/main/wiki/index.md) | +| Add a capability or contribute a fix | [Contributor guide](https://github.com/magiccodingman/MagicQuant/blob/main/CONTRIBUTING.md) | ## License -MagicQuant's original code and documentation are licensed under **GNU AGPL version 3 only** (`AGPL-3.0-only`). Commercial use is permitted subject to its terms. Distribution and remote interaction with modified versions carry source-availability obligations; the [license text](https://github.com/magiccodingman/MagicQuant/blob/main/LICENSE) controls the details. +MagicQuant is open source under **GNU AGPL version 3 only** (`AGPL-3.0-only`). Commercial use is permitted subject to its terms; distribution and remote interaction with modified versions carry source-availability obligations. See the [license](https://github.com/magiccodingman/MagicQuant/blob/main/LICENSE). -This does not automatically relicense model weights or generated GGUFs. Model, dataset, external-provider, and third-party dependency licenses still apply. See [third-party notices](https://github.com/magiccodingman/MagicQuant/blob/main/THIRD-PARTY-NOTICES.md). +Model weights, generated GGUFs, datasets, and external dependencies retain their applicable licenses. See [third-party notices](https://github.com/magiccodingman/MagicQuant/blob/main/THIRD-PARTY-NOTICES.md). diff --git a/wiki/index.md b/wiki/index.md index 9da77c4..d736941 100644 --- a/wiki/index.md +++ b/wiki/index.md @@ -6,6 +6,9 @@ MagicQuant is a benchmark-driven GGUF evaluation and hybrid-discovery system. Th ## Start Here +- [Published Models: Results and Reusable Recipes](./showcase.md) +- [Original Research Overview and Worked Example](./overview.md) + - [Learning From Existing Quantizations (Tensor Configuration Sources)](./docs/Learning-From-Existing-Quantizations.md) - [Prediction Engine](./docs/Prediction-Engine.md) - [Nonlinear Winners and Survivor Selection](./docs/Nonlinear-Winners-And-Survivors.md) diff --git a/wiki/showcase.md b/wiki/showcase.md new file mode 100644 index 0000000..09c142a --- /dev/null +++ b/wiki/showcase.md @@ -0,0 +1,69 @@ +# Published MagicQuant models: results and reusable recipes + +The [MagicQuant collection](https://huggingface.co/collections/magiccodingman/magic-quant) contains downloadable releases, model cards, and manifests. These examples were checked against pinned published revisions on September 7, 2026. They are existing release measurements; this documentation update did not run new model benchmarks. + +Sizes use decimal GB (1,000,000,000 bytes), rounded to two decimals. KLD values describe output-distribution divergence on each release's evaluation. They are not task-accuracy scores, and scores from different reference models or evaluation setups are not directly comparable. + +## Discovery: a 35B MoE release + +Qwen3.6-35B-A3B's ten-entry selection contains eight MagicQuant hybrids, a llama.cpp baseline, and an Unsloth-derived baseline. The complete survivor list shows both the low-divergence end and the storage savings that require larger fidelity sacrifices. + +| Recipe | Size (GB) | KLD | +| --- | ---: | ---: | +| LM-Q8_0 | 36.90 | 0.004654 | +| MQ-Q6_K_1 | 31.59 | 0.005149 | +| MQ-Q5_K_1 | 29.19 | 0.005523 | +| MQ-Q5_K_S_1 | 26.33 | 0.006730 | +| MQ-Q4_K_M_1 | 24.82 | 0.007799 | +| MQ-Q4_K_M_2 | 22.32 | 0.011007 | +| MQ-IQ4_NL_1 | 20.89 | 0.013277 | +| MQ-IQ3_M_1 | 17.60 | 0.026330 | +| UD-IQ3_S | 13.68 | 0.068376 | +| MQ-IQ2_XXS_1 | 9.59 | 0.275130 | + +Evidence: [pinned model card](https://huggingface.co/magiccodingman/Qwen3.6-35B-A3B-MagicQuant-GGUF/resolve/6e1771746ee4c9194fe4e5c8bbe9f301efcfd2d5/README.md), [final survivor measurements](https://huggingface.co/magiccodingman/Qwen3.6-35B-A3B-MagicQuant-GGUF/resolve/6e1771746ee4c9194fe4e5c8bbe9f301efcfd2d5/magicquant-manifest/magicquant.final-survivors.json), and [exact clone recipes](https://huggingface.co/magiccodingman/Qwen3.6-35B-A3B-MagicQuant-GGUF/resolve/6e1771746ee4c9194fe4e5c8bbe9f301efcfd2d5/magicquant-manifest/magicquant.clone-configs.json). + +The external baseline's measurements come from a local reconstruction of its tensor recipe. The original release links upstream for that download; these figures do not independently benchmark the provider's original artifact. + +## Cloning: ten recipes on a related model + +The uncensored variant by llmfan46 was rebuilt using the original release's ten configurations, including its external-derived baseline. Fresh clone measurements are listed below. The archived discovery measurements remain evidence about the source campaign; `magicquant.clone-benchmarks.json` supplies results for the new weights. + +| Cloned recipe | Size (GB) | KLD against the target reference | +| --- | ---: | ---: | +| LM-Q8_0 | 36.91 | 0.004771 | +| MQ-Q6_K_1 | 31.59 | 0.005383 | +| MQ-Q5_K_1 | 29.19 | 0.006012 | +| MQ-Q5_K_S_1 | 26.33 | 0.007155 | +| MQ-Q4_K_M_1 | 24.82 | 0.007832 | +| MQ-Q4_K_M_2 | 22.32 | 0.010894 | +| MQ-IQ4_NL_1 | 20.89 | 0.013040 | +| MQ-IQ3_M_1 | 17.60 | 0.026825 | +| UD-IQ3_S | 13.68 | 0.068513 | +| MQ-IQ2_XXS_1 | 9.59 | 0.275805 | + +Evidence: [pinned clone model card](https://huggingface.co/magiccodingman/Qwen3.6-35B-A3B-Uncensored-MagicQuant-GGUF/resolve/aeead92caeffa71f045a58ae546e718c87c7111f/README.md) and [fresh clone benchmarks](https://huggingface.co/magiccodingman/Qwen3.6-35B-A3B-Uncensored-MagicQuant-GGUF/resolve/aeead92caeffa71f045a58ae546e718c87c7111f/magicquant-manifest/magicquant.clone-benchmarks.json). + +This demonstrates recipe reuse across related weights, not proof that the inherited recipes are optimal or that the two models have equivalent capabilities. Follow the [manifest and cloning guide](../docs/manifests-and-cloning.md) to prepare a compatible target, use `--source-repo`, or pin a manifest with `--source-json`. + +## Specialized research: downward-only MXFP4 adaptation + +The Qwen3.8-27B MXFP4 release starts from AMD's Quark AWQ MXFP4 checkpoint. It adapts recipes from a separate MagicQuant release using a strict downward-only storage policy: preserve a source tensor when the proposed replacement would not save space. This required specialized conversion work beyond ordinary CLI cloning. + +| Artifact | Size (GB) | Storage saved | KLD against native MXFP4 | +| --- | ---: | ---: | ---: | +| Native MXFP4 reference | 18.89 | 0% | 0.000000 | +| MQ-IQ4_XS_1 | 14.98 | 20.70% | 0.000940 | +| MQ-IQ2_XXS_1 | 8.22 | 56.47% | 0.321797 | + +The 14.98 GB artifact preserved all 496 native MXFP4 tensors byte-for-byte. The benchmark covers ordinary language logits; it does not establish vision or MTP quality. The reference is already quantized, so these scores do not measure divergence from BF16. + +Eleven adapted recipes were tested; ten were published. The dominated `UD-IQ2_XXS` result remains documented but its GGUF was removed. Keeping failed evidence separate from downloadable selections makes the decision auditable. + +Evidence: [pinned method and model card](https://huggingface.co/magiccodingman/Qwen3.8-27B-MXFP4-MagicQuant-GGUF/resolve/3ae9728099cc03206096d3ffeeb05e8d24b5f62c/README.md), [artifact metrics and publication status](https://huggingface.co/magiccodingman/Qwen3.8-27B-MXFP4-MagicQuant-GGUF/resolve/3ae9728099cc03206096d3ffeeb05e8d24b5f62c/magicquant-manifest/magicquant.final-survivors.json), and [benchmark scope](https://huggingface.co/magiccodingman/Qwen3.8-27B-MXFP4-MagicQuant-GGUF/resolve/3ae9728099cc03206096d3ffeeb05e8d24b5f62c/magicquant-manifest/magicquant.clone-benchmarks.json). Savings percentages are those reported by the release, before displayed sizes were rounded. + +## The original worked example remains available + +The [original research overview](overview.md) retains the 4B survivor table, tensor-group recipe breakdown, nonlinear-trade example, methodology diagram, and context and GPU scheduling discussion. The project README draws on that material and links here for newer published results. + +Its MQ-Q5_K_1 comparison uses the displayed values: `(2.88 / 2.73 - 1) × 100 ≈ 5.5%` additional storage and `(1 - 0.006632 / 0.009839) × 100 ≈ 32.6%` lower KLD. The [survivor-selection guide](docs/Nonlinear-Winners-And-Survivors.md) explains why a useful interior trade can deserve publication.