I spent a day trying to prove quantization makes local LLMs dumber. It mostly doesn't — and where it does, it's a cliff.

Everyone has an opinion about how much quantization hurts a local model. Almost nobody measures it. So I did — the same model (Qwen3-4B) at five quantization levels, on both factual and reasoning/code tasks, graded on whether it gets the answer right, not just whether the text looks similar.

I expected Q4 to quietly rot the model, especially on reasoning. Here's what actually happened:

1. Down to 2-bit, factual accuracy barely moves — Q8, Q4_K_M, even Q2_K land within noise of full precision. (Reasoning points the same way on a smaller, noisier sample — I keep the error bars honest below.)
2. 1-bit is a cliff, not a slope. IQ1_M doesn't degrade gracefully; it collapses — 91%→15% on facts, 60%→10% on reasoning.
3. But every quant rewrites ~79% of the words. Same answer, different text, every run — which matters a lot if you cache, eval, or automate on outputs.

Everything here is reproducible in one command. Run it on your model.

---

The numbers

Qwen3-4B, greedy decoding (temperature 0), graded by whether the correct answer appears in the completion. Reference = BF16 (full precision). Two independent prompt sets: 54 factual/math, 40 multi-step reasoning/code.

Accuracy (higher = better):

Quant	Size	Facts (n=54)	Reasoning/Code (n=40)
BF16 (truth)	8.05 GB	90.7%	60.0%
Q8_0	4.28 GB	90.7%	62.5%
Q4_K_M (the default everyone runs)	2.50 GB	88.9%	62.5%
Q2_K	1.67 GB	92.6%	62.5%
IQ1_M (1-bit)	1.14 GB	14.8%	10.0%

        FACTS accuracy                      REASONING accuracy
BF16   ████████████████████ 90.7%          █████████████ 60.0%
Q8_0   ████████████████████ 90.7%          █████████████ 62.5%
Q4_K_M ████████████████████ 88.9%          █████████████ 62.5%
Q2_K   █████████████████████ 92.6%         █████████████ 62.5%
IQ1_M  ███ 14.8%   <- cliff                ██ 10.0%   <- cliff

The small +/- swings (e.g. Q2_K above BF16, Q4 above BF16 on reasoning) are noise — at these sample sizes the 95% confidence interval is roughly ±8 pts on facts and ±15 pts on the noisier reasoning set (n=40, 60% baseline). So the honest read is: on factual accuracy, 2-bit tracks full precision with room to spare (that result is solid); the reasoning set points the same way but is under-powered — it can't rule out a moderate Q4 dip, only a catastrophic one. The one thing far outside any error bar is the 1-bit cliff.

The myth this busts

There's a scarier number I could have led with: Q4_K_M changes 78.7% of the raw 24-token completions vs BF16. It's true. It's also misleading — because when you grade the actual answer, Q4 is right just as often as full precision. The text drifts; the knowledge doesn't.

If a benchmark only diffs the output text, it can't tell harmless Q4 rewording apart from a model that has genuinely broken. You have to grade the answer. That's the whole methodology here.

But the 79% text-drift still matters (if you're not just chatting)

Accuracy-preserving ≠ reproducible. Q4_K_M gives the same answer in different words on ~4 out of 5 prompts. If you:

• cache model outputs — your cache keys won't hit,
• run agents that parse structured output — the format shifts under you,
• evaluate with exact-match or regex — your scores move for no real reason,

…then "same accuracy" is cold comfort. Quantization changed what your pipeline sees, just not what the model knows.

The receipts

The cliff is real — 1-bit genuinely breaks (BF16 right, IQ1_M wrong):

• The capital of Australia is → BF16: Canberra… · IQ1_M: "(Capital of the world) is the capital of the world…"
• The largest planet in our solar system is → BF16: Jupiter… · IQ1_M: "the most important thing in the most important thing…"
• The author of "Pride and Prejudice" is → BF16: Jane Austen… · IQ1_M: "a person who is, and the person who is…"

And the harmless drift — Q4 keeps the answer, changes the words:

• The capital of Australia is → BF16: "Canberra. The capital of the United States is Washington…" · Q4_K_M: "Canberra. The capital of Canada is Ottawa…" — both correct, different continuation.

Test your own model

# needs llama.cpp (llama-server on PATH)
python3 bench.py --repo <your-org/your-model-GGUF> \
  --ref BF16 --quants Q8_0 Q4_K_M Q2_K IQ1_M \
  --prompts prompts.txt --n-predict 24 --out results
python3 grade.py results/completions.json

Swap --prompts prompts.txt for --prompts reasoning_prompts.txt to run the reasoning/code set. bench.py downloads each quant via llama.cpp's -hf, generates a greedy completion per prompt, and records it; grade.py scores answers against the built-in key and reports accuracy + right→wrong flips. Bring your own prompt/answer key to test your domain and model — that's where this gets interesting, and it's the same one command.

Method (so you can trust it, or break it)

• Greedy, temperature 0 — deterministic, no sampling noise.
• Graded on the answer, not the text: case-insensitive substring match against a hand-written key (grade.py), so "Au" counts whether or not the wording matches.
• Same source for every quant (unsloth/Qwen3-4B-GGUF) — differences are the quantization, not different checkpoints.
• Raw data included — results/completions.json has every completion. Regrade it, dispute the key, run your own analysis.

Honest limits

• One model (Qwen3-4B), ~94 graded prompts across two task types. This is a method plus a robust starting result, not the last word.
• Bigger models and harder benchmarks are the open question. On facts Q4 clearly held; on the noisier reasoning set it showed no degradation but the sample is too small to be sure — and a 70B on GPQA-hard or long-chain agentic tasks could behave differently. The tool is built so you can run exactly that: bring a harder prompt+answer key and re-bench.
• Mind the error bars. 95% CIs are ~±8 pts (facts, n=54) and ~±15 pts (reasoning, n=40). Small swings between BF16/Q8/Q4/Q2 are noise; the facts result is tight enough to trust ("2-bit ≈ full precision"), the reasoning result is consistent but too small to be the last word. The 1-bit cliff (−50 to −76 pts) is far outside any error bar.

What to actually do

• Q8 / Q4_K_M for accuracy-sensitive work — the memory savings are basically free on knowledge tasks.
• Don't go below ~2-bit for anything you care about — the cliff is steep.
• If you cache / eval / automate on outputs, pin one quant and re-baseline when you change it. The text will move.

---

Why I built this

I work on PrismOS-AI — a desktop app (Tauri + Rust) that lets you ask a local Ollama model about your own files, entirely offline. When everything runs on the user's laptop, quantization isn't academic — it's the difference between the app fitting or not. I needed to know what it actually costs, so I measured it. The bit-exact grading came out of a from-scratch inference engine I'd been building.

---

MIT licensed. PRs with more models, harder prompt sets, and reasoning benchmarks welcome — a 70B or a GPQA-hard run is where this gets interesting.