Quant compare

Every GGUF quantization of 21 popular models, by real file size — 527 builds probed from Hugging Face. Updated 2026-09-02.

Which quant should you download? For almost everyone, Q4_K_M: roughly 0.6 GB per billion parameters, with quality loss most users cannot detect. Q6_K and Q8_0 buy measurable quality when you have memory headroom. Below Q4 the tradeoff collapses — and at 2 bits per weight the model stops being the same model, which is why ModelFit lists those builds but never recommends them.

MODELS COVERED
21
GGUF BUILDS
527
FILE SIZES
Real, HF-probed
2-BIT QUANTS
Never ranked

Compare quants for your memory

Best pick for 16 GB: Qwen3 8B Q8_0 — 8.11 GB weights + ~2.0 GB KV-cache at 16k context = 10.1 GB against 14.4 GB usable. Quality: near-lossless.

QuantWeights+ KV (16k)TotalFit on 16 GBQuality
BF1615.26 GB2.0 GB17.3 GBToo bigFull precision (lossless)
Q8_0 *8.11 GB2.0 GB10.1 GBComfortableNear-lossless
Q6_K_L6.54 GB2.0 GB8.5 GBComfortableExcellent
Q6_K6.26 GB2.0 GB8.3 GBComfortableExcellent
Q5_K_L5.81 GB2.0 GB7.8 GBComfortableVery high
Q5_K_M5.45 GB2.0 GB7.5 GBComfortableVery high
Q5_K_S5.33 GB2.0 GB7.3 GBComfortableVery high
Q4_K_M4.68 GB2.0 GB6.7 GBComfortableHigh — the default pick
Q4_K_L5.11 GB2.0 GB7.1 GBComfortableHigh
Q4_14.89 GB2.0 GB6.9 GBComfortableHigh
Q4_K_S4.47 GB2.0 GB6.5 GBComfortableHigh
IQ4_NL4.46 GB2.0 GB6.5 GBComfortableHigh
Q4_04.46 GB2.0 GB6.5 GBComfortableHigh
IQ4_XS4.25 GB2.0 GB6.3 GBComfortableHigh
Q3_K_XL4.63 GB2.0 GB6.6 GBComfortableAcceptable — visible loss
Q3_K_L4.13 GB2.0 GB6.1 GBComfortableAcceptable — visible loss
Q3_K_M3.84 GB2.0 GB5.8 GBComfortableAcceptable — visible loss
IQ3_M3.63 GB2.0 GB5.6 GBComfortableAcceptable — visible loss
Q3_K_S3.51 GB2.0 GB5.5 GBComfortableAcceptable — visible loss
IQ3_XS3.38 GB2.0 GB5.4 GBComfortableAcceptable — visible loss
IQ3_XXS3.14 GB2.0 GB5.1 GBComfortableAcceptable — visible loss

— listed for completeness, never recommended. Why: below.

Weights = real GGUF file sizes from bartowski/Qwen_Qwen3-8B-GGUF on Hugging Face. KV-cache = fp16 estimate (a q8_0 cache roughly halves it). Usable budget = 90% of memory. Estimates, not measurements.

Why we don't rank 2-bit quants

Quants at 2 bits per weight or below (Q2_K, IQ2, IQ1, TQ1) cut file size by roughly half versus Q4, but the quality collapse is steep and non-linear: perplexity spikes, instruction-following degrades, and hallucinations rise. A model that answers faster but wrong is not a smaller model — it is a worse one. ModelFit lists these builds for completeness but never ranks or recommends them.

Some comparators grade a Q1_0 or Q2_K build an "A" because it loads. Loading is not running well. If a 2-bit build is the only thing that fits your memory, the honest answer is a smaller model at Q4 — not a bigger model at 2 bits.

Quant tables by model

Gemma 3 4B Instruct
4B params — 26 quants
DeepSeek-R1 Distill Qwen 7B
7B params — 24 quants
Qwen3 8B
8B params — 24 quants
Llama 3.1 8B Instruct
8B params — 21 quants
Qwen3.5 9B Instruct
9B params — 25 quants
Gemma 4 12B
12B params — 22 quants
Mistral Nemo 12B
12B params — 22 quants
Qwen3 14B
14B params — 26 quants
Phi-4 14B
14B params — 26 quants
DeepSeek-R1 Distill Qwen 14B
14B params — 26 quants
GPT-OSS 20B
21B params — 28 quants
Mistral Small 3.1
24B params — 26 quants
Gemma 4 26B-A4B
26B params — 27 quants
Qwen3.5 27B Instruct
27B params — 27 quants
Qwen3.6 27B
27B params — 27 quants
Qwen3.8 27B
27B params — 25 quants
Qwen3 30B
30B params — 27 quants
Qwen3.5 35B-A3B Instruct
35B params — 28 quants
Qwen3-Next 80B-A3B
80B params — 26 quants
Llama 4 Scout
109B params — 27 quants
GPT-OSS 120B
117B params — 17 quants

Methodology

  • Weights are the real byte sizes of GGUF files on Hugging Face (official and bartowski/unsloth/LM Studio Community repos), summed across shards, probed 2026-09-02.
  • KV-cache assumes an fp16 cache at the selected context: a size-class estimate for standard GQA models, the exact per-token cost from the published config for hybrid linear-attention models (Qwen3.5/3.6, Qwen3-Next).
  • Usable budget is 90% of memory — the OS and inference runtime need their share. The same rule powers our can-i-run verdict pages.
  • MXFP4-native models (GPT-OSS) keep their expert weights in MXFP4 in every build, so heavier quants barely grow the file — only non-expert tensors are re-quantized.

Frequently asked questions

What is the best quantization for a local LLM?

Q4_K_M is the default pick for almost everyone: it cuts memory by ~40% versus Q8_0 with quality loss most users cannot detect in chat or coding. Go Q6_K or Q8_0 when you have memory to spare, and never go below Q3_K_M for real work.

Why does ModelFit not rank 2-bit quants like Q2_K or IQ2?

At 2 bits per weight the quality collapse is steep and non-linear — perplexity spikes and hallucinations rise. A model that answers faster but wrong is a worse model, not a smaller one. We list 1/2-bit builds for completeness but exclude them from every ranking and recommendation.

How much memory does KV-cache add to a model?

For a standard GQA model at 16k context, roughly 1-2 GB for a 7-8B model and 4-6 GB for a 70B. Hybrid linear-attention models (Qwen3.5/3.6, Qwen3-Next) cache far less because only their sparse full-attention layers store KV. A q8_0 KV cache roughly halves any of these figures.

Are these file sizes measured or estimated?

Weights are real GGUF file sizes probed from Hugging Face repositories (updated 2026-09-02). KV-cache and the 90% usable-memory budget are engine estimates, labeled as such — the same math that powers ModelFit's can-i-run verdict pages.

Cite this page

Free to reuse with attribution (CC BY 4.0). Source repos linked from each model table.

ModelFit: LLM Quantization Comparison (real GGUF file sizes).
https://modelfit.io/quant-compare/ (data probed 2026-09-02).