Mistral Small 3.1 quants compared

26 GGUF builds by real file size, probed from unsloth/Mistral-Small-3.1-24B-Instruct-2503-GGUF on Hugging Face (2026-09-02). 24B params.

Download Mistral Small 3.1 Q4_K_M (13.35 GB) — it fits 24 GB of memory with 16k context. With Ollama: ollama run mistral-small3.1:24b

Every Mistral Small 3.1 quant by real file size

QuantWeights+ KV (16k)TotalFits comfortably inQuality
BF1643.92 GB4.0 GB47.9 GB64 GBFull precision (lossless)
Q8_K_XL27 GB4.0 GB31.0 GB48 GBNear-lossless
Q8_023.33 GB4.0 GB27.3 GB32 GBNear-lossless
Q6_K_XL19.36 GB4.0 GB23.4 GB32 GBExcellent
Q6_K18.02 GB4.0 GB22.0 GB32 GBExcellent
Q5_K_M15.61 GB4.0 GB19.6 GB24 GBVery high
Q5_K_XL15.61 GB4.0 GB19.6 GB24 GBVery high
Q5_K_S15.18 GB4.0 GB19.2 GB24 GBVery high
Q4_K_M *13.35 GB4.0 GB17.4 GB24 GBHigh — the default pick
Q4_113.85 GB4.0 GB17.9 GB24 GBHigh
Q4_K_XL13.51 GB4.0 GB17.5 GB24 GBHigh
Q4_K_S12.62 GB4.0 GB16.6 GB24 GBHigh
Q4_012.57 GB4.0 GB16.6 GB24 GBHigh
IQ4_NL12.54 GB4.0 GB16.5 GB24 GBHigh
IQ4_XS11.9 GB4.0 GB15.9 GB24 GBHigh
Q3_K_XL11.04 GB4.0 GB15.0 GB24 GBAcceptable — visible loss
Q3_K_M10.69 GB4.0 GB14.7 GB24 GBAcceptable — visible loss
Q3_K_S9.69 GB4.0 GB13.7 GB16 GBAcceptable — visible loss
IQ3_XXS8.76 GB4.0 GB12.8 GB16 GBAcceptable — visible loss
Q2_K_XL8.65 GB4.0 GB12.7 GB16 GBExperimental — not ranked — never recommended
Q2_K_L8.43 GB4.0 GB12.4 GB16 GBExperimental — not ranked — never recommended
Q2_K8.28 GB4.0 GB12.3 GB16 GBExperimental — not ranked — never recommended
IQ2_M7.68 GB4.0 GB11.7 GB16 GBExperimental — not ranked — never recommended
IQ2_XXS6.29 GB4.0 GB10.3 GB12 GBExperimental — not ranked — never recommended
IQ1_M5.6 GB4.0 GB9.6 GB12 GBExperimental — not ranked — never recommended
IQ1_S5.18 GB4.0 GB9.2 GB12 GBExperimental — not ranked — never recommended

* default pick. Weights = real GGUF file sizes from unsloth/Mistral-Small-3.1-24B-Instruct-2503-GGUF (probed 2026-09-02). KV = fp16 estimate; a q8_0 cache roughly halves it. "Comfortable" = weights + KV within 90% of memory.

Best Mistral Small 3.1 quant by memory

MemoryRecommended quantTotal (16k ctx)
16 GBIQ3_XXS12.8 GB
24 GBQ5_K_M19.6 GB
32 GBQ8_027.3 GB
64 GBBF1647.9 GB

Why we don't rank Mistral Small 3.1's 2-bit quants

Quants at 2 bits per weight or below (Q2_K, IQ2, IQ1, TQ1) cut file size by roughly half versus Q4, but the quality collapse is steep and non-linear: perplexity spikes, instruction-following degrades, and hallucinations rise. A model that answers faster but wrong is not a smaller model — it is a worse one. ModelFit lists these builds for completeness but never ranks or recommends them.

Frequently asked questions

What is the best quantization of Mistral Small 3.1?

Q4_K_M is the default pick: 13.35 GB of weights, high — the default pick quality, fitting comfortably in 24 GB of memory (weights + 16k context KV-cache). Go Q6_K or Q8_0 if you have headroom.

How much memory does Mistral Small 3.1 need?

At Q4_K_M, Mistral Small 3.1 needs 13.35 GB for the weights plus ~4.0 GB of KV-cache at 16k context — about 17.4 GB total, so a 24 GB card or Mac (90% usable budget) runs it comfortably.

Should I use a Q2_K or IQ2 quant of Mistral Small 3.1?

No. Mistral Small 3.1 at 2 bits per weight is a visibly worse model — quality collapse at that bitrate is steep, not gradual. If only a 2-bit build fits your memory, run a smaller model at Q4_K_M instead. ModelFit lists these builds but never recommends them.

Cite this page

ModelFit: Mistral Small 3.1 quantization comparison (real GGUF file sizes).
https://modelfit.io/quant-compare/mistral-small-3.1-24b/ (data probed 2026-09-02, CC BY 4.0).