Qwen3.5 4B Instruct quants compared

24 real GGUF builds of Qwen3.5 4B Instruct measured from bartowski/Qwen_Qwen3.5-4B-GGUF. Download the Q4_K_M (2.81 GB) unless you know why you need heavier. Probed 2026-09-02.

Download Qwen3.5 4B Instruct Q4_K_M (2.81 GB). It fits 8 GB of memory with 16k context. With Ollama: ollama run qwen3.5:4b

Every Qwen3.5 4B Instruct quant by real file size

QuantWeights+ KV (16k)TotalFits comfortably inQuality
BF168.07 GB0.5 GB8.6 GB12 GBFull precision (lossless)
Q8_04.3 GB0.5 GB4.8 GB8 GBNear-lossless
Q6_K_L3.69 GB0.5 GB4.2 GB8 GBExcellent
Q6_K3.54 GB0.5 GB4.0 GB8 GBExcellent
Q5_K_L3.35 GB0.5 GB3.9 GB8 GBVery high
Q5_K_M3.21 GB0.5 GB3.7 GB8 GBVery high
Q5_K_S3.03 GB0.5 GB3.5 GB8 GBVery high
Q4_K_M *2.81 GB0.5 GB3.3 GB8 GBHigh: the default pick
Q4_K_L2.95 GB0.5 GB3.5 GB8 GBHigh
Q4_12.78 GB0.5 GB3.3 GB8 GBHigh
Q4_K_S2.65 GB0.5 GB3.1 GB8 GBHigh
Q4_02.59 GB0.5 GB3.1 GB8 GBHigh
IQ4_NL2.58 GB0.5 GB3.1 GB8 GBHigh
IQ4_XS2.49 GB0.5 GB3.0 GB8 GBHigh
Q3_K_XL2.63 GB0.5 GB3.1 GB8 GBAcceptable: visible loss
Q3_K_L2.48 GB0.5 GB3.0 GB8 GBAcceptable: visible loss
Q3_K_M2.4 GB0.5 GB2.9 GB8 GBAcceptable: visible loss
IQ3_M2.31 GB0.5 GB2.8 GB8 GBAcceptable: visible loss
Q3_K_S2.29 GB0.5 GB2.8 GB8 GBAcceptable: visible loss
IQ3_XS2.24 GB0.5 GB2.7 GB8 GBAcceptable: visible loss
IQ3_XXS2.09 GB0.5 GB2.6 GB8 GBAcceptable: visible loss
Q2_K_L2.18 GB0.5 GB2.7 GB8 GBExperimental: not ranked: never recommended
Q2_K2.04 GB0.5 GB2.5 GB8 GBExperimental: not ranked: never recommended
IQ2_M1.82 GB0.5 GB2.3 GB8 GBExperimental: not ranked: never recommended

* default pick. Weights = real GGUF file sizes from bartowski/Qwen_Qwen3.5-4B-GGUF (probed 2026-09-02). KV = fp16 estimate; a q8_0 cache roughly halves it. "Comfortable" = weights + KV within 90% of memory.

Best Qwen3.5 4B Instruct quant by memory

MemoryRecommended quantTotal (16k ctx)
8 GBQ8_04.8 GB
12 GBBF168.6 GB

Why we don't rank Qwen3.5 4B Instruct's 2-bit quants

Quants at 2 bits per weight or below (Q2_K, IQ2, IQ1, TQ1) cut file size by roughly half versus Q4, but the quality collapse is steep and non-linear: perplexity spikes, instruction-following degrades, and hallucinations rise. A model that answers faster but wrong is not a smaller model: it is a worse one. ModelFit lists these builds for completeness but never ranks or recommends them.

Run Qwen3.5 4B Instruct on your GPU

RTX 3060
12 GB: verdict
RTX 4060
8 GB: verdict
RTX 4060 Ti
16 GB: verdict
RTX 4070
12 GB: verdict
RTX 4070 Ti SUPER
16 GB: verdict
RTX 3090
24 GB: verdict
RTX 4090
24 GB: verdict
RTX 5070 Ti
16 GB: verdict
RTX 5080
16 GB: verdict
RTX 5090
32 GB: verdict

Frequently asked questions

What is the best quantization of Qwen3.5 4B Instruct?

Q4_K_M is the default pick: 2.81 GB of weights, high: the default pick quality, fitting comfortably in 8 GB of memory (weights + 16k context KV-cache). Go Q6_K or Q8_0 if you have headroom.

How much memory does Qwen3.5 4B Instruct need?

At Q4_K_M, Qwen3.5 4B Instruct needs 2.81 GB for the weights plus ~0.5 GB of KV-cache at 16k context, about 3.3 GB total, so a 8 GB card or Mac (90% usable budget) runs it comfortably.

Should I use a Q2_K or IQ2 quant of Qwen3.5 4B Instruct?

No. Qwen3.5 4B Instruct at 2 bits per weight is a visibly worse model. Quality collapse at that bitrate is steep, not gradual. If only a 2-bit build fits your memory, run a smaller model at Q4_K_M instead. ModelFit lists these builds but never recommends them.

Cite this page

ModelFit: Qwen3.5 4B Instruct quantization comparison (real GGUF file sizes).
https://modelfit.io/quant-compare/qwen3.5-4b (data probed 2026-09-02, CC BY 4.0).