VRAM Calculator for Local LLMs
Enter your graphics card memory and see which local models fit, the single best pick for that card, and what long context costs on top.
How much VRAM does a local LLM need? About 0.6 GB per billion parameters at Q4 quantization for the weights alone, against a usable budget of roughly 90% of the card. On the RTX 4090, that is ~22 GB of weights, which the engine pairs with Gemma 4 26B-A4B at about ~85 tok/s. Context is charged separately: the KV cache grows with every token you keep, and on a 70B model it adds around 20GB at 64k context.
With 24 GB of VRAM, ModelFit budgets about 22 GB for the model and comfortably runs local LLMs up to ~35B parameters at Q4. The best single pick is Gemma 4 26B-A4B. Estimates use a representative 24 GB card (RTX 4090).
Tokens/sec are ModelFit estimates from memory bandwidth and model size, not measured benchmarks. Ollama commands are registry-verified.
What fits at each VRAM tier
Each row is computed by the same engine the calculator runs, using one representative card per tier and the 90% weights budget.
| Card | Usable for weights | Largest model that fits | Best pick | Models that fit |
|---|---|---|---|---|
| RTX 4060 | ~7 GB | Ornith 1.0 9B (9B) | Gemma 4 E4B | 38 |
| RTX 3060 | ~11 GB | Gemma 4 12B (12B) | Qwen3.5 9B Instruct | 47 |
| RTX 5080 | ~14 GB | LFM2 24B-A2B Instruct (24B, 2B active) | Gemma 4 E4B (Q8) | 57 |
| RTX 4090 | ~22 GB | Qwen3.5 35B-A3B Instruct (35B, 3B active) | Gemma 4 26B-A4B | 75 |
| RTX 5090 | ~29 GB | Qwen3.6 35B-A3B (35B, 3B active) | Qwen3.6 35B-A3B | 84 |
| RTX PRO 6000 | ~86 GB | Qwen3.5 122B-A10B Instruct (122B, 10B active) | GPT-OSS 120B | 100 |
Q4_K_M builds from ModelFit's 106-model catalog. Fit and tok/s are ModelFit estimates from memory bandwidth and model size, not measured benchmarks. Updated 2026-09-15.
Context is charged separately
Weights are only the first bill. The KV cache holds the conversation and grows with every token of context you keep, which is why a card that fits a model at 8k can run out of memory at 64k. These rows are computed from each model's own catalog entry.
| Model | Weights at Q4 | KV per 1,000 tokens | KV at 16k | KV at 64k |
|---|---|---|---|---|
| Qwen3.5 9B Instruct (9B) | ~7 GB | ~32 KB (exact config) | 0.5 GB | 2 GB |
| Qwen3.5 27B Instruct (27B) | ~16 GB | ~64 KB (exact config) | 1 GB | 4 GB |
| Llama 3.3 70B Instruct (70B) | ~42 GB | ~320 KB | 5 GB | 20 GB |
KV-cache figures assume an fp16 cache, the llama.cpp/Ollama default. Standard GQA models use a size-class estimate (8 KV heads x 128 head dim class); hybrid linear-attention models (Qwen3.5/3.6, Qwen3-Next) use the exact per-token cost from their published config, since only their sparse full-attention layers cache KV. A q8_0 KV cache roughly halves either figure. Estimates, not measurements.
The practical rule: subtract the KV cache first, then fit the weights in what is left. Quantizing the KV cache to q8_0 halves it. If you are choosing between a bigger model at short context and a smaller one at long context, the quantization comparison breaks that trade down per memory tier.
Every card, page by page
Each card page carries its own bandwidth, estimated token rates by model size, and buy links. The hub lists the same cards side by side.
8 GB · ~7 GB usable · VRAM
12 GB · ~11 GB usable · VRAM
16 GB · ~14 GB usable · VRAM
16 GB · ~14 GB usable · VRAM
12 GB · ~11 GB usable · VRAM
12 GB · ~11 GB usable · VRAM
12 GB · ~11 GB usable · VRAM
16 GB · ~14 GB usable · VRAM
16 GB · ~14 GB usable · VRAM
16 GB · ~14 GB usable · VRAM
16 GB · ~14 GB usable · VRAM
24 GB · ~22 GB usable · VRAM
24 GB · ~22 GB usable · VRAM
32 GB · ~29 GB usable · VRAM
96 GB · ~86 GB usable · VRAM
24 GB · ~22 GB usable · VRAM
20 GB · ~18 GB usable · VRAM
All GPU pages · Can I run it? · Best hardware for a model · Open compatibility dataset
Frequently asked questions
How much VRAM do I need to run a local LLM?
Budget about 0.6 GB of VRAM per billion parameters at Q4 quantization for the weights, and treat roughly 90% of the card as usable so the display and the context cache still have room. In practice an 8GB card runs up to an 8B model, 12GB reaches 9B, 16GB unlocks 14B-class models, 24GB runs 32B-class models, and 32GB reaches 70B-class mixture-of-experts builds that keep only a few billion parameters active per token.
Why can I only use about 90% of my VRAM?
The card also has to hold the CUDA or Metal context, the KV cache for your conversation, and whatever your desktop is displaying. ModelFit budgets 90% of VRAM for model weights so the fit test is honest on a real machine rather than matching a spec sheet. On Apple Silicon the equivalent budget is 70% of unified memory up to 32GB, rising to about 85% at 128GB and above, because the GPU shares that memory with macOS.
Does context length change how much VRAM I need?
Yes, and it is the most common surprise. The KV cache grows linearly with context length, from about 112 KB per 1,000 tokens on small models to over 320 KB per 1,000 tokens on a 70B model with an fp16 cache. A 70B model at 64k tokens of context therefore needs roughly another 20GB on top of its 42GB of weights. Quantizing the KV cache to q8_0 roughly halves that, and hybrid linear-attention models such as Qwen3.5 cache only a fraction of what a classic attention layout needs.
Can I run a 70B model on a 24GB graphics card?
Not at full speed, and not as a dense model. A dense 70B at Q4 needs about 42GB of weights, which exceeds the 22GB usable budget of a 24GB card. The workable paths are a mixture-of-experts model of that total size with a few billion active parameters, a smaller quantization with some layers offloaded to system RAM, or pooling two cards. Pooled fit is what the multi-GPU rig mode of the calculator computes; expect throughput below a single card with the same total VRAM because layer transfers add overhead.
Go deeper
Working with Apple unified memory instead of a discrete card? Start there.
All three modes in one page: unified memory, single card, and multi-GPU rigs.
The model-size-to-memory matrix, tier by tier, for unified memory machines.
The full walkthrough with per-model tables and quantization ladders.
Per-model VRAM breakdowns: DeepSeek V4 · GPT-OSS 20B · Kimi K3 · 8GB VRAM picks · all model pages · models for a 32GB tier · hardware stats
The weekly local-AI refresh
New open-weight models, real Apple Silicon benchmarks, and the one model worth running on your Mac this week. Free, one email a week, unsubscribe anytime.
By subscribing you agree to our Privacy Policy and to receive the weekly email. Unsubscribe anytime.