Ornith 1.0 9B
Local OKOKB · 22
Best for: Agentic coding on small machines·Pop: 76/100
Perf: est. 73 tok/s, typical error 5% · first token ~0.4s
Fits in 8 GB VRAM with room to spare. Best for agentic coding on small machines on RTX 5060.
Best Local AI Models for RTX 5060 (8GB)
The RTX 5060 pairs 8GB of GDDR7 with 448 GB/s, well above the RTX 4060 (272 GB/s), so an 8B model runs at est. 68 tok/s, typical error 5% against est. 40 tok/s. Capacity is the limit: 8GB holds one model up to 9B at Q4 with little room for context. The RTX 5060 Ti 16GB has the same bandwidth and twice the memory.
Quick answer
The best local LLM for the RTX 5060 is Ornith 1.0 9B (est. 73 tok/s, typical error 5%) on its 8GB VRAM. It uses ~5.6GB of VRAM; the RTX 5060 handles dense models up to ~9B at Q4. A 14B model at Q4 with CPU offload runs at est. 8 tok/s, rough estimate, offload not measured. (ModelFit, 2026)
Sizing rule: a Q4 model needs about 0.6 GB of VRAM per billion parameters, and ModelFit budgets 90% of this card's 8GB for weights, context, and KV-cache. The per-size table below uses that same budget. Other strong fits: Qwen3.5 4B Instruct (Q8) (4B, ~4.3GB) and LFM2.5 8B-A1B (8.3B, ~5.5GB). On this card Ornith 1.0 9B runs at est. 73 tok/s, typical error 5%. What it will not run: a 14B model does not fit in VRAM alone, so it spills to system RAM over PCIe and slows sharply.
$ollama run ornith:9b
- Top pick
- Ornith 1.0 9B
- Decode speed
- est. 73 tok/s, typical error 5%
- VRAM needed
- ~5.6 GB
Speeds are ModelFit estimates from memory bandwidth and model size, not measured benchmarks. Typical error: the median gap between the ModelFit speed model and measured runs on hardware the model was not fitted on, for this backend and model size.
Cite this page: ModelFit, RTX 5060 8GB for Local LLMs (2026): 8B at est. 68 tok/s, https://modelfit.io/gpu/rtx-5060/, updated October 2026, CC BY 4.0.
Specs
VRAM
8 GB GDDR7
Speed (8B Q4)
est. 68 tok/stypical error 5%
Bandwidth
448 GB/s
Architecture
Blackwell
Max model size
Up to ~9B dense
Compatibility
10 excellent, 0 workable
Vendor
Table 1
RTX 5060 estimated tokens/sec by model size
Q4_K_M · ModelFit estimate| Model size | Est. speed | Fit on 8GB |
|---|---|---|
| 7B | est. 78 tok/stypical error 5% | Fits in VRAM |
| 14B | est. 8 tok/srough estimate, offload not measured | CPU offload (slow) |
| 20B MoE (3.6B active) | est. 32 tok/srough estimate, offload not measured | CPU offload (slow) |
| 32B | est. 2 tok/srough estimate, offload not measured | CPU offload (slow) |
| 35B MoE (3B active) | est. 37 tok/srough estimate, offload not measured | CPU offload (slow) |
| 70B | est. 1 tok/srough estimate, offload not measured | CPU offload (slow) |
| 120B MoE (5.1B active) | est. 23 tok/srough estimate, offload not measured | CPU offload (slow) |
Fig. 1RTX 5060: estimated speed by model size
Decode speed, generic Q4_K_M model of each size at 4K context, 448 GB/s, typical error 5%
Fits in 8 GBCPU offload (slow)
7B78
14B8
20B MoE (3.6B active)32
32B2
35B MoE (3B active)37
70B1
120B MoE (5.1B active)23
tok/s est.ModelFit estimates, not measured benchmarks: decode speed of a generic Q4_K_M model of each size at 4K context on the RTX 5060's 448 GB/s bandwidth. MoE rows scale by active parameters (decode reads only the active experts), so a 35B MoE runs far faster than a dense 32B. "CPU offload" sizes exceed the 8GB VRAM; dense models slow to a crawl there, MoE models degrade less because hot experts stay GPU-resident. Offload rows are rough estimates: no measured run covers offload, so they carry no typical error. What typical error means.
Fig. 2
Context costs VRAM too.
Ornith 1.0 9B loads ~5.6 GB of weights; at 16k context the KV cache adds ~2.0 GB (exceeds the ~7 GB usable VRAM), and at 64k it adds ~8.0 GB (exceeds the budget, use a smaller quant or a q8_0 KV cache).
Weights 5.6 GBKV cache · 16k context 2.0 GBRuntime reserve 0.8 GB
This configuration exceeds the comfortable VRAM budget by about 0.4 GB. Expect offload pressure or a shorter safe context.
KV-cache figures assume an fp16 cache, the llama.cpp/Ollama default. Standard GQA models use a size-class estimate (8 KV heads x 128 head dim class); hybrid linear-attention models (Qwen3.5/3.6, Qwen3-Next) use the exact per-token cost from their published config, since only their sparse full-attention layers cache KV. A q8_0 KV cache roughly halves either figure. Estimates, not measurements.
Get told when a model beats Ornith 1.0 9B on your RTX 5060
One email when a new open-weight model is a better fit for 8 GB of VRAM, plus the Thursday weekly.
You'll also get the weekly ModelFit email. By signing up you agree to our Privacy Policy. Unsubscribe from either email anytime, in one click.
Where to buy the RTX 5060
Storage & accessories for your model library
- Internal NVMe SSD · 2TBA Gen4 M.2 drive keeps your whole GGUF and quant collection on fast local storage, loading models straight off NVMe.Check price on Amazon
- USB4 NVMe Enclosure40Gbps external storage fast enough to run models from. Pair it with an M.2 drive for a portable model vault.Check price on Amazon
ModelFit may earn a commission on purchases through these links, at no extra cost to you.
RTX 5060 VRAM for AI: what actually fits?
ModelFit budgets about 7.2GB of the 8GB for weights and context. Qwen3 8B at Q4 (~6.5GB) fits with little room for the KV cache, and a 14B model spills into system RAM, where the estimate falls to est. 8 tok/s, rough estimate, offload not measured. The RTX 5060 Ti 16GB streams the same 448 GB/s, so it matches the 5060 on 8B models (est. 68 tok/s, typical error 5%) and also holds a 14B model in VRAM at est. 41 tok/s, typical error 15%.
Table 2
What does not fit in 8GB (and what it costs you)
With 8GB, the honest ceiling is a single 7B model at Q4. Anything larger does not simply run slower: the layers that do not fit spill to system RAM over PCIe, and decode speed falls off a cliff. These are the same bandwidth-derived estimates used in the table above, shown here as the penalty rather than the headline.
| Model size | Fits in 8GB? | Est. speed | Slowdown vs 7B |
|---|---|---|---|
| 7B | Yes | est. 78 tok/stypical error 5% | n/a |
| 14B | No, needs ~2GB more in system RAM | est. 8 tok/srough estimate, offload not measured | ~10x slower |
| 20B MoE (3.6B active) | No, needs ~5GB more in system RAM | est. 32 tok/srough estimate, offload not measured | ~2x slower |
| 32B | No, needs ~12GB more in system RAM | est. 2 tok/srough estimate, offload not measured | ~39x slower |
| 35B MoE (3B active) | No, needs ~14GB more in system RAM | est. 37 tok/srough estimate, offload not measured | ~2x slower |
Mixture-of-experts models are the exception worth knowing: they activate only a fraction of their parameters per token, so a large MoE can stay usable on a small card where a dense model of the same total size will not. Speeds are ModelFit estimates from memory bandwidth and model size, not measured benchmarks.
Table 3
RTX 5060 vs similar GPUs
| Hardware | Memory | Speed | Bandwidth |
|---|---|---|---|
| RTX 4060 | 8 GB | est. 40 tok/stypical error 5% | 272 GB/s |
| RTX 5060 | 8 GB | est. 68 tok/stypical error 5% | 448 GB/s |
| RTX 3060 | 12 GB | est. 50 tok/stypical error 5% | 360 GB/s |
| RTX 5070 | 12 GB | est. 98 tok/stypical error 5% | 672 GB/s |
| RTX 5060 Ti | 16 GB | est. 68 tok/stypical error 5% | 448 GB/s |
Recommended models
10 modelsQwen3.5 4B Instruct (Q8)
Local OKExcellentA · 40
Best for: Coding, Agents, Multimodal·Pop: 88/100
Perf: est. 79 tok/s, typical error 5% · first token ~1.2s
Fits in 8 GB VRAM with room to spare. Best for coding, agents, multimodal on RTX 5060.
LFM2.5 8B-A1B
Local OKOKB · 24
Best for: On-device agents, tool calling, multilingual chat·Pop: 72/100
Perf: est. 185 tok/s, typical error 15% · first token ~0.1s
Fits in 8 GB VRAM with room to spare. Best for on-device agents, tool calling, multilingual chat on RTX 5060.
Gemma 4 E4B
Local OKExcellentA · 44
Best for: On-device, Mobile, Chat·Pop: 82/100
Perf: est. 102 tok/s, typical error 5% · first token ~0.4s
Fits in 8 GB VRAM with room to spare. Best for on-device, mobile, chat on RTX 5060.
MiMo-V2.6-Distill-Qwen-9B
Local OKOKB · 19
Best for: Agentic coding, Small machines·Pop: 60/100
Perf: est. 71 tok/s, typical error 5% · first token ~0.4s
Fits in 8 GB VRAM with room to spare. Best for agentic coding, small machines on RTX 5060.
Gemma 4 E2B (Q8)
Local OKExcellentA · 36
Best for: IoT, Mobile, Edge·Pop: 76/100
Perf: est. 155 tok/s, typical error 5% · first token ~1.3s
Fits in 8 GB VRAM with room to spare. Best for iot, mobile, edge on RTX 5060.
Granite 4.2 8B
Local OKExcellentA · 31
Best for: Chat, Coding·Pop: 66/100
Perf: est. 65 tok/s, typical error 5% · first token ~0.5s
Fits in 8 GB VRAM with room to spare. Best for chat, coding on RTX 5060.
Qwen3.5 9B Instruct
Local OKOKB · 3
Best for: Quality, Coding, Reasoning·Pop: 86/100
Perf: est. 73 tok/s, typical error 5% · first token ~0.4s
Fits in 8 GB VRAM with room to spare. Best for quality, coding, reasoning on RTX 5060.
Ministral 3 8B
Local OKOKB · 17
Best for: Chat, Coding, Vision·Pop: 74/100
Perf: est. 60 tok/s, typical error 5% · first token ~0.5s
Fits in 8 GB VRAM with room to spare. Best for chat, coding, vision on RTX 5060.
Granite Switch 4.1 8B
Local OKOKB · 11
Best for: Chat, Decisions, Safety checks·Pop: 64/100
Perf: est. 53 tok/s, typical error 5% · first token ~0.5s
Fits in 8 GB VRAM with room to spare. Best for chat, decisions, safety checks on RTX 5060.
Video and image models on the RTX 5060
Runs · Stated: the linked source says a GPU with that much VRAM runs the model. Runs · Published: a dated run on the named card with that much VRAM. unknown: no source yet. Fits with offload: the main model weights fit in VRAM with a 15% margin; the text encoder and VAE run from system RAM, which is slower. Exceeds VRAM: even the main weights are too large for 8 GB. Every VRAM tier: video · image. Ranked with reasons: video models for 8 GB of VRAM · image models for 8 GB of VRAM.
Video
| Model | Fits | Quality | App |
|---|---|---|---|
| LTX-2.3LTX-2 Community License Agreement | Runs · PublishedRTX 4060 · 8 GB VRAM · ComfyUI GGUF workflow, 720p 5 s in about 190 s, 32 GB RAM · ComfyUI issue #14345 | No independent score yet | Lightricks/LTX-2 ltx-pipelines, ComfyUI |
| MiniMax H3MiniMax H3 Community License Agreement | Runs · PublishedRTX 4050 · 6 GB VRAM · ComfyUI int8 convrot, 0.4 megapixel, 5 s, 16 GB RAM · ComfyUI issue #15482 | #8 of 48 in text to video11,417 votessnapshot 2026-09-22 | stable-diffusion.cpp, ComfyUI |
| Wan2.2 A14BApache License 2.0 | Runs · PublishedRTX 2060 · 6 GB VRAM · ComfyUI Q8 GGUF, 480x720, 81 frames, 7 to 10 min, 28 GB RAM plus a 60 GB page file · ComfyUI issue #12451 | #42 of 48 in text to video10,458 votessnapshot 2026-09-22 | ComfyUI |
| Wan2.2 TI2V 5BApache License 2.0 | Runs · Stated8 GB VRAM · ComfyUI native offloading · ComfyUI docs: Wan2.2Complete: Comfy-Org repackaged FP16 with FP8 scaled UMT518.1 GB download | No independent score yet | ComfyUI |
| Mochi 1 previewApache License 2.0 | unknownComplete: calcuis GGUF Q3_K_M with Q4_0 T5 and FP8 VAE7.7 GB download | #48 of 48 in text to video5,893 votessnapshot 2026-09-22 | Diffusers, ComfyUI |
Quality: blind human votes from Arena (arena.ai), CC BY 4.0, snapshot 2026-09-22. Scores are for the Arena checkpoint, not your local quantized build.
Image
| Model | Fits | Quality | App |
|---|---|---|---|
| FLUX 3 Image KleinFLUX Kommunity License v1.0 | Runs · Stated8 GB VRAM · official ComfyUI int8 files · FLUX 3 Image Klein README (Black Forest Labs) | No independent score yet | Diffusers |
| FLUX.1 [dev]FLUX.1 [dev] Non-Commercial License v1.1.1 | Runs · Stated6 GB VRAM · stable-diffusion.cpp with GGUF weights · stable-diffusion.cpp FLUX docs | #78 of 82 in text to image49,221 votessnapshot 2026-10-07 | Diffusers, ComfyUI |
| Qwen-Image-2.1Qwen Research License Agreement | Runs · Stated7 GB VRAM · DiffSynth-Studio, weights streamed from disk · DiffSynth-Studio docs: Qwen-Image-2.1 | #19 of 82 in text to image11,441 votessnapshot 2026-10-07 | Diffusers, stable-diffusion.cpp |
| SDXL Base 1.0CreativeML Open RAIL++-M License | Runs · Stated8 GB VRAM · SDXL 1.0, no app named · SDXL 1.0 announcement (Stability AI) | No independent score yet | ComfyUI, stable-diffusion.cpp |
| Z-Image-TurboApache 2.0 | Runs · Stated4 GB VRAM · stable-diffusion.cpp with GGUF weights · stable-diffusion.cpp Z-Image docs | #59 of 82 in text to image24,808 votessnapshot 2026-10-07 | Diffusers, stable-diffusion.cpp |
| FLUX.2 [klein]Apache 2.0 (4B), FLUX Non-Commercial License (9B) | Fits with offloadComplete: Klein 4B Q8_0 GGUF12.5 GB download | No independent score yet | Diffusers, stable-diffusion.cpp |
| MOSS-ImageApache 2.0 | Fits with offloadComplete: UD-Q4_K_XL GGUF13.2 GB download | No independent score yet | stable-diffusion.cpp |
| Stable Diffusion 3.5 LargeStability AI Community License | Fits with offloadComplete: Q4_0 GGUF16.4 GB download | #81 of 82 in text to image23,396 votessnapshot 2026-10-07 | ComfyUI |
| FLUX.2 [dev]FLUX Non-Commercial License | Exceeds VRAMComplete: Q4_K_M GGUF34.8 GB download | #38 of 82 in text to image77,988 votessnapshot 2026-10-07 | Diffusers, ComfyUI, stable-diffusion.cpp |
| Krea 2 TurboKrea 2 Community License Agreement v.1 | Exceeds VRAMComplete: Turbo Q4_K_M GGUF16.9 GB download | No independent score yet | Diffusers, stable-diffusion.cpp |
Quality: blind human votes from Arena (arena.ai), CC BY 4.0, snapshot 2026-10-07. Scores are for the Arena checkpoint, not your local quantized build.
Similar GPUs for local AI
- RTX 5060 Ti16GB · est. 68 tok/s
- RTX 40608GB · est. 40 tok/s
- RTX 306012GB · est. 50 tok/s
- RTX 507012GB · est. 98 tok/s
Speeds are estimates for Qwen3 8B Q4_K_M, typical error 5%.
Compatible model families
- QwenAlibaba Cloud: Qwen3.8 27B local flagship (Apache 2.0, vision)
- LlamaMeta: Most popular open-weight model family
- DeepSeekDeepSeek AI: V4.1 Flash: multimodal 552B MoE, API only
- MistralMistral AI: Excellent performance-per-parameter ratio
- GemmaGoogle DeepMind: Gemma 4 E4B: 4.5B effective, runs on 8GB Macs
- PhiMicrosoft: Best quality-per-gigabyte at small sizes
RTX 5060 FAQ: common questions
How much VRAM does the RTX 5060 have for LLMs?
The RTX 5060 has 8GB of GDDR7 VRAM at 448 GB/s. ModelFit budgets about 7.2GB of it for weights and context, enough for one model up to 9B at Q4 quantization with little room left for a long context.
What size LLM can I run on an RTX 5060?
Up to 9B at Q4 quantization: Qwen3 8B (~6.5GB), Llama 3.1 8B (~6.5GB) or Mistral 7B (~5.5GB). A 14B model spills into system RAM, where the estimate falls to est. 8 tok/s, rough estimate, offload not measured.
RTX 5060 vs RTX 5060 Ti for local AI?
Get the 5060 Ti 16GB. Both stream 448 GB/s, so they run 8B models at the same speed (est. 68 tok/s, typical error 5% vs est. 68 tok/s), but the Ti has twice the memory: it holds 14B models in VRAM, where the 5060 offloads them to system RAM.
RTX 5060 vs RTX 4060 for local LLMs?
Both have 8GB, so they load the same models. The 5060 streams 448 GB/s against 272 GB/s, which makes it 70% faster on 8B models (est. 68 tok/s, typical error 5% vs est. 40 tok/s).
How fast is a 27B-class model on the RTX 5060?
The RTX 5060's 8GB of VRAM cannot fit a 32B model comfortably. The largest size class it fits is 7B, at est. 78 tok/s, typical error 5%. The largest individual model in the catalog that fits is GLM-4-9B-Chat (9B).
Related guides & benchmarks
- Best AI Models for 8GB VRAM (2026)Ranked 3B-8B picks for 8GB cards like the RTX 5060.
- How Much VRAM Do You Need for Local LLMs?A tier-by-tier table from 8GB to 96GB: what each size of card can run.
- Gemma 4 GPU Requirements: Every VariantWhich Gemma 4 variants fit an 8GB GPU, and which need 12GB or more.
More local AI guides
- Best LLM for 8GB VRAMEvery 8 GB card side by side, each with its best pick and speed.
- How Much RAM (or VRAM) Do You Need for a Local LLM?The model-size-to-memory matrix: what each VRAM and RAM tier actually runs.
- Best LLM for MacBook (Apple Silicon)Unified-memory Macs run bigger models per dollar. Picks by RAM tier, M1 to M5.
- Local AI image generator on MacCompare image models by RAM and app.
- Local image generation on MacImage models and apps by Mac RAM.
Browse all NVIDIA GPUs for AI
Real-world calibration
Have this hardware? Share a measured run.
Time a model that fits your machine with Ollama, using the same prompts as every other run. The CLI shows your result and asks before it sends anything; your machine then joins the speed test leaderboard. It does not change the quality benchmark.
$ npx @modelfit/cli@latest bench
Want personalized recommendations?
Use our interactive wizard to compare models across Apple Silicon and NVIDIA GPUs.