Best NVIDIA GPUs for Local AI

Find the right GPU for running AI models locally with Ollama. From budget 12GB cards to the 32GB RTX 5090, compare speeds, VRAM, and model compatibility.

NVIDIA GPUs ranked: 16
VRAM, GB: 8 to 96
Memory bandwidth, GB/s: 272 to 1,792

Fig. 1

Estimated speed

Qwen3 8B Q4_K @ 4K context, typical error 5%

RTX 5090est. 213 tok/s

RTX PRO 6000est. 213 tok/s

RTX 4090est. 133 tok/s

RTX 5080est. 132 tok/s

RTX 5070 Tiest. 125 tok/s

RTX 3090est. 117 tok/s

RTX 4080 SUPERest. 102 tok/s

RTX 5070est. 98 tok/s

RTX 4070 Ti SUPERest. 94 tok/s

RTX 4070est. 73 tok/s

RTX 4070 SUPERest. 73 tok/s

RTX 5060 Tiest. 68 tok/s

RTX 3070est. 61 tok/s

RTX 3060est. 50 tok/s

RTX 4060 Tiest. 43 tok/s

RTX 4060est. 40 tok/s

Table 1

GPUs by tier

GPU	Memory	Speed (8B)	Bandwidth	Arch	Largest model
Budget
RTX 4060	8 GB GDDR6	est. 40 tok/stypical error 5%	272 GB/s	Ada Lovelace	Up to 8B parameter models
RTX 3060	12 GB GDDR6	est. 50 tok/stypical error 5%	360 GB/s	Ampere	Up to 9B parameter models
RTX 3070	8 GB GDDR6	est. 61 tok/stypical error 5%	448 GB/s	Ampere	Up to 8B parameter models
RTX 4060 Ti	16 GB GDDR6	est. 43 tok/stypical error 5%	288 GB/s	Ada Lovelace	Up to 14B parameter models
RTX 5060 Ti	16 GB GDDR7	est. 68 tok/stypical error 5%	448 GB/s	Blackwell	Up to 14B parameter models
Mid-Range
RTX 4070	12 GB GDDR6X	est. 73 tok/stypical error 5%	504 GB/s	Ada Lovelace	Up to 9B parameter models
RTX 4070 SUPER	12 GB GDDR6X	est. 73 tok/stypical error 5%	504 GB/s	Ada Lovelace	Up to 9B parameter models
RTX 5070	12 GB GDDR7	est. 98 tok/stypical error 5%	672 GB/s	Blackwell	Up to 9B parameter models
RTX 5070 Ti	16 GB GDDR7	est. 125 tok/stypical error 5%	896 GB/s	Blackwell	Up to 14B parameter models
RTX 4070 Ti SUPER	16 GB GDDR6X	est. 94 tok/stypical error 5%	672 GB/s	Ada Lovelace	Up to 14B parameter models
High-End
RTX 4080 SUPER	16 GB GDDR6X	est. 102 tok/stypical error 5%	736 GB/s	Ada Lovelace	Up to 14B parameter models
RTX 5080	16 GB GDDR7	est. 132 tok/stypical error 5%	960 GB/s	Blackwell	Up to 14B parameter models
RTX 3090	24 GB GDDR6X	est. 117 tok/stypical error 5%	936 GB/s	Ampere	Up to 32B parameter models
RTX 4090	24 GB GDDR6X	est. 133 tok/stypical error 5%	1008 GB/s	Ada Lovelace	Up to 32B parameter models
Ultra
RTX 5090	32 GB GDDR7	est. 213 tok/stypical error 5%	1792 GB/s	Blackwell	Up to 70B parameter models
RTX PRO 6000	96 GB GDDR7	est. 213 tok/stypical error 5%	1792 GB/s	Blackwell	Up to 120B parameter models

Running two or three cards? Pool the VRAM

GPU 1GPU 2GPU 3

Pick at least two cards (duplicates allowed, e.g. 2x RTX 5060 Ti) to see what the pooled VRAM runs.

Pooling works: Ollama and llama.cpp split layers across cards automatically. With the default layer split, throughput stays near single-card speed and mixed cards run at the slower card's pace. Two identical cards using llama.cpp's row (tensor) split can decode faster than one card, since both memory buses read weights in parallel, though transfer overhead keeps it below 2x. Mixing NVIDIA and AMD in one rig requires the Vulkan build. Fit is what pooling reliably buys you; per-card speed estimates on the GPU pages are single-card only. Long context also eats into the budget (KV-cache), so leave headroom.

Table 2

VRAM guide: what models can you run?

VRAM	Max model size	Example models
12 GB	Up to 9B (Q4)	Qwen2.5 7B, Llama 3.2 8B, Mistral 7B
16 GB	Up to 14B-27B (Q4)	Qwen2.5 14B, DeepSeek-R1 14B
24 GB	Up to 32B (Q4)	Qwen2.5 32B, DeepSeek-R1 32B
32 GB	Up to 70B (Q4)	Llama 3.1 70B, Qwen2.5 72B

Explore more

Have an Apple Silicon Mac instead?

ModelFit also supports MacBook Air, MacBook Pro, Mac Studio, Mac Mini, and iPhone.

Open ModelFit Wizard