The Tesla T4 is a Turing data-center card with 16GB of GDDR6 at 300 GB/s, in a 70 W low-profile, passively cooled package. It holds dense models up to 14B at Q4: an 8B model runs at est. 37 tok/s, typical error 5% and a 14B model at est. 22 tok/s, typical error 15%. Ollama supports it under compute capability 7.5.
The best local LLM for the Tesla T4 is Mellum2.1 12B-A2.5B Thinking (est. 75 tok/s, typical error 15%) on its 16GB VRAM. It uses ~8.1GB of VRAM; the Tesla T4 handles dense models up to ~14B at Q4. A 14B model at Q4 runs at est. 22 tok/s, typical error 15%. (ModelFit, 2026)
Sizing rule: a Q4 model needs about 0.6 GB of VRAM per billion parameters, and ModelFit budgets 90% of this card's 16GB for weights, context, and KV-cache. The per-size table below uses that same budget. Other strong fits: Gemma 4 12B (12B, ~8GB) and NVIDIA Nemotron 3.5 Bolt E4B (4B, ~7.9GB). On this card Mellum2.1 12B-A2.5B Thinking runs at est. 75 tok/s, typical error 15%. What it will not run: a 32B model does not fit in VRAM alone, so it spills to system RAM over PCIe and slows sharply.
Speeds are ModelFit estimates from memory bandwidth and model size, not measured benchmarks. Typical error: the median gap between the ModelFit speed model and measured runs on hardware the model was not fitted on, for this backend and model size.
Cite this page: ModelFit, NVIDIA T4 VRAM: 16GB for Local LLMs, 14B at est. 22 tok/s, https://modelfit.io/gpu/tesla-t4/, updated October 2026, CC BY 4.0.
| Model size | Est. speed | Fit on 16GB |
|---|---|---|
| 7B | est. 43 tok/stypical error 5% | Fits in VRAM |
| 14B | est. 22 tok/stypical error 15% | Fits in VRAM |
| 20B MoE (3.6B active) | est. 50 tok/stypical error 15% | Fits in VRAM |
| 32B | est. 2 tok/srough estimate, offload not measured | CPU offload (slow) |
| 35B MoE (3B active) | est. 21 tok/srough estimate, offload not measured | CPU offload (slow) |
| 70B | est. 0 tok/srough estimate, offload not measured | CPU offload (slow) |
| 120B MoE (5.1B active) | est. 13 tok/srough estimate, offload not measured | CPU offload (slow) |
Decode speed, generic Q4_K_M model of each size at 4K context, 300 GB/s, typical error 5-15%
ModelFit estimates, not measured benchmarks: decode speed of a generic Q4_K_M model of each size at 4K context on the Tesla T4's 300 GB/s bandwidth. MoE rows scale by active parameters (decode reads only the active experts), so a 35B MoE runs far faster than a dense 32B. "CPU offload" sizes exceed the 16GB VRAM; dense models slow to a crawl there, MoE models degrade less because hot experts stay GPU-resident. Offload rows are rough estimates: no measured run covers offload, so they carry no typical error. What typical error means.
Mellum2.1 12B-A2.5B Thinking loads ~8.1 GB of weights; at 16k context the KV cache adds ~0.2 GB (still fits the ~14 GB usable VRAM), and at 64k it adds ~0.9 GB (still fits).
The model leaves about 6.1 GB of the usable VRAM budget free after weights and 16k context.
KV-cache figures assume an fp16 cache, the llama.cpp/Ollama default. Standard GQA models use a size-class estimate (8 KV heads x 128 head dim class); hybrid linear-attention models (Qwen3.5/3.6, Qwen3-Next) use the exact per-token cost from their published config, since only their sparse full-attention layers cache KV. A q8_0 KV cache roughly halves either figure. Estimates, not measurements.
One email when a new open-weight model is a better fit for 16 GB of VRAM, plus the Thursday weekly.
You'll also get the weekly ModelFit email. By signing up you agree to our Privacy Policy. Unsubscribe from either email anytime, in one click.
ModelFit may earn a commission on purchases through these links, at no extra cost to you.
ModelFit budgets about 14.4GB of the 16GB for weights and context, so 14B models at Q4 (~11GB) load with room for context, and 20B-class mixture-of-experts models fit too. Bandwidth is the limit: 300 GB/s is close to the RTX 4060 Ti 16GB (288 GB/s), so decode speed lands in the same range (est. 43 tok/s, typical error 5% on 8B for the 4060 Ti). Its Turing tensor cores (65 TFLOPS FP16) keep prompt reading quick for a 70 W card. Dense 27B and 32B models spill into system RAM.
| Hardware | Memory | Speed | Bandwidth |
|---|---|---|---|
| RTX 3060 | 12 GB | est. 50 tok/stypical error 5% | 360 GB/s |
| Tesla T4 | 16 GB | est. 37 tok/stypical error 5% | 300 GB/s |
| Tesla P100 | 16 GB | est. 42 tok/stypical error 5% | 732 GB/s |
| RTX 4060 Ti | 16 GB | est. 43 tok/stypical error 5% | 288 GB/s |
| RTX 5060 Ti | 16 GB | est. 68 tok/stypical error 5% | 448 GB/s |
Best for: Agentic coding, Small machines·Pop: 76/100
Perf: est. 75 tok/s, typical error 15% · first token ~0.2s
Fits in 16 GB VRAM with room to spare. Best for agentic coding, small machines on Tesla T4.
Best for: Chat, Coding, Multimodal·Pop: 80/100
Perf: est. 26 tok/s, typical error 15% · first token ~0.7s
Fits in 16 GB VRAM with room to spare. Best for chat, coding, multimodal on Tesla T4.
Best for: Voice assistants, Tool calling, Chat·Pop: 75/100
Perf: est. 51 tok/s, typical error 5% · first token ~0.5s
Fits in 16 GB VRAM with room to spare. Best for voice assistants, tool calling, chat on Tesla T4.
Best for: On-device, Mobile, Chat·Pop: 82/100
Perf: est. 40 tok/s, typical error 5% · first token ~2.7s
Fits in 16 GB VRAM with room to spare. Best for on-device, mobile, chat on Tesla T4.
Best for: Quality, Coding, Reasoning·Pop: 86/100
Perf: est. 24 tok/s, typical error 5% · first token ~3.1s
Fits in 16 GB VRAM with room to spare. Best for quality, coding, reasoning on Tesla T4.
Best for: Local AI agents, privacy-first tool calling, MCP workflows·Pop: 80/100
Perf: est. 84 tok/s, typical error 15% · first token ~0.2s
Fits in 16 GB VRAM with room to spare. Best for local ai agents, privacy-first tool calling, mcp workflows on Tesla T4.
Best for: Chat, Coding, Multimodal·Pop: 80/100
Perf: est. 16 tok/s, typical error 15% · first token ~4.4s
Fits in 16 GB VRAM with room to spare. Best for chat, coding, multimodal on Tesla T4.
Best for: Quality, Coding, Reasoning·Pop: 86/100
Perf: est. 40 tok/s, typical error 5% · first token ~0.5s
Fits in 16 GB VRAM with room to spare. Best for quality, coding, reasoning on Tesla T4.
Best for: On-device agents, tool calling, multilingual chat·Pop: 72/100
Perf: est. 110 tok/s, typical error 15% · first token ~0.1s
Fits in 16 GB VRAM with room to spare. Best for on-device agents, tool calling, multilingual chat on Tesla T4.
Best for: Agentic coding on small machines·Pop: 76/100
Perf: est. 40 tok/s, typical error 5% · first token ~0.5s
Fits in 16 GB VRAM with room to spare. Best for agentic coding on small machines on Tesla T4.
Runs · Stated: the linked source says a GPU with that much VRAM runs the model. Runs · Published: a dated run on the named card with that much VRAM. unknown: no source yet. Fits with offload: the main model weights fit in VRAM with a 15% margin; the text encoder and VAE run from system RAM, which is slower. Exceeds VRAM: even the main weights are too large for 16 GB. Every VRAM tier: video · image. Ranked with reasons: video models for 16 GB of VRAM · image models for 16 GB of VRAM.
| Model | Fits | Quality | App |
|---|---|---|---|
| HunyuanVideo-1.5Tencent Hunyuan Community License | Runs · Stated14 GB VRAM · official runner with model offloading · HunyuanVideo-1.5 model card | #38 of 48 in text to video4,292 votessnapshot 2026-09-22 | ComfyUI |
| LTX-2.3LTX-2 Community License Agreement | Runs · Stated16 GB VRAM · distilled with FP8 cast, previews only · LTX blog: hardware for AI video models | No independent score yet | Lightricks/LTX-2 ltx-pipelines, ComfyUI |
| LTX-2.5LTX-2.x Community License Agreement | Runs · Stated12 GB VRAM · LTX Desktop on Windows, weight streaming, 40 GB RAM, 540p · LTX Desktop compatibility matrix (Lightricks) | No independent score yet | Lightricks/LTX-2 ltx-pipelines, ComfyUI |
| MiniMax H3MiniMax H3 Community License Agreement | Runs · PublishedRTX 5060 · 16 GB VRAM · ComfyUI int8 convrot with int4 text encoder, 1024x768, 10 s in 24 min · ComfyUI issue #15274 | #8 of 48 in text to video11,417 votessnapshot 2026-09-22 | stable-diffusion.cpp, ComfyUI |
| Wan2.1Apache License 2.0 | Runs · Stated13 GB VRAM · T2V 14B in Diffusers, bf16 with group offloading to system RAM · Diffusers docs: Wan pipeline | No independent score yet | ComfyUI |
| Wan2.2 A14BApache License 2.0 | Runs · PublishedRTX 5060 Ti · 16 GB VRAM · ComfyUI fp16 I2V with 4-step LoRAs, 64 GB system RAM · ComfyUI issue #10662 | #42 of 48 in text to video10,458 votessnapshot 2026-09-22 | ComfyUI |
| Wan2.2 TI2V 5BApache License 2.0 | Runs · Stated8 GB VRAM · ComfyUI native offloading · ComfyUI docs: Wan2.2Complete: Comfy-Org repackaged FP16 with FP8 scaled UMT518.1 GB download | No independent score yet | ComfyUI |
| Mochi 1 previewApache License 2.0 | unknownComplete: calcuis GGUF Q3_K_M with Q4_0 T5 and FP8 VAE7.7 GB download | #48 of 48 in text to video5,893 votessnapshot 2026-09-22 | Diffusers, ComfyUI |
Quality: blind human votes from Arena (arena.ai), CC BY 4.0, snapshot 2026-09-22. Scores are for the Arena checkpoint, not your local quantized build.
| Model | Fits | Quality | App |
|---|---|---|---|
| FLUX 3 Image KleinFLUX Kommunity License v1.0 | Runs · Stated8 GB VRAM · official ComfyUI int8 files · FLUX 3 Image Klein README (Black Forest Labs) | No independent score yet | Diffusers |
| FLUX.1 [dev]FLUX.1 [dev] Non-Commercial License v1.1.1 | Runs · Stated6 GB VRAM · stable-diffusion.cpp with GGUF weights · stable-diffusion.cpp FLUX docs | #78 of 82 in text to image49,221 votessnapshot 2026-10-07 | Diffusers, ComfyUI |
| FLUX.2 [klein]Apache 2.0 (4B), FLUX Non-Commercial License (9B) | Runs · Stated13 GB VRAM · Klein 4B · FLUX.2 [klein] 4B model cardComplete: Klein 4B BF1616.0 GB download | No independent score yet | Diffusers, stable-diffusion.cpp |
| Krea 2 TurboKrea 2 Community License Agreement v.1 | Runs · PublishedRTX 4070 · 12 GB VRAM · ComfyUI fp8, dynamic VRAM paging, 8 steps · ComfyUI issue #15365 | No independent score yet | Diffusers, stable-diffusion.cpp |
| Qwen-Image-2.1Qwen Research License Agreement | Runs · PublishedRTX 5080 · 16 GB VRAM · stable-diffusion.cpp Q8_0, 1024x1024 · stable-diffusion.cpp issue #2041 | #19 of 82 in text to image11,441 votessnapshot 2026-10-07 | Diffusers, stable-diffusion.cpp |
| SDXL Base 1.0CreativeML Open RAIL++-M License | Runs · Stated8 GB VRAM · SDXL 1.0, no app named · SDXL 1.0 announcement (Stability AI) | No independent score yet | ComfyUI, stable-diffusion.cpp |
| Z-Image-TurboApache 2.0 | Runs · Stated16 GB VRAM · stated for consumer GPUs · Z-Image-Turbo model card | #59 of 82 in text to image24,808 votessnapshot 2026-10-07 | Diffusers, stable-diffusion.cpp |
| MOSS-ImageApache 2.0 | unknownComplete: UD-Q4_K_XL GGUF13.2 GB download | No independent score yet | stable-diffusion.cpp |
| Stable Diffusion 3.5 LargeStability AI Community License | Fits with offloadComplete: Q4_0 GGUF16.4 GB download | #81 of 82 in text to image23,396 votessnapshot 2026-10-07 | ComfyUI |
| FLUX.2 [dev]FLUX Non-Commercial License | Exceeds VRAMComplete: Q4_K_M GGUF34.8 GB download | #38 of 82 in text to image77,988 votessnapshot 2026-10-07 | Diffusers, ComfyUI, stable-diffusion.cpp |
Quality: blind human votes from Arena (arena.ai), CC BY 4.0, snapshot 2026-10-07. Scores are for the Arena checkpoint, not your local quantized build.
Speeds are estimates for Qwen3 8B Q4_K_M, typical error 5%.
The Tesla T4 has 16GB of GDDR6 at 300 GB/s (NVIDIA datasheet). ModelFit budgets about 14.4GB of it for weights and context, enough for dense models up to 14B at Q4 quantization.
Yes. Ollama lists the T4 under compute capability 7.5 in its NVIDIA support table and needs driver version 550 or newer. It runs the card through CUDA with no extra setup.
Dense models up to 14B at Q4, such as Qwen3 14B (~11GB) at est. 22 tok/s, typical error 15%, plus 20B-class mixture-of-experts models. 8B models run at est. 37 tok/s, typical error 5%. Dense 27B and 32B models spill into system RAM.
The T4 has 16GB against 12GB, so it holds 14B models with longer contexts and 20B-class mixture-of-experts models the 3060 cannot. The RTX 3060 streams more memory (360 vs 300 GB/s), so it is faster on models both fit (est. 50 tok/s, typical error 5% vs est. 37 tok/s on 8B). The T4 is passively cooled (NVIDIA datasheet) and needs server airflow.
The Tesla T4's 16GB of VRAM cannot fit a 32B model comfortably. The largest size class it fits is 20B MoE (3.6B active), at est. 50 tok/s, typical error 15%. The largest individual model in the catalog that fits is LFM2 24B-A2B Instruct (24B).
Time a model that fits your machine with Ollama, using the same prompts as every other run. The CLI shows your result and asks before it sends anything; your machine then joins the speed test leaderboard. It does not change the quality benchmark.
Use our interactive wizard to compare models across Apple Silicon and NVIDIA GPUs.