How Ollama Decides GPU vs CPU
Ollama scans your system at startup and picks the fastest available backend without you having to configure anything. It checks for a supported Apple Metal, Nvidia CUDA, or AMD ROCm device first, with Vulkan as an additional path on Windows and Linux for GPUs those two do not cover. If none is found, or none is compatible with your specific card, it runs entirely on the CPU.
GPU found and model fits
The entire model loads into VRAM (or unified memory on Apple Silicon) and every layer runs on the GPU. This is the fastest path and what ollama ps reports as 100% GPU.
GPU found but model too large
Ollama offloads as many layers as fit into available GPU memory and keeps the remainder on the CPU. ollama ps reports a split such as 48%/52% CPU/GPU.
No compatible GPU
Every layer runs on the CPU, reported as 100% CPU. Inference still works, it is simply much slower than any GPU path.
Multiple GPUs
If a model fits entirely on one GPU, Ollama loads it there to avoid cross-GPU traffic. If it does not fit on any single GPU, Ollama spreads it across all available GPUs instead.
None of this requires a flag or a config file for the common case. GPU acceleration is the default behavior, not an opt-in feature. See our Ollama setup guide for the full install walkthrough.
Apple Silicon: Metal and Unified Memory
On Mac, Ollama accelerates inference on the GPU using Apple's Metal API. There is no separate "VRAM" pool to run out of the way a discrete graphics card has one. Apple Silicon uses unified memory: the CPU and GPU share the same physical RAM, so the practical ceiling for a model is simply how much of your Mac's total RAM the system leaves available.
Why this matters: a MacBook Air, MacBook Pro, or Mac Studio with more RAM can comfortably fit larger models entirely on the GPU side of that shared pool, without the CPU/GPU split that discrete-VRAM systems run into. Check exactly how much RAM a given model needs on your Mac with the RAM sizing guide or the hardware calculator.
Nvidia (CUDA) and AMD (ROCm) Support
On Windows and Linux, GPU acceleration depends on which card you have and which backend Ollama uses to talk to it.
Nvidia
Ollama supports Nvidia GPUs with compute capability 5.0 or higher, requiring driver version 550 or newer. Cards in the compute capability 5.0 through 6.2 range specifically require driver version 570 or newer. That covers everything from older Maxwell-era cards through the current RTX 50-series and datacenter cards like the H100 and H200. You can look up your card's compute capability on Nvidia's developer site to confirm support.
AMD
Ollama supports a defined list of AMD Radeon RX cards, Radeon PRO and Radeon AI PRO cards, Ryzen AI unified-memory chips (including the Ryzen AI Max+ 395), and Instinct accelerators through the ROCm library. Linux requires the ROCm v7 driver; Windows requires a ROCm v7 or HIP7-capable driver stack. Unsupported AMD GPUs can sometimes be forced to run against the closest supported target using the HSA_OVERRIDE_GFX_VERSION environment variable.
Vulkan (additional coverage)
For GPUs outside the CUDA and ROCm support lists, including many Intel GPUs, Ollama offers an additional Vulkan backend on Windows and Linux, enabled by default when installed. See our GPU hub for per-card recommendations.
What Happens When a Model Does Not Fit?
A large model does not simply fail to load when it exceeds your GPU's memory. Ollama splits it: as many layers as fit go onto the GPU, and the rest run on the CPU using system RAM. This is called partial offloading, and it happens automatically with no flag required.
On a single GPU
The model is divided between VRAM and system memory. The ollama ps PROCESSOR column reports the exact ratio, for example a model split roughly half and half will show as 48%/52% CPU/GPU.
On multiple GPUs
Ollama evaluates the required VRAM against what is available. If the model fits entirely on any single GPU, it loads there for the best performance (less data crossing the PCI bus). If it does not fit on one GPU, it is spread across every available GPU instead.
Practical takeaway: a partial CPU/GPU split still works, but it is markedly slower than a full GPU load. If your goal is peak speed, pick a model sized to fit entirely within your VRAM or unified memory rather than relying on the split. Our calculator snaps recommendations to what actually fits.
How to Check What Ollama Is Using
You do not need to guess. Ollama ships a single command that reports exactly what a loaded model is running on.
Run ollama ps
While a model is loaded, run:
ollama ps
Sample output:
NAME ID SIZE PROCESSOR UNTIL llama3:70b bcfb190ca3a7 42 GB 100% GPU 4 minutes from now
The PROCESSOR column tells you everything: 100% GPU means fully accelerated, 100% CPU means no GPU offload at all, and a mixed value like 48%/52% CPU/GPU means the model is split across both.
Newer Ollama builds also add a CONTEXT column to the same output, so you can confirm your context window is loaded alongside the processor split, not just the model weights.
Why Speed Collapses on CPU
A GPU (or Apple Silicon's unified memory system) has far more memory bandwidth than a typical CPU and system RAM. Token generation with a large language model is a memory-bandwidth-bound operation: the processor has to stream the entire set of weights through memory for every token it produces. When that bandwidth drops from GPU-class to CPU-class, so does generation speed, and it drops by a lot, not a little.
This is why a partial CPU/GPU split, or a full CPU fallback, feels dramatically slower than a model that fits entirely on the GPU, even though both are technically "working." Exact tokens-per-second figures vary by model, quantization, and hardware, so treat any number you see (including on this site) as an estimate rather than a guarantee.
Rule of thumb: pick a model whose size comfortably fits your GPU's VRAM or your Mac's unified memory. Our best LLM for MacBook guide and run AI offline guide both size picks this way rather than chasing the largest model your machine can technically load.
GPU Settings and Environment Variables
Ollama's defaults work for almost everyone. These documented settings exist for the edge cases: multi-GPU machines, memory-constrained systems, or unusual hardware.
| Setting | What it does |
|---|---|
CUDA_VISIBLE_DEVICES | Limits Ollama to a subset of Nvidia GPUs on a multi-GPU system. Set to -1 to force CPU. |
ROCR_VISIBLE_DEVICES | Same idea for AMD GPUs via ROCm. |
GGML_VK_VISIBLE_DEVICES | Selects which GPU(s) the Vulkan backend uses; set to -1 to disable Vulkan GPUs. |
OLLAMA_CONTEXT_LENGTH | Overrides the context window. Ollama's own VRAM-based defaults are under 24 GiB VRAM: 4k context, 24 to 48 GiB: 32k, 48 GiB or more: 256k. |
OLLAMA_FLASH_ATTENTION | Forces Flash Attention on (1) or off (0); it reduces memory usage as context grows and is used automatically when supported. |
OLLAMA_KV_CACHE_TYPE | Quantizes the K/V cache (f16 default, q8_0, or q4_0) to cut memory use with Flash Attention enabled. |
OLLAMA_MAX_LOADED_MODELS | Caps how many models can be loaded at once; defaults to 3 times the number of GPUs, or 3 for CPU-only inference. |
The API also accepts num_gpu and main_gpu as runtime request options alongside standard generation parameters, for cases where you need per-request control instead of a server-wide environment variable.
Fixing "Ollama Not Using GPU"
If ollama ps reports 100% CPU on a machine with a GPU you expect Ollama to use, work through these in order.
1. Confirm the card is on the supported list
Check your exact card against Ollama's Nvidia compute-capability table or its AMD ROCm card list. Unsupported cards fall back to CPU (or Vulkan, where available) by design, not by bug.
2. Update your GPU driver
Nvidia needs driver 550+ (or 570+ for older compute-capability 5.0 to 6.2 cards). AMD needs the ROCm v7 driver on both Linux and Windows. Running the latest driver resolves most discovery failures.
3. Check the server logs
On macOS: cat ~/.ollama/logs/server.log. On Linux with systemd: journalctl -u ollama --no-pager --follow --pager-end. GPU discovery failures are logged there with specific error codes.
4. Reload the driver after a suspend/resume (Linux Nvidia)
Ollama can lose track of an Nvidia GPU after a laptop sleep cycle. Reload it with sudo rmmod nvidia_uvm && sudo modprobe nvidia_uvm.
5. Docker on macOS has no GPU passthrough
Running Ollama in a Docker container gets GPU acceleration on Linux and Windows (via WSL2) with the nvidia-container-toolkit installed, but Docker Desktop on macOS has no GPU passthrough or emulation, so a containerized Ollama on Mac always runs on CPU regardless of your hardware.
Frequently Asked Questions
Does Ollama use the GPU by default?
Yes. Ollama automatically detects compatible GPU hardware and uses it by default, no configuration needed. On Apple Silicon it uses the Metal API. On systems with a supported Nvidia GPU it uses CUDA. On supported AMD GPUs it uses ROCm, with Vulkan as an additional path on Windows and Linux. If no compatible GPU is found, Ollama falls back to running on the CPU.
How do I check if Ollama is using my GPU?
Run ollama ps while a model is loaded. The PROCESSOR column shows exactly what is being used: 100% GPU means the model runs entirely on the GPU, 100% CPU means it runs entirely in system memory, and a split like 48%/52% CPU/GPU means the model is partially offloaded across both.
What happens if a model does not fit in VRAM?
Ollama splits the model between GPU and system memory, offloading as many layers as fit in VRAM and running the rest on the CPU. The ollama ps PROCESSOR column reports the exact split, for example 48%/52% CPU/GPU. On multi-GPU systems, a model that does not fit on a single GPU is spread across all available GPUs instead.
Why is Ollama running slowly on my machine?
The most common cause is a partial or full CPU fallback, which you can confirm with ollama ps. CPU-only inference is far slower than GPU inference because system RAM bandwidth is a fraction of GPU VRAM bandwidth. Picking a smaller model, or one that fully fits your available VRAM or unified memory, keeps generation on the GPU.
Which GPUs does Ollama support?
Ollama supports Nvidia GPUs with compute capability 5.0 or higher (driver 550 or newer, with cards in the 5.0 to 6.2 range requiring driver 570 or newer), a list of AMD Radeon, Radeon PRO, Ryzen AI, and Instinct GPUs and accelerators via ROCm, all Apple Silicon GPUs via Metal, and additional Windows and Linux GPUs via Vulkan. Check our GPU hub for specific card recommendations.