Run Concurrent LLM Requests on Your Mac: vllm-metal 0.30.0

Every local LLM setup on a Mac has been a single-user machine. One prompt, one agent, one stream of tokens. vllm-metal is the community plugin that puts vLLM's scheduler, paged KV cache and OpenAI-compatible server on Apple Silicon, with MLX and Metal doing the execution. Version 0.30.0 shipped as a stable release on September 23, 2026, and the interesting part is not the version number. It is that a Mac can now queue, batch and admit several requests at once. That turns local inference into a serving problem you can actually measure, and it is the same problem our recommendation engine solves every time it computes a memory budget for a specific machine.

Does it run on your Mac?

Yes, if you are on Apple Silicon. However, the requirements are short and strict.

• Apple Silicon, any M-series chip.
• macOS 15 or newer.
• Native arm64 Python 3.12. Homebrew installs it for you.
• No C++ or Metal compiler needed. The installer downloads prebuilt wheels.

The documented stable path is Homebrew, not the curl script:

brew tap vllm-project/vllm-metal https://github.com/vllm-project/vllm-metal
brew install vllm-project/vllm-metal/vllm-metal

Then run the server directly, with no environment to activate:

vllm serve Qwen/Qwen3.5-0.8B --gpu-memory-utilization 0.5

The curl installer, by contrast, is still the channel for the latest development build. It creates a virtual environment at ~/.venv-vllm-metal that you must activate in every new shell. Adding --stable to it installs a stable release instead. Both paths pull the same wheels, so Homebrew is not a reduced version. Notably, pip install vllm-metal is explicitly not supported, so you cannot skip the installer and hope for the best. Which path should you take? Pick Homebrew if you want the release described below, and pick the curl script with --stable if you want to track unreleased work. If you are new to running models locally, the Ollama install guide for Mac covers the simpler single-user path first.

What version 0.30.0 actually shipped

The release tracks the vLLM 0.30.0 core and contains 39 merged pull requests from 13 contributors since v0.29.0 (release notes, September 23, 2026). The repository had 1,779 stars on September 26, 2026, according to the GitHub API. So this is a young but fast-moving community plugin, not an Apple product.

The headline change for anyone serving on a Mac is unified KV storage. The KV cache is the memory that holds the attention keys and values of every token already processed, so its layout decides how many requests fit at once. Since 0.30.0, vLLM owns the allocation and physical layout of every scheduler-managed KV and state cache, and Metal reads that storage through zero-copy MLX views. Before this, hybrid models kept separate state pools. Now they use the full cache budget, which matters because the hybrid architectures (Qwen3.5 through 3.8, LFM2, Granite 4.0) are the ones that actually fit on a Mac.

Meanwhile, the rest of the release is a long list of small wins:

• Granite 4.0 hybrid checkpoints, dense and MoE, serve on paged state.
• min_p sampling runs natively in MLX. Batches that mix greedy and random requests stay on the MLX sampler.
• A zero-copy MLX to PyTorch bridge over DLPack, so tensors cross without copying.
• The legacy non-paged serving path is gone. Serving always uses paged attention now.
• Quantized KV cache dtypes such as fp8, int8_per_token_head and nvfp4 are rejected at startup, because Metal stores the paged KV cache in the model dtype.

That last bullet is the kind of detail that decides whether your config works. If you were planning to shrink KV memory with an fp8 cache, you cannot. The supported alternatives are auto or TurboQuant, which compresses KV by 2.56x at its default settings (TurboQuant docs, 2026).

One caveat worth knowing before you read the docs: they lag the release. The configuration page still documents VLLM_METAL_MEMORY_FRACTION and the paged-attention switch, both of which 0.30.0 removed. Therefore, trust the release notes over the docs when the two disagree. In our experience, that single rule saves an afternoon of debugging flags that no longer exist.

Why concurrent serving is the real change

Single-request inference is a model execution problem. Several overlapping requests is a serving problem: time to first token, memory growth and admission control take over. Continuous batching is the scheduler behaviour that admits new requests into a batch while the current one is still decoding, and it is why a queue of prompts behaves nothing like the same prompts run one after another. That is the gap vllm-metal closes, and the project's own comparison table shows how differently Mac engines handle it.

So the honest framing is not that other Mac engines serve one user only. llama.cpp batches decode across four server slots, oMLX batches, and Splash batches up to four. The difference is what they pair it with. Every other engine prefills each prompt in its own call, and most keep KV in contiguous buffers or fixed token cells. Paged KV storage is a layout where the cache is split into fixed-size blocks, so a request reserves only the blocks it needs instead of one contiguous buffer sized for the worst case. vllm-metal packs prompt and decode tokens from every request into one model forward, with request boundaries marked by cu_seqlens, and reads KV from that paged storage, addressed by per-request block tables.

That combination is why ragged batches stop hurting. In the project's own padding comparison on Qwen3.6-35B-A3B at 4-bit, with eight requests and 20 output tokens each, a batch of eight similar prompts cost vllm-metal 3.87 seconds of wall time. A batch with one long prompt and seven short ones cost 3.64 seconds, slightly less. The same ragged batch cost mlx_lm 10.99 seconds, a 143% penalty, and llama.cpp 5.33 seconds (vllm-metal announcement, September 22, 2026).

Why does that matter for your workload? Because a real agent session is ragged by nature: one long document context, then seven short follow-ups from a chat UI. If ragged batches cost your engine the kind of 143% penalty mlx_lm pays, the mix of prompts is what your users wait on.

What a Mac sustains at concurrency 8

The project published cross-machine numbers at concurrency 1 and 8 on four Macs. The workload was Gemma 4 E4B at 4-bit, with 100 prompts of about 1,024 input and 128 output tokens, a fresh server per level and a memory budget of 0.5. These are the v0.29.0 numbers, not v0.30.0, and they are the project's measurements rather than ours. One definition matters before you read the table. Time to first token (TTFT) is the delay between submitting a prompt and seeing the first token of the reply. That is the figure your users feel, while throughput is the one your scheduler brags about.

Read the throughput column carefully, because that is where the memory tiers separate. Going from one request to eight multiplies output throughput by 2.58 on the M5 Pro 64 GB, by 1.63 on the M1 Max and by 1.50 on the M2 Max. On the M1 Pro 32 GB it multiplies by 1.26. [ORIGINAL DATA] Those four multipliers are our arithmetic on the project's table, not a published figure. Still, the 32 GB machine does hold eight concurrent requests without falling over, but it barely gains aggregate speed from doing so.

As a result, first-token latency degrades by roughly 4.5 to 4.9 times at concurrency 8 on every machine in that table, and that is the number your users feel. If you are serving an interactive agent, concurrency 8 on any of these Macs means waiting seconds, not milliseconds, for the first token.

Two things are missing from the published record. There is no concurrency sweep on a 128 GB or 512 GB Mac, and there is no published measurement of the point where memory pressure starts killing throughput. So if you want to know what your 128 GB Studio sustains, you will have to run vllm bench serve yourself.

The memory budget is now one number you set

vllm-metal runs a warmup pass at startup, accounts for weights, activations and temporary buffers, and hands the remainder to a fixed KV pool. Requests that do not fit wait until pages free up. Unified memory is the single pool that the CPU, GPU and Neural Engine share on Apple Silicon, which is why a Mac has no separate VRAM to spill into. Since 0.30.0 the knob is the same one upstream vLLM uses:

vllm serve mlx-community/Qwen3.8-27B-4bit --gpu-memory-utilization 0.7

That fraction applies to total unified memory, and it is a hard ceiling for the whole engine, not just KV. Here is what it resolves to per RAM tier.

The official examples make the arithmetic concrete. The Qwen3.8-27B hybrid at 4-bit has 16.08 GB of checkpoint files on Hugging Face, and the project recommends serving it on a 64 GB Mac at 0.7. That leaves about 28.7 GB of the 44.8 GB budget for KV pages and activations. The same model at 0.5 on a 32 GB Mac gets a 16 GB budget, which is smaller than the weights alone, so it does not fit at all. For the small end, Gemma 4 E4B at 4-bit totals 5.18 GB and fits comfortably in a 16 GB budget.

[ORIGINAL DATA] The per-tier table above is our arithmetic: we multiply the machine's unified memory by the fraction you set, and our recommendation engine applies that same rule whenever it calculates whether a model fits a given Mac. In our experience, that calculation is the step that decides the argument, because the fraction you choose moves the ceiling more than any flag you pass after it.

In practice, this is the trade-off to internalize. Raising --gpu-memory-utilization buys context and concurrency, and it takes headroom away from macOS and your apps on the same physical memory. There is no separate VRAM pool to spill into. That is the same constraint the bottleneck breakdown for local AI hardware describes, and it is why the 27B Qwen3.8 guide lands on 64 GB as the realistic floor.

Where a Mac still loses to a GPU box

The scheduler is now real. The hardware underneath it is still a laptop.

• No tensor parallelism. --tensor-parallel-size above 1 is not implemented, so you cannot split one layer's weights across two Macs.
• No quantized KV cache. Metal stores KV in the model dtype, so fp8 and nvfp4 caches are rejected at startup.
• One shared memory pool. Weights, KV, activations and macOS compete for the same silicon. A discrete GPU has its own memory that the desktop cannot touch.
• Memory bandwidth caps concurrency gains. The 32 GB M1 Pro gained 26% throughput at concurrency 8, against 158% on the M5 Pro.
• No published concurrency ceiling. Nobody has measured where a 128 GB Mac saturates, or how many concurrent sessions a 64 GB Mac holds before admission control starts queueing.

Where a Mac wins is the part a GPU box cannot copy: one unified pool with no PCIe transfer, silent operation, and a machine that already has your files on it. For the comparison against the older local stacks, MLX versus Ollama on Mac in 2026 covers what each runtime is actually good at.

Multi-Mac serving is groundwork, not a product

vllm-metal can split a model across two Macs with pipeline parallelism, sending the cross-stage activations over the MLX ring on a Thunderbolt cable. Pipeline parallelism is a split where each Mac holds a contiguous set of layers and passes activations to the next stage, while data parallelism is a scale-out where every Mac runs a full replica behind one endpoint. The pipeline path is validated end to end, and the limits are worth reading before you buy a second machine.

• Pipeline parallelism is numerically validated as bit-exact on a single node, on Qwen3-0.6B, 1.7B and 4B, with a maximum absolute difference of zero.
• The only published end-to-end run across two Macs uses Qwen3-0.6B over Thunderbolt. A 0.6B model is a plumbing test, not a workload.
• There is no published throughput or latency measurement for the multi-Mac path at all. No numbers, no chart.
• Co-located stages double-claim the memory budget, because each stage applies --gpu-memory-utilization to the whole device without knowing the other exists.
• Hybrid, YOCO, MLA, pooling and vision models are rejected under pipeline parallelism, which excludes most of the models you would want to split.

Data parallelism is the more practical multi-Mac option: one full replica per Mac behind a single endpoint, validated at two Macs for dense models. It is a pure throughput scale-out and stays below the ideal of doubling, because the head Mac also runs the API server, the coordinator and the load balancer. It cannot serve a model larger than one Mac.

So the realistic multi-Mac answer today is: it works, it is new, and you should measure it on your own models before relying on it. If you want to know what a large unified-memory Mac does on its own, the Mac Studio M5 Ultra 512 GB breakdown is the reference point for the other end of the scale. Meanwhile, SlotStream on a 48 GB Mac shows the alternative strategy of streaming experts from SSD instead of splitting across machines.

How much of vLLM really works on Metal

Text serving is the solid part. The server, the V1 scheduler, chunked prefill, paged KV, prefix caching, LoRA adapters, structured outputs and speculative decoding all work for supported text models. The supported-model matrix lists 30-plus families, including Qwen3, Qwen3.5 through 3.8, Gemma 3 and 4, Llama 3, Mistral, GPT-OSS, OLMo and LFM2.

By contrast, everything else is marked experimental in the project's own docs, and the marker is not decorative:

• Vision-language models are limited to image input on two families, Qwen3-VL and PaddleOCR-VL.
• Text embeddings and reranking cover four families and require special handling for the reranker.
• Speech-to-text received fixes in this release, but non-greedy transcription is rejected at admission.
• GLM-4.5 is listed as not verified, because its MLA path has no Metal kernel and falls back to the slow MLX path.
• --watermark-config requires Model Runner V2 and is not supported on Metal.
• GGUF is narrower than the word suggests. Dense qwen2, qwen3, llama and mistral checkpoints load with per-tensor Q8_0, Q4_0 or Q4_1. K-quants, fused-QKV, MoE, hybrid and sharded remote files are rejected with an explicit error.

That GGUF boundary is the one most likely to bite you, because the format is the default in most Mac tutorials. GGUF is llama.cpp's single-file format for quantized checkpoints, and it is what most Mac users already have on disk. If your model is a K-quant GGUF, vllm-metal will refuse it rather than run it slowly. The 0.30.0 release adds groundwork for repacking Q4_K and Q5_K blocks into MLX affine tensors, but the loader does not route those types yet.

FAQ

Can I run vllm-metal on an Intel Mac?

No. It requires Apple Silicon and macOS 15 or newer. There is no x86 path, and the plugin depends on MLX and Metal throughout.

How much RAM do I need for concurrent serving?

It depends on the model, not on the concurrency alone. The engine caps itself at --gpu-memory-utilization times total memory, and weights come out of that same budget. A 27B model at 4-bit has about 16 GB of checkpoint files, so the 44.8 GB budget of a 64 GB Mac at 0.7 leaves roughly 28.7 GB for KV and activations. On a 32 GB Mac at 0.5 the budget is 16 GB, which is smaller than the weights.

Does vllm-metal replace Ollama or llama.cpp on my Mac?

It replaces them for serving, not for everything. It gives you an OpenAI-compatible endpoint, continuous batching, chunked prefill and paged KV, which the other Mac engines do not combine. It also rejects model formats and quantization types that llama.cpp accepts happily. If you want one prompt at a time with zero configuration, the older tools are still simpler.

How many concurrent requests can my Mac handle?

No one has published that number for the current release. The only per-tier concurrency data is at concurrency 1 and 8, on an M1 Pro 32 GB and three 64 GB machines. The project has not published a saturation point or a memory-pressure ceiling for 128 GB and larger Macs. Therefore, run vllm bench serve on your own machine with a fresh server per concurrency level.

Can I split one model across two Macs?

Yes, with pipeline parallelism over a Thunderbolt cable, but only on the narrow set of models it supports. The only published end-to-end run is Qwen3-0.6B, and there is no published throughput measurement for the multi-Mac path. Treat it as validated groundwork.

The bottom line

The shift is real: a Mac can now serve several requests at once with a scheduler and paged KV cache behind an OpenAI-compatible API. However, the practical ceiling is the machine you already own. A 64 GB Mac at concurrency 8 delivers 51 to 154 tokens per second depending on the chip, while a 32 GB Mac gains 26% over single-stream. Set --gpu-memory-utilization deliberately, check your model format before you install, and measure your own concurrency ceiling, because the published record stops at eight.

Every number in this article is either a figure from a source we opened ourselves or our own arithmetic, labelled as such. How we work: about ModelFit. Corrections welcome: contact.