llama.cpp 0.4.0 landed on September 4, 2026, and three of its changes directly affect what fits on your Mac. First, lazy tensor reading lets oversized embedding tables stream from SSD instead of sitting in RAM. That matters most for Qwen3.8-Flash-Next and its 97.7 GiB n-gram table. Second, sparse flash attention, including a Metal backend, speeds up long-context DeepSeek-V4 and GLM inference by up to 3x on an M2 Ultra. Third, Apple RDMA over Thunderbolt 5 turns the RPC backend into a real multi-Mac cluster transport (GitHub release notes, 2026).
So which of these actually changes your setup? Let's break it down tier by tier.
Does 0.4.0 change anything for your Mac?
Short answer by tier. If you run models that already fit in RAM, lazy mode is a net loss. However, if you run huge MoE models or long contexts, this release is the biggest llama.cpp update of the year for Apple Silicon.
| Your setup | Lazy tensors | Sparse flash attention | Apple RDMA |
|---|---|---|---|
| 16-24 GB, small dense or MoE models | Skip it: measured 8-11% token-generation slowdown on a small model | Only if you run DeepSeek-V4/GLM-class models at 32K+ context | Not relevant |
| 32-64 GB, large MoE attempts | --lazy-mode auto removes the n-gram table from resident RAM (est. saving: tens of GB on Qwen3.8-Flash-Next) | Same, applies to Qwen4exp too | Not relevant |
| 96-256 GB, long-context DeepSeek-V4/GLM | Useful for the embedding table | Up to ~3x prefill speedup at 64K context on M2 Ultra (measured by the PR author) | Not relevant |
| 2+ Macs with Thunderbolt 5, macOS 26.2 | No impact | No impact | Measured 2.26x decode vs TCP on a 2-node M3 Ultra test rig |
[ORIGINAL DATA] In our experience at modelfit, the tier table above mirrors exactly what our recommendation engine computes: when we calculate the model ladder for each Mac, the RAM ceiling — not the chip — is what decides whether a release like 0.4.0 matters to you.
Now the details, feature by feature.
What is lazy tensor reading and when should you enable it?
Lazy tensor reading is a new llama.cpp mode (--lazy-mode on|auto|off, PR #27794) that stops the engine from loading the whole embedding table into RAM at startup. mmap (memory mapping) is the underlying mechanism: rows are read on demand from the SSD instead of being copied into memory up front. As a result, models with PLE and n-gram embedding tables — meaning Qwen3.8-Flash-Next and the Gemma-4 E-series — never need the full table resident. In auto mode, only tensors larger than 4 GiB go lazy, so small models keep the old eager behavior (PR #27794, 2026).
The target is obvious once you look at the model. Qwen3.8-Flash-Next is a 125B MoE with 6B active parameters. Its n-gram embedding table alone is 51B parameters: 97.7 GiB unquantized, per byteiota's release analysis (2026). Without lazy mode, that table sits in RAM for the entire session. With it, only the rows a token touches are paged in. For context, our Qwen3.8-Flash-Next open-weights breakdown covers the architecture behind that table.
The trade-off, however, is real and measured. On a Gemma-4 E4B Q4_K_M, the PR author's test cut peak RSS from 7.37 GB to 6.16 GB. The cost: 8 to 10.7% of token-generation speed. Why? Because for a small model, disk read latency is significant relative to per-token compute. For large models like Qwen3.8-Flash-Next, by contrast, the effect is minor, per the same PR. So:
./llama-server --model qwen3.8-flash-next-q4_k_m.gguf --lazy-mode autoStart with auto. It is the conservative setting that still removes the giant table from RAM. One honest caveat: llama.cpp's optimization of the qwen4exp architecture is still in progress per the release notes, so expect speed to improve in later builds.
And what if your goal is running the 125B on a 48 GB Mac today? In that case, the MLX-based SlotStream approach we covered last week is the measured path. Its author sustains 17.3 GB/s of expert-sized SSD reads, which shows Apple SSDs have the bandwidth headroom this lazy mode relies on.
Does sparse flash attention work on Mac, or is it NVIDIA-only?
It works on Mac. The CUDA implementation landed first (PR #27970), and a Metal implementation followed in the same release cycle (PR #28098, 2026). Sparse flash attention is an attention mode where DeepSeek-V4, GLM, and Qwen4exp models attend to only the most relevant KV entries per token instead of the full context. KV cache is the per-token memory that stores keys and values from previous tokens so the model doesn't recompute them. On the implementation side, both backends go through the new ggml_flash_attn_ext_set_n_kv_max API in ggml 0.23.0.
The Metal numbers, measured by the PR author on an M2 Ultra with DeepSeek-V4, scale with context length:
| Context | Prefill before | Prefill with sparse FA | Speedup |
|---|---|---|---|
| 2,048 | 401.6 t/s | 401.0 t/s | 1.0x |
| 8,192 | 323.6 t/s | 373.5 t/s | 1.15x |
| 32,768 | 170.4 t/s | 347.9 t/s | 2.04x |
| 65,536 | 107.1 t/s | 323.5 t/s | 3.02x |
Notice the pattern: the longer the context, the bigger the win. Token generation, meanwhile, stays roughly flat (27.5 vs 27.2 t/s at 2K context, 20.4 vs 23.8 t/s at 65K). On CUDA with a DGX Spark and a 2-bit DeepSeek-V4 quant, the same pattern extends further: prefill at 1M context goes from 33.99 to 74.90 t/s, a 2.2x speedup (PR #27970, 2026).
That said, one claim needs correcting. Summaries of this release say sparse attention cuts the KV cache by 4x. We could not verify that figure in the release notes, the ggml 0.23.0 changelog, or either PR. What the primary sources show instead is an attention-compute speedup that grows with context length. If a KV-memory reduction lands with it, it is not documented in the 0.4.0 materials. For what DeepSeek-V4 actually needs in memory, see our DeepSeek V4 VRAM requirements guide.
What does Apple RDMA over Thunderbolt 5 change for Mac clusters?
RDMA (Remote Direct Memory Access) is a networking technique where one machine reads and writes another machine's memory directly, without going through the OS network stack. In practice, it makes llama.cpp's RPC backend viable across Macs without the TCP penalty. PR #26421 adds Apple's RDMA implementation as an RPC transport, built on Apple's tech note TN3205.
The requirements, however, are strict. You need macOS 26.2 or later, Apple silicon, and Thunderbolt 5 ports. In addition, RDMA must first be enabled from macOS Recovery with rdma_ctl enable (Apple TN3205, 2026). For reference, Thunderbolt 5 links run at 80 Gb/s bidirectional, double Thunderbolt 4 (AppleInsider, 2025).
The PR author's measurements on M3 Ultra nodes, layer-parallel with a small Qwen3-0.6B test model:
| Nodes | Decode t/s over TCP | Decode t/s over Apple RDMA | Gain |
|---|---|---|---|
| 2 | 115.6 | 261.3 | 2.26x |
| 3 | 101.1 | 164.2 | 1.62x |
| 4 | 87.7 | 133.5 | 1.52x |
Prefill shows the same pattern (5,296 to 7,227 t/s on 2 nodes, 1.36x). Importantly, the author is explicit that these gains are inflated because the test model is tiny and RPC calls dominate per-token time. Therefore, expect smaller but real gains on large models. If you are building a two-Mac rig, this is the first release where the transport is not the obvious bottleneck. Before going multi-node, our best LLMs for Mac Studio M5 Ultra 512GB guide covers the single-machine ceiling, and the M6 Mac mini preview is the budget entry point.
What else is in 0.4.0?
Three smaller items worth knowing:
- Per-slot context limits on the server (PR #24124): each slot gets its own context budget instead of an even split. For example, a multi-user API server can now cap context per client without spawning separate processes.
- Video input through the mtmd multimodal system: llama.cpp runs FFmpeg as a subprocess rather than bundling codecs. Consequently, FFmpeg must be installed or video input silently does nothing (byteiota, 2026).
- ggml 0.23.0 underneath: async and event RPC backend APIs plus allocation-dependency tracking in graph optimization (ggml release notes, 2026).
Finally, new model support rounds out the release: initial Qwen3.8-Flash-Next (qwen4exp) and NVIDIA Nemotron-3-Puzzle-75B-A9B, plus nanbeige4.2-3B and DSpark support for Nemotron 3.5.
Should you upgrade now?
Yes if you build llama.cpp yourself, with one warning: the session and state format versions were bumped for KV-cell token tracking, so saved states from older builds will not load cleanly. To upgrade, rebuild from source with git pull && cmake -B build && cmake --build build --config Release. On the other hand, if you run through Ollama or LM Studio, you wait until those tools pull in 0.4.0 before --lazy-mode and per-slot limits reach you (byteiota, 2026).
Still unsure whether your Mac fits a given model? That is exactly the question our modelfit checker answers: pick your machine, and we calculate the model ladder for you. [ORIGINAL DATA] Every fit verdict in that ladder comes from the same calculation method we used to sanity-check the RAM claims in this article.
FAQ
Does llama.cpp 0.4.0 let a 48GB Mac run Qwen3.8-Flash-Next?
Not comfortably by itself. Lazy mode removes the 97.7 GiB unquantized n-gram table from resident RAM. However, the qwen4exp architecture is only initially supported, and optimization is still pending per the release notes. Today, the measured 48 GB path is SlotStream on MLX at about 12 tok/s warm. The GGUF path in llama.cpp should catch up as qwen4exp matures.
Is --lazy-mode slower for small models?
Yes, measurably. In the PR author's Gemma-4 E4B test, lazy loading cut peak RSS from 7.37 GB to 6.16 GB but reduced token generation by 8 to 10.7%. The auto setting exists for exactly this reason: it only applies lazy reading to tensors above 4 GiB, so small models keep eager loading.
Does sparse flash attention cut the KV cache by 4x?
Not verified. That figure circulates in release summaries, but neither the llama.cpp 0.4.0 notes nor the ggml 0.23.0 changelog nor the implementation PRs document it. What is measured, by contrast, is prefill speedups that grow with context: up to 3.02x at 65K context on an M2 Ultra over Metal and 2.2x at 1M context on a DGX Spark over CUDA. Token generation is roughly unchanged at short context.
What do I need for a Thunderbolt RDMA Mac cluster?
Per Apple TN3205: Macs with Apple silicon and Thunderbolt 5, running macOS 26.2 or later, with RDMA enabled from macOS Recovery via rdma_ctl enable. Then run ggml-rpc-server on the workers and point GGML_RDMA_DEV at the correct rdma_enX interface. In short, llama.cpp 0.4.0 is the first release with the Apple RDMA transport built in.
Will Ollama and LM Studio get these features?
Eventually, not today. Both bundle their own llama.cpp builds, so lazy mode, per-slot limits, and sparse attention arrive only when they pull in 0.4.0. Direct llama.cpp users who build from source, meanwhile, can use everything now.
Sources
- llama.cpp v0.4.0 release notes (GitHub, 2026)
- ggml v0.23.0 release notes (GitHub, 2026)
- PR #27794: lazy tensor reading (GitHub, 2026)
- PR #27970: sparse flash attention for DeepSeek-V4/GLM (GitHub, 2026)
- PR #28098: Metal sparse flash attention (GitHub, 2026)
- PR #26421: Apple RDMA RPC transport (GitHub, 2026)
- Apple TN3205: RDMA over Thunderbolt (Apple, 2026)
- llama.cpp 0.4.0: Lazy Tensors, Video, and Server Fixes (byteiota, 2026)
- llama.cpp 0.4.0 overview (Freedom.Tech, 2026)
Match this model to a machine that can run it: by RAM tier for Apple Silicon, or by VRAM for an NVIDIA GPU.
The weekly local-AI refresh
New open-weight models, real Apple Silicon benchmarks, and the one model worth running on your Mac this week. Free, one email a week, unsubscribe anytime.
By subscribing you agree to our Privacy Policy and to receive the weekly email. Unsubscribe anytime.
Have questions? Reach out on X/Twitter