Run a 125B MoE on a 48GB Mac: SlotStream Streams Experts from SSD at ~12 tok/s

SlotStream is a native Swift/MLX engine that runs Qwen3.8-Flash-Next — a 125B-parameter mixture-of-experts model that weighs 103.8 GB at 4-bit — on ordinary Apple Silicon Macs by keeping only a bounded pool of experts in RAM and streaming the rest from the SSD. On the author's M5 Pro MacBook Pro with 48 GB it decodes at about 12 tokens per second once warm, and community reports put a 16 GB Mac mini at 1.41 tok/s. It exposes Ollama-compatible and OpenAI-compatible APIs and works offline after a one-time download.

Does it run on your Mac?

That is the only question that matters, so here is the answer first. Requirements: any Apple Silicon Mac, macOS 14 or later (model runs tested on macOS 26), and about 110 GB of free SSD space (SlotStream README, 2026).

Two honest caveats on this table. First, every speed above is the author's or a community member's measurement, not a modelfit benchmark — we have not rerun them. Second, the planner's estimates run optimistic on small machines: the 16 GB tier is estimated at ~4 tok/s but the one real 16 GB report came in at 1.41 tok/s, on a base-storage M2 mini whose slow SSD is likely part of the gap (docs/HARDWARE.md, 2026). Treat the estimate column as a ceiling, not a promise.

What SlotStream actually is

SlotStream is an open-source (MIT) inference engine written in Swift on top of MLX, built by Carlos Galarza specifically to run one model: Qwen3.8-Flash-Next in the 4-bit MLX conversion by pipenetwork (GitHub, 324 stars as of September 9, 2026). It launched as a Show HN that reached 237 points.

The engineering finding behind it is worth understanding, because it explains why you cannot just mlx_lm.load() this model. The author's measurements show that MLX cannot sparsely materialize a memory-mapped tensor: any gather over a lazily loaded array evaluates the whole source tensor. A 63-token prompt through the stock path pulls in roughly the full 104 GB checkpoint, pushes the machine into 48.8 GB of swap, and dies before emitting a token. SlotStream's answer is a slot pool: a pre-allocated, fixed-size pool of expert slots that never grows, filled by direct pread calls of exactly the 2.76 MB expert records each token needs, with an eviction policy keeping the working set bounded forever.

The model itself is 48 layers of 512 routed experts with 10 active per token. Of the 103.77 GB total (byte-measured from the safetensors headers), 67.9 GB is routed experts, 32 GB is an n-gram/PLE store, and only 3.8 GB must stay resident. That 3.8 GB floor is why a 16 GB Mac can run it at all. On the author's 2 TB Apple SSD, expert-sized reads sustain 17.3 GB/s, enough that even a zero-hit cache would theoretically sustain ~13 tok/s of decode I/O — the binding constraint on small Macs is memory and compute, not bandwidth (MEASUREMENTS.md, 2026).

Cold behavior, from the same measurements: the engine starts in 2.3 s (1.1 s when page-cached), and each request to a cold expert cache carries a fixed ~6 s refill floor before tokens flow. The ~12 tok/s figure is the warm number; expect the first reply of a session to be slower.

The download shrank to 88.3 GB

The raw 4-bit checkpoint is about 105 GB on the wire. Since v0.2.10, SlotStream downloads a lossless, quantization-aware package of 88,294,086,225 bytes — 16.12% fewer bytes — that reconstructs the exact original files with hash verification at every stage, resumable after interruption. v0.2.11 (September 6, 2026) moved hosting of that package to a public Hugging Face mirror, so the publisher no longer pays per download (CHANGELOG.md). Transfer time is roughly 2 hours at 100 Mbps or 8 hours at 25 Mbps, per the author's arithmetic. You still need ~110 GB free on disk for the reconstructed model.

SlotStream vs a dense 27B: the honest trade-off

The obvious question: why stream 104 GB from SSD when a dense 27B at 4-bit loads entirely in RAM? On the same 48 GB M5 Pro class machine, our tier guide puts a dense Qwen 27B at Q4_K_M around ~15 tok/s (modelfit estimate) with an ~18 GB footprint — see our best LLMs for the Mac mini M5 Pro 48GB guide. SlotStream's ~12 tok/s is nearly that speed, for a 125B-class MoE instead of a 27B dense.

So the trade is not speed, it is quality-per-watt versus simplicity:

• Choose the dense 27B if you want zero disk dependency, instant cold starts, and a model that survives being loaded once and left alone. It is the lower-risk daily driver.
• Choose SlotStream if you want the largest model that physically runs on your machine, accept SSD streaming and a warm-up cost, and have 110 GB of disk to spare. For coding agents and batch work where a frontier-adjacent open model matters more than the first-token wait, 12 tok/s of 125B beats 15 tok/s of 27B.

If your Mac has 16 GB, neither the measured 1.41 tok/s nor the ~4 tok/s estimate is pleasant for interactive chat. At that tier a smaller MoE that fits in RAM is the better answer — our 35B-on-a-16GB-Mac-mini mmap guide shows a 35B MoE hitting 17 tok/s with zero swap on exactly that machine. And if you are shopping for a Mac to run models like this, our M6 Mac mini local LLM preview covers what the next generation changes.

API and tooling

SlotStream serves an Ollama-compatible API (/api/chat, /api/generate, /api/tags) and an OpenAI Chat Completions endpoint with function tools, streamed calls, and reasoning history (v0.2.8), plus vision support for the model's image tower (v0.2.7). There is a documented setup guide for running it as the local backend of the Hermes agent. Install is a single curl | sh from the repo, and slotstream doctor checks your machine before you download anything.

FAQ

Can a 48GB Mac really run a 125B model?

Yes, measured. The author runs Qwen3.8-Flash-Next (103.8 GB at 4-bit) on an M5 Pro MacBook Pro 48 GB at ~12 tok/s warm, with a 33 GB memory target and about 152 of 512 experts cached per layer. The trick is that only ~3.8 GB of the model must stay resident; the expert pool is bounded and the rest streams from SSD. It works because the model is a mixture-of-experts that activates only 10 of 512 experts per token.

How much disk space does SlotStream need?

About 110 GB free. The download itself is an 88.3 GB compressed package (since v0.2.10) that reconstructs into the full ~104 GB model, verified by hashes. Interrupted downloads resume.

Is 12 tok/s fast enough for real use?

For chat and agent loops, yes — that is reading-aloud speed and roughly what a dense 27B at 4-bit achieves on the same hardware. The cost is a cold-cache penalty of around 6 seconds per fresh request before generation warms up, and slower first replies in a new session.

Does it work on 16GB or 24GB Macs?

It runs on 16 GB, but slowly: the one measured report is 1.41 tok/s on a base-storage M2 Mac mini, below the author's ~4 tok/s estimate for the tier. The 24 GB tier has no measured report yet (estimate ~8 tok/s). On 8 GB Macs it requires swap and can slow the whole machine — not recommended.

Can I use it with Ollama or OpenAI clients?

Yes. SlotStream exposes both an Ollama-compatible API and an OpenAI Chat Completions endpoint with tool calling, so existing clients, agents, and SDKs can point at it as a drop-in local backend. Note it supports exactly one model today — Qwen3.8-Flash-Next — and the author has stated the goal is memory optimization for this specific model, not a general inference engine.

---

Sources: github.com/carloslfu/slotstream (README, MEASUREMENTS.md, docs/HARDWARE.md, CHANGELOG.md, all retrieved September 9, 2026) and the Show HN thread. All tok/s figures are the author's measurements or credited community reports, not modelfit benchmarks.