Swiftlet: 80B MoE in 4.3 GB of RAM on a Mac

An 80-billion-parameter model running on a MacBook with 24 GB of unified memory, at a peak of 4.3 GB of RAM, is not supposed to be possible. The rule everyone repeats is that a model must fit in memory or it does not run. Swiftlet breaks that rule by keeping only the dense core resident and reading the routed experts from your SSD on every token. The project is Apache-2.0 and was created on August 3, 2026. As of September 26 it sits at 647 stars and 37 forks (GitHub). Meanwhile, its Show HN hit 312 points and 141 comments on the day it launched. Below, we work through what actually fits on which Mac, what it costs in disk, and what 4.5 tokens per second feels like in practice.

What Swiftlet actually claims

Swiftlet is a Swift and Metal runtime for the Qwen3-Next and Qwen3.5/3.6 hybrid MoE families. In addition, the author publishes three container figures, all measured on an M5 Mac:

Those numbers come from the repository README, and they are consistent with the model cards on Hugging Face for both the 80B container and the 35B container. However, every decode speed in this article is the author's own measurement on one machine. No independent reproduction has been published, and we have not run Swiftlet ourselves.

One discrepancy is worth flagging. A digest of this launch circulated with a decode figure of 4.5 to 5.3 tok/s for the 80B. The published table says 4.5 to 5 tok/s, and that is the number we use below.

Why an 80B model fits in 4.3 GB

The mechanism is not compression. It is routing plus storage layout.

A mixture-of-experts (MoE) model is a network whose layers hold many parallel expert sub-networks, of which only a few fire for each token. Qwen3-Next-80B-A3B activates roughly 3B parameters per token. For example, each layer routes a token to 10 of 512 experts in the 80B, or 8 of 256 experts in the 35B. Therefore the overwhelming majority of the weights are never needed for a given token. Better still, the ones that are needed are known before the read happens.

Next, Swiftlet splits the model into two parts:

• The dense core stays resident in memory: attention, DeltaNet projections, routers, the shared expert, embeddings. That is about 1.3 GB at 4-bit for the 35B, and 2.5 GB for the 80B.
• The tens of thousands of routed experts go into a .qpack container as fixed-stride blobs. Consequently, fetching one expert is exactly one pread from SSD. No mmap, no page-cache thrash.
• Hot experts are held in a bounded pool with LFU plus recency eviction.

Fixed stride is the load-bearing detail. When every expert occupies the same number of bytes at a known offset, one expert fetch becomes one positioned read instead of a scatter of page faults. That is what separates .qpack from mmap, where the kernel decides what stays cached. There, reads arrive in 16 KB pages that rarely line up with an expert boundary.

A third structural fact matters here. About 75 percent of the layers in these models use Gated DeltaNet linear attention with a fixed-size recurrent state. As a result, those layers carry no growing KV cache at any context length. Why does that matter? A KV cache that grows with context would eat into the RAM budget that the streaming design is trying to protect.

What 4.5 tokens per second actually feels like

First, some context for the number: average reading speed is roughly 4 to 5 words per second, and a token is about 0.75 of a word. In other words, 4.5 tok/s is slower than you read aloud, and much slower than you read silently.

In practice:

• A 200-word answer takes around a minute of generation.
• Code feels worse than prose, because you are waiting on exact syntax rather than skimming.
• It is genuinely usable for asynchronous work: draft, summarize, rewrite, translate, and leave it running while you do something else.
• It is not usable for interactive agent loops that make a dozen dependent calls per task.

For comparison, the same runtime decodes the 4-bit 35B at about 19.5 tok/s on an M4 Max with a 40-core GPU and 64 GB. That is the same container at roughly four times the speed, purely on GPU cores.

The honest trade: prefill, cache, and SSD

Read this section slowly, because the decode speed is only half the story.

Prefill is the weak spot. Prefill is the phase where the model processes your whole prompt before it emits the first token. The author states the problem directly in the Hacker News thread: prefill currently goes through the same per-token path as decode, which he calls dumb for long prompts. His planned fix is to batch expert reads for the whole prompt per layer instead of per token. Until that lands, he says, the runtime is good for chat-length input rather than for feeding it a 10,000-token document. In the meantime, the README lists batched prefill as the top item on the roadmap.

How bad is long-prompt prefill today? One commenter in that thread estimated half an hour to process 10,000 tokens on an M5. That figure is a commenter's estimate, not a published benchmark, and we have not reproduced it. Treat it as an order of magnitude, not a datasheet. In addition, the README notes that prefill runs at roughly decode speed on a base M1 with 8 GPU cores and 16 GB. That makes a long system prompt expensive on that class of machine.

Counter-intuitively, the cache barely moves throughput. In the author's sweep on the M5, a 1 GB cache hit 43 percent of the time and a 6 GB cache hit 70 percent, at nearly the same tokens per second. The bottleneck is GPU dispatch rather than SSD throughput. On the M1 16 GB datapoint, --cache-gb 2 matched --cache-gb 8 on a short prompt while saving several GB of RAM. The one place a bigger cache helps is long context: on a 503-token prompt, going from 2 GB to 6 GB lifted decode by about 8 percent by trimming the CPU gap between command buffers.

So the 4.3 GB peak RAM figure is achievable. However, which cache size produced it is not published in the README or the model cards. That specific pairing is not verified.

SSD endurance is the unresolved question. Expert streaming reads hundreds of blobs per token, continuously, for as long as generation runs. As it stands, Apple does not document endurance for this workload, and the HN thread did not converge on an answer. The common reading is that NAND reads are far less destructive than writes, and that drives are rated by TBW, which is the total bytes written a vendor guarantees inside a warranty window. The counter-argument raised in the thread is read disturb, where enough reads on a block force the controller to rewrite it and its neighbours. Notably, nobody in that thread had measured it on a Mac. If you run this daily, it is an open risk, not a solved one.

Mac compatibility by RAM tier

In short, the floor here is disk, not RAM. You need Apple Silicon, macOS 14 or newer, and free SSD space for the container.

In fact, that last row is the clearest demonstration of why the design matters. The 8-bit 80B has 78.8 GiB of weights, which cannot fit in a 64 GB machine under any conventional scheme, yet Swiftlet serves it in 6.9 GiB. The same measurements, credited to @Avicennasis in the repo, put the 4-bit 397B at 207.6 GiB of weights served in 12.6 GiB at about 1.4 tok/s.

[ORIGINAL DATA] We pushed every published container size in this article through the same memory model that powers our Mac compatibility recommendations. That model holds the dense core resident (1.3 GB for the 35B at 4-bit, 2.5 GB for the 80B) and treats the routed experts as a streamed pool. In our experience, the 16 GB question is almost never about the resident footprint. It is about what happens during a long prefill, when prompt processing and generation compete for the same GPU.

If you want the conventional physics on the same hardware, modelfit has already published the honest mmap math for a 35B on a 16 GB Mac Mini and the 744B-in-25GB arithmetic behind Colibri. Swiftlet is the same family of trick with a stricter layout discipline. If your budget stops around 48 GB, see why 64 GB is the local-LLM dead zone, and if you are still sizing a machine, how much unified memory an LLM actually needs is the prerequisite read.

How it compares to mmap-based llama.cpp streaming

This is the comparison modelfit cares about most, and the honest answer is that nobody has published it.

Historically, llama.cpp has used mmap by default for years. mmap is the Unix memory-mapping call that lets the kernel page file contents in on demand. That path has real published numbers on the same class of hardware: we measured 17.3 tok/s with zero swap for the 35B MoE on a base Mac Mini M4 16 GB, in our Mac Mini M4 16 GB article. That is substantially faster than the author's 2.45 tok/s for the same 35B class of model on a base M1. However, the comparison is not apples to apples. The chip, the quantization, the runtime, and the prompt length all differ.

What does Swiftlet change technically? The read pattern. mmap hands placement to the kernel page cache, so a miss costs a page fault and the fault granularity does not match expert granularity. By contrast, Swiftlet reads at fixed expert stride into a pool it controls. It also reports cache hit rates that mmap cannot report at all. Whether that layout advantage beats a mature mmap implementation on the same machine is exactly the experiment nobody has run. If you have an M-series Mac and both runtimes, that is the test worth publishing, and we would link it from here the day it exists.

The iPhone claim, and what it does not cover

The 35B runs on an iPhone 17 in about 2.5 GB of RAM at about 1 tok/s, and it ships inside the Priv AI app on the App Store, with the app open source on GitHub. For context, the author credits ANEMLL for demonstrating a 397B MoE streaming on an iPhone 17 Pro as a proof of concept in early 2026.

Meanwhile, two clarifications are worth making, because the HN title folds both claims together:

• The phone container is the 35B one. The 80B model card states plainly that iPhones do not have the storage for it, and points at the 35B container as the phone model. In other words, the 80B-in-4.3-GB headline is a Mac claim, and the phone claim is a separate 35B container.
• The iPhone support targets Apple Silicon iOS 17 or newer, and the 1 tok/s figure is the author's, not a benchmark suite.

Which checkpoints the containers were repacked from

The 35B container is repacked from mlx-community/Qwen3.6-35B-A3B-4bit, with weights byte-identical and no re-quantization, which the model card states explicitly. The 80B card describes the source as Qwen3-Next-80B-A3B-Instruct at MLX 4-bit with group size 64, but it does not name the upstream MLX repository. The repacker can also consume raw MLX checkpoints directly with --from-hf mlx-community/..., and the repo ships a verify_container.py script to check a finished download against published hashes.

Finally, one practical caveat comes from the issue tracker. A closed issue documents a case where a mixed-precision checkpoint (8-bit dense, 4-bit experts) decoded garbage silently, because the repacker recorded the checkpoint default quantization for the expert blobs. The fix is in, but it is a reminder that the container is a repack. In short, repacking has failure modes that a plain weight file does not.

FAQ

Can an 80B model really run on a 24 GB Mac?

Yes, according to the author, if you have 42 GB of free SSD space. Swiftlet keeps about 2.5 GB of dense weights resident and streams the routed experts from disk, peaking at 4.3 GB of RAM. That figure is the author's measurement on an M5 and has not been independently reproduced.

Is 4.5 tokens per second usable?

In short, it is slower than reading aloud. It works for asynchronous drafting, summarizing, and rewriting, and it does not work for interactive agent loops with many dependent calls. For comparison, the equivalent 35B container runs at 7-11 tok/s on the same M5, which is a much more comfortable pace.

What does expert streaming cost in disk wear?

Nobody has measured it on a Mac. Reads are far less destructive than writes on NAND, but every token triggers hundreds of expert reads, and read disturb is a real mechanism. Apple does not publish endurance figures for this workload, so treat it as an open risk.

Does Swiftlet beat llama.cpp mmap on the same machine?

Not established. That said, llama.cpp mmap already runs a 35B MoE at 17.3 tok/s on a base Mac Mini M4 16 GB, per the test we ran for our own Mac Mini article. No head-to-head against Swiftlet on identical hardware, quantization, and prompt has been published. Swiftlet's claimed edge is the fixed-stride read pattern and the controllable cache, not raw tokens per second.

What hardware do I need?

Apple Silicon, macOS 14 or newer, and free SSD space: 18 GB for the 4-bit 35B, 34 GB for the 8-bit 35B, 42 GB for the 4-bit 80B. The runtime also targets iOS 17 or newer for the 35B.

Sources and verification status

Verified on September 26, 2026:

• Repository metadata: leonickson1/Swiftlet, created August 3, 2026, Apache-2.0, Swift, 647 stars and 37 forks, last push September 14, 2026.
• Show HN discussion, August 3, 2026: 312 points, 141 comments, including the author's own statements on prefill.
• Model cards: Qwen3-Next-80B-A3B-qpack and Qwen3.6-35B-A3B-qpack.

Not verified, and labelled as such above: time to first token on a short prompt (no figure is published by the author), the 4.5 to 5.3 tok/s variant that appeared in third-party coverage, the commenter's half-hour estimate for 10,000 prefill tokens, the specific cache size behind the 4.3 GB peak, the upstream MLX repository for the 80B container, and SSD endurance under sustained expert streaming.

If you are choosing a machine rather than a runtime, start with the best LLM for a 16 GB Mac or the 125B-on-48GB SlotStream breakdown, which covers the same streaming-from-SSD trade with published throughput numbers.