Run 80B Qwen on 4.3GB RAM with Expert Streaming on Mac

The day “local 80B” stopped being a fantasy

There’s a particular kind of disappointment that hits right after you learn what modern language models cost to run. You download a model, you start the server, and then the memory warning arrives like a weather alert. Even with aggressive quantization, dense 80B-style models feel like they want more RAM than most machines can afford.

So the interesting twist in Swiftlet isn’t “new quantization tricks” or “a miracle allocator.” It’s a different strategy: run a sparse model so that only a fraction of its weights do real work per token, and stream the rest from storage only when the model routes to them. Swiftlet reports running:

• Qwen3-Next-80B-A3B (4-bit) in about 4.3GB RAM on an M5 Mac, at roughly 4.5–5 tok/s.
• Qwen3.6-35B-A3B (4-bit) in about 2.6GB RAM on the same kind of setup, at roughly 7–11 tok/s.
• The 35B model running on an iPhone 17 using about 2.5GB RAM, around ~1 tok/s “today.” (github.com)

That’s the headline. The rest of this post is the mechanics behind it, explained from the ground up.

Two ideas that make “huge on small devices” plausible

Swiftlet’s approach is basically the collision of two engineering ideas:

1. Mixture of Experts (MoE): a model with many “experts,” but only a few are used per token.
2. Expert streaming: keep only what’s needed in RAM, and fetch the routed experts from disk on demand.

Let’s unpack those without assuming any background.

Mixture of Experts (MoE), in plain language

A normal (dense) transformer has one big feed-forward network per layer. Those networks have weights that are all “in play” every time you generate a token.

MoE changes that. Instead of one feed-forward block, it uses many smaller feed-forward blocks called experts. A learned router looks at the current token’s hidden state and decides which experts should process it. In Swiftlet’s setup, it routes:

• For the 80B model: each token goes to 10 experts out of 512 per MoE layer.
• For the 35B model: each token goes to 8 experts out of 256 per MoE layer. ()

This is why MoE models can have a large “total parameter count” while needing far fewer “active parameters” during a forward pass.

Why RAM is still a problem

Even if only a handful of experts are active per token, the model still has to have the expert weights available somewhere. In typical local inference, you preload all weights into RAM (or GPU memory). For a sparse model, that wastes memory on experts the router rarely uses.

Swiftlet’s move is to stop treating experts as resident memory.

Expert streaming: keep the dense core, page in experts

Swiftlet is described as a “Swift + Metal runtime” that does exactly this: keep only the small dense core resident, while streaming routed MoE weights from storage on demand. ()

Here’s the key mental model:

• The model has parts that are used on every token (dense attention components, routers, embeddings, and any “shared” components).
• The MoE experts are selected dynamically by routing.
• If only a few experts are needed per token, then it’s wasteful to keep all expert weights in RAM.

Swiftlet reports keeping the dense weights resident at roughly:

• ~1.3GB for the 35B model at 4-bit
• ~2.5GB for the 80B model at 4-bit ()

…and then fetching experts from disk when routing selects them.

“How do you fetch from disk without killing performance?”

This is where .qpack enters.

Swiftlet repacks the routed experts into a .qpack container so that fetching an expert becomes “predictable I/O.” The README describes packing experts into fixed-stride blobs, so that grabbing one expert is “exactly one pread from SSD,” avoiding the chaos of memory-mapping and OS page-cache thrash. ()

A quick definition:

• pread is a POSIX file call that reads data from a file at a given offset without changing the file’s current read position.
• mmap (memory-mapped files) maps file contents into your process address space. It can be fast, but it also makes paging behavior harder to control.

Swiftlet’s point is simple: make expert reads behave like a controlled sequence of SSD reads, not a “trust the OS” situation.

Caching: because the same experts repeat

If every routed expert were truly random, streaming would be hopeless. In practice, language generation has structure: the router’s choices across adjacent tokens aren’t perfectly uniform.

So Swiftlet uses a bounded expert cache, described as a pool with LFU plus recency eviction. ()

Definitions:

• LFU (Least Frequently Used) means evict experts that have been requested the least.
• Recency means prefer keeping experts that were requested more recently (helps when frequencies change mid-chat).

Swiftlet notes that cache size doesn’t dramatically affect speed for misses, because modern Apple SSDs absorb the penalty—yet the cache still raises hit rates (it mentions measured hit rates in a fairly wide band). ()

Running the forward pass: Metal for the tensor math

Once experts are available, the next bottleneck is compute: multiplying matrices quickly.

Swiftlet runs the forward pass on Metal, Apple’s GPU/accelerator framework on macOS and iOS. It also mentions runtime-compiled shaders, meaning the same code path can ship to iOS without requiring a Metal toolchain at build time. ()

Even if you’ve never touched Metal, the takeaway is: the “decode loop” can stay on the device fast enough that storage reads don’t fully dominate.

A detail that matters a lot: no exploding KV cache

A lot of local LLM performance pain comes from KV cache, which is the saved intermediate attention state used to generate the next token without recomputing the entire history.

Swiftlet reports that about 75% of its layers use Gated DeltaNet linear attention with a fixed-size recurrent state, so there’s “no growing KV cache for those layers at any context length.” ()

That’s a subtle but huge difference for long chats: you don’t want memory to grow with every generated token.

What it looks like to run it on a Mac

Swiftlet’s README includes a straightforward workflow: build the Swift package, repack a model into .qpack, then chat or serve.

At a high level:

1. Build the runtime:

git clone https://github.com/leonickson1/Swiftlet.git && cd Swiftlet
swift build -c release

2. Repack the model container from Hugging Face:

.build/release/swiftlet-repack \
 --from-hf Leonickson/Qwen3.6-35B-A3B-qpack \
 --output ~/models/qwen3.6-35b.qpack

For the 80B container, the README shows the analogous command with Qwen3-Next-80B-A3B-qpack. (github.com)

3. Chat with streaming:

.build/release/swiftlet chat ~/models/qwen3.6-35b.qpack \
 "Who wrote One Hundred Years of Solitude?" \
 "What language did he write it in?"

4. Optionally run an OpenAI-compatible loopback server:

.build/release/swiftlet-server --model ~/models/qwen3.6-35b.qpack --port 8080

(github.com)

The tradeoffs you should expect

It’s tempting to hear “80B on a phone” and assume “it’s like an 80B in every sense.” Swiftlet sets expectations more honestly:

• Only about 3B parameters are active per token in this family.
• The models “chat and write like large models” but recall “facts like small ones.” ()

In other words, sparsity helps memory feasibility, but it doesn’t magically remove architectural differences.

And one question-shaped thought that comes up immediately for local inference: What happens the moment your SSD is slower than your cache hit rate? Swiftlet’s caching + fixed-stride container design aims to keep that from becoming the dominant bottleneck, but it’s still the central systems question.

Where this points the next engineering wave

Swiftlet doesn’t just show “a model works.” It argues for an architecture where:

• MoE experts are treated like on-demand data,
• I/O behavior is engineered (container layout, fixed-stride reads),
• compute stays GPU-accelerated (Metal kernels),
• correctness is validated against reference implementations layer-by-layer. ()

That combination is what makes the RAM numbers believable.

Closing thought

The surprising part of expert streaming is that it feels less like “squeezing” a huge model into a small box, and more like admitting a truth: most of the weights don’t need to be in memory at the same time. Once you design around that—routing, packing, caching, and GPU compute—the impossible starts to look like disciplined engineering.