Streaming Gemma 4 26B on 2GB RAM: TurboFieldfare on M-series Macs

The first time “26B on 8GB” stopped sounding like a lie

There’s a particular moment with local LLMs that feels a little magical and a little suspicious at the same time. You see “26B parameters,” then you remember your Mac has 8GB of RAM, and your brain starts doing the back-of-the-napkin math: no way. So you scroll for a while… and then something clicks into place because the trick isn’t magic at all. It’s memory engineering.

That’s the spirit behind TurboFieldfare: a Swift + Metal runtime that runs Gemma 4 26B-A4B inference with an approximate ~2GB RAM budget on Apple Silicon Macs—even the smaller “8GB” models. The project’s core promise is blunt: it does not load the entire model into memory; it keeps a “core” in RAM and streams only the expert weights needed for each token from SSD. (github.com)

Why can a model labeled “26B” fit into a couple gigabytes? Because Gemma 4’s 26B variant is a Mixture-of-Experts (MoE) model: it contains many parameters overall, but for each token it activates only a small subset.

Gemma 4 26B-A4B: what “MoE” really means

A transformer LLM is mostly a stack of repeating layers. In a dense model, every parameter is “active” for every token. In an MoE model, the layer contains multiple “experts” (think: specialized sub-networks), and a router decides which experts to run for the current token.

Gemma 4’s 26B-A4B has 26B total parameters but only ~4B active parameters per forward pass. In the MoE terminology used by the model cards, it routes to 8 active experts out of 128 plus a separate shared expert branch. (huggingface.co)

That matters for memory because “which experts are needed right now?” becomes the key question. If the runtime can predict (or at least quickly determine) which experts will be needed next, it can avoid bringing every expert into RAM.

The real memory villain: weights + KV cache

When people say “an LLM doesn’t fit,” they often mean “the weights don’t fit.” That part is true, but it’s not the whole story.

Two big memory buckets show up in practice:

1. Weights: the trained parameters of the model (embeddings, attention projections, expert networks, etc.).
2. KV cache: when generating text token-by-token, the model repeatedly reuses internal attention context. Instead of recomputing everything from scratch, implementations store intermediate “keys” and “values” (often called K and V) for previous tokens. That stored context is the KV cache.

KV cache is especially sneaky because it grows with context length. TurboFieldfare uses a FP16 KV cache (FP16 = 16-bit floating point numbers) and supports different context windows by choosing how the cache is laid out in memory rings vs. linear storage. (github.com)

In its “production” design, TurboFieldfare keeps:
- about 1.35GB of common resident weights in memory
- about ~305 MiB for the FP16 KV cache at 4K context
- plus streamed expert slots that are filled on demand ()

So even before streaming experts, the runtime is making a deliberate bet: “KV cache and core weights are allowed, everything else must be bounded.”

Quantization: turning “gigabytes” into “manageable”

A major reason local LLMs got practical is quantization. Quantization stores weights with fewer bits than full precision. TurboFieldfare’s README summarizes its weight representation as:
- MLX affine 4-bit weights (4-bit quantized values packed with BF16 scale and bias per group of 64 weights)
- an 8-bit router
- 4-bit shared and routed experts ()

The important beginner takeaway: quantization is not “smaller model.” It’s “same model behavior, compressed representation.” The runtime still has to do attention, MoE routing, etc.—but it reads and multiplies compressed tensors.

TurboFieldfare’s central idea: keep the core, stream the experts

Here’s where TurboFieldfare becomes an engineering story instead of a marketing claim.

TurboFieldfare runs Gemma 4 without loading the full 14.3GB model into memory. Its README describes the strategy like this: it keeps the shared 1.35GB core and FP16 KV cache in memory, then streams the experts needed for each token from SSD. ()

Under the hood: expert caching with LFU eviction

Streaming from SSD sounds slow, but it doesn’t have to be catastrophic if you reuse what you recently loaded.

TurboFieldfare opens routed-expert “streamers” per transformer layer and maintains a 16-slot LFU (Least Frequently Used) cache per layer streamer. LFU means: when the cache is full, it evicts the experts used least often. ()

When a token arrives, the router predicts the top-8 expert IDs. The CPU uses those IDs to plan reads against the cache, then fills misses with bounded parallel pread reads into Metal-visible buffers, while Metal computes the shared expert branch in parallel. ()

The system design also notes a useful optimization for MoE routing in batches: prompt prefill processes known prompt tokens in chunked blocks, so one fetched expert can serve multiple rows (TurboFieldfare mentions chunks up to 128 tokens for prefill). ()

The .gturbo format: why the installer is as important as the runtime

Streaming at inference time is one half. The other half is getting the model into a layout that’s friendly to bounded streaming.

TurboFieldfare installs a pinned text-only Gemma 4 instruction checkpoint into a custom on-disk directory called .gturbo. In the system design doc, the installer is described as a bounded repack process:
- It does not download a full Hugging Face snapshot into a giant local heap buffer.
- It reads an index and tensor metadata.
- It requests only bounded remote byte ranges.
- It copies quantized values directly into their final packed locations.
- It validates using manifests and hashes, and promotes an atomic install when done. ()

That design choice is what makes it possible to keep the runtime’s RAM predictable, even though the source checkpoint is large.

Running it on your Mac: a practical walkthrough

TurboFieldfare’s “Quick start” is a straightforward path for builders and tinkerers.

Requirements

The README calls out validated targets and toolchain constraints:
- Apple Silicon Mac (validated with an 8GB M2 MacBook Air)
- macOS 26 with Metal 4
- Xcode 26 and Swift 6.2+
- enough storage for about 14.3GB of installed model ()

Build and run

From the repository root:

git clone https://github.com/drumih/turbo-fieldfare.git
cd turbo-fieldfare
swift build -c release
.build/release/TurboFieldfareMac

On first launch, the Mac app downloads and repacks the pinned model into the .gturbo layout (the README describes it as streaming downloads and then validating before promoting the install). (github.com)

After that, you choose Load Model, type a prompt, and press Generate.

A command-line option

The CLI expects an installed .gturbo directory (for example, scratch/gemma4.gturbo after installing via the app): ()

swift run -c release TurboFieldfareCLI \
 --model scratch/gemma4.gturbo \
 --prompt "The capital of France is" \
 --max-new 64 \
 --temperature 0

Setting --temperature 0 requests deterministic greedy decoding.

Tuning without breaking the memory budget

Local LLM demos often ignore what matters: the slow parts and the memory-sensitive knobs.

TurboFieldfare’s runtime controls define:
- Maximum context via --max-context (the app supports multiple sizes; default is 4K) (github.com)
- Temperature (default 0.2), where 0 is greedy and positive values sample ()
- Top-K and Top-P truncation (default Top-K 64, Top-P 0.95) ()
- Expert-cache slots (8, 16, 24, 32; production default 16), where more slots reduce reads but use more RAM ()

One subtle but important rule: changing context length or expert-cache slots requires a reload, because those settings reshape the FP16 KV memory and cache capacity. ()

TurboFieldfare’s benchmarks docs also define throughput-style metrics that help you interpret results:
- Decode rate = tokens/sec after prompt prefill
- TTFT = time-to-first-token including prefill + wait for the first generated token ()

What kind of speed should you expect?

TurboFieldfare includes benchmark numbers that are worth treating as reference points, not universal guarantees.

In its results table:
- On an 8GB M2 with TurboFieldfare: 5.10–6.30 tok/s and about ~1.9–2.1 GB footprint (github.com)
- On a 24GB M5 Pro with TurboFieldfare: 31–35 tok/s and about ~2.1 GB footprint ()

Those numbers line up with the design goal: keep the runtime’s resident memory small while accepting that MoE routing will trade compute time against SSD-backed streaming and cache misses.

The mental model that makes this all feel coherent

When you zoom out, TurboFieldfare looks less like “a new model” and more like “a new schedule.” It’s a runtime that coordinates:
- quantized resident weights
- a bounded KV cache strategy
- a router-driven expert selection
- an LFU expert cache
- Metal compute overlapped with bounded pread reads

Once you see those pieces as a pipeline, the 2GB claim stops being a mystery and becomes a consequence of engineering constraints.

Closing thought: local LLMs as a systems problem

The real lesson from TurboFieldfare is that running large language models locally isn’t only about model quality. It’s about designing around memory, I/O, and compute overlap.

Gemma 4’s MoE structure gives the runtime a lever (“only a few experts per token”), and TurboFieldfare’s streaming .gturbo layout gives it a way to pull those experts in without overflowing RAM. That combination turns “impossible on 8GB” into “measurable, tweakable, and—most importantly—repeatable.”