Run GLM-5.2 on a Slow PC with colibrì (Disk-Streamed MoE)

Imagine you’ve got a laptop that feels like it was built for spreadsheets in 2017. Then you hear about GLM-5.2—a 744B-parameter frontier model—and the thought lands like a brick: how could anything that big ever run locally?

The trick is that GLM-5.2 is a Mixture-of-Experts (MoE) model: instead of running all parameters for every token, it activates only a subset. The project we’re exploring here, colibrì, leans into that design so hard that it can keep the “always-needed” parts in RAM and stream the rest from disk—written in a single C file, with no Python at runtime.

By the end of this walkthrough, the core idea should click: you’re not loading 744B weights into RAM—you’re loading ~10GB into RAM and paging the rest through a cache.

The real puzzle: “744B” does not mean “744B per token”

Most big language models are dense: every layer computes with essentially all parameters every time. That means memory and compute costs scale pretty directly with total model size.

MoE changes the cost model. In an MoE layer, a router (a learned gating function) decides which experts (specialized feed-forward sub-networks) should process the current token. Only those selected experts run; the other experts are skipped.

For GLM-5.2, the important headline is:

• Total size: 744B parameters
• Active per token: about ~40B parameters (because only a routed subset of experts is used)

Even that still sounds huge—until you zoom in further.

How colibrì fits GLM-5.2 into ~25GB RAM

colibrì’s strategy is a three-part memory story:

1. Keep the dense “base” in RAM
   - This includes attention components, shared experts, embeddings—parts that are used every token.
   - In the provided setup, this resident portion is compressed to int4 (explained next).
   - The README-style figures describe resident dense RAM around ~9.9GB when using int4.

2. Store the routed experts on disk
   - GLM-5.2 has many routed experts across multiple MoE layers.
   - colibrì keeps those experts in a large on-disk container (~370GB for the int4 container), and loads only what the router needs.

3. Use an LRU expert cache (plus optional “pinning”)
   - An LRU cache means “least recently used”: when RAM is full, the cache evicts experts that haven’t been used recently.
   - A pinned hot-store is an optional region of RAM reserved for the hottest experts so they stop getting evicted during interactive chatting.
   - On top of that, the operating system’s page cache acts like a second-level cache “for free,” because reads are still file-backed.

Quantization: int4 and int8 are not just “smaller numbers”

To make disk-streaming practical, colibrì uses quantization, meaning it stores model weights with fewer bits.

• int4: 4-bit integers (very compact). This is how most weights in the resident dense set and streamed expert set are stored.
• int8: 8-bit integers (less compact, more accurate).

A detail that matters for speed/behavior: colibrì expects the Multi-Token Prediction (MTP) head (a speculative decoding component) to be int8. If it gets down-quantized too far (e.g., int4), speculative decoding stops being useful.

The architecture features that make “slow but possible”

Let’s connect the runtime design to the model design.

MLA attention and compressed KV cache

During generation, a transformer needs to remember prior token information. This is stored in a KV cache, where K = keys and V = values used by attention.

Keeping a KV cache for very long contexts can explode memory. GLM-5.2 uses MLA (Multi-head Latent Attention), plus a more compact approach to KV storage. colibrì implements a compressed KV cache, described as dramatically smaller than a naive float KV cache. That’s one reason the engine can stay within a reasonable RAM budget.

A sigmoid router (DeepSeek-V3-style)

The router decides the expert(s) per token. In this implementation, colibrì uses a DeepSeek-V3-style sigmoid router flavor (including routed scaling logic and shared expert behavior). For you as a user, the practical result is simple: expert choice drives what gets streamed from disk.

Speculative decoding with MTP

Speculative decoding is a performance trick: a smaller/cheaper head proposes multiple future tokens, and the main model checks them.

In GLM-5.2, this is built in via an MTP (multi-token prediction) head (at layer 78). colibrì implements it so that drafts are verified in a batched forward pass.

Why this matters on a slow machine:

• Speculation only helps if the draft head works well.
• That’s why the MTP head must be int8 in the intended setup.
• When everything is “right,” the README describes acceptance rates and effective tokens per verification forward in a useful range.

Why it’s slow on cold start (and what “warm cache” means)

When you first run a chat session, your expert cache is empty, so the router repeatedly asks for experts that aren’t in RAM yet.

That creates the dominant bottleneck: random disk reads.

The README-style numbers summarize it bluntly:

• cold decode costs are dominated by disk I/O
• throughput is on the order of ~0.05–0.1 tokens/second in the “cold” baseline case

After a while, things improve because:

• the LRU cache fills with experts that are actually being routed to
• OS page cache begins serving repeated reads faster
• speculative decoding can reduce the number of expensive verified steps

In other words, colibrì isn’t “fast”—it’s eventually interactive once your workload warms the cache.

Getting GLM-5.2 running: build, get weights, run chat

Below is the “walk with the reader” version of the workflow.

1) Prepare the machine

At a minimum, colibrì targets:

• Linux or WSL2
• gcc with OpenMP support
• AVX2 CPU instructions
• at least 16GB RAM (higher is better)
• about ~370–400GB of free local NVMe storage for the int4 container

One important practical note: the README warns to avoid network mounts (like 9p) because the I/O pattern is brutal.

2) Build the engine

Inside the repository, the runtime is a small C program plus utilities. The typical flow is:

cd c
./setup.sh

This checks your toolchain and builds the engine, plus runs architecture self-tests.

3) Choose your weights path

There are two approaches.

Option A: Use pre-converted int4 weights (fastest path).

A Hugging Face repo provides colibrì-native int4 containers meant to be dropped onto a fast local disk. Once the files are downloaded, you point colibrì at that directory:

COLI_MODEL=/path/to/GLM-5.2-colibri-int4 ./coli chat

Option B: Convert FP8 → int4 yourself (heavier, but self-contained).

This uses the one-time converter. The README describes that conversion downloads shards (so you don’t need the full FP8 checkpoint available at once), then requantizes into the engine’s container format.

./coli convert --model /path/to/model_dir

At runtime afterward, the engine stays pure C; Python is only for the conversion step.

4) Run chat and use the tuning knobs

Running chat typically looks like:

COLI_MODEL=/path/to/GLM-5.2-colibri-int4 ./coli chat

The engine auto-detects a RAM budget and configures expert cache behavior. Still, some knobs are especially relevant on a slow machine:

• --temp and nucleus/top-p: token sampling settings tuned for int4 reality
• DRAFT=n: disables speculative decoding depth (useful when MTP isn’t helping)
• --ngen: limits max tokens per response so you don’t accidentally push into a long, expensive generation

A “human” way to think about these options: on slow hardware, the goal is not perfect throughput—it’s stable progress without turning every answer into a disk I/O marathon.

The mental model to keep you sane

When things feel slow, the system isn’t “misconfigured.” It’s behaving as designed.

The model’s workload splits into:

• resident compute: dense attention + shared pieces in RAM (int4)
• routed compute: selected experts streamed from disk
• cache dynamics: your throughput improves as the cache captures expert reuse patterns

So the best performance comes from “staying in the same neighborhood” (similar prompts and conversation structure) long enough for the cache to reflect those routing patterns.

Conclusion: disk-streamed experts turn a 744B model into a local experiment

Running GLM-5.2 on a slow computer is possible because MoE makes the per-token compute selective, and colibrì exploits that selectivity with a practical systems design: keep dense weights in int4 RAM, stream routed experts from disk, and accelerate repeated routing with an LRU cache (plus optional pinning). The engine also integrates GLM-5.2-specific details like MLA attention and native MTP speculative decoding, with quantization choices that preserve speculation usefulness.

The result is not high-speed inference—it’s something more interesting: a real frontier-class model that can answer on modest hardware, as long as you respect the disk bottleneck and let the cache warm up.