11–16× Faster LLMs in macOS VMs on Apple Silicon

You fire up a macOS VM and run your favorite local LLM. The model weights load fine, but performance feels… off. Prompt processing crawls, token generation crawls slower than it should, and you start suspecting the VM is missing “real GPU access.”

That suspicion is understandable. But the surprising part of the story behind 11–16× faster LLM inference on Apple Silicon isn’t raw GPU passthrough. It’s that the VM’s GPU capability reporting is conservative, and libraries like llama.cpp react to those reported capabilities by picking slower Metal kernels.

Below is a beginner-friendly (but technically honest) walkthrough of what’s happening, why it matters, and how a small, process-scoped “capability shim” can steer llama.cpp onto faster Metal paths inside a macOS VM. (github.com)

The VM that “has a GPU” but still runs slow

On Apple Silicon, macOS virtual machines typically use Apple’s Virtualization.framework rather than approaches like PCI passthrough. Virtualization.framework includes a paravirtualized graphics device: a virtual GPU exposed to the guest that routes graphics work through Apple’s GPU stack while the host retains control. (developer.apple.com)

Here’s the key twist: the guest doesn’t just see “a GPU,” it also sees a set of answers about what that GPU supports.

Modern GPU APIs (like Apple Metal) are designed around capability discovery. Metal applications query the device at runtime and then select optimized kernels and code paths based on what features appear available. ()

So if the paravirtualized GPU reports older or more conservative limits, the application can do the correct thing—just not the fastest thing.

Why does a GPU that’s fast on bare metal suddenly feel sluggish in a VM? Because the VM’s guest software believes certain Metal features are unavailable, and llama.cpp plays by those rules. ()

A quick mental model: llama.cpp, Metal backends, and kernel selection

llama.cpp is a local LLM inference engine that can run across multiple hardware backends. On Apple Silicon, the relevant backend is the Metal backend, which runs core tensor operations as Metal compute shaders (kernels). ()

The “kernel selection” idea is crucial: once the Metal device reports support (or non-support) for something like “SIMD-group matrices” or bfloat16, llama.cpp can choose a path with faster shaders—or fall back to a safer/older one.

So the problem wasn’t “the VM can’t execute newer Metal kernels.” The problem was that the guest’s capability answers were too low, pushing llama.cpp onto slower kernels even when the physical GPU could run the faster ones. ()

Where the slowdown comes from: conservative GPU capability answers

Inside a stock Tahoe VM, the paravirtualized GPU reports a limited profile (including older assumptions about Apple-family support and maximum threadgroup memory). That shapes what Metal applications believe is safe to do. ()

Two examples from the experiment:

• The guest reports that supportsFamily: for a high Apple GPU family is false.
• The guest reports maximum threadgroup memory as 32 KB, which can block certain optimized reductions/matrix paths.

Metal terms, translated:

• Apple GPU family: a family identifier used by Metal to describe feature sets; one way to check it is supportsFamily(MTLGPUFamily) -> Bool. ()
• Threadgroup memory: fast on-chip shared memory for a group of GPU threads; if the reported limit is too small, some kernels won’t be selected. ()

The solution: a process-scoped Metal capability shim

Instead of trying to remap hardware or truly “passthrough” a physical GPU, the research build uses a small compatibility layer inserted between the application and Metal.

In plain language, it works like this:

1. The VM and host GPU pipeline stay the same.
2. Only one guest process (one workload) gets different answers to specific capability queries.
3. llama.cpp then selects the faster Metal kernels that match those now-“enabled” capabilities.

The shim intercepts Metal capability queries and returns tuned values for the tested profile. For their tested configuration, it:

• Changes the reported supportsFamily: outcome via an Apple-family maximum override (to Apple-family 9 / 1009).
• Raises max threadgroup memory from 32 KB to 64 KB.

Those changes are narrowly targeted. Other capability values stay “stock” during the benchmark. ()

What features become available?

In their ablation table, the toggled capability profile switches on paths for:

• SIMD-group matrix support
• SIMD-group reduction
• bfloat16

…and increases the threadgroup memory limit that those kernels may require. ()

Benchmarking in a way you can actually trust

The experiment uses llama-bench, a llama.cpp benchmarking tool that reports throughput in tokens per second (t/s).

• Prompt processing measures how fast the model reads the prompt and performs the initial computations.
• Token generation measures how fast it produces new tokens step-by-step. (mintlify.com)

The command shape matters too: the test holds context and workload structure constant so speedups are attributable to kernel-path selection, not random configuration drift. ()

The headline result: M1 Ultra + TinyLlama

Their “minimal artifact” test setup was:

• Host: Apple M1 Ultra (48-core GPU)
• Host macOS: 26.6.1
• Guest: Tahoe VM image with macOS 26.5.2, 8 vCPU, 16 GiB RAM
• Lume version: 0.5.1
• llama.cpp build: release b10167
• Model: TinyLlama 1.1B Chat Q4_K_M

The benchmark command was:

llama-bench -m tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf \
 -p 512 -n 128 -r 10 -t 8 -ngl -1 -o json

Results (medians of ten samples per row):

• Prompt processing: 431.86 tok/s (stock) → 4,786.70 tok/s (unlocked)
• 11.08× faster, reaching 98.25% of bare-metal
• Token generation: 12.63 tok/s (stock) → 206.60 tok/s (unlocked)
• 16.36× faster, reaching 72.06% of bare-metal

So prompt throughput nearly matches bare metal, while decode/generation remains more sensitive to remaining VM overheads. (github.com)

A second data point: Gemma 4 12B

To show this wasn’t a one-model fluke, they repeated the same approach with Gemma 4 12B using a QAT Q4_0 GGUF variant.

Reported results:

• Prompt processing: 7.20× faster, reaching 99.59% of bare-metal
• Token generation: 14.54× faster, reaching 94.82% of bare-metal ()

Important sanity check: why MLX-LM didn’t benefit

They also tested MLX-LM 0.31.3 with an example 4-bit model on MLX 0.32.0.

The unlocked-vs-stock ratio stayed basically flat:

• Prompt processing ~1.005×
• Token generation ~0.993×

The interpretation is that MLX-LM’s existing Metal strategy was already fast enough in the stock VM, and/or the shim doesn’t provide the same useful “capability leverage” for MLX’s kernel/residency decisions. ()

How this “shim” gets activated (DYLD_INSERT_LIBRARIES)

The mechanism to scope behavior per process relies on environment-based dynamic library injection.

• DYLD_INSERT_LIBRARIES: on macOS, this environment variable asks the dynamic loader to load an extra .dylib into a process at startup.
• Because injection is per process, the capability override can be limited to just the LLM server/worker you care about.

In their “Try it in a Lume VM” section, the workflow includes:

1. Build the shim dylibs from libs/lume/metal-capability-shim.
2. Enable an “unrestricted device feature level” setting for paravirtualized graphics at the VM level.
3. Inject the matching shim dylib into the guest process.

The VM setting was enabled with a defaults write command setting ForceUnrestrictedDeviceFeatureLevel to -bool true. ()

The injected activation looked like:

lume ssh my-vm \
 "DYLD_INSERT_LIBRARIES=/path/to/LumeMetalCapabilities-arm64.dylib \
 LUME_METAL_APPLE_FAMILY_MAX=1009 \
 /path/to/metal-capabilities 1009"

This keeps other processes on the stock capability profile, and keeps the change as narrow as possible. (github.com)

Limitations you shouldn’t ignore

The technique is powerful, but it’s not a universal “fix all VMs” button.

Their documented limitations include:

• Version-sensitive: it depends on private, guest-side Metal behavior that may change between macOS releases.
• Per-process: hardened or platform-protected executables may refuse library injection.
• Narrowly validated: their evidence covers specific probing and a small set of workloads on one host/guest combination.
• Still a VM: the broader Virtualization.framework rendering/virtualization limits remain. ()

Wrapping up: the fastest kernel is the one your runtime believes exists

The biggest takeaway is almost philosophical: performance problems in virtualization often don’t come from raw compute being absent. They come from disagreement.

In this case, the guest’s Metal stack reported conservative capabilities, so llama.cpp chose conservative kernels. The shim doesn’t change the physical GPU or the VM’s graphics pipeline. It changes what capability answers the app sees, nudging Metal kernel selection toward faster paths.

Two tightly scoped capability edits moved TinyLlama prompt throughput from hundreds of tok/s to several thousand tok/s in the VM—and improved generation even more dramatically for their tested configurations. ()

If you build systems that mix virtualization and hardware acceleration, this is a useful pattern to remember: the “GPU you have” and the “GPU you’re allowed to use” are not always the same thing. Inference speed often starts with the answers your software believes first.