The CPU Is Back in LLM Inference—Rethink the Split

A few months ago, the system diagram for an LLM service looked almost too simple: “CPU does the request, GPU does the generation.” Then the metrics started lying a little.

The GPU utilization graph stayed stubbornly low, while the CPU cores spiked in bursts. Latency to the first meaningful token (the moment the model starts “speaking”) got worse exactly when the product started using more tool calls, more multi-step reasoning, and more small, specialized models. What changed wasn’t the model size. It was the shape of inference.

That’s the heart of why the CPU is back: inference isn’t one big compute action anymore. It’s a loop.

CPU vs GPU: not a popularity contest, a division of labor

A CPU (central processing unit) is designed for general-purpose work: branching logic, system calls, memory management, networking, and coordinating many moving parts. A GPU (graphics processing unit) is built for highly parallel arithmetic, especially the kind of math that shows up repeatedly in neural networks.

In an LLM inference pipeline (inference = running a trained model to generate output, without updating weights), these strengths line up with two different needs:

• The GPU is great at the transformer forward pass: the bulk computation that turns hidden states into the next token. Transformers use attention, a mechanism that blends information from earlier tokens.
• The CPU is great at everything around the model: tokenization (turning text into token IDs), scheduling, batching decisions, and orchestrating tool calls (the model asking the application to fetch data, run code, or consult external services).

For years, the stack often behaved like a relay race: CPU hands off, GPU races, CPU catches. But modern assistants aren’t single-shot. They’re iterative.

Why the “CPU is just a passenger” assumption breaks

The old mental model matched a narrow workload: one prompt in, tokens out, maybe with a little post-processing. Today’s workload often looks like this instead:

1. Generate a plan or intermediate reasoning.
2. Decide on tool calls.
3. Call tools, parse results, retry on partial failures.
4. Feed results back into a subsequent model call.
5. Repeat until the user-facing response is done.

In this world, the model becomes one component inside a larger agentic control loop. That loop is heavy on conditional branching (“if the tool result looks like X, do Y”), parsing, and network/I/O waits. CPUs are architected to handle those patterns well. The article that inspired this piece argues that CPU-side work can become a significant fraction of end-to-end latency in agentic deployments, even reaching ranges reported by Intel/academic work. (redhat.com)

So where should the CPU-GPU split move: toward raw compute, or toward orchestration?

It turns out the answer depends on which part of the loop is dominating your bottleneck (the slowest stage that caps throughput).

The prefill/decode split: where time goes per request

Even for a “normal” single prompt, LLM inference is usually split into two phases:

• Prefill: process the entire prompt. This is compute-heavy because attention must incorporate all prompt tokens.
• Decode: generate tokens one at a time. Each step uses attention over the previously seen context, but doing it naively would be wasteful.

To make decode fast, systems use a KV cache (key-value cache): during attention, each token produces intermediate tensors called keys and values. Caching them avoids recomputing attention history every time a new token is generated.

Here’s the twist for CPU/GPU partitioning: KV cache management isn’t “free.” Memory movement, allocation strategy, and batching all affect whether the GPU stays fed.

That’s also why CPU inference became more credible: software teams started treating KV cache management and scheduling as first-class engineering problems—not just GPU kernels.

When CPU orchestration matters most: tool calls and branching

In an agentic system, CPU overhead shows up in places that don’t look like “AI compute,” but still directly impact user-perceived latency:

• Parsing and routing: turning model outputs into structured actions.
• Tool dispatch: making API calls, running internal services, or executing code in a sandbox (a restricted execution environment).
• Retry and fallback: handling partial failures without derailing the whole response.
• Multi-model coordination: swapping between smaller models for sub-tasks.

These steps are often dominated by instruction latency—how quickly a CPU core can follow an unpredictable chain of work—rather than by peak FLOPS (floating-point operations per second). The Red Hat article frames this as the CPU being a “logic engine” and the GPU being the “math muscle,” but the key takeaway is operational: CPU work can become the critical path. ()

A second driver: smaller models closer to the data

The other reason the CPU is back is architectural economics.

A small language model (SLM) is cheaper to run and easier to deploy near where data is generated: on-prem, in factories, at the edge, or in privacy-sensitive environments. Instead of shipping every request to a GPU cluster, teams can run local inference with the same orchestration logic—often on CPU-centric infrastructure.

This is also where retrieval-augmented generation (RAG) (combining model generation with retrieved documents) changes the equation. RAG lets smaller models punch above their weight by grounding answers in retrieved context.

In practice, the moment you care about milliseconds, privacy boundaries, or offline availability, “send everything to the GPU” becomes less attractive.

What changed in software: continuous batching + better KV cache handling

CPU inference wasn’t always competitive. Two things shifted:

1. Scheduling and batching got smarter so GPUs (and CPUs) don’t waste cycles.
2. KV cache handling became efficient enough that decode didn’t collapse under memory fragmentation.

A well-known example is vLLM, an inference engine that introduced techniques like continuous batching, paged attention, prefix caching, and chunked prefill.

Let’s define the key pieces:

Continuous batching

Continuous batching means the server keeps mixing requests at different generation steps into a single running schedule, instead of waiting for a “batch” to finish entirely. This boosts overall throughput because idle gaps shrink.

PagedAttention (KV cache as memory “pages”)

Paged attention is a KV cache memory-management strategy. Instead of allocating a huge contiguous buffer (which wastes memory and causes fragmentation), the cache is organized like pages, so variable-length requests fit more efficiently.

vLLM’s design is built around this idea; the original PagedAttention work explains how serving efficiency improves when KV cache memory is managed like pages to reduce waste. (arxiv.org)

Prefix caching

Prefix caching stores computed KV states for prompt prefixes that repeat across requests (for example, the same system prompt, similar instructions, or shared retrieved context). Then later requests can reuse those cached states.

vLLM describes “automatic prefix caching” as an approach to map cache entries to prompt prefixes to avoid recomputation. (docs.vllm.ai)

Chunked prefill

For long prompts, chunked prefill splits the prompt into smaller chunks, processes them, and blends them into the overall scheduling. vLLM documentation notes that chunked prefill exists specifically to handle large prefills without stalling other requests. (github.com)

CPU performance knobs: threading, affinity, and math libraries

Even with good scheduling, CPU inference is sensitive to how threads run and where memory lives.

Two concepts matter immediately:

• OpenMP is a popular way to parallelize CPU code using threads.
• Thread pinning / affinity means binding threads to specific CPU cores to reduce cache thrashing and unpredictable migrations.

vLLM’s CPU deployment guidance (as described in the Red Hat article) includes environment variables for controlling KV cache allocation and OpenMP thread binding. ()

A typical example of the kind of knobs you’ll see looks like this:

# Size KV cache memory region (GB) on the CPU
export VLLM_CPU_KVCACHE_SPACE=32

# Pin OpenMP threads for more predictable cache locality
export VLLM_CPU_OMP_THREADS_BIND=true

On the compute side, CPUs still need optimized math kernels. Frameworks often use deep learning libraries such as oneDNN (an optimized library of deep-learning primitives for Intel and other architectures). (intel.com)

For maximum performance, inference stacks also take advantage of instruction set features (special CPU instructions), like AVX-style vector operations and newer matrix extensions on supported chips.

Putting it together: a practical way to decide the split

Instead of asking “CPU or GPU?”, ask “which stage is stalling my pipeline?” That leads to a more useful test:

• If your service spends time in tool calling, orchestration, parsing, sandboxing, and branching, CPU-heavy serving is more likely to pay off.
• If your requests are dominated by compute-heavy transformer execution with minimal orchestration complexity, GPUs stay the right default.
• If your GPU is underutilized while CPUs are busy, that’s often a sign the CPU work is critical-path latency.

The modern CPU-GPU split isn’t a fixed ratio. It’s a moving boundary around the critical path.

Conclusion: iterative inference changes what “serving” means

The CPU isn’t “replacing” the GPU. The GPU still wins at large-model token generation under heavy concurrency. What changed is that inference increasingly looks like an iterative control system: orchestration, tool calls, multi-step loops, and repeated model invocations.

Once you view serving as a pipeline with phases (prefill, decode, KV cache management, plus orchestration between calls), it becomes easier to place work on the right hardware. CPUs are back because the perimeter of “where AI compute lives” has expanded—and, in agentic workloads, the CPU often becomes part of the engine, not the receptionist.