Qwen3.8-27B is now listed on Cerebras public endpoints at roughly 1,500 generated tokens per second. This guide explains what the speed means, how reasoning and context limits affect real latency, and how to make a first API call.
Small language models are becoming capable enough to handle the repetitive work that fills inboxes, support queues, and business systems. Lower inference costs, faster responses, local deployment, and careful routing can make AI products viable at much larger scale without retiring frontier models for difficult cases.
LLM inference no longer looks like “CPU submits, GPU generates.” As assistants become agentic, tool calls and orchestration move onto the critical path, making CPU-heavy serving practical. This post breaks down prefill/decode, KV cache, and the software techniques (continuous batching, prefix caching, chunked prefill) that make CPU inference viable.
Kimi K3 is huge enough that serving it on commodity setups quickly becomes an HBM problem. On AMD MI355X, better $/token comes from fixing two ROCm bottlenecks: speculative decoding verification and prefill attention kernel selection for TTFT.
TurboFieldfare runs Gemma 4 26B-A4B on Apple Silicon with ~2GB RAM by keeping core weights + an FP16 KV cache resident and streaming only the routed MoE experts from SSD. This post explains the memory pieces (MoE routing, KV cache, quantization) and walks through building, installing, and tuning the runtime.
Open-weight models aren’t “free” once they enter production: inference turns token generation into real COGS. In a commodity-like market, margins come from better cost-per-success, including efficient serving and prefix caching—not just from cheaper weights.
Running SOTA LLMs locally is a systems problem, not just a model download. VRAM and quantization must fit, and multi-GPU speed depends on PCIe topology, P2P routing, and NCCL stability.
Qwen 3.6 27B stands out as a practical local model: high enough quality for day-to-day development and strong performance with llama.cpp. This guide explains why 27B is the sweet spot and shows how to run it (with MTP) and integrate it into coding tools.