DeepSeek’s DeepInfer release opens the runtime layer behind its large-scale V4.1 Flash service. The article explains its hardware limits, speculative decoding, asynchronous scheduling, cache-aware replay, and why the public engine is not yet a complete API service.
A batched compact-Householder QR kernel is tricky on GPUs because Householder reflectors form a dependency chain. This post explains how blocked Householder QR with compact WY updates turns the trailing work into GEMM-shaped operations, and how an auto-research loop with Codex drove fast, correct kernel iterations to a 232× speedup over baseline.
LLM inference no longer looks like “CPU submits, GPU generates.” As assistants become agentic, tool calls and orchestration move onto the critical path, making CPU-heavy serving practical. This post breaks down prefill/decode, KV cache, and the software techniques (continuous batching, prefix caching, chunked prefill) that make CPU inference viable.
Shitty is a terminal emulator built for low latency and high throughput, keeping terminal state on the CPU while rendering cells with GPU compute backends (Vulkan/Metal). It also warns that it’s memory-unsafe, making testing and sanitizer-based verification central to its design.