Why the “free” price tag on open-weight Chinese models isn’t the point

A few years into MBA school, one idea sticks to the walls: when marginal cost is effectively zero, business models don’t behave like the textbooks. Software can be copied cheaply; distribution can be near-instant; transaction costs can shrink until they feel like rounding error. In that world, value chains look centralized, and the winner isn’t necessarily the cheapest builder—it’s the best at controlling demand.

Then the AI era arrived, and the conversation got weird again.

Because AI changed the unit of production.

In generative AI, the “work” isn’t done once when you ship a model checkpoint. The work happens every time a user asks a question. That means the economics of running models—costs that scale with usage—matter much more than people initially assumed.

So when the debate turns to open-weight models (including top-performing ones released from China), the loud question becomes: “Why are people afraid of these models?” But beneath the politics is a more technical story: open weights make R&D look optional, and that’s true… until inference cost comes due.

R&D isn’t the same as serving costs (and this distinction drives everything)

Start with two accounting terms that sound similar but behave very differently.

R&D (research and development) is the up-front expense to create the model: training runs, experimentation, engineering, evaluation, and iteration. It’s largely fixed in the short term.

COGS (cost of goods sold) is the expense tied to delivering the product for each unit of revenue.

For AI, COGS is dominated by inference.

Inference is the “run the model” phase: the model reads input tokens, does computation, and emits output tokens.

Here’s the key economic observation: if you triple revenue but the model generates ~3× the tokens, COGS usually scales with that same token volume. So when people celebrate “free” weights, what they often mean is: “You don’t have to pay R&D.” That’s real. But the model still charges (or still costs) you to run.

You can see this tension in publicly listed API pricing. For example, Moonshot’s Kimi K3 lists $3 per 1M input tokens (cache miss) and $15 per 1M output tokens. (tomshardware.com) Meanwhile OpenAI’s GPT-5.6 Sol lists $5 per 1M input tokens and $30 per 1M output tokens. (developers.openai.com)

Even if two models have different input/output rates, the direction is clear: output tokens are where costs grow, and “free weights” don’t eliminate those costs.

Token factories: the industry is back to measuring production, not prototypes

Nvidia has popularized the framing of AI factories: infrastructure that converts energy into tokens, where tokens are the basic pieces of text the model processes during inference. (blogs.nvidia.com) In this view, competitiveness depends heavily on cost per token and throughput under power constraints.

That framing makes sense for the first wave of AI: chat.

But reasoning and agents complicate the measurement.

What makes reasoning expensive?

A model that performs reasoning typically uses more computation per user request, often expressed as producing a longer chain-of-thought—meaning more intermediate reasoning tokens are generated before the final answer.

An agent goes further: it may run multiple steps, call tools, check results, and loop until it has enough confidence to finish. Every loop is more tokens (and often more total tokens than “chat mode”).

So the “token price” isn’t a commodity the way gasoline is.

Tokens aren’t fungible; intelligence is

A commodity is a good where units are interchangeable. A gallon of oil is a gallon of oil.

But a token from Model A isn’t the same thing as a token from Model B. Two models can produce the same final answer with different internal paths, and that difference shows up as different token usage.

That means the real economic unit is closer to intelligence per dollar, not “tokens per dollar” in a vacuum.

A more technical way to say it:

• Some models need fewer reasoning tokens to get to the correct answer.
• Some models get more done per output token (better token efficiency).
• Some architectures reduce compute per token.

Those differences become COGS differences. So even when two providers quote “similar” per-token rates, their customers can still see radically different costs per successful task.

When open weights spread, AI services start behaving like commodity markets

Now connect this back to the open-weight story.

Open-weight models reduce the barrier to entry: you can download weights instead of paying for model access.

But if lots of teams can build similar apps, customers stop caring about tiny quality differences and start shopping on price and cost structure.

At that point, AI starts looking like a commodity market, where:

• Everyone can sell something “good enough” (the product becomes standardized).
• Price gravitates toward what the least efficient seller can sustain.
• Profit survives not by charging more, but by delivering at a lower COGS per unit value.

So why does open-weight adoption often trigger fear in incumbents? Because it weakens the classic pricing moat.

If the same CRUD-style app can be implemented against multiple models, the durable advantage shifts to who can run inference more efficiently.

That’s a business problem, but the mechanism is technical.

The real COGS for intelligence: a checklist of what costs money

The “token price” you see in documentation is only the visible part. The invisible part is how inference is engineered at scale.

In practice, COGS for intelligence is driven by several factors:

1) Model footprint

A model’s footprint is how much memory (and accelerator resources) it needs to serve requests concurrently. Even if the weights are the same for everyone, runtime state and serving layout can differ.

2) Inference efficiency

Architectural choices can reduce computation per generated token. One commonly discussed technique is Mixture-of-Experts (MoE): rather than activating every sub-network for every token, MoE activates only a small subset of experts per token. That can reduce compute, improving cost per token.

3) Memory efficiency and KV cache pressure

Every transformer model uses attention, and during inference it stores intermediate attention data in KV cache.

KV cache (key-value cache) is a memory structure that stores representation “summaries” of past tokens so the model doesn’t recompute everything from scratch at every decode step.

Without KV cache, decoding would repeatedly reprocess the entire prompt for each new token. With KV cache, decoding can reuse prior work.

And when prompts get long—like with million-token contexts—KV cache becomes a dominant cost and bottleneck.

4) Serving efficiency: batching and scheduling

Even if the model is fixed, the serving stack matters. Batching is when an inference server groups multiple requests so it can process them together more efficiently. Scheduling decides which requests get processed when, based on resource availability and expected latency.

5) Token efficiency

If reasoning requires fewer output tokens to reach a correct answer, COGS drops even when per-token pricing looks similar.

Prefix caching: why “same beginning” is a real engineering win

One optimization captures the commodity-market mindset especially well: prefix caching (also called prompt caching or context caching).

Prefix caching stores computed KV cache states for a shared prefix of tokens, so new requests that share that same beginning can reuse earlier computation.

Modular’s LLM inference handbook describes the approach clearly: during prefill, the model builds KV cache; during decode, it uses the cached states; and when a new request arrives that shares an identical prefix, the server can skip recomputing that prefix and only process the remaining suffix. (handbook.modular.com)

This is where open weights (and open ecosystems) create a subtle advantage for teams that build production-grade serving stacks.

Because as you serve millions of requests, even a small improvement in cache hit rate can translate into huge cost savings. And cache hits depend on operational realities:

• Systems that keep system prompts and formatting stable get better hits.
• Pipelines that avoid introducing per-request randomness early in the prompt do better.

The technical moral is blunt: commodity pressure pushes competitors toward better infrastructure tricks, not just better research ideas.

So who should you worry about—and what should you actually measure?

The fear around “Chinese models” is often shorthand for a deeper worry: that more capable open-weight models will compress margins by turning intelligence delivery into a cost race.

But the measurable story isn’t national origin. It’s whether a provider (or your own serving stack) can control COGS.

In a world where open weights make model procurement less proprietary, the differentiators become:

• lower cost per successful task (not just per token)
• better inference efficiency (architecture + serving stack)
• better cache utilization (like prefix caching)
• better token efficiency under real reasoning workloads

Why does this matter for builders? Because the “unit economics” finally looks like manufacturing again.

If AI factories convert energy into tokens, then the competitive edge is how many tokens (and how much useful work) you can squeeze out of that energy at scale. (blogs.nvidia.com)

Conclusion: the real shift is from shipping models to operating token production

Open-weight models didn’t make R&D irrelevant. They made R&D less visible at the front door.

Once everyone can get the weights, the long-term advantage shifts to inference economics: COGS, cost per intelligence, and the serving engineering that keeps GPUs busy without letting KV cache and token generation explode your bills.

And that’s the real reason this debate feels tense. It isn’t about whether the models are good enough—it’s about whether anyone can run them cheaply enough to profit when intelligence becomes the commodity everyone can assemble.