From Softmax to Kimi Delta Attention: the “obvious” path you missed

A long-context generation run that feels unfair

There’s a particular kind of frustration you only get with long prompts: everything is working… until you push context length further and the system suddenly gets slower and hungrier for memory. That “hungry KV cache” is the culprit in standard transformer attention.

In ordinary causal softmax attention, each new token at position (t) compares its query vector (q_t) against every previous key vector (k_i) (for (i\le t)). The resulting attention weights are normalized with a softmax, and the token output is a weighted sum of value vectors (v_i). The price is quadratic compute ((\mathcal{O}(T^2))) and a KV cache that grows with (T).

Long-context efficiency is where linear attention earns its keep: you trade some of softmax’s selectivity for an attention mechanism that updates a fixed-size memory state as you scan forward. Then the real question becomes: how do you regain accuracy without giving up the efficiency win?

Kimi Delta Attention (often abbreviated KDA) is one answer. It’s part of a family that starts with linear attention, then adds the delta rule idea (online correction), then adds gating (controlled forgetting), and finally makes the forgetting fine-grained.

Start with quadratic attention, then peel off the hard part

For a single head, softmax attention is easiest to picture like this:

• For each earlier index (i\le t), compute a similarity score (s\langle k_i, q_t\rangle).
• Turn scores into a probability distribution with softmax.
• Output is the probability-weighted sum of values:

[
\lvert o_t\rangle = \sum_{i\le t} a_{ti}\, \lvert v_i\rangle.
]

Where the weights (a_{ti}) depend on a denominator that touches all (j\le t). That denominator is exactly what blocks rearranging the computation into a constant-memory scan.

So we do the classic warm-up: remove the softmax denominator for a moment. Absorb the usual scaling factor (s=d_k^{-1/2}) into the query (it’s just a constant rescaling). Then the “bare” attention becomes:

[
\lvert o_t\rangle = \sum_{i\le t} \langle k_i, q_t\rangle\, \lvert v_i\rangle.
]

Now comes the key algebra trick: the inner product (\langle k_i, q_t\rangle) is a number, so it can slide around. Written with standard vectors (keys and queries are column vectors), the score is (k_i^\top q_t). Then

[
\mathbf{o}t
= \sum{i\le t} \mathbf{v}i\,(\mathbf{k}_i^\top\mathbf{q}_t)
= \left(\sum{i\le t} \mathbf{v}_i\mathbf{k}_i^\top\right)\mathbf{q}_t.
]

Outer product intuition (why this suddenly looks “recurrent”)

An outer product (\mathbf{v}_i\mathbf{k}_i^\top) produces a matrix. That means we can pack all past key/value associations into one matrix:

[
\mathbf{S}t = \sum{i\le t} \mathbf{v}_i\mathbf{k}_i^\top.
]

Then output is just a read from the state:

[
\mathbf{o}_t = \mathbf{S}_t\mathbf{q}_t.
]

And because (\mathbf{S}_t) is a cumulative sum, it has the simple incremental update form:

[
\mathbf{S}t = \mathbf{S}{t-1} + \mathbf{v}_t\mathbf{k}_t^\top.
]

This is the heart of linear attention: the attention mechanism turns into a scan that repeatedly performs a rank-1 matrix update (rank-1 means “constructed from one vector–vector interaction”).

The hidden problem: additive memory never says “forget”

At this stage, linear attention is computationally attractive, but it behaves like an always-on associative memory. It never forgets.

That matters because the matrix (\mathbf{S}_t) keeps accumulating past associations, so later tokens see an increasingly tangled mix of everything that came before. The mechanism has no learned policy for:

• which memories should decay,
• which should be overwritten,
• and which should be protected.

So even though we achieved constant-memory inference, we likely lost the training signal softmax attention naturally provides via normalization.

A natural next idea is to treat the state update as an online learning step. That’s where the delta rule enters.

DeltaNet: update the memory by “correcting toward the right value”

Think of (\mathbf{S}) as a learnable mapping from keys to values. When a new key (\mathbf{k}_t) arrives with an associated target value (\mathbf{v}_t), we want the state to satisfy:

[
\mathbf{S}^\top\mathbf{k}_t \approx \mathbf{v}_t.
]

A beginner-friendly way to say that: “If we query the memory with (\mathbf{k}_t), the result should look like (\mathbf{v}_t).”

Now define a reconstruction loss:

[
\mathcal{L}_t(\mathbf{S}) = \tfrac{1}{2}\left\lVert \mathbf{S}^\top\mathbf{k}_t - \mathbf{v}_t\right\rVert^2.
]

Take a gradient descent step with a learning-rate-like scalar (\beta_t). The delta-rule structure produces a memory update of the form:

[
\mathbf{S}t = (\mathbf{I} - \beta_t\,\mathbf{k}_t\mathbf{k}_t^\top)\mathbf{S}{t-1} + \beta_t\, \mathbf{k}_t\mathbf{v}_t^\top.
]

So we get a two-part update:

1. Correction / erase along a key direction via ((\mathbf{I}-\beta_t\,\mathbf{k}t\mathbf{k}_t^\top)\mathbf{S}{t-1}).
2. Write the new association via (\beta_t\,\mathbf{k}_t\mathbf{v}_t^\top).

This is already more controlled than plain additive linear attention: the update is shaped by the current key.

And as a practical note: this line of work has a specific motivation around hardware efficiency and parallelizable chunkwise training. (arxiv.org)

Gated DeltaNet: now give the model a forgetting knob

The delta-rule update still doesn’t fully answer one question: How quickly should old information fade?

Gated DeltaNet introduces a forget gate (\alpha_t\in[0,1]). The idea is that memory should be scaled down over time so interference doesn’t build up.

For the gated delta rule, the state update becomes:

[
\mathbf{S}t = \alpha_t\,(\mathbf{I}-\beta_t\,\mathbf{k}_t\mathbf{k}_t^\top)\mathbf{S}{t-1} + \beta_t\,\mathbf{k}_t\mathbf{v}_t^\top.
]

This looks simple, but it’s conceptually huge. (\alpha_t) is a learned, data-dependent “keep amount.” If (\alpha_t) is near 0, the memory almost wipes itself before writing the new association. If (\alpha_t) is near 1, the model keeps most of the previous state.

Why this still isn’t enough

A scalar (\alpha_t) says “forget at one rate for the whole memory.” But the memory matrix (\mathbf{S}) contains many feature dimensions—some patterns might need to persist longer than others.

So the forgetting policy should be able to act per feature.

That’s exactly what Kimi Delta Attention improves.

Kimi Delta Attention (KDA): diagonal, channel-wise forgetting

Kimi Delta Attention refines the Gated DeltaNet forget gate by making it fine-grained.

Instead of a scalar forget gate, KDA uses a diagonal gate (\mathrm{Diag}(\alpha_t)). A quick definition: a diagonal gate is a diagonal matrix whose diagonal entries are the components of (\alpha_t). Multiplying by it scales each feature dimension independently.

The KDA recurrent state update (single head) is:

[
\mathbf{S}t = (\mathbf{I} - \beta_t\,\mathbf{k}_t\mathbf{k}_t^\top)\,\mathrm{Diag}(\alpha_t)\,\mathbf{S}{t-1} + \beta_t\,\mathbf{k}_t\mathbf{v}_t^\top
]

and the output is a read:

[
\mathbf{o}_t = \mathbf{S}_t^\top\,\mathbf{q}_t.
]

That’s the clean “shape” of KDA: the delta-rule correction still happens, but the previous memory is first filtered through a per-channel diagonal decay.

You can interpret this as positional awareness and memory lifespan control being encoded directly into the recurrence, not bolted on as an afterthought. (arxiv.org)

Why you could have come up with it

Once you’ve seen the chain

[
\text{linear attention (fast weights)} \rightarrow \text{delta rule (online correction)} \rightarrow \text{gating (controlled forgetting)}
]

KDA is almost inevitable. The only remaining “unforced error” is the granularity mismatch: scalar forgetting is too blunt, so make forgetting diagonal.

So the name “Kimi Delta Attention” is, in a sense, less important than the design pattern:

• Store key/value associations in a matrix state.
• Correct that state toward the newest association using a delta-rule update.
• Filter old state through learned gates so only useful memories survive.
• Refine the filter granularity when interference control needs to be sharper.

Where this ends up in real models

Kimi Delta Attention doesn’t live alone. Real architectures often use hybrids: mostly linear attention layers for speed and memory, with periodic full attention layers to keep global retrieval pathways alive.

Kimi Linear’s technical report describes KDA as extending Gated DeltaNet with channel-wise decay (diagonal gating) and reports a hybrid setup that alternates KDA and full attention in a fixed ratio. (arxiv.org)

This hybrid strategy echoes what’s been adopted in other recent model families using the same DeltaNet-era building blocks. (github.com)

So if KDA looks like “a memory recurrence with a smarter erase,” that’s not far from the truth. The recurrence is the mechanism; the gates are the policy.

The mental model that makes KDA stop feeling mysterious

A fully intuitive summary of the recurrence looks like this:

• (\mathbf{S}_t) is an associative table that stores transient key→value relationships.
• The delta term ((\mathbf{I}-\beta_t\mathbf{k}_t\mathbf{k}_t^\top)) partially clears directions aligned with the current key, so the new association can take over rather than blend forever.
• (\mathrm{Diag}(\alpha_t)) performs feature-wise forgetting, so different subspaces can retain different lifetimes.
• The write term (\beta_t\mathbf{k}_t\mathbf{v}_t^\top) inserts the newest association.
• Reading is always (\mathbf{o}_t = \mathbf{S}_t^\top\mathbf{q}_t): ask the state a question and get a value vector.

And that last part leads to a search-friendly question people often have right away: Why does linear attention feel less selective than softmax? The short answer is that softmax’s normalization couples all past tokens at once, while linear attention replaces that coupling with a compressed state update. KDA’s job is to make that compressed update smarter—through correction (delta rule) and selective forgetting (diagonal gating).

KDA doesn’t magically resurrect softmax’s exact behavior. Instead, it rebuilds the missing behaviors as learned dynamics in the recurrence.

If there’s a single lesson to carry out of this derivation, it’s that the equations aren’t pulled from a hat. They fall out of a simple design loop: compress attention into a state, interpret updates as online learning, and then add just enough forgetting granularity to prevent interference from running the show.