How Linear Attention Evolved from Softmax to Kimi Delta Attention

This technical walkthrough demystifies the DeltaNet family of linear attention mechanisms by deriving them step by step, showing that each variant is a small, motivated fix to the one before it. It starts with ordinary softmax attention, whose cost grows quadratically with sequence length because every query must compare against every past key. Dropping the softmax lets the past collapse into a single fixed-size state matrix built from summed key-value outer products, making the computation linear in sequence length. The catch is that this recurrent write behaves like a += accumulation rather than an assignment, so overlapping keys interfere and corrupt stored associations.

DeltaNet fixes the write by applying a delta rule: instead of blindly adding a value, it first reads what the memory currently returns for the new key and writes only the error, scaled by a learned strength. The author shows this is equivalent to a single gradient-descent step on a reconstruction loss, unifying three views—memory replacement, online learning, and a rank-one linear-algebra update. Gated DeltaNet adds a scalar retention gate that erases stale state globally before the delta correction, while Kimi Delta Attention (KDA) promotes that scalar to a per-channel vector, so different key channels can be forgotten at different rates. This produces a diagonal-plus-low-rank state transition.

The significance is practical: Gated DeltaNet and KDA now power the latest Qwen and Kimi model families, and the piece argues these seemingly inaccessible equations are reachable through simple, intuitive assumptions about how a hidden state should behave. It closes by pointing toward the recurrent and chunkwise Triton kernels that actually execute KDA efficiently, grounding the theory in real implementation.