Kimi Linear: Hybrid Attention That Beats Full Attention With 75% Less KV Cache

Moonshot AI’s Kimi Linear is a hybrid attention architecture that its authors claim is the first linear-attention design to beat full attention in head-to-head tests across short-context, long-context, and reinforcement-learning scaling regimes. The core component is Kimi Delta Attention (KDA), a refinement of Gated DeltaNet that adds finer-grained gating to squeeze more out of the fixed-size memory that recurrent-style linear attention relies on. A custom chunkwise algorithm built on a constrained Diagonal-Plus-Low-Rank transition matrix keeps the math hardware-efficient while staying close to the classical delta rule.

The team pretrained a mixture-of-experts model with 3B active and 48B total parameters, interleaving KDA layers with Multi-Head Latent Attention (MLA). Under an identical training recipe, the hybrid outperformed a pure-MLA baseline on every evaluated task while cutting KV cache usage by up to 75% and delivering up to 6x faster decoding at a 1M-token context. The efficiency gains grow with sequence length, which is the main selling point: the architecture is pitched as a drop-in replacement for full attention that gets cheaper precisely where transformers get most expensive.

The practical significance is that KV cache and decoding throughput are the dominant costs for long-context inference, so a design that reduces both without sacrificing quality is directly relevant to anyone serving large models. Moonshot has open-sourced the KDA kernel and vLLM integration and released both pretrained and instruction-tuned checkpoints, making the claims independently testable rather than paper-only.