Kimi K3: The First Frontier Model to Drop RoPE Entirely for NoPE

Kimi K3, released as an open-weight model, is a production-scale version of last year’s Kimi Linear, blown up from 48B to 2.8 trillion parameters — currently the largest open-weight model available. Rather than a ground-up redesign, it swaps in efficiency-focused replacements for standard components: LatentMoE stands in for a conventional mixture-of-experts (down-projecting large linear layers, the same trick seen in Nemotron 3 Ultra), while multi-head latent attention and Kimi Delta Attention replace regular attention. The through-line matches what DeepSeek V4 and Nemotron 3 are doing — squeezing better inference efficiency out of each block.

Two design choices stand out beyond raw efficiency. The architecture uses ‘attention residuals,’ which link residual paths across layers and weight each connection by an attention score; the technical report claims small but consistent gains in validation loss and downstream performance for roughly 4% added training cost and 2% added inference cost. More notably, K3 abandons rotary positional embeddings (RoPE) across every layer in favor of NoPE (no positional embeddings). While a few smaller models have gone fully NoPE, K3 appears to be the first frontier-scale model to do so, cutting against the recent trend of RoPE in local-attention layers and NoPE in global ones.

K3 also ships with native multimodal support. Both the NoPE-everywhere decision and the attention-residual mechanism are inherited from Kimi Linear rather than new to K3, underscoring that the release is more about scaling a proven design than inventing a new one.