Kimi K2

All models, with their blocks →

Size
1T
Published
Moonshot AI · Jul 2025
Line
Latent attention and experts
View
Architecture
Source

moonshotai/Kimi‑K2‑Instruct @ fd1984e · parameter count checked against the checkpoint’s tensor headersHow to read

Query heads

Full sequencePrefillDecode

61 transformer blocks › 60 MoE blocks › Latent attention › Query heads

Split rotary heads

×

no parametersq.view(bsz, q_len, self.num_heads, self.q_head_dim).transpose(1, 2)

Partition the rotary features into 64 heads.

[B, T, 4096]  →  [B, 64, T, 64][B,\,T,\,4096]\;\to\;[B,\,64,\,T,\,64]
in
qpeq^{\mathrm{pe}} [B batch, T positions, 4096 heads × rotary]
out
QpeQ^{\mathrm{pe}} [B batch, 64 heads, T positions, 64 rotary]
Codemodeling_deepseek.py 767–774
modeling_deepseek.py 767–774
        if self.q_lora_rank is None:
            q = self.q_proj(hidden_states)
        else:
            q = self.q_b_proj(self.q_a_layernorm(self.q_a_proj(hidden_states)))
        q = q.view(bsz, q_len, self.num_heads, self.q_head_dim).transpose(1, 2)
        q_nope, q_pe = torch.split(
            q, [self.qk_nope_head_dim, self.qk_rope_head_dim], dim=-1
        )
Sources1

Direct evidence

Queries are produced through a low-rank latent and split into a non-rotated and a rotated part.

mla-queries · moonshotai/Kimi-K2-Instruct / modeling_deepseek.py
Pinned revision fd1984e2b7a3 ↗ · lines 767–774

Read code excerpt
        if self.q_lora_rank is None:
            q = self.q_proj(hidden_states)
        else:
            q = self.q_b_proj(self.q_a_layernorm(self.q_a_proj(hidden_states)))
        q = q.view(bsz, q_len, self.num_heads, self.q_head_dim).transpose(1, 2)
        q_nope, q_pe = torch.split(
            q, [self.qk_nope_head_dim, self.qk_rope_head_dim], dim=-1
        )