Kimi K2

All models, with their blocks →

Size

Published

Moonshot AI · Jul 2025

Line

Latent attention and experts

View

Architecture

Source

moonshotai/Kimi‑K2‑Instruct @ fd1984e · parameter count checked against the checkpoint’s tensor headersHow to read

Mix and project

Full sequence Prefill Decode

A[B, 64, T, T]V[B, 64, T, 128]Retrieve

AV

O[B, 64, T, 128]Merge heads o[B, T, 8192]Output projection

\Delta_{\mathrm{attn}}=oW_{O}

Δ_attn[B, T, 7168]

61 transformer blocks › 60 MoE blocks › Latent attention › Causal attention › Mix and project

Merge heads

no parametersattn_output.reshape(bsz, q_len, self.num_heads * self.v_head_dim)

Concatenate the heads at each position.

[B,\,64,\,T,\,128]

\to\;[B,\,T,\,64,\,128]

\to\;[B,\,T,\,8192]

in: $O$ [B batch, 64 heads, T positions, 128 value]
out: $o$ [B batch, T positions, 8192 heads × value]

Explanation

Heads are concatenated in order: head $i$ fills features ${128i},\dots,{128i+127}$ .

Codemodeling_deepseek.py 848–852

modeling_deepseek.py 848–852

        attn_output = attn_output.transpose(1, 2).contiguous()

        attn_output = attn_output.reshape(bsz, q_len, self.num_heads * self.v_head_dim)

        attn_output = self.o_proj(attn_output)

Sources1

Direct evidence

Head outputs are flattened and projected back to the model width.

mla-output · moonshotai/Kimi-K2-Instruct / modeling_deepseek.py
Pinned revision fd1984e2b7a3 ↗ · lines 848–852

Read code excerpt

        attn_output = attn_output.transpose(1, 2).contiguous()

        attn_output = attn_output.reshape(bsz, q_len, self.num_heads * self.v_head_dim)

        attn_output = self.o_proj(attn_output)