Kimi K2

All models, with previews →

Moonshot AI · Jul 2025Latent attention and experts

Transformer blocks

Checkpoint configuration and implementation moonshotai/Kimi-K2-Instruct @ fd1984e · parameter count checked against the checkpoint’s tensor headers · How to read

61 transformer blocks

MoE transformer block

Open inside →×

learnedDeepseekV3DecoderLayer(config, layer_idx)

MLA and a mixture of experts, each added to the residual stream; blocks 2–61.

xn′=xn+Δattnx'_n=x_n+\Delta_{\mathrm{attn}}
xn+1=xn′+Δffnx_{n+1}=x'_n+\Delta_{\mathrm{ffn}}
All 39 steps
xattn=RMSNorm⁡(xn)x_{\mathrm{attn}}=\operatorname{RMSNorm}(x_n)
Multi-head latent attention:\text{Multi-head latent attention:}
cQ=xWDQc^{{\htmlClass{vl-stream-q}{Q}}}=xW_{DQ}
c^Q=RMSNorm⁡(cQ)\hat c^{{\htmlClass{vl-stream-q}{Q}}}=\operatorname{RMSNorm}(c^{{\htmlClass{vl-stream-q}{Q}}})
cKV=xWDKVc^{KV}=xW_{DKV}
c^KV=RMSNorm⁡(cKV)\hat c^{KV}=\operatorname{RMSNorm}(c^{KV})
qC=c^QWUQ{\htmlClass{vl-stream-q}{q^{C}}}=\hat c^{{\htmlClass{vl-stream-q}{Q}}}W_{UQ}
QC=reshape⁡(qC){\htmlClass{vl-stream-q}{Q^{C}}}=\operatorname{reshape}({\htmlClass{vl-stream-q}{q^{C}}})
qpe=c^QWQR{\htmlClass{vl-stream-q}{q^{\mathrm{pe}}}}=\hat c^{{\htmlClass{vl-stream-q}{Q}}}W_{QR}
Qpe=reshape⁡(qpe){\htmlClass{vl-stream-q}{Q^{\mathrm{pe}}}}=\operatorname{reshape}({\htmlClass{vl-stream-q}{q^{\mathrm{pe}}}})
QtR=RoPE⁡(Qtpe,Rt){\htmlClass{vl-stream-q}{Q^{R}_{t}}}=\operatorname{RoPE}({\htmlClass{vl-stream-q}{Q^{\mathrm{pe}}_{t}}},{\htmlClass{vl-stream-context}{R_t}})
Q=[QC; QR]{\htmlClass{vl-stream-q}{Q}}=[{\htmlClass{vl-stream-q}{Q^{C}}};\,{\htmlClass{vl-stream-q}{Q^{R}}}]
kC=c^KVWUK{\htmlClass{vl-stream-k}{k^{C}}}=\hat c^{KV}W_{UK}
KC=reshape⁡(kC){\htmlClass{vl-stream-k}{K^{C}}}=\operatorname{reshape}({\htmlClass{vl-stream-k}{k^{C}}})
kpe=xWKR{\htmlClass{vl-stream-k}{k^{\mathrm{pe}}}}=xW_{KR}
ktR=RoPE⁡(ktpe,Rt){\htmlClass{vl-stream-k}{k^{R}_{t}}}=\operatorname{RoPE}({\htmlClass{vl-stream-k}{k^{\mathrm{pe}}_{t}}},{\htmlClass{vl-stream-context}{R_t}})
Ki=[KiC; kR]{\htmlClass{vl-stream-k}{K_i}}=[{\htmlClass{vl-stream-k}{K^{C}_i}};\,{\htmlClass{vl-stream-k}{k^{R}}}]
v=c^KVWUV{\htmlClass{vl-stream-v}{v}}=\hat c^{KV}W_{UV}
V=reshape⁡(v){\htmlClass{vl-stream-v}{V}}=\operatorname{reshape}({\htmlClass{vl-stream-v}{v}})
xn′=xn+Δattnx'_n=x_n+\Delta_{\mathrm{attn}}
xffn=RMSNorm⁡(xn′)x_{\mathrm{ffn}}=\operatorname{RMSNorm}(x'_n)
Mixture of experts:\text{Mixture of experts:}
z=xffnWrz=x_{\mathrm{ffn}}W_{r}
s=σ(z)s=\sigma(z)
Eisel=[ i∈top⁡8(s+b) ]E^{\mathrm{sel}}_i=\big[\,i\in\operatorname{top}_{8}(s+b)\,\big]
Gi=2.827 si Eisel∑jsj EjselG_{i}=2.827\,\frac{s_i\,E^{\mathrm{sel}}_i}{\sum_{j}s_j\,E^{\mathrm{sel}}_j}
g=xffnW1g=x_{\mathrm{ffn}}W_{1}
u=xffnW3u=x_{\mathrm{ffn}}W_{3}
a=g σ(g)a=g\,\sigma(g)
h=a⊙uh=a\odot u
Ei=hW2E_i=hW_{2}
yrouted=∑i=1384Gi Eiy^{\mathrm{routed}}=\sum_{i=1}^{384}G_i\,E_i
g=xffnW1g=x_{\mathrm{ffn}}W_{1}
u=xffnW3u=x_{\mathrm{ffn}}W_{3}
a=g σ(g)a=g\,\sigma(g)
h=a⊙uh=a\odot u
yshared=hW2y^{\mathrm{shared}}=hW_{2}
Δffn=yrouted+yshared\Delta_{\mathrm{ffn}}=y^{\mathrm{routed}}+y^{\mathrm{shared}}
xn+1=xn′+Δffnx_{n+1}=x'_n+\Delta_{\mathrm{ffn}}
in
x1x_{1} [B batch, T positions, 7168 d]RR [T positions, 32 pairs]MM [T queries, T keys]
out
xn+1x_{n+1} [B batch, T positions, 7168 d]
parameters
17,059,365,248 · × 60 MoE blocks = 1,023,561,914,880
Parameters

17,059,365,248 parameters per MoE block, across its learned components

Exact per-instance accounting
ComponentCalculationParameters
Attention RMSNorm7,1687{,}1687,168
Multi-head latent attention101,124,096 across its components101,124,096
MoE RMSNorm7,1687{,}1687,168
Mixture of experts16,958,226,816 across its components16,958,226,816
Per-instance total17,059,365,248

Across 60 independent MoE blocks: 17,059,365,248×60=1,023,561,914,88017{,}059{,}365{,}248 \times 60 = 1{,}023{,}561{,}914{,}880 parameters.

Derived from the pinned configuration and implementation, and equal to the count of the checkpoint’s tensors read from its safetensors headers (FP8 block scales and stored rotary buffers excluded).

Codemodeling_deepseek.py 1152–1166, 1198–1218
modeling_deepseek.py 1152–1166
        self.mlp = (
            DeepseekV3MoE(config)
            if (
                config.n_routed_experts is not None
                and layer_idx >= config.first_k_dense_replace
                and layer_idx % config.moe_layer_freq == 0
            )
            else DeepseekV3MLP(config)
        )
        self.input_layernorm = DeepseekV3RMSNorm(
            config.hidden_size, eps=config.rms_norm_eps
        )
        self.post_attention_layernorm = DeepseekV3RMSNorm(
            config.hidden_size, eps=config.rms_norm_eps
        )
modeling_deepseek.py 1198–1218
        residual = hidden_states

        hidden_states = self.input_layernorm(hidden_states)

        # Self Attention
        hidden_states, self_attn_weights, present_key_value = self.self_attn(
            hidden_states=hidden_states,
            attention_mask=attention_mask,
            position_ids=position_ids,
            past_key_value=past_key_value,
            output_attentions=output_attentions,
            use_cache=use_cache,
            **kwargs,
        )
        hidden_states = residual + hidden_states

        # Fully Connected
        residual = hidden_states
        hidden_states = self.post_attention_layernorm(hidden_states)
        hidden_states = self.mlp(hidden_states)
        hidden_states = residual + hidden_states
Sources1

Direct evidence

A block applies input RMSNorm, attention and a residual addition, then post-attention RMSNorm, the FFN and a second residual addition; the first first_k_dense_replace blocks use the dense MLP, the rest MoE.

block · moonshotai/Kimi-K2-Instruct / modeling_deepseek.py
Pinned revision fd1984e2b7a3 ↗ · lines 1152–1218

Read code excerpt
        self.mlp = (
            DeepseekV3MoE(config)
            if (
                config.n_routed_experts is not None
                and layer_idx >= config.first_k_dense_replace
                and layer_idx % config.moe_layer_freq == 0
            )
            else DeepseekV3MLP(config)
        )
        self.input_layernorm = DeepseekV3RMSNorm(
            config.hidden_size, eps=config.rms_norm_eps
        )
        self.post_attention_layernorm = DeepseekV3RMSNorm(
            config.hidden_size, eps=config.rms_norm_eps
        )

… lines 1167–1197 omitted …

        residual = hidden_states

        hidden_states = self.input_layernorm(hidden_states)

        # Self Attention
        hidden_states, self_attn_weights, present_key_value = self.self_attn(
            hidden_states=hidden_states,
            attention_mask=attention_mask,
            position_ids=position_ids,
            past_key_value=past_key_value,
            output_attentions=output_attentions,
            use_cache=use_cache,
            **kwargs,
        )
        hidden_states = residual + hidden_states

        # Fully Connected
        residual = hidden_states
        hidden_states = self.post_attention_layernorm(hidden_states)
        hidden_states = self.mlp(hidden_states)
        hidden_states = residual + hidden_states