Kimi K2 · MoE transformer block

in: $x_n$ [B batch, T positions, 7168 d] $R$ [T positions, 32 pairs] $M$ [T queries, T keys]
out: $x_{n+1}$ [B batch, T positions, 7168 d]
parameters: 17,059,365,248 · × 60 MoE blocks = 1,023,561,914,880

Parameters

17,059,365,248 parameters per MoE block, across its learned components

Exact per-instance accounting
Component	Calculation	Parameters
Attention RMSNorm	$7{,}168$	7,168
Multi-head latent attention	101,124,096 across its components	101,124,096
MoE RMSNorm	$7{,}168$	7,168
Mixture of experts	16,958,226,816 across its components	16,958,226,816
Per-instance total		17,059,365,248

Across 60 independent MoE blocks: $17{,}059{,}365{,}248 \times 60 = 1{,}023{,}561{,}914{,}880$ parameters.

Derived from the pinned configuration and implementation, and equal to the count of the checkpoint’s tensors read from its safetensors headers (FP8 block scales and stored rotary buffers excluded).

In numbers

Weights · one block, by sublayer

MoE block · FP815.9 GiB

Attention · FP896.4 MiB

MoE · FP815.8 GiB

Two RMSNorms · bfloat1628 KiB

17,059,365,248 parameters per block: attention 101,124,096 (0.59%), the MoE 16,958,226,816 (99.4%), the two RMSNorms 14,336; × 60 MoE blocks = 1,023,561,914,880.

config.json 20–23 config.json 29 modeling_deepseek.py 1152–1166 modeling_deepseek.py 1198–1218 config.json 64 config.json 38–46 modeling_deepseek.py 409–415

Codemodeling_deepseek.py 1152–1166, 1198–1218

modeling_deepseek.py 1152–1166

        self.mlp = (
            DeepseekV3MoE(config)
            if (
                config.n_routed_experts is not None
                and layer_idx >= config.first_k_dense_replace
                and layer_idx % config.moe_layer_freq == 0
            )
            else DeepseekV3MLP(config)
        )
        self.input_layernorm = DeepseekV3RMSNorm(
            config.hidden_size, eps=config.rms_norm_eps
        )
        self.post_attention_layernorm = DeepseekV3RMSNorm(
            config.hidden_size, eps=config.rms_norm_eps
        )

modeling_deepseek.py 1198–1218

        residual = hidden_states

        hidden_states = self.input_layernorm(hidden_states)

        # Self Attention
        hidden_states, self_attn_weights, present_key_value = self.self_attn(
            hidden_states=hidden_states,
            attention_mask=attention_mask,
            position_ids=position_ids,
            past_key_value=past_key_value,
            output_attentions=output_attentions,
            use_cache=use_cache,
            **kwargs,
        )
        hidden_states = residual + hidden_states

        # Fully Connected
        residual = hidden_states
        hidden_states = self.post_attention_layernorm(hidden_states)
        hidden_states = self.mlp(hidden_states)
        hidden_states = residual + hidden_states

Sources1

Direct evidence

A block applies input RMSNorm, attention and a residual addition, then post-attention RMSNorm, the FFN and a second residual addition; the first first_k_dense_replace blocks use the dense MLP, the rest MoE.

block · moonshotai/Kimi-K2-Instruct / modeling_deepseek.py
Pinned revision fd1984e2b7a3 ↗ · lines 1152–1218

Read code excerpt

        self.mlp = (
            DeepseekV3MoE(config)
            if (
                config.n_routed_experts is not None
                and layer_idx >= config.first_k_dense_replace
                and layer_idx % config.moe_layer_freq == 0
            )
            else DeepseekV3MLP(config)
        )
        self.input_layernorm = DeepseekV3RMSNorm(
            config.hidden_size, eps=config.rms_norm_eps
        )
        self.post_attention_layernorm = DeepseekV3RMSNorm(
            config.hidden_size, eps=config.rms_norm_eps
        )

… lines 1167–1197 omitted …

        residual = hidden_states

        hidden_states = self.input_layernorm(hidden_states)

        # Self Attention
        hidden_states, self_attn_weights, present_key_value = self.self_attn(
            hidden_states=hidden_states,
            attention_mask=attention_mask,
            position_ids=position_ids,
            past_key_value=past_key_value,
            output_attentions=output_attentions,
            use_cache=use_cache,
            **kwargs,
        )
        hidden_states = residual + hidden_states

        # Fully Connected
        residual = hidden_states
        hidden_states = self.post_attention_layernorm(hidden_states)
        hidden_states = self.mlp(hidden_states)
        hidden_states = residual + hidden_states