Kimi K2 · Dense transformer block

in: $x_n$ [B batch, T positions, 7168 d] $R$ [T positions, 32 pairs] $M$ [T queries, T keys]
out: $x_{n+1}$ [B batch, T positions, 7168 d]
parameters: 497,500,160

Parameters

497,500,160 parameters per dense block, across its learned components

Exact per-instance accounting
Component	Calculation	Parameters
Attention RMSNorm	$7{,}168$	7,168
Multi-head latent attention	101,124,096 across its components	101,124,096
MLP RMSNorm	$7{,}168$	7,168
SwiGLU MLP	396,361,728 across its components	396,361,728
Per-instance total		497,500,160

Across 1 independent dense blocks: $497{,}500{,}160 \times 1 = 497{,}500{,}160$ parameters.

Derived from the pinned configuration and implementation, and equal to the count of the checkpoint’s tensors read from its safetensors headers (FP8 block scales and stored rotary buffers excluded).

In numbers

Weights · one block, by sublayer

Dense block · FP8474.5 MiB

Attention · FP896.4 MiB

MLP · FP8378 MiB

Two RMSNorms · bfloat1628 KiB

497,500,160 parameters per block: attention 101,124,096 (20.3%), the MLP 396,361,728 (79.7%), the two RMSNorms 14,336; × 1 dense block = 497,500,160.

config.json 19–20 config.json 29 modeling_deepseek.py 1152–1166 modeling_deepseek.py 1198–1218 config.json 64 config.json 38–46

Codemodeling_deepseek.py 1152–1166, 1198–1218

modeling_deepseek.py 1152–1166

        self.mlp = (
            DeepseekV3MoE(config)
            if (
                config.n_routed_experts is not None
                and layer_idx >= config.first_k_dense_replace
                and layer_idx % config.moe_layer_freq == 0
            )
            else DeepseekV3MLP(config)
        )
        self.input_layernorm = DeepseekV3RMSNorm(
            config.hidden_size, eps=config.rms_norm_eps
        )
        self.post_attention_layernorm = DeepseekV3RMSNorm(
            config.hidden_size, eps=config.rms_norm_eps
        )

modeling_deepseek.py 1198–1218

        residual = hidden_states

        hidden_states = self.input_layernorm(hidden_states)

        # Self Attention
        hidden_states, self_attn_weights, present_key_value = self.self_attn(
            hidden_states=hidden_states,
            attention_mask=attention_mask,
            position_ids=position_ids,
            past_key_value=past_key_value,
            output_attentions=output_attentions,
            use_cache=use_cache,
            **kwargs,
        )
        hidden_states = residual + hidden_states

        # Fully Connected
        residual = hidden_states
        hidden_states = self.post_attention_layernorm(hidden_states)
        hidden_states = self.mlp(hidden_states)
        hidden_states = residual + hidden_states

Sources1

Direct evidence

A block applies input RMSNorm, attention and a residual addition, then post-attention RMSNorm, the FFN and a second residual addition; the first first_k_dense_replace blocks use the dense MLP, the rest MoE.

block · moonshotai/Kimi-K2-Instruct / modeling_deepseek.py
Pinned revision fd1984e2b7a3 ↗ · lines 1152–1218

Read code excerpt

        self.mlp = (
            DeepseekV3MoE(config)
            if (
                config.n_routed_experts is not None
                and layer_idx >= config.first_k_dense_replace
                and layer_idx % config.moe_layer_freq == 0
            )
            else DeepseekV3MLP(config)
        )
        self.input_layernorm = DeepseekV3RMSNorm(
            config.hidden_size, eps=config.rms_norm_eps
        )
        self.post_attention_layernorm = DeepseekV3RMSNorm(
            config.hidden_size, eps=config.rms_norm_eps
        )

… lines 1167–1197 omitted …

        residual = hidden_states

        hidden_states = self.input_layernorm(hidden_states)

        # Self Attention
        hidden_states, self_attn_weights, present_key_value = self.self_attn(
            hidden_states=hidden_states,
            attention_mask=attention_mask,
            position_ids=position_ids,
            past_key_value=past_key_value,
            output_attentions=output_attentions,
            use_cache=use_cache,
            **kwargs,
        )
        hidden_states = residual + hidden_states

        # Fully Connected
        residual = hidden_states
        hidden_states = self.post_attention_layernorm(hidden_states)
        hidden_states = self.mlp(hidden_states)
        hidden_states = residual + hidden_states