Kimi K2 · Query content up-projection

61 transformer blocks › 60 MoE blocks › Latent attention › Query heads

Query content up-projection

learnedself.q_b_proj(self.q_a_layernorm(self.q_a_proj(hidden_states)))

Map the latent to 64 × 128 content features.

{\htmlData{light=content-flat content-up}{{\htmlClass{vl-stream-q}{q^{C}}}}}={\htmlData{light=latent}{\hat c^{{\htmlClass{vl-stream-q}{Q}}}}}W_{UQ}

in: $\hat c^{Q}$ [B batch, T positions, 1536 q latent]
out: $q^{C}$ [B batch, T positions, 8192 heads × content]
parameters: 12,582,912 · × 60 MoE blocks = 754,974,720

Parameters

12,582,912 parameters per instance.

Exact per-instance accounting
Component	Calculation	Parameters
Weight matrix	$1{,}536 \times 8{,}192$	12,582,912
Per-instance total		12,582,912

This projection has no bias parameter.

The reference module holds the weight output by input, [out, in].

Across 60 independent MoE blocks: $12{,}582{,}912 \times 60 = 754{,}974{,}720$ parameters.

Derived from the configured matrix dimensions; the model total matches the checkpoint’s tensors.

Explanation

$W_{UQ}$ is output features 0–127 of each head’s 192 in q_b_proj.

Codemodeling_deepseek.py 662–688, 767–774

modeling_deepseek.py 662–688

            self.q_a_proj = nn.Linear(
                self.hidden_size, config.q_lora_rank, bias=config.attention_bias
            )
            self.q_a_layernorm = DeepseekV3RMSNorm(config.q_lora_rank)
            self.q_b_proj = nn.Linear(
                config.q_lora_rank, self.num_heads * self.q_head_dim, bias=False
            )

        self.kv_a_proj_with_mqa = nn.Linear(
            self.hidden_size,
            config.kv_lora_rank + config.qk_rope_head_dim,
            bias=config.attention_bias,
        )
        self.kv_a_layernorm = DeepseekV3RMSNorm(config.kv_lora_rank)
        self.kv_b_proj = nn.Linear(
            config.kv_lora_rank,
            self.num_heads
            * (self.q_head_dim - self.qk_rope_head_dim + self.v_head_dim),
            bias=False,
        )

        self.o_proj = nn.Linear(
            self.num_heads * self.v_head_dim,
            self.hidden_size,
            bias=config.attention_bias,
        )
        self._init_rope()

modeling_deepseek.py 767–774

        if self.q_lora_rank is None:
            q = self.q_proj(hidden_states)
        else:
            q = self.q_b_proj(self.q_a_layernorm(self.q_a_proj(hidden_states)))
        q = q.view(bsz, q_len, self.num_heads, self.q_head_dim).transpose(1, 2)
        q_nope, q_pe = torch.split(
            q, [self.qk_nope_head_dim, self.qk_rope_head_dim], dim=-1
        )

Sources2

Direct evidence

MLA owns a query down-projection, query RMSNorm and up-projection, a joint key–value down-projection that also emits the shared rotary key, a latent RMSNorm, a key–value up-projection and an output projection.

mla-init · moonshotai/Kimi-K2-Instruct / modeling_deepseek.py
Pinned revision fd1984e2b7a3 ↗ · lines 662–688

Read code excerpt

            self.q_a_proj = nn.Linear(
                self.hidden_size, config.q_lora_rank, bias=config.attention_bias
            )
            self.q_a_layernorm = DeepseekV3RMSNorm(config.q_lora_rank)
            self.q_b_proj = nn.Linear(
                config.q_lora_rank, self.num_heads * self.q_head_dim, bias=False
            )

        self.kv_a_proj_with_mqa = nn.Linear(
            self.hidden_size,
            config.kv_lora_rank + config.qk_rope_head_dim,
            bias=config.attention_bias,
        )
        self.kv_a_layernorm = DeepseekV3RMSNorm(config.kv_lora_rank)
        self.kv_b_proj = nn.Linear(
            config.kv_lora_rank,
            self.num_heads
            * (self.q_head_dim - self.qk_rope_head_dim + self.v_head_dim),
            bias=False,
        )

        self.o_proj = nn.Linear(
            self.num_heads * self.v_head_dim,
            self.hidden_size,
            bias=config.attention_bias,
        )
        self._init_rope()

Direct evidence

Queries are produced through a low-rank latent and split into a non-rotated and a rotated part.

mla-queries · moonshotai/Kimi-K2-Instruct / modeling_deepseek.py
Pinned revision fd1984e2b7a3 ↗ · lines 767–774

Read code excerpt

        if self.q_lora_rank is None:
            q = self.q_proj(hidden_states)
        else:
            q = self.q_b_proj(self.q_a_layernorm(self.q_a_proj(hidden_states)))
        q = q.view(bsz, q_len, self.num_heads, self.q_head_dim).transpose(1, 2)
        q_nope, q_pe = torch.split(
            q, [self.qk_nope_head_dim, self.qk_rope_head_dim], dim=-1
        )