DeepSeek-V3.2-Exp · Absorb key weights

61 transformer blocks › 58 MoE blocks › Sparse latent attention › Query heads

Absorb key weights

reuses weightsq_nope = torch.einsum("bshd,hdc->bshc", q_nope, wkv_b[:, :self.qk_nope_head_dim])

Fold $W_{UK}$ into each head’s content query: 128 content features become 512 latent features.

{\htmlData{light=absorbed absorb}{\tilde {\htmlClass{vl-stream-q}{Q^{C}_i}}}}={\htmlData{light=content content-heads}{{\htmlClass{vl-stream-q}{Q^{C}_i}}}}\,W_{UK,i}^{\mathsf T}

in: $Q^{C}$ [B batch, 128 heads, 1 position, 128 content]
out: $\tilde Q^{C}$ [B batch, 128 heads, 1 position, 512 kv latent]

Applies $W_{UK}$ , owned by the key heads’ content up-projection (wkv_b), dequantized once for decoding (model.py 591–595); it owns no weights of its own.

Parameters

Reuses Key content up-projection: weights are counted there, with no additional parameter ownership here.

Explanation

The content keys carry no rotation, so $Q^{C}_i(\hat c^{KV}W_{UK,i})^{\mathsf T}=(Q^{C}_iW_{UK,i}^{\mathsf T})\,\hat c^{KV\mathsf T}$ : the scores are the same without reconstructing any key.

Codemodel.py 590–606

model.py 590–606

        else:                   # MQA decode
            if self.dequant_wkv_b is None and self.wkv_b.scale is not None:
                self.dequant_wkv_b = weight_dequant(self.wkv_b.weight, self.wkv_b.scale)
            wkv_b = self.wkv_b.weight if self.dequant_wkv_b is None else self.dequant_wkv_b
            wkv_b = wkv_b.view(self.n_local_heads, -1, self.kv_lora_rank)
            q_nope = torch.einsum("bshd,hdc->bshc", q_nope, wkv_b[:, :self.qk_nope_head_dim])
            scores = (torch.einsum("bshc,btc->bsht", q_nope, self.kv_cache[:bsz, :end_pos]) +
                      torch.einsum("bshr,btr->bsht", q_pe, self.pe_cache[:bsz, :end_pos])) * self.softmax_scale

            # indexer
            topk_indices = self.indexer(x, qr, start_pos, freqs_cis, mask)
            index_mask = torch.full((bsz, 1, end_pos), float("-inf"), device=x.device).scatter_(-1, topk_indices, 0)
            scores += index_mask.unsqueeze(2)

            scores = scores.softmax(dim=-1)
            x = torch.einsum("bsht,btc->bshc", scores, self.kv_cache[:bsz, :end_pos])
            x = torch.einsum("bshc,hdc->bshd", x, wkv_b[:, -self.v_head_dim:])

Sources1

Direct evidence

When decoding one token, the implementation absorbs the key–value up-projection into queries and outputs and attends over the cached latent directly.

mla-decode · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 590–606

Read code excerpt

        else:                   # MQA decode
            if self.dequant_wkv_b is None and self.wkv_b.scale is not None:
                self.dequant_wkv_b = weight_dequant(self.wkv_b.weight, self.wkv_b.scale)
            wkv_b = self.wkv_b.weight if self.dequant_wkv_b is None else self.dequant_wkv_b
            wkv_b = wkv_b.view(self.n_local_heads, -1, self.kv_lora_rank)
            q_nope = torch.einsum("bshd,hdc->bshc", q_nope, wkv_b[:, :self.qk_nope_head_dim])
            scores = (torch.einsum("bshc,btc->bsht", q_nope, self.kv_cache[:bsz, :end_pos]) +
                      torch.einsum("bshr,btr->bsht", q_pe, self.pe_cache[:bsz, :end_pos])) * self.softmax_scale

            # indexer
            topk_indices = self.indexer(x, qr, start_pos, freqs_cis, mask)
            index_mask = torch.full((bsz, 1, end_pos), float("-inf"), device=x.device).scatter_(-1, topk_indices, 0)
            scores += index_mask.unsqueeze(2)

            scores = scores.softmax(dim=-1)
            x = torch.einsum("bsht,btc->bshc", scores, self.kv_cache[:bsz, :end_pos])
            x = torch.einsum("bshc,hdc->bshd", x, wkv_b[:, -self.v_head_dim:])