61 transformer blocks › 58 MoE blocks › Sparse latent attention
learnedself.kv_norm(kv)
Compress each position’s 7,168 features into a 512-feature key–value latent.
Why
One 512-feature latent per position stands in for the keys and values of all 128 heads. Standard attention has to cache every head’s key and value for every token, which the authors name as the bottleneck of serving. This latent and the rotary key part are what the attention caches for a position; the indexer caches one key more (model.py 453). DeepSeek-V2 §2.1.1–2.1.3
- in
- [B batch, T positions, 7168 d]
- out
- [B batch, T positions, 512 kv latent]
- parameters
- 3,670,528 · × 58 MoE blocks = 212,890,624
Parameters
3,670,528 parameters per instance, across its learned components
| Component | Calculation | Parameters |
|---|---|---|
| Key–value down-projection | 3,670,016 | |
| Key–value latent RMSNorm | 512 | |
| Per-instance total | 3,670,528 | |
Across 58 independent MoE blocks: parameters.
Derived from the pinned reference configuration and inference code. Checkpoint tensors have not been independently counted; FP8 block scales and the multi-token-prediction module are not counted.
Explanation
The reference caches this latent and the rotary key (model.py 572–573): 576 values per position. Full per-head keys (192 features) and values (128) would take 40,960. The lightning indexer caches one more key per position, of 128 features.
One matrix, wkv_a, produces this latent and the 64-feature rotary key together, 576 output features in all; the reference splits them after the product (model.py 565–566).
Codemodel.py 523–539, 565–568
self.qk_head_dim = args.qk_nope_head_dim + args.qk_rope_head_dim
self.v_head_dim = args.v_head_dim
self.wq_a = Linear(self.dim, self.q_lora_rank)
self.q_norm = RMSNorm(self.q_lora_rank)
self.wq_b = ColumnParallelLinear(self.q_lora_rank, self.n_heads * self.qk_head_dim)
self.wkv_a = Linear(self.dim, self.kv_lora_rank + self.qk_rope_head_dim)
self.kv_norm = RMSNorm(self.kv_lora_rank)
self.wkv_b = ColumnParallelLinear(self.kv_lora_rank, self.n_heads * (self.qk_nope_head_dim + self.v_head_dim))
self.wo = RowParallelLinear(self.n_heads * self.v_head_dim, self.dim)
self.softmax_scale = self.qk_head_dim ** -0.5
self.scale_fmt = args.scale_fmt
if args.max_seq_len > args.original_seq_len:
mscale = 0.1 * args.mscale * math.log(args.rope_factor) + 1.0
self.softmax_scale = self.softmax_scale * mscale * mscale
self.indexer = Indexer(args) kv = self.wkv_a(x)
kv, k_pe = torch.split(kv, [self.kv_lora_rank, self.qk_rope_head_dim], dim=-1)
kv = self.kv_norm(kv)
k_pe = apply_rotary_emb(k_pe.unsqueeze(2), freqs_cis)Sources2
Direct evidence
MLA owns a query down-projection, query RMSNorm and up-projection, a joint key–value down-projection that also emits the shared rotary key, a latent RMSNorm, a key–value up-projection and an output projection.
mla-init · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 523–539
Read code excerpt
self.qk_head_dim = args.qk_nope_head_dim + args.qk_rope_head_dim
self.v_head_dim = args.v_head_dim
self.wq_a = Linear(self.dim, self.q_lora_rank)
self.q_norm = RMSNorm(self.q_lora_rank)
self.wq_b = ColumnParallelLinear(self.q_lora_rank, self.n_heads * self.qk_head_dim)
self.wkv_a = Linear(self.dim, self.kv_lora_rank + self.qk_rope_head_dim)
self.kv_norm = RMSNorm(self.kv_lora_rank)
self.wkv_b = ColumnParallelLinear(self.kv_lora_rank, self.n_heads * (self.qk_nope_head_dim + self.v_head_dim))
self.wo = RowParallelLinear(self.n_heads * self.v_head_dim, self.dim)
self.softmax_scale = self.qk_head_dim ** -0.5
self.scale_fmt = args.scale_fmt
if args.max_seq_len > args.original_seq_len:
mscale = 0.1 * args.mscale * math.log(args.rope_factor) + 1.0
self.softmax_scale = self.softmax_scale * mscale * mscale
self.indexer = Indexer(args)Direct evidence
The joint down-projection yields the key–value latent and one rotary key shared by all heads; the latent is normalized and the key rotated.
mla-kv · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 565–568
Read code excerpt
kv = self.wkv_a(x)
kv, k_pe = torch.split(kv, [self.kv_lora_rank, self.qk_rope_head_dim], dim=-1)
kv = self.kv_norm(kv)
k_pe = apply_rotary_emb(k_pe.unsqueeze(2), freqs_cis)