DeepSeek-V3.2-Exp

All models, with their blocks →

Size
671B
Published
DeepSeek · Sep 2025
Line
Latent attention and experts
View
Source

deepseek‑ai/DeepSeek‑V3.2‑Exp @ 87e509a · checkpoint, tokenizer and tensors not verifiedHow to read

Query heads

Full sequencePrefillDecode

61 transformer blocks › 58 MoE blocks › Sparse latent attention › Query heads

Split content heads

×

no parametersq.view(bsz, seqlen, self.n_local_heads, self.qk_head_dim)

Partition the content features into 128 heads.

[B, T, 16384]  →  [B, 128, T, 128][B,\,T,\,16384]\;\to\;[B,\,128,\,T,\,128]
in
qCq^{C} [B batch, T positions, 16384 heads × content]
out
QCQ^{C} [B batch, 128 heads, T positions, 128 content]

The code keeps queries, keys, values and scores position-first, [B, T, heads, …] (model.py 562, 577, 580), and the indexer’s queries too (461); the atlas writes them head-first.

Codemodel.py 560–564
model.py 560–564
        qr = self.q_norm(self.wq_a(x))
        q = self.wq_b(qr)
        q = q.view(bsz, seqlen, self.n_local_heads, self.qk_head_dim)
        q_nope, q_pe = torch.split(q, [self.qk_nope_head_dim, self.qk_rope_head_dim], dim=-1)
        q_pe = apply_rotary_emb(q_pe, freqs_cis)
Sources1

Direct evidence

Queries are produced through a low-rank latent, split into a non-rotated and a rotated part, and rotated.

mla-queries · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 560–564

Read code excerpt
        qr = self.q_norm(self.wq_a(x))
        q = self.wq_b(qr)
        q = q.view(bsz, seqlen, self.n_local_heads, self.qk_head_dim)
        q_nope, q_pe = torch.split(q, [self.qk_nope_head_dim, self.qk_rope_head_dim], dim=-1)
        q_pe = apply_rotary_emb(q_pe, freqs_cis)