DeepSeek-V3.2-Exp

All models, with their blocks →

Size
671B
Published
DeepSeek · Sep 2025
Line
Latent attention and experts
View
Source

deepseek‑ai/DeepSeek‑V3.2‑Exp @ 87e509a · checkpoint, tokenizer and tensors not verifiedHow to read

Query heads

Full sequencePrefillDecode

61 transformer blocks › 58 MoE blocks › Sparse latent attention › Query heads

Split rotary heads

×

no parametersq.view(bsz, seqlen, self.n_local_heads, self.qk_head_dim)

Partition the rotary features into 128 heads.

[B, T, 8192]  →  [B, 128, T, 64][B,\,T,\,8192]\;\to\;[B,\,128,\,T,\,64]
in
qpeq^{\mathrm{pe}} [B batch, 1 position, 8192 heads × rotary]
out
QpeQ^{\mathrm{pe}} [B batch, 128 heads, 1 position, 64 rotary]
Codemodel.py 560–564
model.py 560–564
        qr = self.q_norm(self.wq_a(x))
        q = self.wq_b(qr)
        q = q.view(bsz, seqlen, self.n_local_heads, self.qk_head_dim)
        q_nope, q_pe = torch.split(q, [self.qk_nope_head_dim, self.qk_rope_head_dim], dim=-1)
        q_pe = apply_rotary_emb(q_pe, freqs_cis)
Sources1

Direct evidence

Queries are produced through a low-rank latent, split into a non-rotated and a rotated part, and rotated.

mla-queries · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 560–564

Read code excerpt
        qr = self.q_norm(self.wq_a(x))
        q = self.wq_b(qr)
        q = q.view(bsz, seqlen, self.n_local_heads, self.qk_head_dim)
        q_nope, q_pe = torch.split(q, [self.qk_nope_head_dim, self.qk_rope_head_dim], dim=-1)
        q_pe = apply_rotary_emb(q_pe, freqs_cis)