DeepSeek-V3.2-Exp

All models, with their blocks →

Size

671B

Published

DeepSeek · Sep 2025

Line

Latent attention and experts

View

Source

deepseek‑ai/DeepSeek‑V3.2‑Exp @ 87e509a · checkpoint, tokenizer and tensors not verifiedHow to read

Query heads

Full sequence Prefill Decode

ĉ^Q[B, T, 1536]R[T, 32]Query content up-projection

{\htmlClass{vl-stream-q}{q^{C}}}=\hat c^{{\htmlClass{vl-stream-q}{Q}}}W_{UQ}

q^C[B, T, 16384]Split content heads Q^C[B, 128, T, 128]Query rotary up-projection

{\htmlClass{vl-stream-q}{q^{\mathrm{pe}}}}=\hat c^{{\htmlClass{vl-stream-q}{Q}}}W_{QR}

q^pe[B, T, 8192]Split rotary heads Q^pe[B, 128, T, 64]Rotate query part

{\htmlClass{vl-stream-q}{Q^{R}}}=\operatorname{RoPE}({\htmlClass{vl-stream-q}{Q^{\mathrm{pe}}}})

Q^R[B, 128, T, 64]Join query parts

{\htmlClass{vl-stream-q}{Q}}=[{\htmlClass{vl-stream-q}{Q^{C}}};\,{\htmlClass{vl-stream-q}{Q^{R}}}]

Q[B, 128, T, 192]

61 transformer blocks › 58 MoE blocks › Sparse latent attention › Query heads

Split rotary heads

no parametersq.view(bsz, seqlen, self.n_local_heads, self.qk_head_dim)

Partition the rotary features into 128 heads.

[B,\,T,\,8192]\;\to\;[B,\,128,\,T,\,64]

in: $q^{\mathrm{pe}}$ [B batch, T positions, 8192 heads × rotary]
out: $Q^{\mathrm{pe}}$ [B batch, 128 heads, T positions, 64 rotary]

Codemodel.py 560–564

model.py 560–564

        qr = self.q_norm(self.wq_a(x))
        q = self.wq_b(qr)
        q = q.view(bsz, seqlen, self.n_local_heads, self.qk_head_dim)
        q_nope, q_pe = torch.split(q, [self.qk_nope_head_dim, self.qk_rope_head_dim], dim=-1)
        q_pe = apply_rotary_emb(q_pe, freqs_cis)

Sources1

Direct evidence

Queries are produced through a low-rank latent, split into a non-rotated and a rotated part, and rotated.

mla-queries · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 560–564

Read code excerpt

        qr = self.q_norm(self.wq_a(x))
        q = self.wq_b(qr)
        q = q.view(bsz, seqlen, self.n_local_heads, self.qk_head_dim)
        q_nope, q_pe = torch.split(q, [self.qk_nope_head_dim, self.qk_rope_head_dim], dim=-1)
        q_pe = apply_rotary_emb(q_pe, freqs_cis)