DeepSeek-V3.2-Exp

All models, with their blocks →

Size
671B
Published
DeepSeek · Sep 2025
Line
Latent attention and experts
View
Source

deepseek‑ai/DeepSeek‑V3.2‑Exp @ 87e509a · checkpoint, tokenizer and tensors not verifiedHow to read

Mix and project

Full sequencePrefillDecode

61 transformer blocks › 58 MoE blocks › Sparse latent attention › Sparse attention › Mix and project

Merge heads

×

no parametersx.flatten(2)

Concatenate the heads at each position.

[B, 128, T, 128][B,\,128,\,T,\,128]
→  [B, T, 128, 128]\to\;[B,\,T,\,128,\,128]
→  [B, T, 16384]\to\;[B,\,T,\,16384]
in
OO [B batch, 128 heads, T positions, 128 value]
out
oo [B batch, T positions, 16384 heads × value]
Explanation

Heads are concatenated in order: head ii fills features 128i,…,128i+127{128i},\dots,{128i+127}.

Codemodel.py 607–608
model.py 607–608
        x = self.wo(x.flatten(2))
        return x
Sources1

Direct evidence

Head outputs are flattened and projected back to the model width.

mla-output · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 607–608

Read code excerpt
        x = self.wo(x.flatten(2))
        return x