DeepSeek-V3.2-Exp

All models, with their blocks →

Size
671B
Published
DeepSeek · Sep 2025
Line
Latent attention and experts
View
Source

deepseek‑ai/DeepSeek‑V3.2‑Exp @ 87e509a · checkpoint, tokenizer and tensors not verifiedHow to read

Routed expert

61 transformer blocks › 58 MoE blocks › Mixture of experts › 256 routed experts

Element-wise product

×

no parametersF.silu(self.w1(x).float()) * self.w3(x).float()

Gate the up features by the activated gate, feature by feature.

h=a⊙u{\htmlData{light=h product}{h}}={\htmlData{light=a silu}{a}}\odot {\htmlData{light=u up}{u}}
in
aa [B batch, T positions, 2048 expert d_ff]uu [B batch, T positions, 2048 expert d_ff]
out
hh [B batch, T positions, 2048 expert d_ff]
Codemodel.py 730–732, 744
model.py 730–732
        self.w1 = Linear(dim, inter_dim)
        self.w2 = Linear(inter_dim, dim)
        self.w3 = Linear(dim, inter_dim)
model.py 744
        return self.w2((F.silu(self.w1(x).float()) * self.w3(x).float()).type_as(x))
Sources1

Direct evidence

Each routed expert is its own SwiGLU MLP of moe_inter_dim hidden features.

expert · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 730–744

Read code excerpt
        self.w1 = Linear(dim, inter_dim)
        self.w2 = Linear(inter_dim, dim)
        self.w3 = Linear(dim, inter_dim)

… lines 733–743 omitted …

        return self.w2((F.silu(self.w1(x).float()) * self.w3(x).float()).type_as(x))