DeepSeek-V3.2-Exp

Size

671B

Published

DeepSeek · Sep 2025

Line

Latent attention and experts

View

Source

deepseek‑ai/DeepSeek‑V3.2‑Exp @ 87e509a · checkpoint, tokenizer and tensors not verifiedHow to read

Routed expert

g=x_{\mathrm{ffn}}W_{1}

u=x_{\mathrm{ffn}}W_{3}

a = \operatorname{SiLU}(g)

h = a \odot u

E_i=hW_{2}

61 transformer blocks › 58 MoE blocks › Mixture of experts › 256 routed experts

tensorF.silu(self.w1(x).float())

The SiLU-activated gate features.

Sources

Contextual evidence from SiLU

Each routed expert is its own SwiGLU MLP of moe_inter_dim hidden features.

expert · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 730–744

Read code excerpt

        self.w1 = Linear(dim, inter_dim)
        self.w2 = Linear(inter_dim, dim)
        self.w3 = Linear(dim, inter_dim)

… lines 733–743 omitted …

        return self.w2((F.silu(self.w1(x).float()) * self.w3(x).float()).type_as(x))