DeepSeek-V3.2-Exp

All models, with previews →

DeepSeek · Sep 2025Latent attention and experts

Shared expert

Reference code deepseek-ai/DeepSeek-V3.2-Exp @ 87e509a · checkpoint, tokenizer and tensors not verified · How to read

x_ffn[B, T, 7168]Gate projection

g=x_{\mathrm{ffn}}W_{1}

g[B, T, 2048]Up projection

u=x_{\mathrm{ffn}}W_{3}

u[B, T, 2048]SiLU

a = \operatorname{SiLU}(g)

a[B, T, 2048]Element-wise product

h = a \odot u

h[B, T, 2048]Down projection

y^{\mathrm{shared}}=hW_{2}

y^shared[B, T, 7168]

61 transformer blocks › 58 MoE blocks › Mixture of experts › Shared expert

Activated gate

tensorF.silu(self.w1(x).float())

The SiLU-activated gate features.

shape: $a$ [B batch, T positions, 2048 shared d_ff]

Sources

Contextual evidence from SiLU

The dense feed-forward network is a SwiGLU MLP: w2(silu(w1 x) ⊙ w3 x).

mlp · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 629–643

Read code excerpt

        self.w1 = ColumnParallelLinear(dim, inter_dim)
        self.w2 = RowParallelLinear(inter_dim, dim, reduce_output=reduce_output)
        self.w3 = ColumnParallelLinear(dim, inter_dim)

… lines 632–642 omitted …

        return self.w2((F.silu(self.w1(x).float()) * self.w3(x).float()).type_as(x))