DeepSeek-V3.2-Exp · SiLU · OpenModelAtlas

61 transformer blocks › 3 dense blocks › SwiGLU

SiLU

no parametersF.silu(self.w1(x).float())

Activate each gate feature independently.

{\htmlData{light=a silu}{a}}={\htmlData{light=g gate}{g}}\,\sigma({\htmlData{light=g gate}{g}})

in: $g$ [B batch, T positions, 18432 d_ff]
out: $a$ [B batch, T positions, 18432 d_ff]

SiLU and the product run in float32 and are cast back before the down projection (model.py 643).

Explanation

${\sigma(z)=1/(1+e^{-z})}$ is the logistic sigmoid.

Codemodel.py 629–631, 643

model.py 629–631

        self.w1 = ColumnParallelLinear(dim, inter_dim)
        self.w2 = RowParallelLinear(inter_dim, dim, reduce_output=reduce_output)
        self.w3 = ColumnParallelLinear(dim, inter_dim)

model.py 643

        return self.w2((F.silu(self.w1(x).float()) * self.w3(x).float()).type_as(x))

Sources1

Direct evidence

The dense feed-forward network is a SwiGLU MLP: w2(silu(w1 x) ⊙ w3 x).

mlp · deepseek-ai/DeepSeek-V3.2-Exp / inference/model.py
Pinned revision 87e509a2e5a1 ↗ · lines 629–643

Read code excerpt

        self.w1 = ColumnParallelLinear(dim, inter_dim)
        self.w2 = RowParallelLinear(inter_dim, dim, reduce_output=reduce_output)
        self.w3 = ColumnParallelLinear(dim, inter_dim)

… lines 632–642 omitted …

        return self.w2((F.silu(self.w1(x).float()) * self.w3(x).float()).type_as(x))