Llama 3

All models, with previews →

Meta · Apr 2024Dense decoder

SwiGLU MLP

Reference code meta-llama/llama3 @ a0940f9, configuration meta-llama/llama-models @ 0e0b8c5 · checkpoint, tokenizer and tensors not verified · How to read

32 independent transformer blocks › SwiGLU

SiLU

×

no parametersF.silu(self.w1(x))

Apply a smooth nonlinear transformation to each gate feature.

a=g σ(g){\htmlData{light=a silu}{a}}={\htmlData{light=g gate}{g}}\,\sigma({\htmlData{light=g gate}{g}})
in
gg [B batch, T positions, 14336 d_ff]
out
aa [B batch, T positions, 14336 d_ff]
Explanation

σ(z)=1/(1+e−z){\sigma(z)=1/(1+e^{-z})} is the logistic sigmoid.

SiLU is unbounded above and dips to about −0.28 near g≈−1.28g\approx-1.28, so a gate value can amplify an up feature or flip its sign.

Codemodel.py 208–219 · activation.py 397–403
model.py 208–219
        self.w1 = ColumnParallelLinear(
            dim, hidden_dim, bias=False, gather_output=False, init_method=lambda x: x
        )
        self.w2 = RowParallelLinear(
            hidden_dim, dim, bias=False, input_is_parallel=True, init_method=lambda x: x
        )
        self.w3 = ColumnParallelLinear(
            dim, hidden_dim, bias=False, gather_output=False, init_method=lambda x: x
        )

    def forward(self, x):
        return self.w2(F.silu(self.w1(x)) * self.w3(x))
activation.py 397–403
class SiLU(Module):
    r"""Applies the Sigmoid Linear Unit (SiLU) function, element-wise.

    The SiLU function is also known as the swish function.

    .. math::
        \text{silu}(x) = x * \sigma(x), \text{where } \sigma(x) \text{ is the logistic sigmoid.}
Sources2

Direct evidence

The MLP owns three bias-free projections. Its forward expression applies SiLU to w1(x), multiplies w3(x) element-wise and projects through w2.

mlp · meta-llama/llama3 / llama/model.py
Pinned revision a0940f9cf706 ↗ · lines 208–219

Read code excerpt
        self.w1 = ColumnParallelLinear(
            dim, hidden_dim, bias=False, gather_output=False, init_method=lambda x: x
        )
        self.w2 = RowParallelLinear(
            hidden_dim, dim, bias=False, input_is_parallel=True, init_method=lambda x: x
        )
        self.w3 = ColumnParallelLinear(
            dim, hidden_dim, bias=False, gather_output=False, init_method=lambda x: x
        )

    def forward(self, x):
        return self.w2(F.silu(self.w1(x)) * self.w3(x))

Dependency operation context

SiLU acts element-wise as z times its logistic sigmoid.

silu · pytorch/pytorch / torch/nn/modules/activation.py
Pinned revision ba56102387ef ↗ · lines 397–403

Read code excerpt
class SiLU(Module):
    r"""Applies the Sigmoid Linear Unit (SiLU) function, element-wise.

    The SiLU function is also known as the swish function.

    .. math::
        \text{silu}(x) = x * \sigma(x), \text{where } \sigma(x) \text{ is the logistic sigmoid.}