32 independent transformer blocks › SwiGLU
Activated gate
×tensorF.silu(self.w1(x))
Multiplicative coefficients for the parallel up features.
- shape
- [B batch, T positions, 14336 d_ff]
Codemodel.py 208–219 · activation.py 397–403
self.w1 = ColumnParallelLinear(
dim, hidden_dim, bias=False, gather_output=False, init_method=lambda x: x
)
self.w2 = RowParallelLinear(
hidden_dim, dim, bias=False, input_is_parallel=True, init_method=lambda x: x
)
self.w3 = ColumnParallelLinear(
dim, hidden_dim, bias=False, gather_output=False, init_method=lambda x: x
)
def forward(self, x):
return self.w2(F.silu(self.w1(x)) * self.w3(x))class SiLU(Module):
r"""Applies the Sigmoid Linear Unit (SiLU) function, element-wise.
The SiLU function is also known as the swish function.
.. math::
\text{silu}(x) = x * \sigma(x), \text{where } \sigma(x) \text{ is the logistic sigmoid.}Sources2
Contextual evidence
The MLP owns three bias-free projections. Its forward expression applies SiLU to w1(x), multiplies w3(x) element-wise and projects through w2.
mlp · meta-llama/llama3 / llama/model.py
Pinned revision a0940f9cf706 ↗ · lines 208–219
Read code excerpt
self.w1 = ColumnParallelLinear(
dim, hidden_dim, bias=False, gather_output=False, init_method=lambda x: x
)
self.w2 = RowParallelLinear(
hidden_dim, dim, bias=False, input_is_parallel=True, init_method=lambda x: x
)
self.w3 = ColumnParallelLinear(
dim, hidden_dim, bias=False, gather_output=False, init_method=lambda x: x
)
def forward(self, x):
return self.w2(F.silu(self.w1(x)) * self.w3(x))Contextual evidence
SiLU acts element-wise as z times its logistic sigmoid.
silu · pytorch/pytorch / torch/nn/modules/activation.py
Pinned revision ba56102387ef ↗ · lines 397–403
Read code excerpt
class SiLU(Module):
r"""Applies the Sigmoid Linear Unit (SiLU) function, element-wise.
The SiLU function is also known as the swish function.
.. math::
\text{silu}(x) = x * \sigma(x), \text{where } \sigma(x) \text{ is the logistic sigmoid.}