Token decoder › 3 blocks › Gated MLP
Input features
×tensor
The normalized features entering this SwiGLU.
- shape
- [B batch, T positions, 768 d]
Sources
Contextual evidence from LayerNorm
The denoiser’s and the token decoder’s LayerNorm computes in float32 and owns a gain and no bias.
The implementation, pinned at revision e18c42acdfb7, not yet public
Contextual evidence from LayerNorm
PyTorch 2.4.1 functional LayerNorm defaults to epsilon 1e-5.
torch-layer-norm · pytorch/pytorch / torch/nn/functional.py
Pinned revision ee1b6804381c ↗ · lines 2561–2577
Read code excerpt
def layer_norm(
input: Tensor,
normalized_shape: List[int],
weight: Optional[Tensor] = None,
bias: Optional[Tensor] = None,
eps: float = 1e-5,
) -> Tensor:
r"""Apply Layer Normalization for last certain number of dimensions.
See :class:`~torch.nn.LayerNorm` for details.
"""
if has_torch_function_variadic(input, weight, bias):
return handle_torch_function(
layer_norm, (input, weight, bias), input, normalized_shape, weight=weight, bias=bias, eps=eps
)
return torch.layer_norm(input, normalized_shape, weight, bias, eps, torch.backends.cudnn.enabled)
Contextual evidence from Gate projection
The gated MLP: down(SiLU(gate x) · up x), three projections without bias.
The implementation, pinned at revision e18c42acdfb7, not yet public