LDLM · Final LayerNorm · OpenModelAtlas

Sampler › Denoiser

Final LayerNorm

learned

Center and scale each position’s features and apply a learned gain; there is no bias.

{\htmlData{light=final-norm}{\bar x}}=\gamma_{\mathrm{out}}\odot\frac{{\htmlData{light=stack-output blocks}{x_{12}}}-\mu({\htmlData{light=stack-output blocks}{x_{12}}})}{\sqrt{\sigma^2({\htmlData{light=stack-output blocks}{x_{12}}})+\varepsilon}}

\varepsilon=10^{-5}

in: $x_{12}$ [B batch, T positions, 768 d]
out: $\bar x$ [B batch, T positions, 768 d]
parameters: 768

Computed in float32 with autocast off; ε is PyTorch’s default.

Parameters

768 parameters owned by this component

Exact per-instance accounting
Component	Calculation	Parameters
gain only	$768$	768
Per-instance total		768

Across 1024 shared calls: $768 \times 1 = 768$ parameters.

Derived from the pinned implementation’s constructors and configuration. The checkpoint’s tensors have not been counted. 306,011,162 of these parameters are trained; frozen GPT-2 and the decoder’s mask vector, 124,442,112, are not.

One gain for each of the 768 features; no bias.

Codefunctional.py 2561–2577

functional.py 2561–2577

def layer_norm(
    input: Tensor,
    normalized_shape: List[int],
    weight: Optional[Tensor] = None,
    bias: Optional[Tensor] = None,
    eps: float = 1e-5,
) -> Tensor:
    r"""Apply Layer Normalization for last certain number of dimensions.

    See :class:`~torch.nn.LayerNorm` for details.
    """
    if has_torch_function_variadic(input, weight, bias):
        return handle_torch_function(
            layer_norm, (input, weight, bias), input, normalized_shape, weight=weight, bias=bias, eps=eps
        )
    return torch.layer_norm(input, normalized_shape, weight, bias, eps, torch.backends.cudnn.enabled)

Sources2

Direct evidence

The denoiser’s and the token decoder’s LayerNorm computes in float32 and owns a gain and no bias.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

PyTorch 2.4.1 functional LayerNorm defaults to epsilon 1e-5.

torch-layer-norm · pytorch/pytorch / torch/nn/functional.py
Pinned revision ee1b6804381c ↗ · lines 2561–2577

Read code excerpt

def layer_norm(
    input: Tensor,
    normalized_shape: List[int],
    weight: Optional[Tensor] = None,
    bias: Optional[Tensor] = None,
    eps: float = 1e-5,
) -> Tensor:
    r"""Apply Layer Normalization for last certain number of dimensions.

    See :class:`~torch.nn.LayerNorm` for details.
    """
    if has_torch_function_variadic(input, weight, bias):
        return handle_torch_function(
            layer_norm, (input, weight, bias), input, normalized_shape, weight=weight, bias=bias, eps=eps
        )
    return torch.layer_norm(input, normalized_shape, weight, bias, eps, torch.backends.cudnn.enabled)

LDLMIn construction