LDLMIn construction

Contextual evidence from Stop-gradient

Reconstruction: Gaussian noise of fixed standard deviation is added to the latents, the latent decoder runs, the token decoder reads its output detached, and the cross-entropy against the token IDs is taken; the latent decoder’s output is also compared with the standardized features by a mean squared error.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Stop-gradient

Joint training: decoder noise of standard deviation 3; weights 1 for the three losses; the diffusion gradient into the encoder rises from 0.001 over 50,000 steps by a straight-through scaling; the token decoder’s input is detached; AdamW at 3·10⁻⁴ for 1,000,000 steps of 512 sequences, gradients clipped at norm 1.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from LayerNorm

The denoiser’s and the token decoder’s LayerNorm computes in float32 and owns a gain and no bias.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from LayerNorm

PyTorch 2.4.1 functional LayerNorm defaults to epsilon 1e-5.

torch-layer-norm · pytorch/pytorch / torch/nn/functional.py
Pinned revision ee1b6804381c ↗ · lines 2561–2577

def layer_norm(
    input: Tensor,
    normalized_shape: List[int],
    weight: Optional[Tensor] = None,
    bias: Optional[Tensor] = None,
    eps: float = 1e-5,
) -> Tensor:
    r"""Apply Layer Normalization for last certain number of dimensions.

    See :class:`~torch.nn.LayerNorm` for details.
    """
    if has_torch_function_variadic(input, weight, bias):
        return handle_torch_function(
            layer_norm, (input, weight, bias), input, normalized_shape, weight=weight, bias=bias, eps=eps
        )
    return torch.layer_norm(input, normalized_shape, weight, bias, eps, torch.backends.cudnn.enabled)

Contextual evidence from Add attention update

The residual update is scale · dropout(x) added to the residual; modulation is x · (1 + scale) + shift.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Add attention update

A block normalizes, modulates by the time condition’s shift and scale, projects to queries, keys and values, rotates queries and keys, attends over all positions with softcap, and adds the projected output scaled by a gate; the MLP branch does the same with its own shift, scale and gate.

The implementation, pinned at revision e18c42acdfb7, not yet public