LDLM · MLP · OpenModelAtlas

Sampler › Denoiser › 12 denoiser blocks

MLP

learned

A two-layer MLP with 3,072 hidden features and the tanh form of GELU.

g=\frac{h}{2}\left[1+\tanh\!\left(\sqrt{\frac{2}{\pi}}\left(h+0.044715h^3\right)\right)\right]

\Delta_{\mathrm{mlp}}=gW_{2}+b_{2}

All 3 steps

h=x_{\mathrm{mlp}}W_{1}+b_{1}

g=\frac{h}{2}\left[1+\tanh\!\left(\sqrt{\frac{2}{\pi}}\left(h+0.044715h^3\right)\right)\right]

\Delta_{\mathrm{mlp}}=gW_{2}+b_{2}

in: $x_{\mathrm{mlp}}$ [B batch, T positions, 768 d]
out: $\Delta_{\mathrm{mlp}}$ [B batch, T positions, 768 d]
parameters: 4,722,432 · × 12 blocks = 56,669,184

Parameters

4,722,432 parameters per instance, across its learned components

Exact per-instance accounting
Component	Calculation	Parameters
Expansion	$768 \times 3{,}072 + 3{,}072$	2,362,368
Contraction	$3{,}072 \times 768 + 768$	2,360,064
Per-instance total		4,722,432

Across 12 independent blocks: $4{,}722{,}432 \times 12 = 56{,}669{,}184$ parameters.

Derived from the pinned implementation’s constructors and configuration. The checkpoint’s tensors have not been counted. 306,011,162 of these parameters are trained; frozen GPT-2 and the decoder’s mask vector, 124,442,112, are not.

Explanation

Both layers have a bias; the block’s normalization and time modulation come before it.

Sources2

Direct evidence

The standard MLP: an affine layer, GELU with the tanh approximation, an affine layer.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

A block owns two gain-only LayerNorms, a fused query–key–value projection and an output projection without bias, an MLP of the configured kind, and, with modulation, an affine layer from the time condition to six vectors, initialized at zero.

The implementation, pinned at revision e18c42acdfb7, not yet public

LDLMIn construction

Denoiser block

MLP