LDLM · RMSNorm · OpenModelAtlas

Latent encoder › 6 encoder blocks › Perceiver attention

RMSNorm

learned

Rescale each position’s 768 features by their root mean square and apply a learned gain.

{\htmlData{light=query-norm}{\bar u_n}}=\gamma_{a}\odot\frac{{\htmlData{light=x}{u_n}}}{\max\!\big(\lVert {\htmlData{light=x}{u_n}}\rVert_2/\sqrt{768},\;\varepsilon\big)}

\varepsilon=10^{-5}

in: $u_n$ [B batch, T positions, 768 d]
out: $\bar u_n$ [B batch, T positions, 768 d]
parameters: 768 · × 6 blocks = 4,608

ε bounds the divisor from below; it is not added under the root.

Parameters

768 parameters owned by this component

Exact per-instance accounting
Component	Calculation	Parameters
RMSNorm gain	$768$	768
Per-instance total		768

Across 6 independent blocks: $768 \times 6 = 4{,}608$ parameters.

Derived from the pinned implementation’s constructors and configuration. The checkpoint’s tensors have not been counted. 306,011,162 of these parameters are trained; frozen GPT-2 and the decoder’s mask vector, 124,442,112, are not.

One gain for each of the 768 features.

Sources3

Direct evidence

RMSNorm divides by the vector’s norm times width^(−1/2), clamped from below at ε, and multiplies by a learned gain.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

Perceiver attention owns an RMSNorm for each of its two streams, a fused query–key–value projection of the query stream, a fused key–value projection of the other stream, RMSNorms for each head’s queries and keys, and an output projection; no bias.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

Both streams are normalized; queries come from one stream, keys and values from both, concatenated along positions; queries and keys are normalized per head; attention runs over all positions with softcap; the heads are merged and projected.

The implementation, pinned at revision e18c42acdfb7, not yet public

LDLMIn construction