LDLM · Decoder block · OpenModelAtlas

Latent decoder

Decoder block

learned

One of 6 blocks: Perceiver attention from the decoder’s state to the latents and that state, then an MLP, each scaled by a learned number and added.

\tilde\Delta_{\mathrm{mlp}}=(1+s_m)\,\Delta_{\mathrm{mlp}}

r_{n+1}=r'_n+\tilde\Delta_{\mathrm{mlp}}

All 29 steps

\text{Perceiver attention:}

\bar r_n=\gamma_{a}\odot\frac{r_n}{\max\!\big(\lVert r_n\rVert_2/\sqrt{768},\;\varepsilon\big)}

\varepsilon=10^{-5}

\bar z^{p}=\gamma_{b}\odot\frac{z^{p}}{\max\!\big(\lVert z^{p}\rVert_2/\sqrt{768},\;\varepsilon\big)}

\varepsilon=10^{-5}

p^{a}=[\,{\htmlClass{vl-stream-q}{q}}\;{\htmlClass{vl-stream-k}{k^{a}}}\;{\htmlClass{vl-stream-v}{v^{a}}}\,]=\bar r_n\,W_{QKV}

p^{b}=[\,{\htmlClass{vl-stream-k}{k^{b}}}\;{\htmlClass{vl-stream-v}{v^{b}}}\,]=\bar z^{p}\,W_{KV}

{\htmlClass{vl-stream-q}{q}}=p^{a}_{1:768}

{\htmlClass{vl-stream-q}{Q}}=\gamma_{{\htmlClass{vl-stream-q}{Q}}}\odot\frac{{\htmlClass{vl-stream-q}{Q^{h}}}}{\max\!\big(\lVert {\htmlClass{vl-stream-q}{Q^{h}}}\rVert_2/\sqrt{64},\;\varepsilon\big)}

\varepsilon=10^{-5}

{\htmlClass{vl-stream-k}{k^{a}}}=p^{a}_{769:1536}

{\htmlClass{vl-stream-k}{k^{b}}}=p^{b}_{1:768}

{\htmlClass{vl-stream-k}{k}}=[\,{\htmlClass{vl-stream-k}{k^{b}}};\;{\htmlClass{vl-stream-k}{k^{a}}}\,]

{\htmlClass{vl-stream-k}{K}}=\gamma_{{\htmlClass{vl-stream-k}{K}}}\odot\frac{{\htmlClass{vl-stream-k}{K^{h}}}}{\max\!\big(\lVert {\htmlClass{vl-stream-k}{K^{h}}}\rVert_2/\sqrt{64},\;\varepsilon\big)}

\varepsilon=10^{-5}

{\htmlClass{vl-stream-v}{v^{a}}}=p^{a}_{1537:2304}

{\htmlClass{vl-stream-v}{v^{b}}}=p^{b}_{769:1536}

{\htmlClass{vl-stream-v}{v}}=[\,{\htmlClass{vl-stream-v}{v^{b}}};\;{\htmlClass{vl-stream-v}{v^{a}}}\,]

S=QK^{\mathsf T}

F=50\tanh\!\Big(\frac{S}{50\sqrt{64}}\Big)

A_{t,s}=\frac{\exp F_{t,s}}{\sum_{u=1}^{2T}\exp F_{t,u}}

O=AV

o=\operatorname{reshape}(O)

\Delta_{\mathrm{attn}}=oW_{O}

\tilde\Delta_{\mathrm{attn}}=(1+s_a)\,\Delta_{\mathrm{attn}}

r'_n=r_n+\tilde\Delta_{\mathrm{attn}}

\text{MLP:}

\bar r'_n=\operatorname{LayerNorm}(r'_n)

h=\bar r'_n\,W_{1}+b_{1}

g=h\,\Phi(h)

\Delta_{\mathrm{mlp}}=g\,W_{2}+b_{2}

\tilde\Delta_{\mathrm{mlp}}=(1+s_m)\,\Delta_{\mathrm{mlp}}

r_{n+1}=r'_n+\tilde\Delta_{\mathrm{mlp}}

in: $r_0$ [B batch, T positions, 768 d] $z^{p}$ [B batch, T positions, 768 d]
out: $r_{n+1}$ [B batch, T positions, 768 d]
parameters: 8,264,580 · × 6 blocks = 49,587,480

Parameters

8,264,580 parameters per block, across its learned components

Exact per-instance accounting
Component	Calculation	Parameters
Perceiver attention	3,540,608 across its components	3,540,608
Residual scale	$1 + 1$	2
MLP	4,723,968 across its components	4,723,968
Residual scale	$1 + 1$	2
Per-instance total		8,264,580

Across 6 independent blocks: $8{,}264{,}580 \times 6 = 49{,}587{,}480$ parameters.

Derived from the pinned implementation’s constructors and configuration. The checkpoint’s tensors have not been counted. 306,011,162 of these parameters are trained; frozen GPT-2 and the decoder’s mask vector, 124,442,112, are not.

Explanation

It is the encoder’s block with the two streams exchanged, and with a learned scale on each update.

Sources4

Direct evidence

A decoder block: Perceiver attention with the queries’ stream first read from the decoder’s state and keys and values from the latents and the state, a scaled residual, an MLP and a second scaled residual.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

Every query is the mask vector: it passes the MLP and a scale, positions are added; the latents take the same position table; the blocks are applied; every position counts as masked.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

The decoder’s residual scale: x · (1 + s), with one learned scalar for masked positions and one for unmasked ones, both zero at initialization.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

The latent decoder: 12 heads of 64 features, query and key normalization, softcap 50, 6 layers, MLP multiplier 4; the token decoder has 3 blocks.

The implementation, pinned at revision e18c42acdfb7, not yet public

LDLMIn construction