LDLMIn construction

All models, with their blocks →

Size
306M trained
Published
Meshchaninov et al. · May 2026
Line
Diffusion language model
View
Architecture
Source

LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verifiedHow to read

Encoder block

TrainingGeneration

Latent encoder › 6 encoder blocks

Perceiver attention

Open inside →×

learned

12-head attention in which the latents ask and both the latents and the features answer: queries from the latents, keys and values from the two streams joined along positions.

pb=[ kb  vb ]=hˉp WKVp^{b}=[\,{\htmlClass{vl-stream-k}{k^{b}}}\;{\htmlClass{vl-stream-v}{v^{b}}}\,]=\bar h^{p}\,W_{KV}
Δattn=oWO\Delta_{\mathrm{attn}}=oW_{O}
All 22 steps
uˉn=γa⊙unmax⁡ ⁣(∥un∥2/768,  ε)\bar u_n=\gamma_{a}\odot\frac{u_n}{\max\!\big(\lVert u_n\rVert_2/\sqrt{768},\;\varepsilon\big)}
ε=10−5\varepsilon=10^{-5}
hˉp=γb⊙hpmax⁡ ⁣(∥hp∥2/768,  ε)\bar h^{p}=\gamma_{b}\odot\frac{h^{p}}{\max\!\big(\lVert h^{p}\rVert_2/\sqrt{768},\;\varepsilon\big)}
ε=10−5\varepsilon=10^{-5}
pa=[ q  ka  va ]=uˉn WQKVp^{a}=[\,{\htmlClass{vl-stream-q}{q}}\;{\htmlClass{vl-stream-k}{k^{a}}}\;{\htmlClass{vl-stream-v}{v^{a}}}\,]=\bar u_n\,W_{QKV}
pb=[ kb  vb ]=hˉp WKVp^{b}=[\,{\htmlClass{vl-stream-k}{k^{b}}}\;{\htmlClass{vl-stream-v}{v^{b}}}\,]=\bar h^{p}\,W_{KV}
Query heads:\text{Query heads:}
q=p1:768a{\htmlClass{vl-stream-q}{q}}=p^{a}_{1:768}
Q=γQ⊙Qhmax⁡ ⁣(∥Qh∥2/64,  ε){\htmlClass{vl-stream-q}{Q}}=\gamma_{{\htmlClass{vl-stream-q}{Q}}}\odot\frac{{\htmlClass{vl-stream-q}{Q^{h}}}}{\max\!\big(\lVert {\htmlClass{vl-stream-q}{Q^{h}}}\rVert_2/\sqrt{64},\;\varepsilon\big)}
ε=10−5\varepsilon=10^{-5}
Key and value heads:\text{Key and value heads:}
ka=p769:1536a{\htmlClass{vl-stream-k}{k^{a}}}=p^{a}_{769:1536}
kb=p1:768b{\htmlClass{vl-stream-k}{k^{b}}}=p^{b}_{1:768}
k=[ ka;  kb ]{\htmlClass{vl-stream-k}{k}}=[\,{\htmlClass{vl-stream-k}{k^{a}}};\;{\htmlClass{vl-stream-k}{k^{b}}}\,]
K=γK⊙Khmax⁡ ⁣(∥Kh∥2/64,  ε){\htmlClass{vl-stream-k}{K}}=\gamma_{{\htmlClass{vl-stream-k}{K}}}\odot\frac{{\htmlClass{vl-stream-k}{K^{h}}}}{\max\!\big(\lVert {\htmlClass{vl-stream-k}{K^{h}}}\rVert_2/\sqrt{64},\;\varepsilon\big)}
ε=10−5\varepsilon=10^{-5}
va=p1537:2304a{\htmlClass{vl-stream-v}{v^{a}}}=p^{a}_{1537:2304}
vb=p769:1536b{\htmlClass{vl-stream-v}{v^{b}}}=p^{b}_{769:1536}
v=[ va;  vb ]{\htmlClass{vl-stream-v}{v}}=[\,{\htmlClass{vl-stream-v}{v^{a}}};\;{\htmlClass{vl-stream-v}{v^{b}}}\,]
Attention over both streams:\text{Attention over both streams:}
S=QKTS=QK^{\mathsf T}
F=50tanh⁡ ⁣(S5064)F=50\tanh\!\Big(\frac{S}{50\sqrt{64}}\Big)
At,s=exp⁡Ft,s∑u=12Texp⁡Ft,uA_{t,s}=\frac{\exp F_{t,s}}{\sum_{u=1}^{2T}\exp F_{t,u}}
O=AVO=AV
o=reshape⁡(O)o=\operatorname{reshape}(O)
Δattn=oWO\Delta_{\mathrm{attn}}=oW_{O}
in
unu_n [B batch, T positions, 768 d]hph^{p} [B batch, T positions, 768 d]
out
Δattn\Delta_{\mathrm{attn}} [B batch, T positions, 768 d]
parameters
3,540,608 · × 6 blocks = 21,243,648

The code keeps head tensors as [B, T, 12, 64] and never exchanges the two axes; the atlas writes heads before positions, as on its other plates.

Parameters

3,540,608 parameters per instance, across its learned components

Exact per-instance accounting
ComponentCalculationParameters
RMSNorm768768768
RMSNorm768768768
Query, key and value projection768×2,304768 \times 2{,}3041,769,472
Key and value projection768×1,536768 \times 1{,}5361,179,648
Query heads64 across its components64
Key and value heads64 across its components64
Output projection768×768768 \times 768589,824
Per-instance total3,540,608

Across 6 independent blocks: 3,540,608×6=21,243,6483{,}540{,}608 \times 6 = 21{,}243{,}648 parameters.

Derived from the pinned implementation’s constructors and configuration. The checkpoint’s tensors have not been counted. 306,011,162 of these parameters are trained; frozen GPT-2 and the decoder’s mask vector, 124,442,112, are not.

Explanation

This is how the encoder reads the frozen features: its latents are the only stream that is updated, and the features enter every block unchanged.

Queries and keys are normalized per head and the scores softcapped at 50; nothing encodes position here, the positions were added to both streams before the blocks.

Sources2

Direct evidence

Perceiver attention owns an RMSNorm for each of its two streams, a fused query–key–value projection of the query stream, a fused key–value projection of the other stream, RMSNorms for each head’s queries and keys, and an output projection; no bias.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

Both streams are normalized; queries come from one stream, keys and values from both, concatenated along positions; queries and keys are normalized per head; attention runs over all positions with softcap; the heads are merged and projected.

The implementation, pinned at revision e18c42acdfb7, not yet public