LDLMIn construction

All models, with previews →

Meshchaninov et al. · May 2026Diffusion language model

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

TrainingGeneration

Latent encoder

Open inside →×

learned

A Perceiver of 6 blocks: 1,024 learned queries read the standardized GPT-2 features and become the latents z0z_0, one per position.

Why

Jointly. The latents come from an encoder that is being trained, so the diffusion objective acts on the latent space itself. The paper names this the key difference from latent diffusion pipelines whose encoder is frozen while the diffusion model trains. LDLM paper §4.2

Shallow token embeddings give a suboptimal latent space even when perfectly tuned for diffusion, and the representations of pre-trained encoders are not optimally suited to diffusion, since those encoders are trained on surrogate tasks. Training the latent encoder jointly with the diffusion model is how the paper addresses this gap. LDLM paper §1

Perceiver. The encoder is a Perceiver Resampler: it keeps a set of trainable latent vectors and updates them by cross-attention to the pre-trained hidden states, which stay fixed across its layers, so the latents aggregate information from the whole sequence. Against a self-attention encoder it has a better generative perplexity (66.3 against 94.7) and diversity and a lower Mauve (89.1 against 91.9). LDLM paper Appendix B.4

u0=γin⊙umax⁡ ⁣(∥u∥2/768,  ε)u_0=\gamma_{\mathrm{in}}\odot\frac{u}{\max\!\big(\lVert u\rVert_2/\sqrt{768},\;\varepsilon\big)}
ε=10−5\varepsilon=10^{-5}
z0=γout⊙u6max⁡ ⁣(∥u6∥2/768,  ε)z_0=\gamma_{\mathrm{out}}\odot\frac{u_{6}}{\max\!\big(\lVert u_{6}\rVert_2/\sqrt{768},\;\varepsilon\big)}
ε=10−5\varepsilon=10^{-5}
All 18 steps
Ep, b,t=768−1/2 PtE_{p,\,b,t}=768^{-1/2}\,P_t
hp=h+Eph^{p}=h+E_{p}
ub,t=Utu_{b,t}=U_t
u0=γin⊙umax⁡ ⁣(∥u∥2/768,  ε)u_0=\gamma_{\mathrm{in}}\odot\frac{u}{\max\!\big(\lVert u\rVert_2/\sqrt{768},\;\varepsilon\big)}
ε=10−5\varepsilon=10^{-5}
Encoder block:\text{Encoder block:}
uˉn=γa⊙unmax⁡ ⁣(∥un∥2/768,  ε)\bar u_n=\gamma_{a}\odot\frac{u_n}{\max\!\big(\lVert u_n\rVert_2/\sqrt{768},\;\varepsilon\big)}
ε=10−5\varepsilon=10^{-5}
hˉp=γb⊙hpmax⁡ ⁣(∥hp∥2/768,  ε)\bar h^{p}=\gamma_{b}\odot\frac{h^{p}}{\max\!\big(\lVert h^{p}\rVert_2/\sqrt{768},\;\varepsilon\big)}
ε=10−5\varepsilon=10^{-5}
pa=[ q  ka  va ]=uˉn WQKVp^{a}=[\,{\htmlClass{vl-stream-q}{q}}\;{\htmlClass{vl-stream-k}{k^{a}}}\;{\htmlClass{vl-stream-v}{v^{a}}}\,]=\bar u_n\,W_{QKV}
pb=[ kb  vb ]=hˉp WKVp^{b}=[\,{\htmlClass{vl-stream-k}{k^{b}}}\;{\htmlClass{vl-stream-v}{v^{b}}}\,]=\bar h^{p}\,W_{KV}
o=reshape⁡(O)o=\operatorname{reshape}(O)
Δattn=oWO\Delta_{\mathrm{attn}}=oW_{O}
un′=un+Δattnu'_n=u_n+\Delta_{\mathrm{attn}}
uˉn′=LayerNorm⁡(un′)\bar u'_n=\operatorname{LayerNorm}(u'_n)
h=uˉn′ W1+b1h=\bar u'_n\,W_{1}+b_{1}
g=h Φ(h)g=h\,\Phi(h)
Δmlp=g W2+b2\Delta_{\mathrm{mlp}}=g\,W_{2}+b_{2}
un+1=un′+Δmlpu_{n+1}=u'_n+\Delta_{\mathrm{mlp}}
z0=γout⊙u6max⁡ ⁣(∥u6∥2/768,  ε)z_0=\gamma_{\mathrm{out}}\odot\frac{u_{6}}{\max\!\big(\lVert u_{6}\rVert_2/\sqrt{768},\;\varepsilon\big)}
ε=10−5\varepsilon=10^{-5}
in
hh [B batch, T positions, 768 d]
out
z0z_0 [B batch, T positions, 768 d]
parameters
51,161,856
Parameters

51,161,856 parameters per instance, across its learned components

Exact per-instance accounting
ComponentCalculationParameters
Position table1,024×7681{,}024 \times 768786,432
Learned latent queries1,024×7681{,}024 \times 768786,432
Input RMSNorm768768768
Encoder block8,264,576×68{,}264{,}576 \times 649,587,456
Output RMSNorm768768768
Per-instance total51,161,856

Derived from the pinned implementation’s constructors and configuration. The checkpoint’s tensors have not been counted. 306,011,162 of these parameters are trained; frozen GPT-2 and the decoder’s mask vector, 124,442,112, are not.

Explanation

The latents do not start from the text: they start from the same 1,024 learned vectors for every sequence, and everything they know of the text they read from the features through attention.

There are as many latents as positions (num_latents follows max_sequence_len), so nothing is compressed along the sequence.

Sources4

Direct evidence

The encoder owns the learned latent queries, a position table, an input and an output RMSNorm and the blocks.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

The encoder adds positions to the features, normalizes the learned queries, applies the blocks, each attending to the features, and normalizes the result.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

The latent encoder: 12 heads of 64 features, query and key normalization, softcap 50, 6 layers, MLP multiplier 4, dropout 0.1, ε = 10⁻⁵.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

The feature model is GPT-2, frozen; the hidden state at index −3 is extracted; the encoder and the decoder are of the Perceiver kind; the file’s vocabulary size and mask token ID are defaults that the trainer overwrites with the tokenizer’s; there are as many latents as positions, of 768 features.

The implementation, pinned at revision e18c42acdfb7, not yet public