LDLMIn construction

All models, with previews →

Meshchaninov et al. · May 2026Diffusion language model

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

Training Generation

GPT-2 Smallhidden state −3124M weightsstandardized Open →

Latent encoder Open →

z₀ Decoder noise

+\,3\,\epsilon

Latent decoder Open →

Token decoder Open →

L Token cross-entropy

\mathcal L_{w}=\operatorname{CE}(L,\,I)

Feature reconstruction

\mathcal L_{h}=\operatorname{MSE}(\hat h,\,h)

z_t=\alpha_t z_0+\sigma_t\epsilon

z_t t z₀

Denoiser Open →

z₀ Diffusion loss

\mathcal L_{\mathrm{diff}}=\operatorname{MSE}(\hat z_0,\,z_0)

learns from

\mathcal L_{h}\cdot\mathcal L_{\mathrm{diff}}

learns from

\mathcal L_{\mathrm{diff}}

learns from

\mathcal L_{h}

learns from

\mathcal L_{w}

joint objective: the three losses are added, and one backward pass trains the four networksfrozenGPT-2 Small →∇ × γ ∇ × 0

Standardized features

tensor

Frozen GPT-2’s hidden state, standardized: the encoder’s input and the reconstruction’s target.

shape: $h$ [B batch, T positions, 768 d]

Sources

Contextual evidence from GPT-2 Small

With the feature model frozen, its hidden state is computed without gradient, standardized, and passed to the encoder; the same tensor is the target of the feature loss.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from GPT-2 Small

The feature model is wrapped to return one entry of its hidden states, chosen by index; GPT-2’s token table is resized to the configured vocabulary.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from GPT-2 Small

With an extraction layer other than −1 the encoder loads the wrapped feature model; when it is frozen it is put in evaluation mode and takes no gradient.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from GPT-2 Small

The feature statistics are a mean and a standard deviation per feature over valid positions of the validation set, the variance clamped at 10⁻⁴; features are standardized with them.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from GPT-2 Small

The feature model is GPT-2, frozen; the hidden state at index −3 is extracted; the encoder and the decoder are of the Perceiver kind; the file’s vocabulary size and mask token ID are defaults that the trainer overwrites with the tokenizer’s; there are as many latents as positions, of 768 features.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Latent encoder

The encoder owns the learned latent queries, a position table, an input and an output RMSNorm and the blocks.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Latent encoder

The encoder adds positions to the features, normalizes the learned queries, applies the blocks, each attending to the features, and normalizes the result.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Latent encoder

The latent encoder: 12 heads of 64 features, query and key normalization, softcap 50, 6 layers, MLP multiplier 4, dropout 0.1, ε = 10⁻⁵.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Feature reconstruction

Reconstruction: Gaussian noise of fixed standard deviation is added to the latents, the latent decoder runs, the token decoder reads its output detached, and the cross-entropy against the token IDs is taken; the latent decoder’s output is also compared with the standardized features by a mean squared error.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Feature reconstruction

Cross-entropy and mean squared error are averaged over valid positions.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Feature reconstruction

Joint training: decoder noise of standard deviation 3; weights 1 for the three losses; the diffusion gradient into the encoder rises from 0.001 over 50,000 steps by a straight-through scaling; the token decoder’s input is detached; AdamW at 3·10⁻⁴ for 1,000,000 steps of 512 sequences, gradients clipped at norm 1.

The implementation, pinned at revision e18c42acdfb7, not yet public