LDLMIn construction

All models, with previews →

Meshchaninov et al. · May 2026Diffusion language model

Latent decoder

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

TrainingGeneration

Latent decoder

Latents to decode

×

tensor

In training, the clean latents with the decoder noise added.

shape
z~0\tilde z_0 [B batch, T positions, 768 d]
Sources

Contextual evidence from Decoder noise

Reconstruction: Gaussian noise of fixed standard deviation is added to the latents, the latent decoder runs, the token decoder reads its output detached, and the cross-entropy against the token IDs is taken; the latent decoder’s output is also compared with the standardized features by a mean squared error.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Decoder noise

Joint training: decoder noise of standard deviation 3; weights 1 for the three losses; the diffusion gradient into the encoder rises from 0.001 over 50,000 steps by a straight-through scaling; the token decoder’s input is detached; AdamW at 3·10⁻⁴ for 1,000,000 steps of 512 sequences, gradients clipped at norm 1.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Positions to latents

Every query is the mask vector: it passes the MLP and a scale, positions are added; the latents take the same position table; the blocks are applied; every position counts as masked.

The implementation, pinned at revision e18c42acdfb7, not yet public