LDLMIn construction

All models, with their blocks →

Size
306M trained
Published
Meshchaninov et al. · May 2026
Line
Diffusion language model
View
Architecture
Source

LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verifiedHow to read

Decoder block

TrainingGeneration

Latent decoder › 6 decoder blocks

Add attention update

×

no parameters

Add the mechanism’s update to the residual stream element by element.

rn′=rn+Δ~attn{\htmlData{light=residual attention-add}{r'_n}}={\htmlData{light=block-input}{r_n}}+{\htmlData{light=attention-scale}{\tilde\Delta_{\mathrm{attn}}}}
in
rnr_n [B batch, T positions, 768 d]Δ~attn\tilde\Delta_{\mathrm{attn}} [B batch, T positions, 768 d]
out
rn′r'_n [B batch, T positions, 768 d]
Sources1

Direct evidence

A decoder block: Perceiver attention with the queries’ stream first read from the decoder’s state and keys and values from the latents and the state, a scaled residual, an MLP and a second scaled residual.

The implementation, pinned at revision e18c42acdfb7, not yet public