LDLMIn construction

All models, with their blocks →

Size

306M trained

Published

Meshchaninov et al. · May 2026

Line

Diffusion language model

View

Architecture

Source

LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verifiedHow to read

Latent decoder

Training Generation

z₀[B, T, 768]Position tableE_p E_p Positions to latents

z^{p}=\tilde z_0+E_{p}

z^p[B, T, 768]Mask vector

E[\texttt{<mask>}]

Query MLP Open →

(1+s_e)

Positions to queries

r_0=\tilde e+E_{p}

r₀[B, T, 768]

6 decoder blocks Open →

Latent decoder

Positions to latents

no parameters

Add each position’s learned vector to its latent.

{\htmlData{light=with-positions add-positions}{z^{p}}}={\htmlData{light=latents}{\tilde z_0}}+{\htmlData{light=positions}{E_{p}}}

in: $\tilde z_0$ [B batch, T positions, 768 d] $E_{p}$ [B batch, T positions, 768 d]
out: $z^{p}$ [B batch, T positions, 768 d]

Sources1

Direct evidence

Every query is the mask vector: it passes the MLP and a scale, positions are added; the latents take the same position table; the blocks are applied; every position counts as masked.

The implementation, pinned at revision e18c42acdfb7, not yet public