LDLMIn construction

All models, with their blocks →

Size
306M trained
Published
Meshchaninov et al. · May 2026
Line
Diffusion language model
View
Architecture
Source

LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verifiedHow to read

Denoiser block

TrainingGeneration

Sampler › Denoiser › 12 denoiser blocks

Gate

×

no parameters

Scale the update, feature by feature, by a vector made from the time.

Δ~attn=ga⊙Δattn{\htmlData{light=attention-gate}{\tilde\Delta_{\mathrm{attn}}}}=g_{a}\odot {\htmlData{light=attention}{\Delta_{\mathrm{attn}}}}
in
Δattn\Delta_{\mathrm{attn}} [B batch, T positions, 768 d]mnm_n [B batch, 1 every position, 4608 six vectors of 768]
out
Δ~attn\tilde\Delta_{\mathrm{attn}} [B batch, T positions, 768 d]
Explanation

gag_{a} is the third of the six vectors of mnm_n.

The projection that makes mnm_n starts at zero, so at initialization every gate is zero and a block changes nothing. In training a dropout of 0.1 acts on the update before the gate.

Sources2

Direct evidence

The residual update is scale · dropout(x) added to the residual; modulation is x · (1 + scale) + shift.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

A block normalizes, modulates by the time condition’s shift and scale, projects to queries, keys and values, rotates queries and keys, attends over all positions with softcap, and adds the projected output scaled by a gate; the MLP branch does the same with its own shift, scale and gate.

The implementation, pinned at revision e18c42acdfb7, not yet public