LDLMIn construction

All models, with their blocks →

Size

306M trained

Published

Meshchaninov et al. · May 2026

Line

Diffusion language model

View

Architecture

Source

LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verifiedHow to read

Self-attention

Training Generation

y_attn[B, T, 768]Q, K, V projection

y_{\mathrm{attn}}\,W_{QKV}

Split query heads Split key heads Split value heads Rotary tableR R RoPE RoPE Q[B, 12, T, 64]K[B, 12, T, 64]V[B, 12, T, 64]

Attend Open →

O[B, 12, T, 64]Merge heads Output projection

\Delta_{\mathrm{attn}}=oW_{O}

Δ_attn[B, T, 768]

Token decoder › 3 blocks › Self-attention

Attention update

tensor

The update handed to the residual addition.

shape: $\Delta_{\mathrm{attn}}$ [B batch, T positions, 768 d]

Sources

Contextual evidence from Add attention update

The residual update is scale · dropout(x) added to the residual; modulation is x · (1 + scale) + shift.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Add attention update

A block normalizes, modulates by the time condition’s shift and scale, projects to queries, keys and values, rotates queries and keys, attends over all positions with softcap, and adds the projected output scaled by a gate; the MLP branch does the same with its own shift, scale and gate.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Output projection

A block owns two gain-only LayerNorms, a fused query–key–value projection and an output projection without bias, an MLP of the configured kind, and, with modulation, an affine layer from the time condition to six vectors, initialized at zero.

The implementation, pinned at revision e18c42acdfb7, not yet public