LDLMIn construction

All models, with previews →

Meshchaninov et al. · May 2026Diffusion language model

Self-attention

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

TrainingGeneration

Token decoder › 3 blocks › Self-attention

Rotate keys

×

no parameters

Rotate each pair of features of the keys by an angle that grows with the position.

Kt=rot⁡ ⁣(Kth; t θ){\htmlData{light=keys rotate-k}{{\htmlClass{vl-stream-k}{K_{t}}}}}=\operatorname{rot}\!\big({\htmlData{light=k-heads}{{\htmlClass{vl-stream-k}{K^{h}_{t}}}}};\,t\,\theta\big)
θj=10,000−2j/64\theta_j=10,000^{-2j/64}
in
KhK^{h} [B batch, 12 heads, T positions, 64 head features]RR [T positions, 32 pairs, 2 cos, sin]
out
KK [B batch, 12 heads, T positions, 64 head features]

Feature j is paired with feature j + 32 of the same head (flash-attn’s packed rotary); the angles are computed in float32.

Sources2

Direct evidence

The rotary table: angles from positions and inverse frequencies of base 10,000, computed in float32; queries and keys are rotated and values take the identity.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

A block normalizes, modulates by the time condition’s shift and scale, projects to queries, keys and values, rotates queries and keys, attends over all positions with softcap, and adds the projected output scaled by a gate; the MLP branch does the same with its own shift, scale and gate.

The implementation, pinned at revision e18c42acdfb7, not yet public