LDLMIn construction

All models, with their blocks →

Size
306M trained
Published
Meshchaninov et al. · May 2026
Line
Diffusion language model
View
Architecture
Source

LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verifiedHow to read

Self-attention

TrainingGeneration

Token decoder › 3 blocks › Self-attention

Attention over all positions

Open inside →×

no parameters

Each query weighs the values at all T positions by its scores against their keys.

At,s=exp⁡Ft,s∑u=1Texp⁡Ft,uA_{t,s}=\frac{\exp F_{t,s}}{\sum_{u=1}^{T}\exp F_{t,u}}
O=AVO=AV
All 4 steps
S=QKTS=QK^{\mathsf T}
F=S64F=\frac{S}{\sqrt{64}}
At,s=exp⁡Ft,s∑u=1Texp⁡Ft,uA_{t,s}=\frac{\exp F_{t,s}}{\sum_{u=1}^{T}\exp F_{t,u}}
O=AVO=AV
in
QQ [B batch, 12 heads, T positions, 64 head features]KK [B batch, 12 heads, T positions, 64 head features]VV [B batch, 12 heads, T positions, 64 head features]
out
OO [B batch, 12 heads, T positions, 64 head features]
Explanation

The fused kernel does not store the score and weight fields; the plate states its arithmetic step by step.

Sources2

Direct evidence

A block normalizes, modulates by the time condition’s shift and scale, projects to queries, keys and values, rotates queries and keys, attends over all positions with softcap, and adds the projected output scaled by a gate; the MLP branch does the same with its own shift, scale and gate.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

The token decoder’s blocks: no softcap, no time conditioning, the gated MLP with the configured multiplier.

The implementation, pinned at revision e18c42acdfb7, not yet public