LDLMIn construction

Meshchaninov et al. · May 2026Diffusion language model

Self-attention

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

Training Generation

y_attn[B, T, 768]Q, K, V projection

y_{\mathrm{attn}}\,W_{QKV}

Split query heads Split key heads Split value heads Rotary tableR R RoPE RoPE Q[B, 12, T, 64]K[B, 12, T, 64]V[B, 12, T, 64]

Attend Open →

O[B, 12, T, 64]Merge heads Output projection

\Delta_{\mathrm{attn}}=oW_{O}

Δ_attn[B, T, 768]

Token decoder › 3 blocks › Self-attention

Split key heads

no parameters

The second 768 features of the fused projection are the keys: read them as 12 heads of 64.

[B,\,T,\,2304]_{769:1536}\to[B,\,12,\,T,\,64]

in: $[\,q\;k\;v\,]$ [B batch, T positions, 2304 q, k, v]
out: $K^{h}$ [B batch, 12 heads, T positions, 64 head features]

Sources1

Direct evidence

A block normalizes, modulates by the time condition’s shift and scale, projects to queries, keys and values, rotates queries and keys, attends over all positions with softcap, and adds the projected output scaled by a gate; the MLP branch does the same with its own shift, scale and gate.

The implementation, pinned at revision e18c42acdfb7, not yet public