LDLMIn construction

All models, with their blocks →

Size
306M trained
Published
Meshchaninov et al. · May 2026
Line
Diffusion language model
View
Architecture
Source

LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verifiedHow to read

Self-attention

TrainingGeneration

Sampler › Denoiser › 12 denoiser blocks › Self-attention

Query, key and value projection

×

learned

One matrix maps the input to queries, keys and values, 768 features each.

[ q  k  v ]=xattn WQKV[\,{\htmlClass{vl-stream-q}{q}}\;{\htmlClass{vl-stream-k}{k}}\;{\htmlClass{vl-stream-v}{v}}\,]={\htmlData{light=x}{x_{\mathrm{attn}}}}\,W_{QKV}
in
xattnx_{\mathrm{attn}} [B batch, T positions, 768 d]
out
[ q  k  v ][\,q\;k\;v\,] [B batch, T positions, 2304 q, k, v]
parameters
1,769,472 · × 12 blocks = 21,233,664
Parameters

1,769,472 parameters per instance.

Exact per-instance accounting
ComponentCalculationParameters
Weight matrix768×2,304768 \times 2{,}3041,769,472
Per-instance total1,769,472

This projection has no bias parameter.

The reference module holds the weight output by input, [out, in].

Across 12 independent blocks: 1,769,472×12=21,233,6641{,}769{,}472 \times 12 = 21{,}233{,}664 parameters.

Derived from the configured matrix dimensions; checkpoint tensors have not been independently counted.

Sources2

Direct evidence

A block owns two gain-only LayerNorms, a fused query–key–value projection and an output projection without bias, an MLP of the configured kind, and, with modulation, an affine layer from the time condition to six vectors, initialized at zero.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

A block normalizes, modulates by the time condition’s shift and scale, projects to queries, keys and values, rotates queries and keys, attends over all positions with softcap, and adds the projected output scaled by a gate; the MLP branch does the same with its own shift, scale and gate.

The implementation, pinned at revision e18c42acdfb7, not yet public