LDLMIn construction

All models, with their blocks →

Size

306M trained

Published

Meshchaninov et al. · May 2026

Line

Diffusion language model

View

Architecture

Source

LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verifiedHow to read

Attention over all positions

Training Generation

Q[B, 12, T, 64]K[B, 12, T, 64]V[B, 12, T, 64]Query–key products Scale

S/\sqrt{64}

Softmax over keys

A=\operatorname{softmax}(F)

A[B, 12, T, T]Weighted value sum O[B, 12, T, 64]

Token decoder › 3 blocks › Self-attention › Attention over all positions

Head outputs

tensor

What each head retrieved from the values.

shape: $O$ [B batch, 12 heads, T positions, 64 head features]

Sources

Contextual evidence from Attention over all positions

A block normalizes, modulates by the time condition’s shift and scale, projects to queries, keys and values, rotates queries and keys, attends over all positions with softcap, and adds the projected output scaled by a gate; the MLP branch does the same with its own shift, scale and gate.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Attention over all positions

The token decoder’s blocks: no softcap, no time conditioning, the gated MLP with the configured multiplier.

The implementation, pinned at revision e18c42acdfb7, not yet public