LDLMIn construction

All models, with their blocks →

Size
306M trained
Published
Meshchaninov et al. · May 2026
Line
Diffusion language model
View
Architecture
Source

LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verifiedHow to read

Token-decoder block

TrainingGeneration
in
yny_n [B batch, T positions, 768 d]
out
yn+1y_{n+1} [B batch, T positions, 768 d]
parameters
9,438,720 · × 3 blocks = 28,316,160
mode
generation
Parameters

9,438,720 parameters per block, across its learned components

Exact per-instance accounting
ComponentCalculationParameters
LayerNorm768768768
Self-attention2,359,296 across its components2,359,296
LayerNorm768768768
Gated MLP7,077,888 across its components7,077,888
Per-instance total9,438,720

Across 3 independent blocks: 9,438,720×3=28,316,1609{,}438{,}720 \times 3 = 28{,}316{,}160 parameters.

Derived from the pinned implementation’s constructors and configuration. The checkpoint’s tensors have not been counted. 306,011,162 of these parameters are trained; frozen GPT-2 and the decoder’s mask vector, 124,442,112, are not.

Explanation

It is the denoiser’s block class without the time: no modulation and no gate, the gated MLP in place of the standard one, and no softcap.

In training a dropout of 0.1 acts on each update before it is added.

Sources5

Direct evidence

The token decoder: a rotary table, the configured number of blocks, and a linear readout to the vocabulary without bias.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

The token decoder’s blocks: no softcap, no time conditioning, the gated MLP with the configured multiplier.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

A block owns two gain-only LayerNorms, a fused query–key–value projection and an output projection without bias, an MLP of the configured kind, and, with modulation, an affine layer from the time condition to six vectors, initialized at zero.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

A block normalizes, modulates by the time condition’s shift and scale, projects to queries, keys and values, rotates queries and keys, attends over all positions with softcap, and adds the projected output scaled by a gate; the MLP branch does the same with its own shift, scale and gate.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

The latent decoder: 12 heads of 64 features, query and key normalization, softcap 50, 6 layers, MLP multiplier 4; the token decoder has 3 blocks.

The implementation, pinned at revision e18c42acdfb7, not yet public