LDLMIn construction

All models, with previews →

Meshchaninov et al. · May 2026Diffusion language model

Gated MLP

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

TrainingGeneration
in
ymlpy_{\mathrm{mlp}} [B batch, T positions, 768 d]
out
Δmlp\Delta_{\mathrm{mlp}} [B batch, T positions, 768 d]
parameters
7,077,888 · × 3 blocks = 21,233,664
mode
generation
Parameters

7,077,888 parameters per instance, across its learned components

Exact per-instance accounting
ComponentCalculationParameters
Gate projection768×3,072768 \times 3{,}0722,359,296
Up projection768×3,072768 \times 3{,}0722,359,296
Down projection3,072×7683{,}072 \times 7682,359,296
Per-instance total7,077,888

Across 3 independent blocks: 7,077,888×3=21,233,6647{,}077{,}888 \times 3 = 21{,}233{,}664 parameters.

Derived from the pinned implementation’s constructors and configuration. The checkpoint’s tensors have not been counted. 306,011,162 of these parameters are trained; frozen GPT-2 and the decoder’s mask vector, 124,442,112, are not.

Explanation

Its hidden width is 4 × 768 = 3,072.

Sources3

Direct evidence

The gated MLP: down(SiLU(gate x) · up x), three projections without bias.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

A block owns two gain-only LayerNorms, a fused query–key–value projection and an output projection without bias, an MLP of the configured kind, and, with modulation, an affine layer from the time condition to six vectors, initialized at zero.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

The token decoder’s blocks: no softcap, no time conditioning, the gated MLP with the configured multiplier.

The implementation, pinned at revision e18c42acdfb7, not yet public