LDLMIn construction

All models, with their blocks →

Size
306M trained
Published
Meshchaninov et al. · May 2026
Line
Diffusion language model
View
Architecture
Source

LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verifiedHow to read

MLP

TrainingGeneration

Latent encoder › 6 encoder blocks › MLP

Contraction

×

learned

Map 3,072 features back to 768.

Δmlp=g W2+b2{\htmlData{light=output down}{\Delta_{\mathrm{mlp}}}}={\htmlData{light=activated gelu}{g}}\,W_{2}+b_{2}
in
gg [B batch, T positions, 3072 d_ff]
out
Δmlp\Delta_{\mathrm{mlp}} [B batch, T positions, 768 d]
parameters
2,360,064 · × 6 blocks = 14,160,384
Parameters

2,360,064 parameters per instance.

Exact per-instance accounting
ComponentCalculationParameters
Weight matrix3,072×7683{,}072 \times 7682,359,296
Output bias768768768
Per-instance total2,360,064

The reference module holds the weight output by input, [out, in].

Across 6 independent blocks: 2,360,064×6=14,160,3842{,}360{,}064 \times 6 = 14{,}160{,}384 parameters.

Derived from the configured matrix dimensions; checkpoint tensors have not been independently counted.

Sources1

Direct evidence

The Perceiver blocks’ MLP: LayerNorm, an affine layer, exact GELU, dropout, an affine layer.

The implementation, pinned at revision e18c42acdfb7, not yet public