LDLMIn construction

All models, with their blocks →

Size
306M trained
Published
Meshchaninov et al. · May 2026
Line
Diffusion language model
View
Architecture
Source

LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verifiedHow to read

Query MLP

TrainingGeneration

Latent decoder › Query MLP

Expansion

×

learned

Map 768 features to 3,072.

h=mˉ W1+b1{\htmlData{light=up}{h}}={\htmlData{light=norm}{\bar m}}\,W_{1}+b_{1}
in
mˉ\bar m [B batch, T positions, 768 d]
out
hh [B batch, T positions, 3072 d_ff]
parameters
2,362,368
Parameters

2,362,368 parameters per instance.

Exact per-instance accounting
ComponentCalculationParameters
Weight matrix768×3,072768 \times 3{,}0722,359,296
Output bias3,0723{,}0723,072
Per-instance total2,362,368

The reference module holds the weight output by input, [out, in].

Derived from the configured matrix dimensions; checkpoint tensors have not been independently counted.

Sources1

Direct evidence

The Perceiver blocks’ MLP: LayerNorm, an affine layer, exact GELU, dropout, an affine layer.

The implementation, pinned at revision e18c42acdfb7, not yet public