LDLMIn construction

All models, with their blocks →

Size
306M trained
Published
Meshchaninov et al. · May 2026
Line
Diffusion language model
View
Architecture
Source

LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verifiedHow to read

Perceiver attention

TrainingGeneration

Latent decoder › 6 decoder blocks › Perceiver attention

Output projection

×

learned

Map the merged features to the update of the decoder state.

Δattn=oWO{\htmlData{light=update output-projection}{\Delta_{\mathrm{attn}}}}={\htmlData{light=merge}{o}}W_{O}
in
oo [B batch, T positions, 768 d]
out
Δattn\Delta_{\mathrm{attn}} [B batch, T positions, 768 d]
parameters
589,824 · × 6 blocks = 3,538,944
Parameters

589,824 parameters per instance.

Exact per-instance accounting
ComponentCalculationParameters
Weight matrix768×768768 \times 768589,824
Per-instance total589,824

This projection has no bias parameter.

The reference module holds the weight output by input, [out, in].

Across 6 independent blocks: 589,824×6=3,538,944589{,}824 \times 6 = 3{,}538{,}944 parameters.

Derived from the configured matrix dimensions; checkpoint tensors have not been independently counted.

Explanation

The dropout that follows it has probability 0 in this configuration.

Sources2

Direct evidence

Perceiver attention owns an RMSNorm for each of its two streams, a fused query–key–value projection of the query stream, a fused key–value projection of the other stream, RMSNorms for each head’s queries and keys, and an output projection; no bias.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

Both streams are normalized; queries come from one stream, keys and values from both, concatenated along positions; queries and keys are normalized per head; attention runs over all positions with softcap; the heads are merged and projected.

The implementation, pinned at revision e18c42acdfb7, not yet public