LDLMIn construction

All models, with their blocks →

Size
306M trained
Published
Meshchaninov et al. · May 2026
Line
Diffusion language model
View
Architecture
Source

LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verifiedHow to read

Perceiver attention

TrainingGeneration

Latent decoder › 6 decoder blocks › Perceiver attention

Latents with positions

×

supplied · no parameters

The latents being decoded, with their positions added: read through keys and values only.

shape
zpz^{p} [B batch, T positions, 768 d]
Sources

Contextual evidence from Positions to latents

Every query is the mask vector: it passes the MLP and a scale, positions are added; the latents take the same position table; the blocks are applied; every position counts as masked.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from RMSNorm

RMSNorm divides by the vector’s norm times width^(−1/2), clamped from below at ε, and multiplies by a learned gain.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from RMSNorm

Perceiver attention owns an RMSNorm for each of its two streams, a fused query–key–value projection of the query stream, a fused key–value projection of the other stream, RMSNorms for each head’s queries and keys, and an output projection; no bias.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from RMSNorm

Both streams are normalized; queries come from one stream, keys and values from both, concatenated along positions; queries and keys are normalized per head; attention runs over all positions with softcap; the heads are merged and projected.

The implementation, pinned at revision e18c42acdfb7, not yet public