LDLMIn construction

All models, with their blocks →

Size
306M trained
Published
Meshchaninov et al. · May 2026
Line
Diffusion language model
View
Architecture
Source

LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verifiedHow to read

Perceiver attention

TrainingGeneration

Latent encoder › 6 encoder blocks › Perceiver attention

Key and value heads

Open inside →×

learned

Take both streams’ keys and values, join each along positions and split into heads; the keys are normalized per head.

k=[ ka;  kb ]{\htmlClass{vl-stream-k}{k}}=[\,{\htmlClass{vl-stream-k}{k^{a}}};\;{\htmlClass{vl-stream-k}{k^{b}}}\,]
K=γK⊙Khmax⁡ ⁣(∥Kh∥2/64,  ε){\htmlClass{vl-stream-k}{K}}=\gamma_{{\htmlClass{vl-stream-k}{K}}}\odot\frac{{\htmlClass{vl-stream-k}{K^{h}}}}{\max\!\big(\lVert {\htmlClass{vl-stream-k}{K^{h}}}\rVert_2/\sqrt{64},\;\varepsilon\big)}
ε=10−5\varepsilon=10^{-5}
All 7 steps
ka=p769:1536a{\htmlClass{vl-stream-k}{k^{a}}}=p^{a}_{769:1536}
kb=p1:768b{\htmlClass{vl-stream-k}{k^{b}}}=p^{b}_{1:768}
k=[ ka;  kb ]{\htmlClass{vl-stream-k}{k}}=[\,{\htmlClass{vl-stream-k}{k^{a}}};\;{\htmlClass{vl-stream-k}{k^{b}}}\,]
K=γK⊙Khmax⁡ ⁣(∥Kh∥2/64,  ε){\htmlClass{vl-stream-k}{K}}=\gamma_{{\htmlClass{vl-stream-k}{K}}}\odot\frac{{\htmlClass{vl-stream-k}{K^{h}}}}{\max\!\big(\lVert {\htmlClass{vl-stream-k}{K^{h}}}\rVert_2/\sqrt{64},\;\varepsilon\big)}
ε=10−5\varepsilon=10^{-5}
va=p1537:2304a{\htmlClass{vl-stream-v}{v^{a}}}=p^{a}_{1537:2304}
vb=p769:1536b{\htmlClass{vl-stream-v}{v^{b}}}=p^{b}_{769:1536}
v=[ va;  vb ]{\htmlClass{vl-stream-v}{v}}=[\,{\htmlClass{vl-stream-v}{v^{a}}};\;{\htmlClass{vl-stream-v}{v^{b}}}\,]
in
pap^{a} [B batch, T positions, 2304 q, k, v]pbp^{b} [B batch, T positions, 1536]
out
KK [B batch, 12 heads, 2T positions, 64 head features]
parameters
64 · × 6 blocks = 384
Parameters

64 parameters per instance, across its learned components

Exact per-instance accounting
ComponentCalculationParameters
Key RMSNorm646464
Per-instance total64

Across 6 independent blocks: 64×6=38464 \times 6 = 384 parameters.

Derived from the pinned implementation’s constructors and configuration. The checkpoint’s tensors have not been counted. 306,011,162 of these parameters are trained; frozen GPT-2 and the decoder’s mask vector, 124,442,112, are not.

Explanation

The keys and the values of the two streams lie in one sequence of 2T positions, so one softmax weighs both. Values are left as they are: only queries and keys are normalized.

Sources5

Direct evidence

Both streams are normalized; queries come from one stream, keys and values from both, concatenated along positions; queries and keys are normalized per head; attention runs over all positions with softcap; the heads are merged and projected.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

An encoder block adds the Perceiver attention’s output to the latents, then adds the MLP’s output.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

A decoder block: Perceiver attention with the queries’ stream first read from the decoder’s state and keys and values from the latents and the state, a scaled residual, an MLP and a second scaled residual.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

RMSNorm divides by the vector’s norm times width^(−1/2), clamped from below at ε, and multiplies by a learned gain.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

Perceiver attention owns an RMSNorm for each of its two streams, a fused query–key–value projection of the query stream, a fused key–value projection of the other stream, RMSNorms for each head’s queries and keys, and an output projection; no bias.

The implementation, pinned at revision e18c42acdfb7, not yet public