LDLMIn construction

All models, with their blocks →

Size

306M trained

Published

Meshchaninov et al. · May 2026

Line

Diffusion language model

View

Architecture

Source

LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verifiedHow to read

Key and value heads

Training Generation

The latents’ queries, keys and valuesp^a p^a The features’ keys and valuesp^b p^b Take the latents’ own keys Take the features’ keys Join keysalong positions Split heads Key RMSNorm

\lVert x\rVert_2/\sqrt d

K[B, 12, 2T, 64]Take the latents’ own values Take the features’ values Join valuesalong positions Split heads V[B, 12, 2T, 64]

Latent encoder › 6 encoder blocks › Perceiver attention › Key and value heads

Join values along positions

no parameters

Set the two streams’ values one after the other, in the keys’ order.

{\htmlData{light=value-concat}{{\htmlClass{vl-stream-v}{v}}}}=[\,{\htmlData{light=take-v-self}{{\htmlClass{vl-stream-v}{v^{a}}}}};\;{\htmlData{light=take-v-context}{{\htmlClass{vl-stream-v}{v^{b}}}}}\,]

in: $v^{a}$ [B batch, T positions, 768 d] $v^{b}$ [B batch, T positions, 768 d]
out: $v$ [B batch, 2T positions of both streams, 768 features]

Sources1

Direct evidence

Both streams are normalized; queries come from one stream, keys and values from both, concatenated along positions; queries and keys are normalized per head; attention runs over all positions with softcap; the heads are merged and projected.

The implementation, pinned at revision e18c42acdfb7, not yet public