LDLMIn construction

All models, with their blocks →

Size

306M trained

Published

Meshchaninov et al. · May 2026

Line

Diffusion language model

View

Architecture

Source

LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verifiedHow to read

Query heads

Training Generation

p^a[B, T, 2304]Take queries Split query heads Query RMSNorm

\lVert x\rVert_2/\sqrt d

Q[B, 12, T, 64]

Latent encoder › 6 encoder blocks › Perceiver attention › Query heads

Split query heads

no parameters

Read the 768 features as 12 heads of 64.

[B,\,T,\,768]\;\to\;[B,\,12,\,T,\,64]

in: $q$ [B batch, T positions, 768 d]
out: $Q^{h}$ [B batch, 12 heads, T positions, 64 head features]

Sources1

Direct evidence

Both streams are normalized; queries come from one stream, keys and values from both, concatenated along positions; queries and keys are normalized per head; attention runs over all positions with softcap; the heads are merged and projected.

The implementation, pinned at revision e18c42acdfb7, not yet public