LDLMIn construction

All models, with their blocks →

Size

306M trained

Published

Meshchaninov et al. · May 2026

Line

Diffusion language model

View

Architecture

Source

LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verifiedHow to read

Key and value heads

Training Generation

The latents’ queries, keys and valuesp^a p^a The features’ keys and valuesp^b p^b Take the latents’ own keys Take the features’ keys Join keysalong positions Split heads Key RMSNorm

\lVert x\rVert_2/\sqrt d

K[B, 12, 2T, 64]Take the latents’ own values Take the features’ values Join valuesalong positions Split heads V[B, 12, 2T, 64]

Latent encoder › 6 encoder blocks › Perceiver attention › Key and value heads

The latents’ queries, keys and values

tensor

Three parts of 768 features side by side: queries, keys and values of the stream that asks.

shape: $p^{a}$ [B batch, T positions, 2304 q, k, v]

Sources

Contextual evidence from Query, key and value projection

Perceiver attention owns an RMSNorm for each of its two streams, a fused query–key–value projection of the query stream, a fused key–value projection of the other stream, RMSNorms for each head’s queries and keys, and an output projection; no bias.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Query, key and value projection

Both streams are normalized; queries come from one stream, keys and values from both, concatenated along positions; queries and keys are normalized per head; attention runs over all positions with softcap; the heads are merged and projected.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Query heads

RMSNorm divides by the vector’s norm times width^(−1/2), clamped from below at ε, and multiplies by a learned gain.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Key and value heads

An encoder block adds the Perceiver attention’s output to the latents, then adds the MLP’s output.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Key and value heads

A decoder block: Perceiver attention with the queries’ stream first read from the decoder’s state and keys and values from the latents and the state, a scaled residual, an MLP and a second scaled residual.

The implementation, pinned at revision e18c42acdfb7, not yet public