LDLMIn construction

All models, with previews →

Meshchaninov et al. · May 2026Diffusion language model

Token decoder

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

Training Generation

3 blocks Open →

y₃[B, T, 768]Vocabulary readout

L=y_{3}W_{\mathrm{out}}

Token decoder

Logits

tensor

Unnormalized vocabulary scores for the token at each position.

shape: $L$ [B batch, T positions, 50259 vocabulary]

Sources

Contextual evidence from Token decoder

The token decoder: a rotary table, the configured number of blocks, and a linear readout to the vocabulary without bias.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Token decoder

The token decoder’s blocks: no softcap, no time conditioning, the gated MLP with the configured multiplier.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Token decoder

The latent decoder: 12 heads of 64 features, query and key normalization, softcap 50, 6 layers, MLP multiplier 4; the token decoder has 3 blocks.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Token decoder

The feature model is GPT-2, frozen; the hidden state at index −3 is extracted; the encoder and the decoder are of the Perceiver kind; the file’s vocabulary size and mask token ID are defaults that the trainer overwrites with the tokenizer’s; there are as many latents as positions, of 768 features.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Vocabulary readout

Before any network is built, the configured vocabulary size and mask token ID are overwritten with the tokenizer’s.

The implementation, pinned at revision e18c42acdfb7, not yet public