LDLMIn construction

All models, with previews →

Meshchaninov et al. · May 2026Diffusion language model

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

TrainingGeneration

Logits

×

tensor

Unnormalized vocabulary scores for the token at each position: position t scores token t.

shape
LL [B batch, T positions, 50259 vocabulary]
Sources

Contextual evidence from Token decoder

The token decoder: a rotary table, the configured number of blocks, and a linear readout to the vocabulary without bias.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Token decoder

The token decoder’s blocks: no softcap, no time conditioning, the gated MLP with the configured multiplier.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Token decoder

The latent decoder: 12 heads of 64 features, query and key normalization, softcap 50, 6 layers, MLP multiplier 4; the token decoder has 3 blocks.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Token decoder

The feature model is GPT-2, frozen; the hidden state at index −3 is extracted; the encoder and the decoder are of the Perceiver kind; the file’s vocabulary size and mask token ID are defaults that the trainer overwrites with the tokenizer’s; there are as many latents as positions, of 768 features.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Choose tokens

Generation: the sampled latents pass the latent decoder and the token decoder, each position takes the token of the largest logit, and the tokens are decoded to text, special tokens kept unless truncation is on.

The implementation, pinned at revision e18c42acdfb7, not yet public