LDLMIn construction

All models, with previews →

Meshchaninov et al. · May 2026Diffusion language model

GPT-2 Small

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

TrainingGeneration

GPT-2 Small

Standardized features

×

tensor

The features the encoder reads and the latent decoder reproduces.

shape
hh [B batch, T positions, 768 d]
Sources

Contextual evidence from GPT-2 Small

With the feature model frozen, its hidden state is computed without gradient, standardized, and passed to the encoder; the same tensor is the target of the feature loss.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from GPT-2 Small

The feature model is wrapped to return one entry of its hidden states, chosen by index; GPT-2’s token table is resized to the configured vocabulary.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from GPT-2 Small

With an extraction layer other than −1 the encoder loads the wrapped feature model; when it is frozen it is put in evaluation mode and takes no gradient.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from GPT-2 Small

The feature statistics are a mean and a standard deviation per feature over valid positions of the validation set, the variance clamped at 10⁻⁴; features are standardized with them.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from GPT-2 Small

The feature model is GPT-2, frozen; the hidden state at index −3 is extracted; the encoder and the decoder are of the Perceiver kind; the file’s vocabulary size and mask token ID are defaults that the trainer overwrites with the tokenizer’s; there are as many latents as positions, of 768 features.

The implementation, pinned at revision e18c42acdfb7, not yet public