LDLMIn construction

All models, with previews →

Meshchaninov et al. · May 2026Diffusion language model

Latent encoder

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

TrainingGeneration

Latent encoder

Latents

×

tensor

The clean latents: what the diffusion learns to generate and the decoder to read.

shape
z0z_0 [B batch, T positions, 768 d]
Explanation

After the output norm every latent has a root mean square near the gain’s size, so the latents keep a fixed scale while the encoder is trained.

Sources

Contextual evidence from Latent encoder

The encoder owns the learned latent queries, a position table, an input and an output RMSNorm and the blocks.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Latent encoder

The encoder adds positions to the features, normalizes the learned queries, applies the blocks, each attending to the features, and normalizes the result.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Latent encoder

The latent encoder: 12 heads of 64 features, query and key normalization, softcap 50, 6 layers, MLP multiplier 4, dropout 0.1, ε = 10⁻⁵.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Latent encoder

The feature model is GPT-2, frozen; the hidden state at index −3 is extracted; the encoder and the decoder are of the Perceiver kind; the file’s vocabulary size and mask token ID are defaults that the trainer overwrites with the tokenizer’s; there are as many latents as positions, of 768 features.

The implementation, pinned at revision e18c42acdfb7, not yet public

Contextual evidence from Output RMSNorm

RMSNorm divides by the vector’s norm times width^(−1/2), clamped from below at ε, and multiplies by a learned gain.

The implementation, pinned at revision e18c42acdfb7, not yet public