LDLMIn construction

Meshchaninov et al. · May 2026Diffusion language model

GPT-2 Small

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

Training Generation

Token IDs[B, T]Frozen GPT-2hidden state −3 h^raw[B, T, 768]Feature statisticsfixed assets Standardize

(h^{\mathrm{raw}}-\mu)/\sigma

h[B, T, 768]

GPT-2 Small

Standardize

no parameters

Subtract each feature’s mean and divide by its standard deviation.

{\htmlData{light=output standardize}{h}}=\frac{{\htmlData{light=raw gpt2}{h^{\mathrm{raw}}}}-\mu}{\sigma}

in: $h^{\mathrm{raw}}$ [B batch, T positions, 768 d]
out: $h$ [B batch, T positions, 768 d]

Sources2

Direct evidence

The feature statistics are a mean and a standard deviation per feature over valid positions of the validation set, the variance clamped at 10⁻⁴; features are standardized with them.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

With the feature model frozen, its hidden state is computed without gradient, standardized, and passed to the encoder; the same tensor is the target of the feature loss.

The implementation, pinned at revision e18c42acdfb7, not yet public