LDLMIn construction

All models, with previews →

Meshchaninov et al. · May 2026Diffusion language model

Latent encoder

Pinned implementation and paper LDLM @ e18c42a, not yet public · arXiv:2605.07933v1 · OpenWebText · 1024 · checkpoint, tokenizer and saved statistics not verified · How to read

TrainingGeneration

Latent encoder

Output RMSNorm

×

learned

Rescale each position’s 768 features by their root mean square and apply a learned gain.

z0=γout⊙u6max⁡ ⁣(∥u6∥2/768,  ε){\htmlData{light=output output-norm}{z_0}}=\gamma_{\mathrm{out}}\odot\frac{{\htmlData{light=stack-output blocks}{u_{6}}}}{\max\!\big(\lVert {\htmlData{light=stack-output blocks}{u_{6}}}\rVert_2/\sqrt{768},\;\varepsilon\big)}
ε=10−5\varepsilon=10^{-5}
in
u6u_{6} [B batch, T positions, 768 d]
out
z0z_0 [B batch, T positions, 768 d]
parameters
768

ε bounds the divisor from below; it is not added under the root.

Parameters

768 parameters owned by this component

Exact per-instance accounting
ComponentCalculationParameters
RMSNorm gain768768768
Per-instance total768

Derived from the pinned implementation’s constructors and configuration. The checkpoint’s tensors have not been counted. 306,011,162 of these parameters are trained; frozen GPT-2 and the decoder’s mask vector, 124,442,112, are not.

One gain for each of the 768 features.

Sources3

Direct evidence

RMSNorm divides by the vector’s norm times width^(−1/2), clamped from below at ε, and multiplies by a learned gain.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

The encoder owns the learned latent queries, a position table, an input and an output RMSNorm and the blocks.

The implementation, pinned at revision e18c42acdfb7, not yet public

Direct evidence

The encoder adds positions to the features, normalizes the learned queries, applies the blocks, each attending to the features, and normalizes the result.

The implementation, pinned at revision e18c42acdfb7, not yet public